Skip to content

validate_and_wait: poll the validation job, fail fast on failed runs - #8

Merged
nikolaos-paximadakis-eea merged 3 commits into
mainfrom
feat/validate-and-wait-job
Nov 20, 2026
Merged

nikolaos-paximadakis-eea merged 3 commits into
mainfrom
feat/validate-and-wait-job

Conversation

@nikolaos-paximadakis-eea

Copy link
Copy Markdown
Collaborator

Until now, validate_and_wait waited for a VALIDATION_FINISHED_EVENT user notification. A failed run creates no such notification, so the helper always ran into its timeout (600 s by default).

The backend's current line (branch MasterOneVersion, JobControllerImpl#addValidationJob) returns the new job's id. That is the bare number (44570) we saw live and didn't understand.

Changes

  • validation.start_validation() (sync and async):
    • returns that id, or None when the body is empty (develop and MasterDataLakes answer void);
    • takes optional dataflow_id and provider_id, which enable the route's ApiKey branch there.
  • easy.validate_and_wait() polls the job through pollForJobStatus, the same route our import and export helpers use:
    • FAILED, REFUSED or CANCELED: raises JobError at once, with status and the job's info (for example "Validation failed due to system error…");
    • FINISHED with an info text (for example "There are canceled validation tasks…"): a RuntimeWarning.
  • When there is no job id, or the poll route answers 403/404 (it is ApiKey-only on MasterDataLakes), it falls back to waiting for the notification. That wait now also stops on VALIDATION_REFUSED_EVENT, VALIDATE_AS_PROVIDER_REFUSED_EVENT, VALIDATION_CANCELED_EVENT and VALIDATION_FAILED_*_EVENT.
  • New keyword provider_id (a reporter ApiKey needs it on both the start and the poll). Dataset.validate() passes the handle's own.

Known gap (backend): one internal-error path in validateDataSetData only marks the dataset as having a validation error. It changes neither the job nor the notifications, so that case still ends in the timeout.

Testing: mocked tests only, covering a job that finishes, fails, or finishes with a note, a refused poll falling back to notifications, a failure notification, and the new resource parameters and return value. This is not yet run live; one sandbox run, including a run meant to fail, would confirm it.

Independent of #7. Whichever merges second needs a small CHANGELOG merge (both add to "Unreleased").

🤖 Generated with Claude Code

…ions

The backend's current line (MasterOneVersion) answers addValidationJob with the
new job's id (the bare `44570` seen live). start_validation now returns it
(None for an empty body) and takes optional dataflow_id/provider_id, which
enable the route's ApiKey check there. validate_and_wait polls the job via
pollForJobStatus, so a FAILED/REFUSED/CANCELED run raises JobError with the
job's status and info at once instead of timing out; a FINISHED job with an
info text warns. Without a job id, or when polling is refused (ApiKey-only on
MasterDataLakes), it falls back to the notification wait, which now also ends
on VALIDATION_REFUSED/CANCELED/FAILED_* notifications. Dataset.validate passes
the handle's provider_id. Read from the backend source; not yet run live.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fall back to notifications also when the job poll answers 401 or a
rejected-credential 500; a poll timeout names the job, dataset and the
caller's timeout; a FINISHED-with-note result carries the note in
attrs['validation_note']; the job id parser accepts only ASCII digits
(also JSON-quoted).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nikolaos-paximadakis-eea
nikolaos-paximadakis-eea merged commit 7dfd944 into main Nov 20, 2026
6 checks passed
@nikolaos-paximadakis-eea
nikolaos-paximadakis-eea deleted the feat/validate-and-wait-job branch November 20, 2026 10:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant