Skip to content

fix(harness): recheck execution limits before worker launch - #552

Draft
Inference1 wants to merge 1 commit into
InferMatrixLab:mainfrom
Inference1:fix/harness-prelaunch-cancellation
Draft

Inference1 wants to merge 1 commit into
InferMatrixLab:mainfrom
Inference1:fix/harness-prelaunch-cancellation

Conversation

@Inference1

Copy link
Copy Markdown
Contributor

Governing Issue and delivery slice

Refs #429

This proposed prerequisite strengthens the shared local CPU launch path used by the deadline and cancellation work in the accepted Harness scope.

Problem

The runner checks its deadline before creating the process domain and opening log files. Cancellation is checked by the caller before setup and by the supervisor after launch. If cancellation is observed or the deadline expires during log setup, the supervisor can still start a worker.

What

Recheck the cancellation handle and monotonic deadline immediately before Popen. Record the refused step and subsequent steps as not_started when no process was launched.

Why needed

Even a short-lived worker can write output or acquire resources. Checking the limits at the launch boundary keeps those effects from occurring when the cancellation or expiry is already observable.

Design and implementation

The existing supervisor returns the cancellation timestamp and source or its timeout result with an empty PID. The existing cleanup path closes the opened streams and any empty Windows Job. The execution service uses its unstarted-step representation and records cancellation independently from process cleanup. The fix adds no schema fields, clocks or dependencies.

Checks and dependencies

  • Both regression cases launch a worker on 9466bc8. With the patch they commit schema-valid, unstarted step records with no process lease or runtime binding.
  • Linux, Python 3.12. Harness execution, legacy local executor, task protocol, generation seal and Apps fixed-record views. 68 passed, 1 skipped. The skip is the Windows Job Object test. Full Ruff and git diff --check passed.
  • Upstream base 9466bc8 already fails test_project_explanation.py::test_recovered_attempt_does_not_trust_a_rewritable_declaration due to its import of the removed inferrsi.infra.artifacts module.

Remaining work and Issue closure

This draft requests maintainer scope confirmation and formal review of the local launcher change. Production evaluation-mode execution remains tracked by the governing Issue.

Recheck the caller cancellation handle and monotonic deadline after log setup. Commit an unstarted step and preserve the cancellation request when no worker was launched.

Agent-assisted-by: Codex (author)
Signed-off-by: Inference1 <3617237115@qq.com>

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NO PASS

首轮独立检视:9466bc8d910c9fef55d4a9e23ee1a82950280c30 → f331999b2dcc0d7f9d9eabed99483bcb3e935f12。这是 #429 的本地启动前置修正,不是 T37/T40 验收。

批准前必修:日志打开期间耗尽步骤限时会被记录成整次执行超期。 execution_service.py:622–640 将 min(整次截止时间, 步骤截止时间) 传入 supervisor。这个 PR 新增的 启动前检查 会正确拒绝启动,但无 PID 的 timeout 分支一律记 deadline_exhausted。用本提交的两个函数及受控时钟验证:整次限时 60 秒、步骤限时 5 秒、打开日志耗时 7 秒,没有启动进程,却报整次期限用尽。请区分步骤限时和整次期限,同时保留 not_started 和零进程启动;补一个非空 timeout_seconds 的回归。

实际验证:独立执行变更后的函数,用启动哨兵确认取消和整次期限在日志打开后均拒绝启动,且日志流关闭;另复现上述有限步骤限时的误报。未运行完整 Harness 测试、原生 Windows Job Object 或生产 T37/T40,未看 CI。

Copy link
Copy Markdown
Collaborator

Thanks for closing the launch-boundary gap. I reviewed 9466bc8d910c9fef55d4a9e23ee1a82950280c30 → f331999b2dcc0d7f9d9eabed99483bcb3e935f12.

Rechecking the cancellation handle and monotonic deadline after the log files are opened and before Popen is the right place. The early return stays inside the _supervise try/finally, so opened streams and an unattached Windows Job are still closed, and _run_steps now treats a pre-launch cancelled result like the existing no-PID timeout path so later steps stay not_started. The new parametrized test covers the cancellation and whole-run deadline cases.

I also checked the current-head review from @zengchuang-hw and agree it is still blocking:

  • _run_steps passes min(attempt deadline, step deadline) into _supervise. The new pre-launch timeout check in local_process.py is correct to refuse the start, but a no-PID timeout is always recorded as deadline_exhausted, and the supervisor error text always says the whole-run deadline was exhausted. If log setup overruns a finite timeout_seconds while the attempt deadline is still in the future, the step is misreported as a whole-run expiry.

Please distinguish a step-timeout refusal from a whole-run deadline refusal, keep the step not_started with zero process starts, and add a regression with a non-null timeout_seconds for that path.

I am not approving this head.

@Dong1017 Dong1017 added area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:task Bounded delivery task with a delegated slice labels Nov 21, 2026

Copy link
Copy Markdown
Contributor

@Inference1 跟进 #371 / #429 的启动路径前置依赖:当前仍为 f331999b,已有 zengchuang-hw 与 Amy 的有限步骤限时误报意见尚未在新提交中处理。请先区分非空 step timeout 与 attempt deadline 在日志打开期间耗尽的拒绝原因,保留 not_started / 零启动,并补非空 timeout_seconds 回归,再推送新 head 请求正式复审。主线的 #555 已由 #556 合入,后续请同步该修复后跑完整检查;旧失败记录仍保留。

@Dong1017 #552 已是有具名作者 Inference1 的具体 PR,无需另开同样实现。请就这份 PR 落实 Harness-owner 范围确认和作者接续,作为你 PR-3 的已知前置。我们交付的 #561(PR-1)已完成最终主线 127 passed / 2 POSIX skips 与 fresh wheel,仍不重复这里的 launcher 代码。此处请在代码和回归具备后进入正式 Ready 检视;不要把 T37/T40 或 #371 总验收与本修正混同。

@Dong1017

Copy link
Copy Markdown
Collaborator

Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

Decision (Dong-bot, authorized delegate): Harness-owner scope is confirmed for #552 and Inference1 continues as author. I am not passing the current head, f331999b.

Recorded

Current head: blocker still open. I checked the diff against the zengchuang-hw and Amy findings.

  • In execution_service.py (_run_steps), any no-PID timeout is still recorded as deadline_exhausted.
  • The new pre-launch check in local_process.py still sets "whole-run deadline exhausted before command start" whenever time.monotonic() >= deadline. Here deadline is min(attempt deadline, step deadline).
  • If log setup overruns a finite timeout_seconds while the attempt deadline is still in the future, the step is misreported as whole-run expiry.
  • The new test only covers timeout_seconds unset, so this path has no regression.

Required before formal Ready review

  1. Report a step-timeout refusal differently from an attempt-deadline refusal, in both the supervisor error and the step reason. Keep not_started, zero process starts, no lease and no runtime binding.
  2. Add a regression with a non-null timeout_seconds where the step limit expires during log opening and the attempt deadline is still in the future.
  3. Sync the Fix the retired digest import in the project-explanation recovery test on main #555/Use the Harness digest in the interrupted recovery test #556 fix from main and run the full checks. The PR text still records the inferrsi.infra.artifacts import failure on 9466bc8.
  4. Push a new head and request formal re-review. The PR is currently still draft.

The cancellation and whole-run deadline refusal paths look right to me from the diff. The early return stays inside the _supervise try/finally, so the log streams still close. I ran no tests and did not look at CI. Once the new head is up, I'll re-read it against the items above.

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

COMMENT (not an approval). Independent read of f331999, as the planned author of #429 PR-3, which builds on this launch path. I agree with the NO PASS above and add where it comes from and one way to fix it.

Why it is new in this PR. _run_steps passes bound = min(deadline, now + timeout_seconds) (execution_service.py:622). Before this PR, _supervise returned a PID-less timeout only from its entry check (local_process.py:285). With a positive step timeout, bound is later than the call time, so that check can only fire on the whole-run deadline, and mapping it to deadline_exhausted was right. The new check after the log files open (local_process.py:300) can also fire on the step bound. The PID-less branch (execution_service.py:631-633) still records deadline_exhausted for both cases, and it stops the remaining steps for that reason too.

Suggested fix. Classify the PID-less timeout the same way the started-step branch three lines below already does ("deadline_exceeded" if bound >= deadline else "step_timeout"): use deadline_exhausted only when bound >= deadline, and otherwise record the step as not_started with step_timeout. Then let the existing fail-fast handling decide what happens to later steps, as it does for a started step that timed out. A regression with timeout_seconds set to a few seconds and the same log-open delay as your deadline case would pin it.

What I checked and found fine.

  • The early returns sit inside the try, so finally still closes the Windows Job (local_process.py:350-352) and the with block closes both log streams.
  • cancellation_source="caller" matches the post-launch path (local_process.py:331-332), and _cancellation reads it from the run (execution_service.py:743-745), so the record's cancellation facts are the same as for a cancel after launch.
  • The new cancelled PID-less branch needs no separate cancel_seen_at, for the same reason.

I ran nothing; this is a read of the diff at f331999 against main 7a7ab84.

Copy link
Copy Markdown
Contributor

@Inference1 @Dong1017 再同步 #371/#429 的依赖状态:#561/#562/#564 和 #565 的 H/B/Apps/CPU worker 切片已提交,开发栈开始真实 CPU 验证;完整 main 集成验收仍等待本 PR。当前仍是 f331999b2dcc0d7f9d9eabed99483bcb3e935f12,有限步骤 timeout 在日志打开期间耗尽被误报成整次 deadline 的检视意见尚未有修复提交。

请原作者按现有边界补该分类修正和非空 timeout_seconds 回归,并更新 Ready 状态/CI。@Dong1017 若原作者暂时无法继续,请明确后续负责人或排期;如需 Codex 协助,请给出这份限定修正的委派。当前不把该启动边界改动夹进其他 H/Apps/worker PR,继续保留 Inference1 的原作者责任。

@Dong1017

Copy link
Copy Markdown
Collaborator

Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

Decision: Inference1 stays the author of #552, and I'm not delegating the fix yet. The head is still f331999b, so my earlier not-pass stands. Dong-bot (authorized delegate) is recording this.

I'll re-read as soon as a new head is pushed.

Details

Checked the diff at f331999b:

  • execution_service.py: the PID-less timeout/cancelled branch still sets stop = "deadline_exhausted" for every timeout.
  • local_process.py: the new pre-launch check still reports "whole-run deadline exhausted" when monotonic() >= deadline, and deadline is min(attempt, step).
  • The new test only covers the case with no step timeout.

No formal APPROVED review exists on this head. The formal reviews are COMMENTED from zengchuang-hw and Dong1017. I ran no tests and did not check CI.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants