Skip to content

feat(harness): verify frozen preparation before registration - #561

Open
zengchuang-hw wants to merge 4 commits into
mainfrom
codex/harness-429-prepared-binding
Open

zengchuang-hw wants to merge 4 commits into
mainfrom
codex/harness-429-prepared-binding

Conversation

@zengchuang-hw

@zengchuang-hw zengchuang-hw commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Refs #429
Refs #371

Harness now verifies an evaluation request against its fixed preparation before registering an attempt. Malformed or inconsistent bindings raise declared errors with no RunStore attempt or record. Valid evaluation bindings still raise UnsupportedOperation until the separate T37 implementation lands; ordinary command execution remains enabled.

Accepted scope: Dong1017's PR-1 delegation. This is PR-1 of the three accepted Harness delivery units, not completion of #429 or #371. Dong1017 remains the accountable Harness parent; the later PR-2/PR-3 implementation slices are also explicitly delegated to this Codex session in #429. This uses and strengthens the owner-supplied, previously unreviewed PR Pool draft; that agent's exact identity is unknown. zengchuang-hw / Codex is the acting implementer. Dong's parent/material-contributor review does not count as independent approval.

Base: main 10ba7a5500bcd95949219b138f9ada5033b922b0, including #556. Head: 66fef981f6893038f3128595ffcdfd12b65bfba2. Initial increment bfa4bc23 was pushed before completing the slice, as requested.

Delivered behavior

  • Public Harness-owned prepared_binding.read / project / verify: read-only stream capability, bounded reads, independent SHA/size checks and canonical v1 projection. Unknown host programming exceptions propagate. No Apps/Benchmark/Knowledge imports and no grading-rule interpretation.
  • Static shape failures precede artifact I/O. Fixed prepared ID, protocol digest, operation/scope, candidate declarations and supported identity rules are checked before RunStore.claim.
  • Freeze worker declarations mechanically: expected code digest, runtime/entry/revision, one step, output metadata, capabilities, input roles, nonoverlapping paths and whole-argument input/output substitution. No actual entry resolution is performed. The code digest is an expected declaration, not an observed runtime fact.
  • Separate D023 null-timeout vector: 2069 canonical bytes, request digest sha256:62ce11e6d458b5c581245d0988189e1bd834f2ed1e1880d15242faf70c5b6fe9. The canonical Apps binder and Harness digest agree; cross-checks call inferrsi.apps.handoff.bridge.verify_protocol directly. Historical/frozen fixtures are unchanged.
  • Registered the new canonical Apps test consumer and added an installed-wheel isolated import probe for the public Harness port. Binding contract documents the ports, error boundary and remaining units.

Validation

  • Current head 66fef981: 78 binding tests passed, including the corrected zero-launch guard on local_process._supervise, the actual entry used by Harness. The earlier guard intercepted the legacy execute_command wrapper and did not directly intercept Harness launches. The no-registration/empty-table assertions remain in place.
  • Previous d67b4131: final binding + ordinary execution + recovery group 127 passed, 2 POSIX skips; earlier consumer/import group 205 passed, 43 subtests passed, 2 skips. Fresh sdist→wheel→new environment outside checkout, including the independent port probe, passed.
  • Production and packaging source are unchanged by the current one-line test correction. Entire-tree Ruff/whitespace passed previously; focused Ruff and whitespace passed again on this head.
  • Current-head Ready CI passed: both Python 3.10/3.11 full-check jobs report 5996 passed, 31 skipped, 674 subtests passed; installed-package, lint/declarations, PR references, Dashboard and advisory host checks succeeded. Amy renewed her independent approval on this head; a second eligible independent approval remains missing. Previous-head CI is not reused as current-head evidence.
  • No dependency, CI-lane, numeric gate, historical evidence or frozen fixture edits.

Remaining work and review

PR-2 supplies registered T37 refusal records and actual entry verification. PR-3 supplies attempt-level deadline/budget and T40 evaluation replay, including the separately owned #552 launch prerequisite. This PR does not register a real evaluation worker, produce a production execution record, prove a service gain or complete shared CPU acceptance.

@Dong1017 please review the owner boundary and use this PR as the PR-2 base, as recorded. @amy-why-3459 @david6666666 @jiangkuaixue123 please independently review the accepted PR-1 behavior, exact D023 bytes, zero-registration failures and ordinary execution compatibility. Integration requires two independent eligible write+ formal approvals with LGTM as the first nonempty line and passing checks; author, parent/material contributor and non-write bot approvals do not count.

Adapt the authorized pool draft and freeze the null-timeout binding vector.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
Verify candidate and worker declarations against fixed preparation bytes. Preserve ordinary command behavior and leave evaluation startup disabled until entry verification is implemented.

Agent-assisted-by: Codex (author)

Agent-assisted-by: Unknown PR Pool agent (draft contributor)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
Dong1017
Dong1017 previously approved these changes Nov 21, 2026

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 10ba7a5 and head d67b413 (full diff, diffComplete: true). prior-reviews.md has no formal reviews and no outstanding CHANGES_REQUESTED on this head, so this Approve is the first formal review. It is one vote and not a merge-rule substitute.

Verified by reading the diff and the callers at the head commit:

  • Ordering. execution_service.validate runs static shape checks first, including the new _evaluation_errors and the V32-N2 "evaluation step needs a non-null expected_code_sha256" rule. Only after those pass does it call prepared_binding.verify(spec, artifact_reader), and only then the UnsupportedOperation checks. execute passes artifact_reader through, and Harness.execute forwards its configured reader (api.py L50-53). Every refusal therefore happens before RunStore.claim. Valid evaluation specs still end in UnsupportedOperation, so no evaluation attempt can start.
  • Bounded read. prepared_binding.read rejects a malformed Ref, copies the Ref before reader.open, closes the stream, and reads at most size_bytes + 1 bytes. It re-checks size and SHA-256 itself. OSError and a missing reader become ArtifactUnavailable, PublicError is re-raised, and other programming errors propagate.
  • Error codes. IdentityMismatch, UnsupportedRuleVersion, ArtifactUnavailable, ArtifactIntegrityError and UnsupportedSchema are all in PUBLIC_ERROR_CODES (errors.py). The helpers it relies on (run_store._has_fields, _is_id, _describe, input_binding.RELATIVE_PATH, json_v1.MAX_SAFE_INTEGER) exist.
  • Binding logic.
    • Identity is compared first, then declarations.
    • Worker checks cover a single step, entry kind/name/revision, outputs, cwd=".", frozen capabilities, whole-argument {input:} / {output:} expansion, and path overlap.
    • The overlap check sorts path tuples and compares neighbours, which is sufficient: if one path is a prefix of another, every path sorted between them shares that prefix.
    • The expected-sha256@1 and captured-sha256@1 rules and the frozen expected-candidate checks match the contract doc.
    • Unknown rule revisions are refused and not downgraded.
  • Isolation. prepared_binding imports only contracts.json_v1 and harness.*. check_installed.py adds a cold-import assertion that it pulls in no apps, benchmark or knowledge modules.
  • Registry. The apps-module-migrations.json consumer entry is needed because the new test imports inferrsi.apps.handoff.bridge. test_active_consumers_use_canonical_paths_and_are_registered is set-based, so the entry's position does not matter.
  • Tests and docs. The new tests assert an empty attempts and records table and patch claim and process start to fail on use. They cover null and wrong code, identity mismatch, worker drift, malformed projection, reader bounds, reader mutation and propagation of reader programming errors. The updated expectation in test_harness_execute.py (InvalidExecutionSpec instead of UnsupportedOperation for the null-digest evaluation spec) follows from V32-N2. The contract doc, operations row and fixture README state the scope honestly: T37 and T40 are later slices and there is no production acceptance claim.

Not verified here: I did not run the test suite or CI. In particular I did not confirm that the D023 vector's bytes (2069 bytes, digest sha256:62ce11e6…6fe9) match bind_execution output. I also did not audit the golden t108-v4 fixtures or the Apps binder behaviour. Please confirm CI is green before merge.

Non-blocking nit: the new consumer entry in docs/apps-module-migrations.json sits between docs/apps-output.md and docs/architecture.md, while the rest of the list is alphabetical. It could move after src/... / before tests/check_installed.py for tidiness.


Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

@Dong1017 Dong1017 added area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:task Bounded delivery task with a delegated slice labels Nov 21, 2026

Copy link
Copy Markdown
Contributor Author

当前 head d67b41312a2b61cfe81ae1dec1fcda715edd4d62 的收尾验证已完成:绑定 + 旧 ordinary execute + #556 recovery 路径 127 passed,2 个 POSIX 平台跳过;fresh sdist→wheel→全新环境、checkout 外 installed checker(含独立绑定端口导入)通过;全树 Ruff 和 whitespace 通过。此前迁移/边界扩大组的 205 passed、43 subtests passed、2 skipped 仍按实际内容/运行关系保留在正文。D023 规范字节长度和完整 SHA 在测试中直接核对 Apps binder 与 H request_digest,不只是文档声明。

@Dong1017 技术复审已收到。清单排序是非阻塞建议,登记和校验为集合语义;暂不为排序单独推新 head 重启 CI,后续实际修改该清单时再归并顺序。你作为 parent / 草稿材料贡献者的 Approve 不计独立审批。

@amy-why-3459 @david6666666 请对同一最终 head 独立检视,有问题我先修复;两份有效独立批准与检查通过后再 squash。#429 PR-2/3 和生产 CPU 验收仍保留,#371 不关闭。

@Dong1017

Copy link
Copy Markdown
Collaborator

Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

Nothing is needed from me beyond my existing Approve. That Approve is on the current head d67b4131, and no CHANGES_REQUESTED is outstanding. This comment is a status update, not a new ask for a re-review.

  • Ordering suggestion: I accept leaving the docs/apps-module-migrations.json consumer-list ordering for a later edit of that list. It is a non-blocking style point, and the registration and validation treat the list as a set. It is not worth a new head and a CI restart.
  • My Approve: I reviewed the full diff at d67b4131 (diffComplete: true) and found nothing to change.
    • Ordering of checks: static shape checks run first, then prepared_binding.verify, then the UnsupportedOperation checks. All of them run before RunStore.claim.
    • Read bounds: prepared_binding.read reads at most size_bytes + 1 bytes and re-checks the size and SHA-256.
    • D023 vector: the 2069-byte vector and its digest are asserted in the tests against both the Apps binder and the Harness request_digest.
    • What I did not do: I did not run the tests or CI myself. The pass counts above are the author's.
  • Merge condition: as you stated, my Approve does not count toward the two independent approvals. Those have to come from @amy-why-3459 and @david6666666 on this same head, with checks passing, before the squash.
  • Scope: task: Harness evaluation-mode CPU execute for Apps #429 PR-2/PR-3 and production CPU acceptance stay open, and task: Apps 应用层详细设计与迁移计划(CLI / Python / Plugin) #371 stays open.

This reply is from Dong-bot (authorized delegate).

amy-why-3459
amy-why-3459 previously approved these changes Nov 21, 2026

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 10ba7a5 and head d67b413.

Verified against the accepted #429 PR-1 scope in 7061372237 (pre-registration refusals with no RunStore attempt/record, a Harness-owned bind port, and the D023 null-timeout vector). This is PR-1 only; T37 entry verification and T40 remain later slices, and valid evaluation bindings still end in UnsupportedOperation.

  • Ordering: execution_service.validate collects static shape errors first, including _evaluation_errors and the V32-N2 rule that an evaluation step must have a non-null expected_code_sha256. Only a clean spec reaches prepared_binding.verify, and only then the UnsupportedOperation checks. Harness.execute forwards the configured artifact_reader. All of this runs before RunStore.claim. Ordinary command mode never reads a preparation.
  • Bounded read: prepared_binding.read copies the ArtifactRef before reader.open, closes the stream, reads at most size_bytes + 1 bytes, and re-checks size and SHA-256. A missing reader or OSError is ArtifactUnavailable; PublicError is re-raised; other host programming errors propagate. The mutation test shows the original Ref is unchanged.
  • Binding: identity (prepared id, protocol digest, operation, scope) is compared before declarations. Worker checks cover a single step, entry kind/name/revision, outputs, cwd=".", frozen capabilities on the spec, whole-argument {input:} / {output:} expansion, and prefix-overlap of materialized paths. expected-sha256@1 requires a declared input hash even without a frozen candidate; unknown rule revisions are UnsupportedRuleVersion and are not downgraded. The new error codes are in PUBLIC_ERROR_CODES.
  • Isolation: prepared_binding imports only contracts.json_v1 and harness.*. check_installed.py adds an isolated-interpreter probe that the public port does not import Apps, Benchmark, or Knowledge.
  • D023: I independently hashed tests/fixtures/harness-prepared-binding/d023-null-timeout.json at this head: 2069 bytes, digest sha256:62ce11e6d458b5c581245d0988189e1bd834f2ed1e1880d15242faf70c5b6fe9, matching the accepted vector. The tests also assert agreement with bind_execution and run_store.request_digest; I did not execute those binders myself.
  • Tests and docs: refusals assert empty attempts/records tables and patch claim / process start to fail if used. The null-digest evaluation case in test_harness_execute.py correctly becomes InvalidExecutionSpec under V32-N2. The contract and operations row state the remaining PR-2/PR-3 work honestly.

lint-and-smoke, declarations, pr-references, installed-package (3.10/3.11), dashboard-browser and host-compatibility were green on this head when I checked; full-check (3.10/3.11) in https://github.com/InferMatrixLab/InferRSI/actions/runs/44900831192 was still running and should pass before merge.

This is an independent write+ review of the accepted PR-1 behavior. The parent/material-contributor Approve does not count toward the two required independent approvals.

Copy link
Copy Markdown
Contributor Author

@jiangkuaixue123 补请你独立检视当前 head d67b41312a2b61cfe81ae1dec1fcda715edd4d62。#429 PR-1:登记前拒绝、固定准备只读绑定、D023 2069 字节摘要、旧 ordinary execute 兼容;T37/T40 仍留后续切片。

当前 head Ready CI 已全部通过,细项本地回归与 fresh wheel 结果见正文。Amy 已正式批准;原 David 请求仍保留。Dong 是本切片的 accountable parent / owner,他的批准不计独立票,作者也不自审。请按实际 diff 和接受范围给出 PASS / DO NOT PASS;符合时正式 Approve 的首个非空行请独立为 LGTM。若参与过本切片的实质材料或实现,请注明,以便准确记录角色。

Copy link
Copy Markdown
Contributor Author

@hsliuustc0106 补请你独立检视本 PR。当前 head 的 Ready CI 全部通过(3.10 / 3.11 各 5980 passed、31 skipped、674 subtests),核心/边界回归与 fresh wheel 结果见正文;Amy 已同 head 正式批准。现有 David / jiangkuaixue123 的请求仍保留。请按实际差异给出结论,有意见会先处理。

本切片 parent 是 Dong1017,他的技术批准不计独立审批;作者不自审。需要另一位独立 write+ 评审正式批准后才可按既定顺序合入。

Intercept the supervisor used by Harness rather than the legacy execution wrapper in zero-start binding tests.

Agent-assisted-by: Codex (author)
Signed-off-by: zengchuang <zengchuang3@huawei.com>
@zengchuang-hw
zengchuang-hw dismissed stale reviews from amy-why-3459 and Dong1017 via 66fef98 November 21, 2026 13:06

Copy link
Copy Markdown
Contributor Author

修正了一处验证缺口,当前 head 66fef981:零启动哨兵先前拦截 legacy execute_command,实际 Harness 调用 local_process._supervise。现已改到真实入口;#562 的匹配入口正向控制还在同一入口观察真实非空 PID,确认拦截点与实际运行一致。

本 PR 当前 78 项核心用例通过,Ruff/whitespace 通过;生产及打包源码未改。既有表计数/固定记录断言保留。旧头测试和安装证据按原 revision 记账,新的 CI 及正式批准必须对应新 head。此前旧头 Approve 不计新头票。

@amy-why-3459 @david6666666 @jiangkuaixue123 @hsliuustc0106 请复审最终头,有意见先处理。@Dong1017 请保留 owner 技术检视及既定前置顺序;你的票仍不计独立批准。

@Dong1017 Dong1017 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 10ba7a5 and head 66fef98. I read the full diff (diffComplete: true), not only the d67b413→66fef981 delta. This is a technical review by Dong-bot (authorized delegate). The #429 parent/material-contributor role makes it one vote that does not count as independent. Your merge rule still needs the two independent write+ approvals on this head and passing checks.

State: my earlier Approve was dismissed with the push, so this one renews it on the current head. prior-reviews.md shows no formal review on 66fef981 and no outstanding CHANGES_REQUESTED.

Verified from the diff and the head source

  • Ordering.
    • validate runs the static shape checks first: _evaluation_errors, the V32-N2 "evaluation step needs non-null expected_code_sha256" rule, and the existing input/step checks.
    • It then calls prepared_binding.verify(spec, artifact_reader).
    • It then raises the UnsupportedOperation errors.
    • execute forwards artifact_reader into validate (L991) and only calls store.claim afterwards (L995). Every refusal therefore happens before registration. A valid evaluation spec still ends in UnsupportedOperation.
  • Bounded read.
    • read validates the ArtifactRef shape, then calls reader.open(dict(ref)) on a copy and closes the stream.
    • It reads at most size_bytes + 1 bytes in chunks and re-checks size and SHA-256 itself.
    • OSError and a missing reader map to ArtifactUnavailable. PublicError is re-raised. Other host exceptions propagate. Tests cover each of these cases, including a reader that mutates the ref.
  • Identity and declarations.
    • Prepared id, protocol digest, operation and scope are compared first and give IdentityMismatch.
    • Worker checks cover the single step, runtime role, entry kind/name/revision, outputs, cwd="." and frozen capabilities.
    • Whole-argument {input:}/{output:} expansion is checked.
    • The input/output path overlap check compares only adjacent sorted tuples. That is sufficient, because under lexicographic order a prefix path forces every path between the two to share that prefix. Equal paths are caught too.
    • Unknown candidate identity rules or revisions give UnsupportedRuleVersion. expected-sha256@1 requires a declared input hash.
  • Test correction in this head.
    • The zero-launch sentinel now patches execution_service.local_process._supervise and store.claim, which are the real call path. The _argv guard stays.
    • Each refusal test still asserts that the attempts and records tables are empty. The changed assertion in test_harness_execute.py is consistent with V32-N2.
  • Packaging and vector.
    • The check_installed.py probe runs in an isolated interpreter (-I -B) and asserts that no Apps, Benchmark or Knowledge modules are imported.
    • prepared_binding is added to NEW_H_CLOSURE, and the new test consumer is registered in apps-module-migrations.json.
    • The D023 vector is asserted at 2069 bytes with digest sha256:62ce11e6…6b5c581245d0988189e1bd834f2ed1e1880d15242faf70c5b6fe9. The test checks it against both request_digest and the Apps bind_execution/verify_protocol.
    • The docs state that the code digest is an expected declaration, not an observed fact, and that T37, T40 and CPU acceptance are not covered.

Not done: I did not run the tests, the wheel build or CI. The 78-test result and Ruff/whitespace results are the author's. Please wait for current-head checks to pass before the squash. The consumers ordering in apps-module-migrations.json remains a non-blocking style point.

Scope: #429 PR-2/PR-3 and the #371 closure are unchanged and stay open. Nothing here should be read as completing them.


Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).

@amy-why-3459 amy-why-3459 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed the increment d67b41312a2b61cfe81ae1dec1fcda715edd4d62 → 66fef981f6893038f3128595ffcdfd12b65bfba2 against base 10ba7a5500bcd95949219b138f9ada5033b922b0.

The only file change is in tests/test_harness_prepared_binding.py: the zero-start sentinel now patches execution_service.local_process._supervise instead of the legacy execute_command. That is the launch path execution_service._run_steps actually uses. Patching _supervise also covers the legacy execute_command wrapper, which calls _supervise in the same module. The _argv and store.claim guards remain. Production and packaging source are unchanged in this increment.

Ready lint-and-smoke, declarations, pr-references, installed-package (3.10/3.11), dashboard-browser and advisory host-compatibility were green on this head when I checked; full-check (3.10/3.11) in https://github.com/InferMatrixLab/InferRSI/actions/runs/44907170850 was still running and should pass before merge. This is an independent write+ review of the test-hook correction on the previously accepted PR-1 behavior; the parent/material-contributor Approve does not count toward the two required independent approvals. Accuracy/performance comparison evidence is not applicable (test-only change).

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

PR review verdict — PASS
Reviewed base 10ba7a5500bcd95949219b138f9ada5033b922b0 and head 66fef981f6893038f3128595ffcdfd12b65bfba2.

Identity: cursor[bot] independent third-party / technical verify only (collaborator permission none). This is not a write+ eligible Approve, dual-tip satisfaction, or merge-ready claim under docs/agent-collaboration.md. Merge still needs write+ independent Approves + integrator checks.

Clean-code / repo style: PASS

  • Layering: prepared_binding imports only contracts + harness (input_binding, run_store, errors) — no Apps/Benchmark/Knowledge imports (docs/architecture.md dependency direction).
  • Naming & APIs: FixedPreparation, read/project/verify, PublicError codes match docs/contracts/harness-prepared-binding.md and operations/migrations consumers.
  • Error handling: identity/declaration failures raise before store.claim; bounded artifact reads (size+1) with independent SHA/size re-check; host errors mapped honestly.
  • No dead code / no forwarder shim; ordinary command path unchanged; valid evaluation still refused until later T37 slice (as documented).
  • Tests/docs aligned (test_harness_prepared_binding.py, D023 fixture, check_installed NEW_H_CLOSURE probe). Non-blocking: apps-module-migrations.json consumers list order is slightly off-alpha.

Verified: tip Dong1017 + amy Approves present; Ready required checks green (declarations/full-check/installed-package). Read tip diff for prepared_binding.py, execution_service.validate ordering, and binding tests — not CI-green alone.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:harness Harness module: profile, A/B loop, quality gates, evidence reports kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants