Repository navigation
feat(benchmark): add frozen P1 offline assessment - #560
zengchuang-hw wants to merge 6 commits into
Conversation
Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
Dong1017
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 7a7ab84c3b5f26b20c098f439470d5bcfa972809 and head 4258c76a84f3c9cf7086a0b7965a7ec17cd445f9 (full diff, diffComplete: true). prior-reviews.md shows no formal review and no outstanding CHANGES_REQUESTED on this head, so this is the first independent review. This is Dong-bot (authorized delegate) acting for Dong1017.
What I verified by reading the diff and the base-tree code it depends on
- Module boundary.
benchmark/assessment.pyimports onlycontracts.json_v1and the siblingbenchmark.preparation. It has no Apps, Harness or Knowledge import, which fits the import gate indocs/module-layout.md. The newtests/check_installed.pyprobe covers this in a fresh-I -Binterpreter. It checks that the module ships in the wheel and that noinferrsi.apps,inferrsi.harnessorinferrsi.knowledgemodule gets loaded. - Helpers reused from preparation. The private helpers it calls exist at the head and have the expected shapes:
_artifact,_id,_SHA,_DECIMAL,_closure,PreparationError(inpreparation.py),validate_prepared,protocol_digestandprepared_id. - Migration inventory. The
docs/apps-module-migrations.jsonedits addtests/test_benchmark_assessment.pyto the consumers ofapps.storage.archive,apps.storage.artifact_readerandapps.handoff.bridge. These are the canonical modules the new test imports.test_active_consumers_use_canonical_paths_and_are_registeredrequires exactly this registration. It only checks that each consumer is unique and is a file, so the append order is fine. - Binding order.
_bindingchecks the full record hash and size, the attempt and revision, the task, operation and scope,prepared_idandprotocol_digest. It does this before any artifact read, andtest_binding_mismatch_is_refused_before_any_artifact_openasserts it. The inputs are snapshotted through a canonical round-trip, which the mutating-reader test covers. - Byte authority.
_RetainedBytes.readreads at mostsize+1bytes, so an oversize artifact is caught. It checks size and SHA-256, closes any stream it opened, and caches the outcome. Only the portablecodevaluesUnsafeArtifactPath,ArtifactUnavailableandArtifactIntegrityErrorbecome claim-level gaps. Other host exceptions propagate, and the tests coverTypeErrorandCanonicalError. A failure to read the protocol itself is a port error, not a guessed assessment. - Trust checks.
_trustedchecks the check, producer, domain, runtime pin (entrypoint, revision and code hash) and a started step bound to the candidate input. The tests cover wrong-code, foreign-check, uncaptured-input and not-started cases. - Claim logic.
- F and C are independent, so one claim's missing evidence does not erase another's result.
- M additionally requires a completed measurement, confirmed cleanup and lease release.
- Contradictory trustworthy proofs, including conflicting sample values, collapse into one INCONCLUSIVE claim with
conflicting_evidence. validate_assessmentenforces exactly three claims, emptymetricsandcomparison, calendar-checked timestamps and the semantic ID. The ID excludes only the clock, the self-ID and typedArtifactReflocations.
- Docs.
docs/contracts/production-assess.mdand the newdocs/operations.mdrow match the code, including the explicit statement that the Harness evaluation worker and T37/T40 (#429) are not claimed.
Non-blocking follow-ups (not gating)
-
_observationsdropsproblemswhenever at least one other proof for the samecheck_idparsed. Two cases follow, and both end up PASS:- A trusted check lists a valid
passedproof plus a second evidence ref that is unreadable or fails its hash. - A second same-
check_idcheck has unreadable evidence.
Neither case is surfaced as a gap or as
INCONCLUSIVE. A corruptedfailedproof could therefore be hidden by a goodpassedone. The doc says an affected proof becomes a claim-level gap, so I suggest either marking the claim INCONCLUSIVE when any trusted declared ref is unusable, or stating the current rule in the contract doc. A test with two refs on one check would pin whichever behavior you choose. - A trusted check lists a valid
-
assessment_id()is public but only checks the top-level fields. A malformed nested result can raiseKeyErrororTypeErrorthere instead of a declaredAssessmentError.validate_assessmentruns the shape checks first, so the supported paths are fine.
Not verified here
- I did not run the tests, CI or the wheel/installed check, and I did not see the CI status.
- I did not diff the code against the t108-v4 golden fixtures or the Apps
assesswiring beyond what the new tests exercise. - I did not audit
preparation.validate_preparedinternals. - Joint production CPU acceptance (#371/#429) remains outside this PR, as the PR states.
- The merge ruleset still needs the second eligible approval and green required checks.
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
|
@Dong1017 请按 7034454814 的已收口检查点 技术检视当前 head @amy-why-3459 @david6666666 请作为独立评审检视同一 head。当前源码对应核心交叉组 114 passed;边界/登记组与隔离安装的实际受测版本关系见正文(无隐藏跳过或弱化检查)。Ready CI 44898147291 已启动,合入前须通过。 角色明确:作者为 zengchuang-hw / Codex,accountable parent 是 Dong1017;Dong 的 owner 技术意见不计独立批准。两位有效独立 write+ 账户通过后,请各自以首个非空行独立为 |
|
Reviewed The module boundary, binding-before-read, portable artifact error handling, F/C/M independence, V21/V22/V23 coverage, and the Apps T35/T39 tests look consistent with the accepted #434 interface and One issue I would like resolved before approving:
Please either treat a trusted check as incomplete/INCONCLUSIVE when any declared ref is unusable (and add a two-ref test), or state the current "any one valid proof is enough" rule in Non-blocking notes: please rebase onto current main before merge. Required CI on this head was still running when I reviewed, and the known |
Keep an affected claim inconclusive when any matching declared proof is unusable, even if another proof is valid. Return declared errors for malformed identity projections. Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
Agent-assisted-by: Codex (author) Signed-off-by: zengchuang <zengchuang3@huawei.com>
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 7a7ab84c3b5f26b20c098f439470d5bcfa972809 and head 4258c76a84f3c9cf7086a0b7965a7ec17cd445f9.
Technical / third-party verify only (cursor App collaborator permission none — not write+ eligible / dual-tip / merge-ready under docs/agent-collaboration.md).
Checked
- New
inferrsi.benchmark.assessmentoffline P1 port: no Apps/Harness/Knowledge imports; binding + retained-byte proof path; independent F/C/M claims;AssessmentErrorportable codes. - Tests cover codec identity, claim gaps vs host errors, binding refusal before artifact open, unsupported rule revision, clock/location projection, no worker/network, Apps exits via fixture path.
- Docs:
production-assess.md+operations.mdrow;check_installedwheel isolation probe; migrations JSON lists the new test.
Ready note (non-code)
4887f1e
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 10ba7a5500bcd95949219b138f9ada5033b922b0 and head 4887f1e94f39c1d4b675e965cfcc0a6e932d50a2 (re-review after tip move dismissed prior Approve on 4258c76a84f3…).
Technical / third-party verify only (cursor App collaborator permission none — not write+ eligible / dual-tip / merge-ready).
Delta since prior tip
- Evidence-gap integrity: empty evidence_refs and any unusable declared proof keep the claim INCONCLUSIVE even when another proof is valid (_observations / _claim); docs note order independence.
- assessment_id maps malformed nested projection to declared InvalidRequest.
- New tests: test_good_proof_cannot_hide_another_declared_evidence_gap (order x gap kinds) and malformed identity projection.
Also re-checked offline port boundaries (no Apps/Harness/Knowledge imports) and installed-package probe from earlier tip.
Ready full-check in progress on post-#556 base at review time — steward merge after green + write+ independent Approve (Dong tip Approve was dismissed by the push).
|
@amy-why-3459 @Dong1017 已采纳多份证明缺口的意见:head 核心交叉组 141 passed、0 skipped,Ruff/whitespace 通过。按主线同步建议,以普通 merge 纳入已合入的 #556 和当前 main @amy-why-3459 @david6666666 请复审新 head。Dong 是 accountable parent,他的技术批准仍不计两份独立审批。 |
Dong1017
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed base 10ba7a5500bcd95949219b138f9ada5033b922b0 and head 4887f1e94f39c1d4b675e965cfcc0a6e932d50a2 (full diff, diffComplete: true). This is a Dong-bot (authorized delegate) technical owner review. It is not one of the two independent write+ approvals the PR requires, because Dong1017 is the accountable parent. I found no outstanding CHANGES_REQUESTED in prior-reviews.md. My earlier Approve on 4258c76a was dismissed by the push, so this is the re-review on the new head.
Earlier blocker (amy-why-3459): a good proof hiding an unreadable or invalid second proof. Resolved on this head.
-
_observationsnow recordsproblemsfor every matching check, whether or not other proofs parsed:- an untrusted or incomplete check;
- an empty
evidence_refs; - an unreadable ref;
- an integrity failure;
- an invalid proof.
-
_claimmarks the check missing whennot proofs or problems(previously, only whenproofswas empty). Valid refs and gap reasons are kept in the claim, and the other claims are not affected. -
test_good_proof_cannot_hide_another_declared_evidence_gapcovers:- the three portable error codes, plus wrong bytes and invalid JSON;
- the same check with two refs, and two checks with the same
check_id; - an empty
evidence_refs; - both orders.
It asserts
INCONCLUSIVE, themissing_evidence == ["format"]list, retention of the good refs, andunknowneligibility. -
production-assess.mdnow says "Every declared proof for a matching check must be usable" and that the result does not depend on order. The contract and the code agree.
assessment_id hardening. assessment_id wraps _projection and catches KeyError, TypeError, CanonicalError and PreparationError. It re-raises them as AssessmentError("InvalidRequest"). test_public_identity_reports_malformed_nested_projection_as_declared_error covers five malformed shapes. Host adapter exceptions on open still propagate; that is pinned by test_host_programming_errors_are_not_disguised_as_missing_evidence.
Re-checked from the earlier review.
- Module boundary:
benchmark.assessmentimports onlycontracts.json_v1andbenchmark.preparation. The-I -Bprobe intests/check_installed.pyasserts that no apps, harness or knowledge modules load. - Binding runs before any artifact read:
test_binding_mismatch_is_refused_before_any_artifact_open. - The three claims are independent, and V21, V22 and V23 are covered.
- The T35/T39 Apps exit-code paths (0/8/9, exit 2, exit 6 with the in-memory result kept) are covered.
- The migration inventory lists
tests/test_benchmark_assessment.pyfor the canonical modules it imports.
Not verified here (no execution).
- I did not run the 141-test core group, the Ruff and whitespace checks, or the fresh-wheel check. Those figures come from the author.
- Ready CI on this head was still running. The author does not claim it passes. It must be green, including
full-checkafter the #556 merge, before squash. - The merge also needs two eligible independent write+ approvals, from Amy and David. cursor[bot]'s Approve does not count toward that, since its collaborator permission is
none. - This does not change the stated scope: it is an offline assessment of retained fixtures. It does not accept #429, #371 or #434 jointly.
Non-blocking: if the production protocol ever needs "any one valid proof is enough" semantics, that would be a deliberate contract change. It should come with its own acceptance, not a silent relaxation of the current strict rule.
Posted by Dong-bot, an automated agent acting on behalf of @Dong1017 (always-on-bot, model claude-sonnet-5-5).
amy-why-3459
left a comment
There was a problem hiding this comment.
LGTM
PR review verdict — PASS
Reviewed 4258c76a84f3c9cf7086a0b7965a7ec17cd445f9 → 4887f1e94f39c1d4b675e965cfcc0a6e932d50a2 (incremental, including the merge of current main 10ba7a5500bcd95949219b138f9ada5033b922b0).
The earlier blocker is resolved on this head:
_observationsnow records a problem for an emptyevidence_refslist, and still records unreadable, integrity, and invalid-proof gaps even when another proof parsed._claimtreats a matching check as missing evidence whenproblemsis non-empty, not only whenproofsis empty. Verified refs are kept; other claims stay independent.test_good_proof_cannot_hide_another_declared_evidence_gapcovers two refs on one check, two checks with the samecheck_id, both orders, the three portable error codes, wrong bytes, invalid JSON, and empty refs, and asserts INCONCLUSIVE /missing_evidence == ["format"]/ retained good refs.production-assess.mdnow states that every declared proof for a matching check must be usable and that order does not change the result.
I also checked the assessment_id() InvalidRequest mapping for malformed nested projections, and the merge with main (#544, #547, #556). The PR-versus-main diff remains the six assessment-port files; the previously unmerged test_recovered_attempt_does_not_trust_a_rewritable_declaration fix is in this head via #556.
Accuracy/performance comparison evidence is still not applicable (offline assessment of retained fixtures; no service-gain claim). Dong1017's owner review should not count as one of the two independent approvals.
Note: Ready full-check on this head was still running when I reviewed (run 44900119820); lint-and-smoke, installed-package, dashboard-browser, and declarations had passed. Merge should wait until the Ready lane is green.
|
@david6666666 请补充当前 head 当前源码扩大回归 435 passed、332 subtests passed、0 skipped,fresh sdist→wheel 隔离安装通过;Ready CI 等两版 full-check 完成,其余安装/浏览器检查已通过。请重点看每个声明 ref 的缺口是否保留、F/C/M 独立性、V21/V22/V23,以及接口没有执行副作用;有意见我先处理。 |
|
@david6666666 当前 head |
|
@jiangkuaixue123 补请你独立检视当前 head 当前 head Ready CI 已全部通过,细项本地回归与 fresh wheel 结果见正文。Amy 已正式批准;原 David 请求仍保留。Dong 是本切片的 accountable parent / owner,他的批准不计独立票,作者也不自审。请按实际 diff 和接受范围给出 PASS / DO NOT PASS;符合时正式 Approve 的首个非空行请独立为 |
|
@hsliuustc0106 补请你独立检视本 PR。当前 head 的 Ready CI 全部通过(3.10 / 3.11 各 5980 passed、31 skipped、674 subtests),核心/边界回归与 fresh wheel 结果见正文;Amy 已同 head 正式批准。现有 David / jiangkuaixue123 的请求仍保留。请按实际差异给出结论,有意见会先处理。 本切片 parent 是 Dong1017,他的技术批准不计独立审批;作者不自审。需要另一位独立 write+ 评审正式批准后才可按既定顺序合入。 |
|
当前 本 PR 还卡住 #566 的同步集成;新的 H capture 已是 Ready #567,真实证据闭环的剩余任务被明确跟踪为 #565。没有把后续实现或 fixture 测试偷算成 #434/#371 完成,也不会绕过两份独立批准合入。 |
Refs #434
Refs #371
Refs #430
Apps can now assess a fixed retained P1 record through the production Benchmark port instead of an injected result fixture. The new offline service verifies protocol/record binding, actual retained candidate bytes and trustworthy check proofs, then returns separate format (F), correctness (C) and measurement-validity (M) claims. Apps retains primary-key selection, archiving and publication, including exits 0/8/9 and archive-failure exit 6 with the in-memory result preserved.
Authority: #434 limited authorship delegation and final interface record. Dong1017 retains bounded delivery accountability and port ownership; zengchuang-hw / Codex authors the implementation. This does not assign S3 module ownership. The implementation baseline is main
9466bc8d910c9fef55d4a9e23ee1a82950280c30, including merged #530/#537/#542; current integration base is main10ba7a5500bcd95949219b138f9ada5033b922b0(including the #556 recovery-test correction), merged without rewriting branch history. Head:4887f1e94f39c1d4b675e965cfcc0a6e932d50a2.Frozen task and public ports
The same accepted D023 /
t108-v4family drives preparation and assessment:p1-fixture@1 / p1-rules@1 / evaluate / p1, explicitly linked in the merged prepare contract. Public ports arebenchmark.assessment.assess(prepared, evidence, *, artifacts), pureencode/decode/validate_assessment, andAssessmentService(clock=...)for reproducible assessment timestamps. See production-assess.md.Benchmark imports no Apps/Harness/Knowledge module, launches no worker/process, uses no network/device, and writes no archives. The installed-wheel check now permanently verifies the public offline port can import independently in a new
-Iinterpreter. The three new canonical Apps test consumers are registered in the migration inventory.Validation and limits
The CPU cross-tests run the real prepare/assess ports and Apps archive/read/publish code over retained software-test record/proof fixtures. They do not claim a production Harness evaluation worker execution, a service/model gain, or completion of #429/#371. Production shared CPU acceptance must still consume #429's actual evaluation target output. Keep #434 open for that joint acceptance after port integration.
Review and integration
Please inspect frozen P1 trust/eligibility rules, the complete-record binding, per-item failures versus top-level errors, pure identity projection, and Apps T35/T39 ordering. Dong1017's owner technical review is requested but cannot count as an independent approval, because he is the accountable parent for this slice. Obtain two current eligible independent write+ approvals (requested from Amy/David; first non-empty line
LGTM), passing required checks and no blockers before squash integration. Author/self-review does not count.