Repository navigation
fix(benchmark): keep comparison reports when host evidence fails - #551
Inference1 wants to merge 1 commit into
Conversation
Report an unverifiable native receipt as a blocked trial and retain the verified trials. Reuse the report-time host views when selecting the next action. Agent-assisted-by: Codex (author) Signed-off-by: Inference1 <3617237115@qq.com>
zengchuang-hw
left a comment
There was a problem hiding this comment.
NO PASS
首轮独立检视:9466bc8d910c9fef55d4a9e23ee1a82950280c30 → 09f88baef34e43939a25c30e64b65a8f153249b3。按 #536 的缺失证据口径审阅;保留另一臂并将配对结果置为未知的方向正确。
批准前必修:结构不完整的原生回执引用仍会中止整份报告。 _host_payload L413–417 直接访问 receipt["path"];非空 last_observation 缺少 path 时会抛 KeyError,报告的容错分支 只捕获 OSError / ValueError,不会生成另外一臂的报告。请在读取原生证据的边界校验结构,并将这类损坏作为单臂证据缺口;不要放宽冻结集合或直接 collect 的完整性校验。补测结构损坏场景下三种报告仍生成、坏臂 blocked、完好臂保留且配对值未知。
实际验证:独立执行该 head 的 _host_payload,确认缺路径会抛出未捕获的 KeyError,并核对生产调用链和新增回归源码。未运行完整 CLI 报告回归,未看 CI。
|
Thanks for keeping verified comparison arms visible when one native receipt fails. I reviewed The report-time I also checked the current-head review from @zengchuang-hw and agree it is still blocking:
Please validate the receipt structure at this boundary and treat a malformed observation as a per-trial evidence gap, without relaxing frozen-collection or direct I am not approving this head. |
Governing Issue and delivery slice
Refs #536
This proposed follow-up applies the missing-evidence rules agreed for versioned metrics to the retained native comparison workflow from #501.
Problem
benchmarks compare --action reportaborts when a single retained native receipt is missing or fails verification. That prevents publication of JSON, Markdown and HTML even when the other trial's evidence is intact.What
Keep the affected trial as blocked with an explicit evidence gap. Publish the verified trials in all three views and leave the formal paired ratio unknown when a required arm is blocked.
Why needed
Each comparison arm can contain several completed evaluation rounds and an independent final run. Keeping verified arms visible preserves their diagnostic value and identifies the receipt that needs attention before the comparison can support a conclusion.
Design and implementation
Apps uses the existing native status verifier and reads its retained observation for each trial. Expected read and verification errors produce a blocked row in report generation. The same host views supply the report response's next action, so a post-publication read cannot abort that response. An uncollected blocked trial points to receipt inspection. Direct collection, frozen inputs and collection checksum checks still enforce their integrity contracts.
Checks and dependencies
9466bc8and pass with this change. They cover a missing receipt after collection and a changed receipt before collection, including the surviving arm and unknown paired result.git diff --checkpassed.9466bc8already failstest_project_explanation.py::test_recovered_attempt_does_not_trust_a_rewritable_declarationdue to its import of the removedinferrsi.infra.artifactsmodule.Remaining work and Issue closure
This draft requests maintainer scope confirmation and formal review of the report follow-up. Metric acceptance remains with the governing Issue owner.