Skip to content

fix(benchmark): keep comparison reports when host evidence fails - #551

Draft
Inference1 wants to merge 1 commit into
InferMatrixLab:mainfrom
Inference1:fix/benchmark-report-missing-receipt
Draft

Inference1 wants to merge 1 commit into
InferMatrixLab:mainfrom
Inference1:fix/benchmark-report-missing-receipt

Conversation

@Inference1

Copy link
Copy Markdown
Contributor

Governing Issue and delivery slice

Refs #536

This proposed follow-up applies the missing-evidence rules agreed for versioned metrics to the retained native comparison workflow from #501.

Problem

benchmarks compare --action report aborts when a single retained native receipt is missing or fails verification. That prevents publication of JSON, Markdown and HTML even when the other trial's evidence is intact.

What

Keep the affected trial as blocked with an explicit evidence gap. Publish the verified trials in all three views and leave the formal paired ratio unknown when a required arm is blocked.

Why needed

Each comparison arm can contain several completed evaluation rounds and an independent final run. Keeping verified arms visible preserves their diagnostic value and identifies the receipt that needs attention before the comparison can support a conclusion.

Design and implementation

Apps uses the existing native status verifier and reads its retained observation for each trial. Expected read and verification errors produce a blocked row in report generation. The same host views supply the report response's next action, so a post-publication read cannot abort that response. An uncollected blocked trial points to receipt inspection. Direct collection, frozen inputs and collection checksum checks still enforce their integrity contracts.

Checks and dependencies

  • Two CLI regressions fail on 9466bc8 and pass with this change. They cover a missing receipt after collection and a changed receipt before collection, including the surviving arm and unknown paired result.
  • Linux, Python 3.12. Benchmark compare and metrics, report rendering, Agent sessions, Apps delivery and import gates. 321 passed, 2 skipped, 43 subtests passed. The skips require Playwright. Full Ruff and git diff --check passed.
  • Upstream base 9466bc8 already fails test_project_explanation.py::test_recovered_attempt_does_not_trust_a_rewritable_declaration due to its import of the removed inferrsi.infra.artifacts module.

Remaining work and Issue closure

This draft requests maintainer scope confirmation and formal review of the report follow-up. Metric acceptance remains with the governing Issue owner.

Report an unverifiable native receipt as a blocked trial and retain the verified trials. Reuse the report-time host views when selecting the next action.

Agent-assisted-by: Codex (author)
Signed-off-by: Inference1 <3617237115@qq.com>

@zengchuang-hw zengchuang-hw left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NO PASS

首轮独立检视:9466bc8d910c9fef55d4a9e23ee1a82950280c30 → 09f88baef34e43939a25c30e64b65a8f153249b3。按 #536 的缺失证据口径审阅;保留另一臂并将配对结果置为未知的方向正确。

批准前必修:结构不完整的原生回执引用仍会中止整份报告。 _host_payload L413–417 直接访问 receipt["path"];非空 last_observation 缺少 path 时会抛 KeyError,报告的容错分支 只捕获 OSError / ValueError,不会生成另外一臂的报告。请在读取原生证据的边界校验结构,并将这类损坏作为单臂证据缺口;不要放宽冻结集合或直接 collect 的完整性校验。补测结构损坏场景下三种报告仍生成、坏臂 blocked、完好臂保留且配对值未知。

实际验证:独立执行该 head 的 _host_payload,确认缺路径会抛出未捕获的 KeyError,并核对生产调用链和新增回归源码。未运行完整 CLI 报告回归,未看 CI。

Copy link
Copy Markdown
Collaborator

Thanks for keeping verified comparison arms visible when one native receipt fails. I reviewed 9466bc8d910c9fef55d4a9e23ee1a82950280c30 → 09f88baef34e43939a25c30e64b65a8f153249b3.

The report-time OSError / ValueError path, reuse of host_views for next_action, and the two new cases in test_bad_native_receipt_keeps_verified_arm_in_report look consistent with the #536 missing-evidence rule: the intact arm stays in JSON/Markdown/HTML, the formal paired ratio is unknown, and collect() still enforces collection integrity. The uncollected arm correctly points at inspect_retained_host_evidence.

I also checked the current-head review from @zengchuang-hw and agree it is still blocking:

  • _host_payload does receipt["path"] once last_observation is truthy. A non-empty observation that lacks path raises KeyError, which report() does not catch, so the whole report still aborts and the verified arm is unpublished.

Please validate the receipt structure at this boundary and treat a malformed observation as a per-trial evidence gap, without relaxing frozen-collection or direct collect checks. Add a regression that a structurally damaged receipt still produces all three reports, blocks only that arm, and leaves the intact arm and an unknown paired value.

I am not approving this head.

@Dong1017 Dong1017 added the kind:task Bounded delivery task with a delegated slice label Nov 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind:task Bounded delivery task with a delegated slice

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants