[Intelligence Research Project (sunghunkwag)]

What this adds

When a multi-edit candidate passes the Sleep gate, the report shows which validation tasks moved, but not which edit moved them. The maintainers' reply in #174 left that open: "true per-rule causal attribution would require separate ablation runs and can be explored as a follow-up." This PR is that follow-up.

With gate_edit_ablation: true (default false), each skill or memory candidate that passes the gate is scored again on the same validation tasks once per edit, each time without that one edit. Every edit gets a row on its trial: the gate score with and without the edit, an effect of positive / none / negative / unmeasured, and the validation tasks whose score differs.

The rows are report-only. No gate decision reads them, so accepted edits and documents are the same with the option on or off.

Example

The #174 shape: one edit lifts a task, another lowers a different task, and the slice mean still rises (0.400 → 0.850). This is what the report renderer produces for that case (the scenario in test_ablation_names_the_edit_behind_a_hidden_regression), shown as source:

### `skill` candidate — accepted (0.400 → 0.850)

| Task | Tags | Baseline | Candidate | Change |
|---|---|---:|---:|---|
| `essay` | essay | 0.000 | 1.000 | improved |
| `router` | router | 0.800 | 0.700 | regressed |

#### Leave-one-out edit ablation

_Each row scores this candidate again without one of its edits, on the same validation tasks. Every comparison is a single replay, so a nondeterministic backend or judge can move a row by noise._

| Edit | With | Without | Effect | Task changes |
|---|---:|---:|---|---|
| 1. \[add\] Rule A: answer the question that was asked. | 0.850 | 0.350 | positive | `essay` 0.000 → 1.000 |
| 2. \[add\] Rule B: always open with a fixed heading. | 0.850 | 0.900 | negative | `router` 0.800 → 0.700 |

The task table already shows that router regressed. The ablation table adds which edit it came from, and that the candidate scores 0.900 without it.

Behavior and cost

• Default off. No extra replay, and report.md, report.json and the CLI --json output are unchanged. The only default-visible change is a gate_edit_ablation: false field in diagnostics.json and in the logged cycle config. A test compares the option on and off: the same decision, documents, edit lists and trial rows, apart from the added edit_ablation key.
• One-edit candidate: no extra replay. The trial already compared the document with and without that edit, so the row reuses it.
• N-edit candidate: up to N validation replays, tagged gate_ablation:<target> in the evidence log. Once max_tokens_per_night is spent they are skipped, the step is listed once in budget_skipped, and the rows are unmeasured.
• Rejected trials are not ablated, including trials blocked by gate_no_regression.
• An edit that another edit of the same trial overrides is reported as none without a replay, because the document is the same without it.
• Edits that only matched through the removed edit (for example a replace anchored on a line that an add in the same trial created) are listed as co_dropped, so the row is not read as the effect of one edit.
• A missing or non-finite ablation score is unmeasured, with no score and no per-task direction.
• gate_mode: off has no validation replay, so the option has no effect there.
• Rows travel through the existing gate_trials path, so they reach report.md, report.json, diagnostics.json, the CLI --json output and the evidence log with the existing redaction. In multi-skill fan-out each group's rows are in the evidence log; group trials are not in report.md today and I did not change that.

Limits

• Each comparison is a single replay. With a nondeterministic backend or judge, a row can differ by noise; the report says so above each table. Repeating each replay would be a separate change.
• Leave-one-out does not measure interactions. Two edits that are each redundant given the other both show none.
• I ran this with the mock backend and stubbed replays only. I have not run it against a live model backend.

Tests

• tests/test_gate_edit_ablation.py, 21 test cases: the default-off path, the two cases above, memory trials, the no-regression gate, the single-edit, overridden-edit and co-dropped paths, a spent budget, NaN / inf / missing scores, the evidence log, report rendering, CLI redaction, a full mock night, and multi-skill fan-out.
• python -m pytest -q: 2118 passed, 13 skipped, 4 xfailed (Python 3.13, project dependencies plus the parallel extra).
• ruff check on the changed files: clean.
• python -m mkdocs build --strict: passes.

Docs: docs/sleep/README.md, the advanced-config key list in plugins/README.md, and CHANGELOG.md. I left the per-plugin SKILL.md key lists alone.

🤖 Generated with Claude Code

https://claude.ai/code/session_015s4y5QVk9hfpeGNR7dzrHb

[Intelligence Research Project (sunghunkwag)]

@microsoft-github-policy-service agree