diff --git a/RESEARCH.md b/RESEARCH.md index 0da17fd..f2c60af 100644 --- a/RESEARCH.md +++ b/RESEARCH.md @@ -43,7 +43,7 @@ Last reviewed: 2026-08-04. **Implementations.** - PR #15 (Reframe self-review as rubric-based and CI-grounded): shipped 2026-05-25. Observed effect: pending — needs N cycles of data. First applied to PR #26 (one cycle, all rubric items PASS); too early to tell whether the rubric catches real defects that free-form review would miss. -- PR #28 (Generalize the Phase 4 external-signal step to support no-CI repos): shipped 2026-05-25. Observed effect: pending — needs N cycles of data. Replaces the hardcoded "wait for CI green" anchor with `{{EXTERNAL_SIGNAL_LABEL}}` / `{{EXTERNAL_SIGNAL_CMD}}`, so a repo without CI still anchors the rubric on *some* external signal instead of falling back to unanchored judgment — what this finding requires is that an anchor exists, not that it is specifically CI. `create-dev-loop` itself is the case in miniature: it gained CI in PR #65 (2026-07-27), but that workflow only runs `scripts/check_docs.py`, so the anchor for a *behavioral* change here is still the manual `/create-dev-loop` validation checklist rather than the green check. +- PR #28 (Generalize the Phase 4 external-signal step to support no-CI repos): shipped 2026-05-25. Observed effect: pending — needs N cycles of data. Replaces the hardcoded "wait for CI green" anchor with `{{EXTERNAL_SIGNAL_LABEL}}` / `{{EXTERNAL_SIGNAL_CMD}}`, so a repo without CI still anchors the rubric on *some* external signal instead of falling back to unanchored judgment — what this finding requires is that an anchor exists, not that it is specifically CI. `create-dev-loop` itself is the case in miniature: it gained CI in PR #65 (2026-07-27), but that workflow only runs `scripts/check_docs.py` and that script's own unit tests, so the anchor for a *behavioral* change here is still the manual `/create-dev-loop` validation checklist rather than the green check. - PR #34 (Structure Phase 9 self-audit prompts and label gap issues): shipped 2026-05-25. Observed effect: pending — needs N cycles of data. Extends the rubric-over-free-form pattern from Phase 4 (self-review) to Phase 9 (self-audit), with a 6-item structured checklist and a 5-label taxonomy for filed gap issues. - PR #54 (Apply consolidated template-rule backlog, #44/#46/#47): shipped 2026-06-14. Observed effect: pending — needs N cycles of data. Sharpens the external-anchor definition (#44: UNVERIFIED handling when the anchor can't run; #46: green CI is not verification when CI's scope excludes the changed files) and grounds the Tests-fix rubric item empirically (#47: stash-and-run FAIL→PASS, not scored from reasoning alone) — directly targeting the self-preference failure mode this finding's evidence documents for judgment-based rubric scoring. - PR #57 (Fix documentation drift across the three sources of truth): shipped 2026-07-25. Observed effect: pending — needs N cycles of data. Restores `{{EXTERNAL_SIGNAL_LABEL}}` in the Phase 1 carried-over-PR bullet, where PR #54 had reintroduced a hardcoded "CI must be green first" after PR #28 generalized the anchor. An anchor named `CI` in a repo that has no CI is an anchor the loop cannot check, which degrades the self-review to exactly the unanchored critique this finding shows is neutral-to-harmful. @@ -204,7 +204,7 @@ Last reviewed: 2026-08-04. - Consider periodic regression checks: dry-run the generated skill against a known-good fixture repo and compare output to a recorded baseline. Any repo you maintain works as the fixture — the value is in the recorded baseline and the diff against it, not in which repo produced it. **Implementations.** -- PR #33 (Template-version header for drift detection): shipped 2026-05-25. Observed effect: pending — needs N cycles of data. Adds `` and `` HTML comments to the top of every newly generated skill, plus `{{TEMPLATE_VERSION}}` and `{{GENERATED_AT}}` rows to the Step 4 substitution table. Implements the second bullet of this finding's implications; the periodic regression-check (third bullet) is a separate future change. Acting on a detected drift is also deferred — that step belongs in this repo's own generated dev-loop skill (its Phase 9 self-audit), not in the template here. +- PR #33 (Template-version header for drift detection): shipped 2026-05-25. Observed effect: pending — needs N cycles of data. Adds `` and `` HTML comments to the top of every newly generated skill (they have since moved directly below the H1 heading, the fix for #36), plus `{{TEMPLATE_VERSION}}` and `{{GENERATED_AT}}` rows to the Step 4 substitution table. Implements the second bullet of this finding's implications; the periodic regression-check (third bullet) is a separate future change. Acting on a detected drift is also deferred — that step belongs in this repo's own generated dev-loop skill (its Phase 9 self-audit), not in the template here. - PR #35 (Update mode — re-derive existing skills from current template): shipped 2026-05-25. Observed effect: pending — needs N cycles of data. Adds an `update` mode to `/create-dev-loop` so the retrofit work that CLAUDE.md's promotion checklist currently mandates can be driven by the skill itself rather than by hand. Complements PR #33's drift detection — #33 detects drift; PR #35 is the remediation channel. ---