Skip to content

merge(stats): land paired statistics - #52

Merged
LittleCoinCoin merged 105 commits into
mainfrom
feat/m6a-paired-stats
Nov 18, 2026
Merged

LittleCoinCoin merged 105 commits into
mainfrom
feat/m6a-paired-stats

Conversation

@LittleCoinCoin

Copy link
Copy Markdown
Member

Lands milestone M6a of candler-v1 (leaf stats/m6a_paired_stats.md, amended by A9 and A10) from feat/m6a-paired-stats, campaign __roadmap__/m6a-paired-stats/.

What changes

  • Blocked tests in analyze (A9). Each declared factor's level pair is compared with the other declared factors as blocks:

    • Fisher mid-p for one block;
    • CMH without continuity correction, the Mantel–Haenszel odds ratio with its RBG interval, and an exact conditional fallback chosen by the Mantel–Fleiss criterion;
    • a stratified permutation test with a stratified BCa interval for weighted outcomes.

    Holm is applied per comparison per role. Every difference is B − A. Each test equals an offline second engine (scipy 1.17.0, statsmodels 0.15.0).

  • Treatment as a factor (A7). request.treatment can be declared next to model.id. Each cell resolves its own treatment, env lock and image; cells with equal inputs share one. Every other setting factor is still refused (I7).

  • Importer. An import declares request.treatment when the typed treatments differ. It drops value paths that map one to one onto it, so the golden runs declare exactly plan.model and request.treatment.

  • Per-arm instrument_error is reported, with the other invalid causes unchanged.

  • Design record. __design__/architecture.md §Analysis statistics records the blocks, the execution order, the validity sources and the importer rule.

Evidence (campaign gates, L3 handover)

Gate Result
STD (810 tests, fmt, clippy, doc), G8 known truth pass at fb691ef
G2 golden design, G12 import, G3, DIGEST, GOLDEN, G10, LINUX pass at 74f12f2 (only one test file and one design sentence changed after)
SUBJ pass from 9560cd3 to 9a5f836

No cell digest, golden or parity fixture moves. There are no candler-schemas commits, and codes.toml has no removed line.

🤖 Generated with Claude Code

LittleCoinCoin and others added 30 commits November 18, 2026 11:30
Lay down every file and append-only line the seven L0 leaves share, so
that no two siblings edit the same line and no leaf waits on another's
code:

- empty analysis modules midp, exact_strata, cmh, permutation, bca and
  blocking, with their mod lines;
- empty guard modules guard_stats_{exact,cmh,resampling,blocking} and
  guard_known_truth, with their mod lines in tests/it/main.rs;
- one marked gen_stats.py section per vector-writing leaf (exact_tests,
  cmh, resampling), each extending FILES;
- the Stratum type, the block every blocked test takes (C1);
- SplitMix64 complete (new, next_u64, below by rejection), which both
  resampling and execution_order use at L0.

GEN still prints six "rows match" lines; clippy, fmt and doc pass on
candler-core.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The leaf names the mid-p definition and the exact conditional test as
the methods to implement. This records the sources and the definition,
and that the sources could not be re-read offline.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
midp.toml holds scipy hypergeom mid-p values for every table with up
to 12 runs per arm plus larger samples; exact.toml holds the mid-p of
the convolution of 2 to 4 blocks. The generator checks each row against
exact rational arithmetic and brute-force enumeration, and asserts no
near-tie falls inside the tie tolerance.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Two guards read midp.toml and exact.toml and compare to a relative
1e-12. The modules hold stubs that return an error, so both guards fail
for the right reason (the statistic is not computed).

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The one-table mid-p and the exact conditional test of K blocks share a
block pmf (statrs hypergeometric, point mass for a zero margin) and one
two-sided mid-p rule with Fisher's tie tolerance. fisher_two_sided is
unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Three injected bugs (the half dropped, a block left out, ties by float
equality) each turn a guard red with the recorded message. The report
also lists the findings: no pmf helper in fisher.rs to share, sources
not re-read offline, and the 0.5 of a degenerate table.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Leaf exact_tests of m6a-paired-stats (L0). Candler computes Fisher's
two-sided mid-p for one 2x2 table and the exact conditional test of a
2x2xK table (the convolution of the blocks' hypergeometrics, as a
mid-p), each equal to scipy's hypergeom and to exact rational brute
force over every margin set up to 12 runs.

Integration: GEN --check prints eight "rows match" lines (midp.toml
8449 rows, exact.toml 1328), fisher.toml unchanged, guard_stats_exact
and guard_stats::t1_ pass (10 tests), ownership check empty.

fisher.rs has no pmf helper to share, so it is untouched; midp.rs
holds the shared block pmf and tie rule. A degenerate table returns
0.5, which analyze_output must not report as evidence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The permutation p-value, the within-block exchangeability and the BCa
formulae each rest on a primary source; the leaf report names them and
flags that they were cited from knowledge, not re-read, for want of a
network.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The resampled statistics and the run order draw from one generator that a
Python twin reproduces bit for bit; `below` rejects instead of taking a
modulo so large moduli stay unbiased, and `shuffle_within` is a
Fisher-Yates shuffle.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Labels are shuffled within each block only, so the null of no difference
in any block is the reference set; p is (b + 1) / (B + 1), counting the
observed labelling once.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Runs are resampled with replacement within each arm of each block; the
bias correction and the leave-one-run-out jackknife acceleration follow
Efron (1987). The normal distribution function is a series accurate to
1e-16, since the library one is off by up to 1e-11.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The jackknife held the block weights fixed in neither language while the
bootstrap does; both now hold them at the observed values. A row whose
ties are only approximate now depends on the tolerance, and a skewed
block lets scipy tell a flipped or zero acceleration from the right one.
The z0 tie choice and the exact-equality degenerate test are documented.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The four bites, the first verifier round and its fixes are recorded; the
re-check verdict is still open.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The re-check found F1 to F6 closed; the multi-block BCa has only the twin
as its engine, and the report says so.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The statistic, the odds ratio, the RBG interval, the Mantel-Fleiss criterion and the
homogeneity test each need a named source before code. No network was available, so the
report states each formula and ties it to the statsmodels check.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
statsmodels writes the statistic, the pooled odds ratio, its interval and the Breslow-Day
test; the Mantel-Fleiss quantity is counted by hand in exact rationals. Degenerate rows
record what Candler returns, and the generator asserts statsmodels agrees where finite.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Five guards read cmh.toml: the statistic, the odds ratio with its interval, the
Mantel-Fleiss count, the homogeneity test and the degenerate-block convention. They fail
against a stub that computes nothing.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The statistic, the Mantel-Haenszel odds ratio with the RBG interval, the Mantel-Fleiss
quantity and the Breslow-Day test, each equal to statsmodels. Blocks with a zero margin are
dropped; the module doc states the convention for every degenerate case.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A criterion of exactly 5 could sum to a few ulps below 5 over several blocks. The vectors
now carry a four-block row on 5, rows at levels 0.90 and 0.99, and an odds ratio of
exactly 1 where statsmodels divides by zero. The Breslow-Day root uses the stable form.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Each injection that turned a guard red, the four verifier findings and their fixes, and
the zero-cell convention are in the leaf report.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The weighted pass rate needs a test and an interval that respect the
blocks. This lands the seeded SplitMix64 with its Python twin, the
stratified permutation test with p = (b+1)/(B+1), and the stratified
BCa interval with fixed jackknife block weights.

A fresh verifier found six gaps in round one; all six are fixed or
recorded and its re-check closed them. The multi-block BCa has the
twin as its only engine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A binary outcome over two or more blocks is tested by CMH without
continuity correction. This lands the statistic, the Mantel-Haenszel
odds ratio of B over A with its RBG interval, the Mantel-Fleiss
criterion and the Breslow-Day homogeneity check.

A fresh verifier found four gaps; all are fixed and its re-check
closed them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The treatment is the one setting that resolves into a hashed typed
root, so an experiment may declare it as a factor. Three rows pin the
change: the treatment factor is not refused as a setting, every other
setting (and the request parent table filling the treatment) still is,
and the experiment digest of a one-treatment file keeps the literal
computed before the change.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The treatment resolves into the hashed typed root `treatment`, so a
factor on `request.treatment` varies no setting that a cell digest
would lose. `setting_of_factor` exempts exactly that path, named
`TREATMENT_FACTOR`; every other setting, and a parent table that fills
the treatment, is still refused.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The report lists the refusal rows, the bites that turned the rows red
and the two that cannot, and the CLI preflight answer: check_factors
shares the plan's refusal, so it now admits a treatment factor and a
two-level plan collapses to one cell until the level resolve lands.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Decision T2 lets an experiment compare treatments as a declared
factor. The plan's refusal now exempts exactly request.treatment
when a value level sets it. Every other setting, and the request
parent table, stay refused, and a one-treatment file's digest is
unchanged.

The level cannot reach the cell yet: fresh_cells drops every
setting path, so plan_runs gives one collapsed cell for two
treatment levels, and the CLI preflight shares the plan's refusal.
level_resolve closes this in the next level, under the ruled
narrowing of this leaf.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Four rows state the properties of the order, not one frozen sequence: each
round holds each cell once, a seed fixes the order, another seed changes it,
and plan_runs seeds from the experiment digest.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
run_order shuffles the cells of each replicate round with the one
SplitMix64 generator, seeded from the first 8 bytes of the experiment
digest, so independence of runs follows from the design.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
plan_runs iterates run_order instead of the nested cell and replicate
loops, so one function decides the order and no copy of the old loop
remains. Run keys and digests are unchanged. run_order and seed_of are
re-exported from runner; the module stays private.

Two guards pinned the declaration order by position, not by what they
check. b27 now finds each plan by its cell digest. b34 reads the pair of
runs in flight from the runs that ended (max_parallel of the unsealed
ones), since the planned order now picks them. What each guard checks is
unchanged.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The report records each injected bug and the guard it turned red, the
two guards changed under the coordinator ruling, and the Scheduler code
that keeps the planned order at execution.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
LittleCoinCoin and others added 22 commits November 18, 2026 19:46
The golden design plans 12 cells and analyze gives 6 version comparisons
blocked by model and 3 model comparisons blocked by version, each equal to
the second engine. The plan row and its experiment file ride with it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Each bite was injected, seen red and restored from a copy; the report also
holds the planted size, its power and the finding that the prefix-object
guard of level_of is unreachable through analyze.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The HTN golden design (3 models x 4 versions x 20 runs) plans 12
cells from one experiment file, and analyze on a store of that shape
gives 6 version comparisons blocked by model and 3 model comparisons
blocked by version, each equal to an independent second engine.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
golden_design found that check_factors refuses the prefix factors
level_of's object branch would read, so the branch is dead behind it.
The coordinator ruled a small leaf to confirm the callers and delete
it, and to state the pair orientation in R01.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The branch returned an object of the values below a declared factor
that is a strict prefix of a recorded path. check_factors refuses such
a factor before any level is read, and the one production path
(analyze) runs it first, so the branch never ran. The level is now the
value at the factor's path, or null.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The report lists each caller and the check_factors that precedes it, and holds the orientation sentence on a spec defect.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A reader of a comparison needs to know which level the B - A
difference subtracts; it follows the recorded key order.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
check_factors refuses a declared factor that is a strict prefix of a
differing path, and every production caller runs it before level_of,
so the object-under-prefix branch could never run. R01 now says which
level of a comparison is A.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The probe found one pinned imported factor list and one hand-declared
factor set; no cell digest, experiment digest or oracle file reads the
factor list.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
An imported experiment now declares request.treatment when its cells
treatments differ, and no value path whose levels map one to one onto
them, so an imported store and its native rerun declare the same
factors. The verdict of an all-imported stratum drops such a path too,
over every arm; live strata are unchanged. maps_one_to_one lives in
manifest/compare.rs and the importer shares it.

Changed rows: guard_legacy_mapping the_mapped_runs_are_the_oracle_runs
and guard_comparability imported_stratum_of_the_mini_corpus_is_comparable.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
An import now declares request.treatment when the cells carry
different typed treatments and drops the value paths that only
restate it, so the golden import declares the model and the
treatment like its native rerun. Imported strata explain those
paths (A12); live strata keep I7. No digest moves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The weighted H0 row rejected at about half of alpha and was relaxed
to a one-sided bound. A rate near alpha/2 is a finding about the test
or the row, so the coordinator ruled the relaxation reverted and the
counted p traced before any gate moves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The weighted H0 rate is 0.024 with 5 weight values and nominal on a
grid of 1000: ties in the statistic, not the engine. G8 keeps the
two-sided interval on the binary, one-block and tie-free weighted rows
and an upper bound on the tied weighted row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Seeded simulations draw stores in the shape analyze reads and check the
blocked tests against planted truth: the binary H0 rate holds 0.05, the
Simpson H0 case holds blocked and breaks pooled, the weighted H0 rate does
not exceed alpha, and the power at a 0.2 difference is reported.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The golden shape (4 treatment levels by 3 models, 4 replicates) is planned
and run on the existing fakes; planted outcomes go through the real report,
which gives 6 treatment comparisons blocked by model and 3 model comparisons
blocked by treatment.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The weighted H0 row counts the raw permutation p of its one quality
expectation and still rejects at about alpha/2 with five weight values
(0.024 at 6 runs per cell, 0.025 at 10). With weights on a grid of 1 000
the same code rejects at 0.059 (0.056 and 0.045 in the probe), so the
code is not at fault: the statistic ties, and an exact permutation test
on tied data is conservative. The row keeps its two-sided gate and is
red; the evidence and the ruling it needs are in the report.

Add a continuous-weights H0 row and a one-block binary H0 row. Fisher's p
in place of the mid-p on the one-block path moves that rate from 0.0447
to 0.0185 and turns the row red. The fakes seal runs as
invalid(infrastructure) because the fake run leaves no container to
inspect: the fakes, not the runner.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
With 5 weight values the permutation statistic ties and the H0 rate is
0.024, conservative and valid; the row now checks the upper limit of
the 99 % interval only, as A15 rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Known-truth rows for the battery under H0 (binary, one block, tie-free
and tied weighted), the Simpson case, power at a 0.2 difference and two
treatment levels end to end on fakes. The tied weighted row is an upper
bound, as A15 rules, and the design record states why.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All five L3 leaves are merged and done; G2, G12, G3, G8, STD, DIGEST,
GOLDEN, G10, LINUX and SUBJ pass at fb691ef. The Progress rows record
each merge and its open findings.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LittleCoinCoin added a commit that referenced this pull request Nov 18, 2026
`main` has been red since 9560cd3 (#51, roadmap-only): 20 candler-core
guards fail on CI with "`.claude/skills/htn/SKILL.md` is not in commit
<main's HEAD>". The code at #50, the last green run, is the same.

## Cause

`Swatinem/rust-cache` saves `target/` after deleting every file outside
the build profiles. It keeps the directories (cleanup rule: "Files other
than `CACHEDIR.TAG` are removed").

So the cache restores the shared fixture `target/tmp/job_skill_repo` as
empty folders, `.git` included. `skill_repo()` only checked that `.git`
exists, so it reused the skeleton. git did not recognise the skeleton as
a repository and searched upward until it reached the checkout's own
repository.

## Fix (test support only)

- The fixture is reused only when `.git/HEAD` is a file.
- A skeleton is pruned by removing empty directories only, deepest
first, so a repository that another nextest process has just placed is
never deleted.
- git runs with `GIT_CEILING_DIRECTORIES` set to the scratch directory,
so a missing repository fails at once instead of reading the checkout's.

## Evidence

With a planted skeleton in an in-repo `target/`:
- the old fixture fails with CI's exact error (2 of 3 sampled guards);
- the fixed one passes 16 of 16 sampled guards;
- the full candler-core suite passes 329 of 329;
- fmt and clippy (`-D warnings`) are clean.

This also unblocks #52, whose 21 failures share
this cause.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
@LittleCoinCoin
LittleCoinCoin merged commit 8e09fcb into main Nov 18, 2026
15 of 16 checks passed
@LittleCoinCoin
LittleCoinCoin deleted the feat/m6a-paired-stats branch November 18, 2026 13:25
LittleCoinCoin added a commit that referenced this pull request Nov 18, 2026
This closes milestone M6a (leaf Step 3), which landed in
#52.

- Ticks the pre-conditions and gates G1 to G12 in
`__roadmap__/m6a-paired-stats/README.md`.
- Ticks the candler-v1 leaf `stats/m6a_paired_stats.md` gates, as A10
rewords them, and marks the leaf `done` with `dirtree-rdm`. `stats/`
stays in progress until M7 (`mcp/`) lands. Both trees pass `dirtree-rdm
validate`.
- Adds `__reports__/m6a-paired-stats/knowledge_transfer_v0.md`,
covering:
  - what shipped and the gates;
- the spend: about $84 in all, with the build at $54.3 against a $105
cap;
  - the amendment classes and the lessons;
  - five follow-ups for Eliott.

This is a docs and roadmap change only.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant