Repository navigation
merge(stats): land paired statistics - #52
Merged
Merged
Conversation
Lay down every file and append-only line the seven L0 leaves share, so
that no two siblings edit the same line and no leaf waits on another's
code:
- empty analysis modules midp, exact_strata, cmh, permutation, bca and
blocking, with their mod lines;
- empty guard modules guard_stats_{exact,cmh,resampling,blocking} and
guard_known_truth, with their mod lines in tests/it/main.rs;
- one marked gen_stats.py section per vector-writing leaf (exact_tests,
cmh, resampling), each extending FILES;
- the Stratum type, the block every blocked test takes (C1);
- SplitMix64 complete (new, next_u64, below by rejection), which both
resampling and execution_order use at L0.
GEN still prints six "rows match" lines; clippy, fmt and doc pass on
candler-core.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The leaf names the mid-p definition and the exact conditional test as the methods to implement. This records the sources and the definition, and that the sources could not be re-read offline. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
midp.toml holds scipy hypergeom mid-p values for every table with up to 12 runs per arm plus larger samples; exact.toml holds the mid-p of the convolution of 2 to 4 blocks. The generator checks each row against exact rational arithmetic and brute-force enumeration, and asserts no near-tie falls inside the tie tolerance. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Two guards read midp.toml and exact.toml and compare to a relative 1e-12. The modules hold stubs that return an error, so both guards fail for the right reason (the statistic is not computed). Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The one-table mid-p and the exact conditional test of K blocks share a block pmf (statrs hypergeometric, point mass for a zero margin) and one two-sided mid-p rule with Fisher's tie tolerance. fisher_two_sided is unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Three injected bugs (the half dropped, a block left out, ties by float equality) each turn a guard red with the recorded message. The report also lists the findings: no pmf helper in fisher.rs to share, sources not re-read offline, and the 0.5 of a degenerate table. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Leaf exact_tests of m6a-paired-stats (L0). Candler computes Fisher's two-sided mid-p for one 2x2 table and the exact conditional test of a 2x2xK table (the convolution of the blocks' hypergeometrics, as a mid-p), each equal to scipy's hypergeom and to exact rational brute force over every margin set up to 12 runs. Integration: GEN --check prints eight "rows match" lines (midp.toml 8449 rows, exact.toml 1328), fisher.toml unchanged, guard_stats_exact and guard_stats::t1_ pass (10 tests), ownership check empty. fisher.rs has no pmf helper to share, so it is untouched; midp.rs holds the shared block pmf and tie rule. A degenerate table returns 0.5, which analyze_output must not report as evidence. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The permutation p-value, the within-block exchangeability and the BCa formulae each rest on a primary source; the leaf report names them and flags that they were cited from knowledge, not re-read, for want of a network. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The resampled statistics and the run order draw from one generator that a Python twin reproduces bit for bit; `below` rejects instead of taking a modulo so large moduli stay unbiased, and `shuffle_within` is a Fisher-Yates shuffle. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Labels are shuffled within each block only, so the null of no difference in any block is the reference set; p is (b + 1) / (B + 1), counting the observed labelling once. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Runs are resampled with replacement within each arm of each block; the bias correction and the leave-one-run-out jackknife acceleration follow Efron (1987). The normal distribution function is a series accurate to 1e-16, since the library one is off by up to 1e-11. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The jackknife held the block weights fixed in neither language while the bootstrap does; both now hold them at the observed values. A row whose ties are only approximate now depends on the tolerance, and a skewed block lets scipy tell a flipped or zero acceleration from the right one. The z0 tie choice and the exact-equality degenerate test are documented. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The four bites, the first verifier round and its fixes are recorded; the re-check verdict is still open. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The re-check found F1 to F6 closed; the multi-block BCa has only the twin as its engine, and the report says so. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The statistic, the odds ratio, the RBG interval, the Mantel-Fleiss criterion and the homogeneity test each need a named source before code. No network was available, so the report states each formula and ties it to the statsmodels check. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
statsmodels writes the statistic, the pooled odds ratio, its interval and the Breslow-Day test; the Mantel-Fleiss quantity is counted by hand in exact rationals. Degenerate rows record what Candler returns, and the generator asserts statsmodels agrees where finite. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Five guards read cmh.toml: the statistic, the odds ratio with its interval, the Mantel-Fleiss count, the homogeneity test and the degenerate-block convention. They fail against a stub that computes nothing. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The statistic, the Mantel-Haenszel odds ratio with the RBG interval, the Mantel-Fleiss quantity and the Breslow-Day test, each equal to statsmodels. Blocks with a zero margin are dropped; the module doc states the convention for every degenerate case. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A criterion of exactly 5 could sum to a few ulps below 5 over several blocks. The vectors now carry a four-block row on 5, rows at levels 0.90 and 0.99, and an odds ratio of exactly 1 where statsmodels divides by zero. The Breslow-Day root uses the stable form. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Each injection that turned a guard red, the four verifier findings and their fixes, and the zero-cell convention are in the leaf report. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The weighted pass rate needs a test and an interval that respect the blocks. This lands the seeded SplitMix64 with its Python twin, the stratified permutation test with p = (b+1)/(B+1), and the stratified BCa interval with fixed jackknife block weights. A fresh verifier found six gaps in round one; all six are fixed or recorded and its re-check closed them. The multi-block BCa has the twin as its only engine. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A binary outcome over two or more blocks is tested by CMH without continuity correction. This lands the statistic, the Mantel-Haenszel odds ratio of B over A with its RBG interval, the Mantel-Fleiss criterion and the Breslow-Day homogeneity check. A fresh verifier found four gaps; all are fixed and its re-check closed them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The treatment is the one setting that resolves into a hashed typed root, so an experiment may declare it as a factor. Three rows pin the change: the treatment factor is not refused as a setting, every other setting (and the request parent table filling the treatment) still is, and the experiment digest of a one-treatment file keeps the literal computed before the change. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The treatment resolves into the hashed typed root `treatment`, so a factor on `request.treatment` varies no setting that a cell digest would lose. `setting_of_factor` exempts exactly that path, named `TREATMENT_FACTOR`; every other setting, and a parent table that fills the treatment, is still refused. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The report lists the refusal rows, the bites that turned the rows red and the two that cannot, and the CLI preflight answer: check_factors shares the plan's refusal, so it now admits a treatment factor and a two-level plan collapses to one cell until the level resolve lands. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Decision T2 lets an experiment compare treatments as a declared factor. The plan's refusal now exempts exactly request.treatment when a value level sets it. Every other setting, and the request parent table, stay refused, and a one-treatment file's digest is unchanged. The level cannot reach the cell yet: fresh_cells drops every setting path, so plan_runs gives one collapsed cell for two treatment levels, and the CLI preflight shares the plan's refusal. level_resolve closes this in the next level, under the ruled narrowing of this leaf. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Four rows state the properties of the order, not one frozen sequence: each round holds each cell once, a seed fixes the order, another seed changes it, and plan_runs seeds from the experiment digest. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
run_order shuffles the cells of each replicate round with the one SplitMix64 generator, seeded from the first 8 bytes of the experiment digest, so independence of runs follows from the design. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
plan_runs iterates run_order instead of the nested cell and replicate loops, so one function decides the order and no copy of the old loop remains. Run keys and digests are unchanged. run_order and seed_of are re-exported from runner; the module stays private. Two guards pinned the declaration order by position, not by what they check. b27 now finds each plan by its cell digest. b34 reads the pair of runs in flight from the runs that ended (max_parallel of the unsealed ones), since the planned order now picks them. What each guard checks is unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The report records each injected bug and the guard it turned red, the two guards changed under the coordinator ruling, and the Scheduler code that keeps the planned order at execution. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The golden design plans 12 cells and analyze gives 6 version comparisons blocked by model and 3 model comparisons blocked by version, each equal to the second engine. The plan row and its experiment file ride with it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Each bite was injected, seen red and restored from a copy; the report also holds the planted size, its power and the finding that the prefix-object guard of level_of is unreachable through analyze. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The HTN golden design (3 models x 4 versions x 20 runs) plans 12 cells from one experiment file, and analyze on a store of that shape gives 6 version comparisons blocked by model and 3 model comparisons blocked by version, each equal to an independent second engine. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
golden_design found that check_factors refuses the prefix factors level_of's object branch would read, so the branch is dead behind it. The coordinator ruled a small leaf to confirm the callers and delete it, and to state the pair orientation in R01. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The branch returned an object of the values below a declared factor that is a strict prefix of a recorded path. check_factors refuses such a factor before any level is read, and the one production path (analyze) runs it first, so the branch never ran. The level is now the value at the factor's path, or null. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The report lists each caller and the check_factors that precedes it, and holds the orientation sentence on a spec defect. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A reader of a comparison needs to know which level the B - A difference subtracts; it follows the recorded key order. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
check_factors refuses a declared factor that is a strict prefix of a differing path, and every production caller runs it before level_of, so the object-under-prefix branch could never run. R01 now says which level of a comparison is A. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The probe found one pinned imported factor list and one hand-declared factor set; no cell digest, experiment digest or oracle file reads the factor list. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
An imported experiment now declares request.treatment when its cells treatments differ, and no value path whose levels map one to one onto them, so an imported store and its native rerun declare the same factors. The verdict of an all-imported stratum drops such a path too, over every arm; live strata are unchanged. maps_one_to_one lives in manifest/compare.rs and the importer shares it. Changed rows: guard_legacy_mapping the_mapped_runs_are_the_oracle_runs and guard_comparability imported_stratum_of_the_mini_corpus_is_comparable. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
An import now declares request.treatment when the cells carry different typed treatments and drops the value paths that only restate it, so the golden import declares the model and the treatment like its native rerun. Imported strata explain those paths (A12); live strata keep I7. No digest moves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The weighted H0 row rejected at about half of alpha and was relaxed to a one-sided bound. A rate near alpha/2 is a finding about the test or the row, so the coordinator ruled the relaxation reverted and the counted p traced before any gate moves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The weighted H0 rate is 0.024 with 5 weight values and nominal on a grid of 1000: ties in the statistic, not the engine. G8 keeps the two-sided interval on the binary, one-block and tie-free weighted rows and an upper bound on the tied weighted row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Seeded simulations draw stores in the shape analyze reads and check the blocked tests against planted truth: the binary H0 rate holds 0.05, the Simpson H0 case holds blocked and breaks pooled, the weighted H0 rate does not exceed alpha, and the power at a 0.2 difference is reported. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The golden shape (4 treatment levels by 3 models, 4 replicates) is planned and run on the existing fakes; planted outcomes go through the real report, which gives 6 treatment comparisons blocked by model and 3 model comparisons blocked by treatment. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The weighted H0 row counts the raw permutation p of its one quality expectation and still rejects at about alpha/2 with five weight values (0.024 at 6 runs per cell, 0.025 at 10). With weights on a grid of 1 000 the same code rejects at 0.059 (0.056 and 0.045 in the probe), so the code is not at fault: the statistic ties, and an exact permutation test on tied data is conservative. The row keeps its two-sided gate and is red; the evidence and the ruling it needs are in the report. Add a continuous-weights H0 row and a one-block binary H0 row. Fisher's p in place of the mid-p on the one-block path moves that rate from 0.0447 to 0.0185 and turns the row red. The fakes seal runs as invalid(infrastructure) because the fake run leaves no container to inspect: the fakes, not the runner. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
With 5 weight values the permutation statistic ties and the H0 rate is 0.024, conservative and valid; the row now checks the upper limit of the 99 % interval only, as A15 rules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Known-truth rows for the battery under H0 (binary, one block, tie-free and tied weighted), the Simpson case, power at a 0.2 difference and two treatment levels end to end on fakes. The tied weighted row is an upper bound, as A15 rules, and the design record states why. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
All five L3 leaves are merged and done; G2, G12, G3, G8, STD, DIGEST, GOLDEN, G10, LINUX and SUBJ pass at fb691ef. The Progress rows record each merge and its open findings. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
LittleCoinCoin
added a commit
that referenced
this pull request
Nov 18, 2026
`main` has been red since 9560cd3 (#51, roadmap-only): 20 candler-core guards fail on CI with "`.claude/skills/htn/SKILL.md` is not in commit <main's HEAD>". The code at #50, the last green run, is the same. ## Cause `Swatinem/rust-cache` saves `target/` after deleting every file outside the build profiles. It keeps the directories (cleanup rule: "Files other than `CACHEDIR.TAG` are removed"). So the cache restores the shared fixture `target/tmp/job_skill_repo` as empty folders, `.git` included. `skill_repo()` only checked that `.git` exists, so it reused the skeleton. git did not recognise the skeleton as a repository and searched upward until it reached the checkout's own repository. ## Fix (test support only) - The fixture is reused only when `.git/HEAD` is a file. - A skeleton is pruned by removing empty directories only, deepest first, so a repository that another nextest process has just placed is never deleted. - git runs with `GIT_CEILING_DIRECTORIES` set to the scratch directory, so a missing repository fails at once instead of reading the checkout's. ## Evidence With a planted skeleton in an in-repo `target/`: - the old fixture fails with CI's exact error (2 of 3 sampled guards); - the fixed one passes 16 of 16 sampled guards; - the full candler-core suite passes 329 of 329; - fmt and clippy (`-D warnings`) are clean. This also unblocks #52, whose 21 failures share this cause. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
LittleCoinCoin
added a commit
that referenced
this pull request
Nov 18, 2026
This closes milestone M6a (leaf Step 3), which landed in #52. - Ticks the pre-conditions and gates G1 to G12 in `__roadmap__/m6a-paired-stats/README.md`. - Ticks the candler-v1 leaf `stats/m6a_paired_stats.md` gates, as A10 rewords them, and marks the leaf `done` with `dirtree-rdm`. `stats/` stays in progress until M7 (`mcp/`) lands. Both trees pass `dirtree-rdm validate`. - Adds `__reports__/m6a-paired-stats/knowledge_transfer_v0.md`, covering: - what shipped and the gates; - the spend: about $84 in all, with the build at $54.3 against a $105 cap; - the amendment classes and the lessons; - five follow-ups for Eliott. This is a docs and roadmap change only. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lands milestone M6a of
candler-v1(leafstats/m6a_paired_stats.md, amended by A9 and A10) fromfeat/m6a-paired-stats, campaign__roadmap__/m6a-paired-stats/.What changes
Blocked tests in
analyze(A9). Each declared factor's level pair is compared with the other declared factors as blocks:Holm is applied per comparison per role. Every difference is B − A. Each test equals an offline second engine (scipy 1.17.0, statsmodels 0.15.0).
Treatment as a factor (A7).
request.treatmentcan be declared next tomodel.id. Each cell resolves its own treatment, env lock and image; cells with equal inputs share one. Every other setting factor is still refused (I7).Importer. An import declares
request.treatmentwhen the typed treatments differ. It drops value paths that map one to one onto it, so the golden runs declare exactlyplan.modelandrequest.treatment.Per-arm
instrument_erroris reported, with the other invalid causes unchanged.Design record.
__design__/architecture.md§Analysis statistics records the blocks, the execution order, the validity sources and the importer rule.Evidence (campaign gates, L3 handover)
No cell digest, golden or parity fixture moves. There are no
candler-schemascommits, andcodes.tomlhas no removed line.🤖 Generated with Claude Code