Repository navigation
perf(check): one upload per push, and the audit ledger closed - #36
Merged
Merged
Conversation
A 942-file, 55 MiB corpus — the size of the estate's biggest tree — packed, validated and unpacked; each phase runs in its own child so the parent can sample its resident set. Ignored by default: wall clock is evidence, not a gate. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: georgiedekker <georgiedekker@gmail.com>
Measured on a 942-file / 55 MiB corpus, each phase in its own process: collect+pack head 882 ms / 137 MiB -> 918 ms / 79 MiB collect+pack worktree 833 ms / 79 MiB -> 839 ms / 78 MiB - pack: move each blob out of the map instead of cloning it. A tree was resident twice on index/head snapshots. Files that share a blob still get their own copy. - client: poll every 250 ms for the first 10 s, then the calm 2 s. A transient 5xx keeps the calm interval so the ~12 s rollout budget is unchanged. - worker: poll a running Pod every 500 ms instead of 2 s. Compression level 6 -> 1 was measured and rejected: 2.6-2.9x faster on CPU, but real trees grow 19 % (conveyor) and 45 % (agent), and the client posts over a WAN. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: georgiedekker <georgiedekker@gmail.com>
nonagenticai-claude
Bot
requested a review
from georgiedekker
as a code owner
November 21, 2026 17:17
Contributor
Author
|
Live re-check with this branch's client, against the deployed service: Exit-code contract intact (0 = every check passed). The number that matters for expectations: |
`is_github_url` split the string on `/` and `:` and accepted any prefix that began with github.com, so `https://github.com:443@evil.example/r.git` passed the guard and the worker attached an installation token to a clone served by evil.example. Plain `http://github.com` also passed, sending the token in cleartext. Parse the URL instead: https only, exact host, no userinfo, default port. The unit and worker-level tests cover the host-confusion forms. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: georgiedekker <georgiedekker@gmail.com>
`validate` inflates an archive up to 128 MiB and `sha256_hex` walks it again, both synchronously on a Tokio worker. Two concurrent 32 MiB uploads could stall every request on a 2-vCPU API pod, including the k8s probes. Move both into one `spawn_blocking` task; the upload slot still bounds the buffered bytes. The slot was also held across an unbounded body read, so a client that connects and trickles held one of the two slots forever. Bound the read at 180 s and answer 408 (a new `request_timeout` envelope) rather than 413. Correct the three comments that still said the archive cap is 16 MiB; it has been 32 MiB since the cap moved. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: georgiedekker <georgiedekker@gmail.com>
`send_with_retry` repeated any transport failure, including a timeout or reset on `POST /api/sources` and `POST /api/jobs`. A response lost after the server committed left a duplicate source row and, worse, a second Pod running a job the client would never poll. Writes now retry only when the connection was never established (`reqwest::Error::is_connect`); reads keep the full retry. A job create also carries an invocation-scoped idempotency key, so a retry that does reach the server returns the original job instead of creating a second one. Tests: a stalled local server sees exactly one write attempt and three read attempts; two calls never share an idempotency key. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: georgiedekker <georgiedekker@gmail.com>
Two worker defects shared a cause: the lease decided nothing once the job was running. - The heartbeat stopped renewing after `Ok(false)` but the executor kept going. After a database outage, the reaper could requeue and a second worker start the same job while the first still ran it. The heartbeat now signals on a lost lease (or one full interval without a successful renewal), and `execute_claim` drops the executor instead of finishing work it no longer owns. - `reap_expired` could requeue a row between the cancel sweep and its own update; `claim_job` skips `cancel_requested` rows forever, so the job stuck in `queued`. The lease sweeps now leave those rows to `reap_cancelled`, which also heals rows a older build already stranded. A failed log read after a successful Pod was degraded to "" and reported as a non-retryable `runner_failed`; it is now `log_unavailable`, retryable, because it is the read that failed, not the code. Tests: lease theft drops the executor and records nothing; an expired cancel-requested claim is cancelled, never requeued or claimable; a succeeded Pod with a 500 log is infrastructure. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: georgiedekker <georgiedekker@gmail.com>
An untrusted openapi.json could make canonical() expand a two-reference diamond at every level: depth 31 is 2^31 nodes inside a 2 GiB Pod, a CPU and memory bomb. A single node budget is now shared by every route and source; when it is exhausted the route is `unreadable` with a named error, never an OOM. While in the same code path: - `$ref` siblings (legal in OpenAPI 3.1) are merged onto the resolved target instead of dropped, which used to compare different schemas as equal; annotative siblings stay ignored. - The difference list stops at 64 per route and target-only routes at 256, with `truncated` set; previously every difference was materialised before the cap. The failure strings behind a boolean were never read, so they are gone too. Tests: a diamond is unreadable under a small budget; siblings compare and differ at `minimum`; annotations beside $ref stay ignored; 200 differences cap at 64 with truncated set. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: georgiedekker <georgiedekker@gmail.com>
…cross a mint The worker-side GitHub client had no connect or request timeout and no deadline of its own. A stalled connection could hold a worker slot for the whole job timeout while the heartbeat kept renewing the lease. The delivery client now has a 10 s connect and a 60 s request timeout, and every step checks the job timeout before it starts. `installation_token` held the cache mutex across `mint().await`, so one hung token request blocked every clone and delivery in the process. The lock now covers only the cache; the mint has its own 10 s connect / 30 s request bounds. Two concurrent callers may mint twice, which is harmless for a non-single-use token. Test: a stub whose first mint hangs; a second caller returns while it does. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: georgiedekker <georgiedekker@gmail.com>
The read retry policy had no production call site (clippy found the variant test-only). Each status poll now goes through `send_with_retry(Reads, ...)`, so a tunnel blip costs 0.5–1 s of backoff inside the poll instead of a whole 2 s tick, and the existing poll budget still bounds the total. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: georgiedekker <georgiedekker@gmail.com>
…ses its branch and PR Blob POSTs were serial: 500 files meant 500 round trips, and one transient 5xx after blob 499 discarded the first 498 uploads. Uploads now run in bounded chunks of 8, and every call retries a transport failure, a 5xx, a 429 or an exhausted rate limit, honouring Retry-After up to a bound. A lease retry used to suffix the branch `-2` and open a duplicate PR. The branch is now deterministic; a 422 moves the existing ref to the new commit, and an open PR for that head is reused. The PR POST is deliberately not auto-retried: a retryable answer looks for the PR first, then tries once. `execute_with_allowlist` is the testable seam; the environment is still read in exactly one place, because set_var is unsafe in edition 2024. New tests/deliver.rs (real Postgres 18 + stub GitHub API): 24 files upload concurrently (2..=8 in flight), the first blob 503 is retried exactly once, and a second attempt reuses the branch and PR with no second POST. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: georgiedekker <georgiedekker@gmail.com>
The pre-push hook invoked `coding check` twice (build, then test), and each invocation re-packed the HEAD snapshot (~840 ms) and re-uploaded the same 12 MB archive over the WAN. `coding check --gates build,test` now packs once, uploads once, and runs the two jobs against the same source id in order; a failed build stops before the test gate, which the client prints as not run. `--kind` and `--gates` are refused together rather than one silently winning. The hook exit taxonomy is unchanged: 1 blocks, anything else fails open with a warning. README and CHANGELOG record the one-upload behaviour and this lanes Signed-off-by: georgiedekker <georgiedekker@gmail.com>
…by the job clock - The worker started its deadline at Pod create, but the runner clock starts when its container does. A cold pull could make the worker delete the Pod before the runner wrote its structured timeout. The Pending interval is now added to the deadline (capped at 120 s) and the runner own margin grows from 5 s to 15 s, so its timeout path (two 2 s pump graces) still fits. - A Pod waiting on ImagePullBackOff/ErrImagePull/InvalidImageName/Create*Error/ CrashLoopBackOff, or one the scheduler calls Unschedulable, is a new PodState::Stuck: the job fails now as retryable infrastructure instead of burning the whole timeout and reporting a generic one. - The non-success source-fetch body is read with the job deadline, SIGTERM and a 4 KiB cap; it was an unbounded text() with no timeout. - sccache --show-stats runs under a 2 s timeout on a kill-on-drop child; a wedged cache server could otherwise cost a passing job its result line. - Postgres/Redis startup is selected against the deadline and cancel; initdb could outlive the job and be killed before the result line. Tests: an ImagePullBackOff and an Unschedulable pod are Stuck, a creating pod stays Pending; the deadline may be extended only up to the startup grace. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: georgiedekker <georgiedekker@gmail.com>
CHANGELOG and ARCHITECTURE describe the one-upload pre-push hook and the runner startup/stuck-pod bounds. The gates resolution and its clap parsing have unit tests in the binary. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: georgiedekker <georgiedekker@gmail.com>
Preflight caught what a direct cargo test does not: clippy::items_after_test_module rejects items declared after a test module. Move the two CLI tests to the end of main.rs. Co-Authored-By: Claude Code <noreply@anthropic.com> Signed-off-by: georgiedekker <georgiedekker@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this lane is
Two things on one branch: the measured check-path perf slice, and a line-by-line audit of the rest
of the repository whose confirmed defects are fixed with regression tests. This is the lane's only
PR; every commit is on
lane/perf-check-path.Perf, measured
Release build, 942-file / 55 MiB corpus, each phase in its own child process:
worktreeheadThe client polls at 250 ms for the first 10 s (then 2 s) and the worker polls a running Pod at
500 ms, so the completion edge is ~0.4 s instead of ~2 s. The pre-push hook now packs and uploads
once for both gates instead of twice.
Rejected with numbers: deflate level 6 → 1 is 2.6–2.9× faster but grows conveyor +19 %
(21.66 → 25.76 MB) and agent +45 % (5.58 → 8.07 MB); the client posts over a WAN, so it loses.
Audit ledger — fixed (test named where one exists)
P0
is_github_urlsplit on/and:, sohttps://github.com:443@evil.example/r.gitpassed and the worker attached the App installationtoken to a clone served by
evil.example;http://github.comalso passed. The guard now parsesthe URL: https, exact host, no userinfo, default port. Unit + worker-level attack tests.
P1
spawn_blocking); the body readis bounded at 180 s (408, new
request_timeoutenvelope); three comments that said 16 MiB nowsay 32 MiB.
an invocation-scoped idempotency key; a poll's transport failure retries inside the poll. A reset
after the server committed can no longer duplicate an upload or a Pod.
$refdiamond used to expand2³¹ nodes in a 2 GiB Pod) → per-route
unreadable; 3.1$refsiblings merged instead ofdropped; differences capped at 64/route and target-only routes at 256 with
truncatedset.lease sweep cannot requeue a cancel-requested row that
claim_jobthen skips forever; a failedlog read after a succeeded Pod is retryable
log_unavailable, notrunner_failed.parallel chunks of 8; retries on 429/5xx honouring
Retry-After; a lease retry moves thedeterministic branch and reuses the open PR instead of creating
-2and a duplicate; the tokencache no longer holds its lock across a mint.
Pendinginterval extends the job deadline (cap 120 s) and the runner's ownmargin grew to 15 s;
PodState::Stuckcovers ImagePullBackOff/ErrImagePull/InvalidImageName/Create*Error/CrashLoopBackOff/Unschedulable as retryable infrastructure; a non-success fetch body
is read under the deadline and SIGTERM with a 4 KiB cap; sccache stats run under a 2 s timeout;
Postgres/Redis startup is bounded by the deadline and SIGTERM.
P2
coding check --gates build,testpacks once, uploads once, and runs both jobsagainst that one source; a build failure stops before test and is printed as not run.
Not fixed (with the check that says why)
retryableexecutor error (fail_jobis terminal); that is ajob-semantics change and needs an owner decision.
Known test gaps (adversarial audit, non-blocking)
paused clock and a new test-only dependency. The 400/413 upload paths stay covered.
--gatesstops after a failed check; that decision is unit-tested (combined_exit) and thehook's exit-1 test covers the block, but there is no end-to-end
run_checktest of the stop.github_opdeadline checks run between steps, so one stalled GitHub request can overrun thejob's deadline by up to 60 s (three in the PR retry path). Bounded, and the worker's lease keeps
renewing while it runs.
content-addressed, so a late write is harmless.
Verification
cargo fmt --check; clippy pedantic via the build guard: clean.sandbox 18, db 1 (+ the binary's 2 gate tests); 0 failed, 1 ignored (the perf instrument).
no-private-refs: 0 violations; preflight green. Preflight caught one pedantic lint(
items_after_test_module) in the new CLI tests; fixed in7e40ceb.7e40ceb(clean):coding check --kind lint: 2/2 passed (fmt 0.6 s, clippy 263 s) in 273 s.coding check --gates build,test: build PASS (237 s compile), then test PASS (doctest 229 s,nextest 99 s) — one uploaded source for both gates, no second pack or upload.
checks,gate, bothimagejobs,secret-scan,no-private-refs,preflightand CodeQL all pass on
7e40ceb.Honest limits
The compile inside the kata Pod still owns the minutes. This removes seconds of app-side latency,
one archive upload per push, and a set of failure modes that used to be silent; it is not a step
change in job wall clock.