Skip to content

perf(check): one upload per push, and the audit ledger closed - #36

Merged
nonagenticai-claude[bot] merged 15 commits into
mainfrom
lane/perf-check-path
Nov 21, 2026
Merged

nonagenticai-claude[bot] merged 15 commits into
mainfrom
lane/perf-check-path

Conversation

@nonagenticai-claude

@nonagenticai-claude nonagenticai-claude Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

What this lane is

Two things on one branch: the measured check-path perf slice, and a line-by-line audit of the rest
of the repository whose confirmed defects are fixed with regression tests. This is the lane's only
PR; every commit is on lane/perf-check-path.

Perf, measured

Release build, 942-file / 55 MiB corpus, each phase in its own child process:

phase before after
collect+pack worktree 833 ms · 79 MiB 839 ms · 78 MiB
collect+pack head 882 ms · 137 MiB 918 ms · 79 MiB
validate (API) 71 ms 72 ms
unpack (Pod) 190 ms 188 ms
archive 12 077 031 B 12 077 031 B

The client polls at 250 ms for the first 10 s (then 2 s) and the worker polls a running Pod at
500 ms, so the completion edge is ~0.4 s instead of ~2 s. The pre-push hook now packs and uploads
once for both gates instead of twice.

Rejected with numbers: deflate level 6 → 1 is 2.6–2.9× faster but grows conveyor +19 %
(21.66 → 25.76 MB) and agent +45 % (5.58 → 8.07 MB); the client posts over a WAN, so it loses.

Audit ledger — fixed (test named where one exists)

P0

  • GitHub token leak. is_github_url split on / and :, so
    https://github.com:443@evil.example/r.git passed and the worker attached the App installation
    token to a clone served by evil.example; http://github.com also passed. The guard now parses
    the URL: https, exact host, no userinfo, default port. Unit + worker-level attack tests.

P1

  • Upload route: validate + sha256 moved off the Tokio reactor (spawn_blocking); the body read
    is bounded at 180 s (408, new request_timeout envelope); three comments that said 16 MiB now
    say 32 MiB.
  • Client writes: retried only when the connection was never established; a job create carries
    an invocation-scoped idempotency key; a poll's transport failure retries inside the poll. A reset
    after the server committed can no longer duplicate an upload or a Pod.
  • parity_openapi: a shared 2 M-node expansion budget (a depth-31 $ref diamond used to expand
    2³¹ nodes in a 2 GiB Pod) → per-route unreadable; 3.1 $ref siblings merged instead of
    dropped; differences capped at 64/route and target-only routes at 256 with truncated set.
  • Worker: losing the lease now drops the executor instead of continuing to duplicate work; the
    lease sweep cannot requeue a cancel-requested row that claim_job then skips forever; a failed
    log read after a succeeded Pod is retryable log_unavailable, not runner_failed.
  • github_op delivery: 10 s connect / 60 s request timeouts plus a job deadline; blob uploads in
    parallel chunks of 8; retries on 429/5xx honouring Retry-After; a lease retry moves the
    deterministic branch and reuses the open PR instead of creating -2 and a duplicate; the token
    cache no longer holds its lock across a mint.
  • Runner: the Pod Pending interval extends the job deadline (cap 120 s) and the runner's own
    margin grew to 15 s; PodState::Stuck covers ImagePullBackOff/ErrImagePull/InvalidImageName/
    Create*Error/CrashLoopBackOff/Unschedulable as retryable infrastructure; a non-success fetch body
    is read under the deadline and SIGTERM with a 4 KiB cap; sccache stats run under a 2 s timeout;
    Postgres/Redis startup is bounded by the deadline and SIGTERM.

P2

  • Pre-push: coding check --gates build,test packs once, uploads once, and runs both jobs
    against that one source; a build failure stops before test and is printed as not run.

Not fixed (with the check that says why)

  • The worker does not auto-requeue a retryable executor error (fail_job is terminal); that is a
    job-semantics change and needs an owner decision.
  • Streaming the tree into the tar instead of holding it in RAM is a refactor, not a defect.
  • The deployed worker polls at 2 s until Flux rolls the merged image; not a code change.

Known test gaps (adversarial audit, non-blocking)

  • The upload read-timeout (408) path has no test: producing a body that arrives over 180 s needs a
    paused clock and a new test-only dependency. The 400/413 upload paths stay covered.
  • --gates stops after a failed check; that decision is unit-tested (combined_exit) and the
    hook's exit-1 test covers the block, but there is no end-to-end run_check test of the stop.
  • github_op deadline checks run between steps, so one stalled GitHub request can overrun the
    job's deadline by up to 60 s (three in the PR retry path). Bounded, and the worker's lease keeps
    renewing while it runs.
  • Blob upload tasks detached when a lease is lost can outlive the claim; the objects are
    content-addressed, so a late write is harmless.

Verification

  • cargo fmt --check; clippy pedantic via the build guard: clean.
  • Full suite against a fresh PostgreSQL 18: lib 155, http 21, worker 11, delivery 1, runner 17,
    sandbox 18, db 1 (+ the binary's 2 gate tests); 0 failed, 1 ignored (the perf instrument).
  • no-private-refs: 0 violations; preflight green. Preflight caught one pedantic lint
    (items_after_test_module) in the new CLI tests; fixed in 7e40ceb.
  • Live service, new client, tree 7e40ceb (clean):
    • coding check --kind lint: 2/2 passed (fmt 0.6 s, clippy 263 s) in 273 s.
    • coding check --gates build,test: build PASS (237 s compile), then test PASS (doctest 229 s,
      nextest 99 s) — one uploaded source for both gates, no second pack or upload.
  • PR checks: checks, gate, both image jobs, secret-scan, no-private-refs, preflight
    and CodeQL all pass on 7e40ceb.

Honest limits

The compile inside the kata Pod still owns the minutes. This removes seconds of app-side latency,
one archive upload per push, and a set of failure modes that used to be silent; it is not a step
change in job wall clock.

georgiedekker and others added 3 commits November 21, 2026 17:00
A 942-file, 55 MiB corpus — the size of the estate's biggest tree — packed,
validated and unpacked; each phase runs in its own child so the parent can
sample its resident set. Ignored by default: wall clock is evidence, not a gate.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: georgiedekker <georgiedekker@gmail.com>
Measured on a 942-file / 55 MiB corpus, each phase in its own process:

  collect+pack head    882 ms / 137 MiB -> 918 ms / 79 MiB
  collect+pack worktree 833 ms /  79 MiB -> 839 ms / 78 MiB

- pack: move each blob out of the map instead of cloning it. A tree was resident
  twice on index/head snapshots. Files that share a blob still get their own copy.
- client: poll every 250 ms for the first 10 s, then the calm 2 s. A transient 5xx
  keeps the calm interval so the ~12 s rollout budget is unchanged.
- worker: poll a running Pod every 500 ms instead of 2 s.

Compression level 6 -> 1 was measured and rejected: 2.6-2.9x faster on CPU, but
real trees grow 19 % (conveyor) and 45 % (agent), and the client posts over a WAN.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: georgiedekker <georgiedekker@gmail.com>
@nonagenticai-claude

Copy link
Copy Markdown
Contributor Author

Live re-check with this branch's client, against the deployed service:

packed 68 files, 909352 bytes (247823 compressed), sha256 dd85e7b0
uploaded source c1f1dd79-…
job 7f72e7a5-… queued
  [  0.8s] queued
  [  1.4s] running
  [246.6s] succeeded

  PASS  cargo fmt (exit 0)      0.5s
  PASS  cargo clippy (exit 0) 236.3s
verdict: 2 of 2 checks passed        real 251.2s

Exit-code contract intact (0 = every check passed). The number that matters for expectations:
236 s of the 251 s is clippy inside the kata Pod; the app's own side — pack, upload, validate,
queue, Pod create, VM boot, unpack — is ~10 s. That is the ceiling this lane is working against,
which is why it is worth seconds and memory, not a step change.

@nonagenticai-claude
nonagenticai-claude Bot removed the request for review from georgiedekker November 21, 2026 17:25
georgiedekker and others added 12 commits November 21, 2026 17:30
`is_github_url` split the string on `/` and `:` and accepted any prefix that
began with github.com, so `https://github.com:443@evil.example/r.git` passed
the guard and the worker attached an installation token to a clone served by
evil.example. Plain `http://github.com` also passed, sending the token in
cleartext.

Parse the URL instead: https only, exact host, no userinfo, default port.
The unit and worker-level tests cover the host-confusion forms.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: georgiedekker <georgiedekker@gmail.com>
`validate` inflates an archive up to 128 MiB and `sha256_hex` walks it again,
both synchronously on a Tokio worker. Two concurrent 32 MiB uploads could stall
every request on a 2-vCPU API pod, including the k8s probes. Move both into one
`spawn_blocking` task; the upload slot still bounds the buffered bytes.

The slot was also held across an unbounded body read, so a client that connects
and trickles held one of the two slots forever. Bound the read at 180 s and
answer 408 (a new `request_timeout` envelope) rather than 413.

Correct the three comments that still said the archive cap is 16 MiB; it has
been 32 MiB since the cap moved.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: georgiedekker <georgiedekker@gmail.com>
`send_with_retry` repeated any transport failure, including a timeout or reset
on `POST /api/sources` and `POST /api/jobs`. A response lost after the server
committed left a duplicate source row and, worse, a second Pod running a job
the client would never poll.

Writes now retry only when the connection was never established
(`reqwest::Error::is_connect`); reads keep the full retry. A job create also
carries an invocation-scoped idempotency key, so a retry that does reach the
server returns the original job instead of creating a second one.

Tests: a stalled local server sees exactly one write attempt and three read
attempts; two calls never share an idempotency key.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: georgiedekker <georgiedekker@gmail.com>
Two worker defects shared a cause: the lease decided nothing once the job was
running.

- The heartbeat stopped renewing after `Ok(false)` but the executor kept going.
  After a database outage, the reaper could requeue and a second worker start
  the same job while the first still ran it. The heartbeat now signals on a lost
  lease (or one full interval without a successful renewal), and `execute_claim`
  drops the executor instead of finishing work it no longer owns.
- `reap_expired` could requeue a row between the cancel sweep and its own update;
  `claim_job` skips `cancel_requested` rows forever, so the job stuck in
  `queued`. The lease sweeps now leave those rows to `reap_cancelled`, which
  also heals rows a older build already stranded.

A failed log read after a successful Pod was degraded to "" and reported as a
non-retryable `runner_failed`; it is now `log_unavailable`, retryable, because
it is the read that failed, not the code.

Tests: lease theft drops the executor and records nothing; an expired
cancel-requested claim is cancelled, never requeued or claimable; a succeeded
Pod with a 500 log is infrastructure.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: georgiedekker <georgiedekker@gmail.com>
An untrusted openapi.json could make canonical() expand a two-reference diamond
at every level: depth 31 is 2^31 nodes inside a 2 GiB Pod, a CPU and memory
bomb. A single node budget is now shared by every route and source; when it is
exhausted the route is `unreadable` with a named error, never an OOM.

While in the same code path:
- `$ref` siblings (legal in OpenAPI 3.1) are merged onto the resolved target
  instead of dropped, which used to compare different schemas as equal;
  annotative siblings stay ignored.
- The difference list stops at 64 per route and target-only routes at 256, with
  `truncated` set; previously every difference was materialised before the cap.
  The failure strings behind a boolean were never read, so they are gone too.

Tests: a diamond is unreadable under a small budget; siblings compare and
differ at `minimum`; annotations beside $ref stay ignored; 200 differences cap
at 64 with truncated set.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: georgiedekker <georgiedekker@gmail.com>
…cross a mint

The worker-side GitHub client had no connect or request timeout and no deadline
of its own. A stalled connection could hold a worker slot for the whole job
timeout while the heartbeat kept renewing the lease. The delivery client now
has a 10 s connect and a 60 s request timeout, and every step checks the job
timeout before it starts.

`installation_token` held the cache mutex across `mint().await`, so one hung
token request blocked every clone and delivery in the process. The lock now
covers only the cache; the mint has its own 10 s connect / 30 s request bounds.
Two concurrent callers may mint twice, which is harmless for a non-single-use
token.

Test: a stub whose first mint hangs; a second caller returns while it does.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: georgiedekker <georgiedekker@gmail.com>
The read retry policy had no production call site (clippy found the variant
test-only). Each status poll now goes through `send_with_retry(Reads, ...)`, so
a tunnel blip costs 0.5–1 s of backoff inside the poll instead of a whole 2 s
tick, and the existing poll budget still bounds the total.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: georgiedekker <georgiedekker@gmail.com>
…ses its branch and PR

Blob POSTs were serial: 500 files meant 500 round trips, and one transient 5xx
after blob 499 discarded the first 498 uploads. Uploads now run in bounded
chunks of 8, and every call retries a transport failure, a 5xx, a 429 or an
exhausted rate limit, honouring Retry-After up to a bound.

A lease retry used to suffix the branch `-2` and open a duplicate PR. The
branch is now deterministic; a 422 moves the existing ref to the new commit,
and an open PR for that head is reused. The PR POST is deliberately not
auto-retried: a retryable answer looks for the PR first, then tries once.

`execute_with_allowlist` is the testable seam; the environment is still read in
exactly one place, because set_var is unsafe in edition 2024.

New tests/deliver.rs (real Postgres 18 + stub GitHub API): 24 files upload
concurrently (2..=8 in flight), the first blob 503 is retried exactly once,
and a second attempt reuses the branch and PR with no second POST.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: georgiedekker <georgiedekker@gmail.com>
The pre-push hook invoked `coding check` twice (build, then test), and each
invocation re-packed the HEAD snapshot (~840 ms) and re-uploaded the same
12 MB archive over the WAN. `coding check --gates build,test` now packs once,
uploads once, and runs the two jobs against the same source id in order; a
failed build stops before the test gate, which the client prints as not run.
`--kind` and `--gates` are refused together rather than one silently winning.

The hook exit taxonomy is unchanged: 1 blocks, anything else fails open with a
warning. README and CHANGELOG record the one-upload behaviour and this lanes

Signed-off-by: georgiedekker <georgiedekker@gmail.com>
…by the job clock

- The worker started its deadline at Pod create, but the runner clock starts
  when its container does. A cold pull could make the worker delete the Pod
  before the runner wrote its structured timeout. The Pending interval is now
  added to the deadline (capped at 120 s) and the runner own margin grows from
  5 s to 15 s, so its timeout path (two 2 s pump graces) still fits.
- A Pod waiting on ImagePullBackOff/ErrImagePull/InvalidImageName/Create*Error/
  CrashLoopBackOff, or one the scheduler calls Unschedulable, is a new
  PodState::Stuck: the job fails now as retryable infrastructure instead of
  burning the whole timeout and reporting a generic one.
- The non-success source-fetch body is read with the job deadline, SIGTERM and
  a 4 KiB cap; it was an unbounded text() with no timeout.
- sccache --show-stats runs under a 2 s timeout on a kill-on-drop child; a
  wedged cache server could otherwise cost a passing job its result line.
- Postgres/Redis startup is selected against the deadline and cancel; initdb
  could outlive the job and be killed before the result line.

Tests: an ImagePullBackOff and an Unschedulable pod are Stuck, a creating pod
stays Pending; the deadline may be extended only up to the startup grace.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: georgiedekker <georgiedekker@gmail.com>
CHANGELOG and ARCHITECTURE describe the one-upload pre-push hook and the
runner startup/stuck-pod bounds. The gates resolution and its clap parsing
have unit tests in the binary.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: georgiedekker <georgiedekker@gmail.com>
Preflight caught what a direct cargo test does not: clippy::items_after_test_module
rejects items declared after a test module. Move the two CLI tests to the end of
main.rs.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Signed-off-by: georgiedekker <georgiedekker@gmail.com>
@nonagenticai-claude nonagenticai-claude Bot changed the title perf(check): stop copying the tree and halve the completion edge perf(check): one upload per push, and the audit ledger closed Nov 21, 2026
@nonagenticai-claude
nonagenticai-claude Bot merged commit d4b6996 into main Nov 21, 2026
10 checks passed
@nonagenticai-claude
nonagenticai-claude Bot deleted the lane/perf-check-path branch November 21, 2026 18:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant