Repository navigation
perf(parser,gosqlx): reserve the parse stack once before tokenizing [W1-04] - #940
Conversation
…W1-04] A parse on a fresh goroutine grew its stack 2 -> 4 -> 8 (-> 16) KiB inside the parse. internal/stackgrow.Reserve (12 KiB frame, lands at 16 KiB) now runs first in tokscratch, four public-tokenizer pipelines and the parser core, so the stack is copied once while shallow. No output, allocs/op or B/op change; no public API. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
|
Claude finished @ajitpratap0's task in 24s —— View job Review of PR #940: reserve the parse stack once before tokenizing [W1-04]I read the diff of the single commit. I did not build or run the tests in this job. The test results below come from the PR description. Verdict: the approach is sound and the change is small and well contained. I have no blocking issues, only a few non-blocking points. What I checked
Non-blocking suggestions
SecurityNo new input handling, no unsafe, and no new allocations of attacker-controlled size. The reserved size is a compile-time constant. No concerns. Test coverageCoverage looks thorough. It includes the entry-point enumeration test ( Overall this looks good to merge once the amd64 perf gate confirms the gain. |
Perf gate: PASS (shadow mode: report only, this check never fails)Base Runner CPU:
Rules: each arm runs 20 randomized code layouts ( Gated suite (GOGC=off, GOMAXPROCS=1)
A/A' control (base revision, disjoint layout seeds): 0 of 65 benchmarks moved more than 3%; 0 would have been flagged by the time rule; suite geomean -0.17%, p 0.242. Low power5 benchmark(s) have under 80% estimated power to be flagged on their own for a +5% slowdown (their spread across layouts is too wide for 20 layouts at the Benjamini-Hochberg level alpha/m = 0.000154). A slowdown below the last column usually goes unflagged there; instructions/op, allocs/op and the suite geomean still guard them.
Power is estimated with the normal approximation for the U test (Noether): power = Phi(ln(1+shift)/s * sqrt(K/2) * sqrt(3/pi) - z(1 - alpha/(2m))), with s the spread and K the layouts per arm. Instructions per operation (cmd/perfdriver)
Secondary metric: default GC, GOMAXPROCS=1 (informational)
Environmentbenchstat (primary metric, samples pooled across layouts: informational, the verdict uses the per-layout test) |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
…and comment fixes [W1-04] Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
|
Claude finished @ajitpratap0's task in 19s —— View job Review of PR #940: reserve the parse stack once before tokenizing [W1-04]I read the diff for Verdict: no blocking issues. The change is small, internal-only, and the amd64 gate confirms it. Evidence
Code
Non-blocking suggestions
SecurityNo new input handling, no Test coverageGood. It includes entry-point enumeration ( Overall this is good to merge once the CHANGELOG numbers and the PR-body/ |
… [W1-04] Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
…-04] Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
|
Claude finished @ajitpratap0's task in 21s —— View job Review of PR #940: reserve the parse stack once before tokenizing [W1-04]I read Verdict: no blocking issues. The change is small and internal-only. The latest amd64 gate run confirms the gain. Evidence
Code
Items to resolve before merge
SecurityNo new input handling, no Test coverageGood. It covers entry-point enumeration ( |
|
Claude finished @ajitpratap0's task in 28s —— View job Review of PR #940: reserve the parse stack once before tokenizing [W1-04]I read the full diff against Verdict: no blocking issues. The earlier review notes about the CHANGELOG placeholder and the Resolved since the earlier reviews
Code
Non-blocking suggestions
SecurityNo new input handling, no Test coverageGood. It includes entry-point enumeration ( Overall this looks good to merge once CI is green. |
…omments [W1-04] Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
… [W1-04] Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
|
Claude finished @ajitpratap0's task in 21s —— View job Review of PR #940: reserve the parse stack once before tokenizing [W1-04]I read the full diff against Verdict: no blocking issues. The change is small and internal-only, and the amd64 gate confirms the gain. Evidence
Code
Items to resolve
SecurityNo new input handling, no Test coverageGood. It includes entry-point enumeration ( Overall this is good to merge once CI is green and the CI step scope is settled. |
…ved [W1-04] A GC cycle can shrink the goroutine stack mid-parse (seen on windows-latest), which the test misread as a growth copy. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
|
Claude finished @ajitpratap0's task in 19s —— View job Review of PR #940: reserve the parse stack once before tokenizing [W1-04]I read the full diff against Verdict: no blocking issues. The change is small and internal-only, and the amd64 gate confirms the gain. Evidence
Code
Items to resolve
SecurityNo new input handling, no Test coverageGood. It includes entry-point enumeration ( Overall this is good to merge once CI is green and the CI step wording is settled. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
|
Claude finished @ajitpratap0's task in 21s —— View job Review of PR #940: reserve the parse stack once before tokenizing [W1-04]I read the full diff against Verdict: no blocking issues. The change is small and internal-only. The amd64 gate shows the gain, with allocs/op and B/op unchanged. Code
Items to resolve
Test coverageGood. It includes entry-point enumeration ( Overall this is good to merge once CI is green and the CI step wording is settled. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
…atch doc [W1-04] Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
|
Claude finished @ajitpratap0's task in 20s —— View job Review of PR #940: reserve the parse stack once before tokenizing [W1-04]I read the full diff against Verdict: no blocking correctness issues. The change is small and internal-only. The amd64 gate shows large Code
Items to resolve
Test coverageGood. It includes entry-point enumeration ( |
#961 replaced direct CHANGELOG.md edits with per-PR changelog.d/ fragments, which removes this PR's CHANGELOG.md merge conflicts. The entry text is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
|
Claude finished @ajitpratap0's task in 22s —— View job Review of PR #940: reserve the parse stack once before tokenizing [W1-04]I read Verdict: no blocking issues. The change is small and internal-only. The gate shows large Code
Items to resolve
Test coverageCoverage is good. It includes entry-point enumeration ( |
|
Claude finished @ajitpratap0's task in 21s —— View job Review of PR #940: reserve the parse stack once before tokenizing [W1-04]I read the full diff against Verdict: no blocking correctness issues. The change is small and internal-only. The amd64 gate shows large Code
Items to resolve
Test coverageGood. It includes entry-point enumeration ( |
|
Claude finished @ajitpratap0's task in 16s —— View job Review of PR #940: reserve the parse stack once before tokenizing [W1-04]I read the diff against Verdict: no blocking correctness issues. The change is small and internal-only. The amd64 gate shows large Code
Items to resolve
Test coverageCoverage is good. It includes entry-point enumeration ( |
Summary
A parse on a brand-new goroutine grew its stack 2 -> 4 -> 8 (-> 16) KiB inside the parse; for a ~1 us statement the 2-3 copies cost more than the parse.
internal/stackgrow.Reserve()(12 KiB frame, lands a fresh goroutine at 16 KiB, no-op ~1 ns afterwards) now runs first intokscratch.Bytes/String/StringContext(covers every internal tokenize-then-parse pipeline), in four public-tokenizer pipelines (ParseBytesWithTokens,Analyze,LintString, LSPparseWithRecoveryForDialect) and as a backstop in the parser core (parseTokens,parseContextTokens,parseWithRecovery). No output, allocs/op or B/op change; no public API (D18).Memory: a goroutine that parses keeps >= 16 KiB stack until exit/GC shrink (before 4-16 KiB; Windows lands at 32 KiB). Coverage over the 2,999-statement corpus: 99.90% need no further copy.
Also:
BenchmarkParseFreshRW(evidence for the 16 KiB class on real-world SQL) and the CI non-race step now runsAllocBudget|StackBudget|StackReservefor./pkg/gosqlx;./internal/stackgrow(includingTestReserveAvoidsLaterCopies, which is not race-skipped) runs undertask test:race, so no workflow change is needed for it.Test Plan
go build ./...,GOOS=js GOARCH=wasm go build ./wasm/: passgo test -count=1 -run 'StackReserve|StackBudget|EntryPoints|NoUnreserved|Reserve' ./internal/stackgrow ./pkg/gosqlx(non-race and-race): pass; real-world coverage 2996/2999go test -count=1 -run 'AllocBudget|StackBudget|StackReserve' ./pkg/gosqlx ./internal/stackgrow: passtask test:cbinding,go test ./pkg/compatibility/..., staticcheck on changed packages: passtask check: format/vet/lint clean; the race run hit the 2m timeout and timing-ratio flakes (TestJoinWithComments_ScalesLinearly,TestTranspileDeepChainLinear, parser/security/tokenizer timeouts) while the M4 was at load 50-114. Re-run: parser, security, tokenizer, transpiler pass;pkg/formatterpasses alone and flaked once more under the full-tree run (B/op ratio 4.31 within bound; only the wall-clock ratio missed). No test, threshold or timeout was changed.Mutation checks (by Codex):
reserveBytes=16<<10fails T2,5<<10fails T4,neveras const fails T1, dropping the Reserve intokscratch.Stringfails T5.Local
task bench:ab BASE=origin/main BENCH='./pkg/gosqlx=^BenchmarkParseFresh$' ROUNDS=8 LAYOUTS=0 BENCHTIME=200ms(arm64 M4, load 50+, indicative only): geomean -64.88%; simple_select -79.8%, aggregate -74.1%, subquery -68.0%; other cells directionally lower but noisy (+-80%). allocs/op and B/op identical in every cell.CI perf-gate (ci(perf): randomize code layout in the A/B perf gate so layout noise cannot read as a regression #934, amd64 EPYC 7763, 20 interleaved rounds x 20 randomized layouts per arm, per-layout Mann-Whitney U, A/A', BH). Authoritative; local M4 numbers are indicative only. The gate suite now includes ParseFresh insert/update/delete (scripts/perf/gate-suite.txt), so the short-DML targets are gated.
Run 1 (base 6399dc0, head merge 58e1cfa): PASS. simple_select -58.73%, join -43.63%, complex -23.54% (all p<0.001, A/A' about -0.2%); instructions/op -60.5%, -42.0%, -24.2%; no gated time regression; allocs/op and B/op unchanged; suite geomean -2.96%; A/A' moved 0 of 52 benchmarks; no target in the Low-power list.
Run 2 (base a72ec74 = current main, head 21e8482, includes insert/update/delete):
Suite geomean -7.69% (p<0.001); A/A' 0 of 55 moved, geomean -0.02%.
Run 3 (base 6d29d68, head merge cedf5d2 = PR head e92c248; latest authoritative run before the final main merge): simple_select -60%, join -39.74%, complex -23%, insert -57%, update -60%, delete -67% (all p<0.001); suite geomean -7.99%; A/A' moved 0 of 55; Low-power list empty; allocs/op and B/op unchanged; instructions/op within +0.28%; ParseE2E within +/-1%.
Run 4 (head 996ce7b, after the final origin/main merge; the PR head 9d316a0 adds only a test-only commit): PASS. simple_select -58.1%, join -43.9%, complex -26.1%, insert -56.0%, update -58.4%, delete -65.3% (ParseFresh); suite geomean -7.56%; allocs/op and B/op unchanged; instructions/op at most +0.39%; A/A' moved 0 of 55; LintRW/LintParallel within +/-1.1%.
Run 5 (PR head 9d316a0, merge 52f4e44 vs base 8889402, run 42332405535, Intel Xeon Platinum 8573C; attempt 1 and re-run attempt 2 both on this CPU type; sticky comment 6611443891, latest = attempt 2): REGRESSION flagged on
PoolGetPut/serial+4.78% (p<0.001, A/A' -0.23%, spread 0.70%; attempt 1 +4.58%; benchstat 95.94n -> 100.40n). Everything else: ParseFresh simple_select -60.08%, join -43.27%, complex -22.98%, insert -57.10%, update -59.70%, delete -67.11% (all p<0.001); suite geomean -7.79%; allocs/op and B/op unchanged; instructions/op at most +0.38% (the instructions table does not cover PoolGetPut); A/A' moved 0 of 55. The regression reproduces on re-run, so it is REPORTED, not hidden. Cause is NOT proven: PoolGetPut only calls GetTokenizer/GetParser/NewAST/Release/Put and none of them calls stackgrow.Reserve; run 4 (AMD EPYC 9V74, same non-test code) showed -0.05% (p=0.579); local M4 A/B (6x3 runs, GOMAXPROCS=1) shows 28.27 vs 28.16 ns (-0.4%). The only data symbol that moved between base and head in an amd64 build isast.poolLeakCount(offset in its 64-byte line 16 -> 48), which sits on the NewAST/ReleaseAST path, so a data-layout effect on Intel is the working hypothesis; -randlayout randomizes code layout only. Merge decision for the maintainer: ParseFresh gains are large and significant, PoolGetPut is a pool micro-benchmark whose code is untouched, but the rule allows no gated time regression.Run 6 (PR head b085e53 = merge a70d80d vs base 79774dd, AMD EPYC 7763, sticky comment 6611443891 updated 2026-11-07T08:53Z; latest authoritative run before the final origin/main re-merge): PASS. ParseFresh simple_select -58.29%, join -43.86%, complex -24.60%, insert -56.62%, update -59.09%, delete -66.17% (all p<0.001, A/A' under 0.6%); PoolGetPut/serial -0.21% (p=0.665), so the Run 5 PoolGetPut flag did not reproduce on EPYC (it was Intel-specific or a data-layout effect and is not a gated regression on the latest run); every ParseE2E cell within +/-1% (PLAN +/-2% condition met); allocs/op and B/op unchanged; A/A' moved 0 of 55; suite geomean -7.95%; the Low-power list holds only LintRW/snowflake, which is not a target. ParseFresh/complex still misses the PLAN -40% threshold and that stays reported.
Run 7 (latest, PR head 270dab8 = merge f398b01 vs base fb63e32, AMD EPYC 7763, after merging chore(changelog): per-PR changelog fragments to stop CHANGELOG.md conflicts #961): PASS. ParseFresh simple_select -58.69%, join -43.72%, complex -24.65%, insert -57.03%, update -59.30%, delete -66.38% (all p<0.001, A/A' under 0.7%); PoolGetPut/serial -0.31% (p=0.172); ParseE2E within +/-1%; TransformParse +1.0% to +2.1% (p 0.01-0.05, below the 3% gate threshold, status ok); allocs/op and B/op unchanged. All CI checks green.
Gate noise, reported not hidden: the shadow verdict reads "regression" in both PR-head runs for reasons unrelated to this change. Run (base a72ec74, head 4e90e34) flagged ParseRW/clickhouse/medium +3.22% (A/A' +2.18%, Low-power list, instructions/op -0.91%); it did not reproduce in run 2 (-0.30%, p=0.735) nor in run 1 (PASS). Run 2 flagged LintRW/postgresql allocs/op 34189 -> 34190 (+1 in 34k; run 1 measured 34189.5 -> 34189, so this is pool/GC amortization jitter in a benchmark whose Lint path this change does not touch).
PLAN W1-04 threshold (
BenchmarkParseFresh/* <= -40%) is NOT fully met: ParseFresh/complex reaches only -22.7% to -26.1% on amd64 across six runs (-24.60% in the latest) (the remaining cost is parse work, not stack growth), and ParseFresh/join is noise-limited around the target: -39.7% (run 3), -43.6%, -43.4% and -43.9% (run 4), -43.3% (run 5, latest) across five runs, so it meets -40% in four of five. simple_select, insert, update and delete meet it in every run. The W3-06 trigger ("complex above 1.0x Vitess") still has to be re-checked against this result.Open decisions (recommended defaults implemented): M1 16 KiB landing; M2(a) implemented:
scripts/perf/gate-suite.txtis widened, so the short-DML cells (insert/update/delete) are in the gated suite.Breaking Changes
None.
🤖 Generated with Claude Code
https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
Merge state: merged with origin/main fb63e32 (changelog fragments #961); entries live in changelog.d/w1-04-stack-pregrow.*.md; all CI checks green, PR mergeable.
Final gate before merge (merge train)
The perf gate ran on the final head
e74a01b1abafmerged with current main: ## Perf gate: PASS (shadow mode: report only, this check never fails), report #940 (comment). This result replaces any earlier gate numbers in this description.