Repository navigation
perf(fingerprint): token-stream normal form as default with v2 prefix [W5-08] - #960
ajitpratap0 wants to merge 10 commits into
Conversation
… [W5-08] Normalize/Fingerprint now tokenize once and build the normal form from the token stream (no parse, no formatter); fingerprints carry a "v2:" prefix. Legacy v1.17 behaviour is kept behind fingerprint.ModeLegacy. The delimited name predicate moves verbatim to internal/namequote and the parser delegates. BREAKING: fingerprint format, grouping and error contract change; see the CHANGELOG Breaking Changes migration note. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
|
Claude finished @ajitpratap0's task in 23s —— View job Review of PR #960: token-stream fingerprint (v2)I read Overall: The design is sound. It tokenizes once, uses a pooled buffer, hashes the buffer, and keeps the old behaviour behind Issues and suggestions
Test coverageThe coverage described in the PR body is thorough. It includes corpus equivalence against legacy, mutation converses, allocation guards, fuzz tests and golden files. The grouping divergences (1 legacy-merged/v2-split pair, 11 v2-merged/legacy-split pairs) are quantified and pinned. I could not independently verify any of it. Before merge
|
Perf gate: PASS (shadow mode: report only, this check never fails)Base Runner CPU:
Rules: each arm runs 20 randomized code layouts ( Gated suite (GOGC=off, GOMAXPROCS=1)
A/A' control (base revision, disjoint layout seeds): 0 of 71 benchmarks moved more than 3%; 0 would have been flagged by the time rule; suite geomean -0.16%, p 0.221. Low power3 benchmark(s) have under 80% estimated power to be flagged on their own for a +5% slowdown (their spread across layouts is too wide for 20 layouts at the Benjamini-Hochberg level alpha/m = 0.000141). A slowdown below the last column usually goes unflagged there; instructions/op, allocs/op and the suite geomean still guard them.
Power is estimated with the normal approximation for the U test (Noether): power = Phi(ln(1+shift)/s * sqrt(K/2) * sqrt(3/pi) - z(1 - alpha/(2m))), with s the spread and K the layouts per arm. Instructions per operation (cmd/perfdriver)
Secondary metric: default GC, GOMAXPROCS=1 (informational)
Environmentbenchstat (primary metric, samples pooled across layouts: informational, the verdict uses the per-layout test) |
#961 replaced direct CHANGELOG.md edits with per-PR changelog.d/ fragments, which removes this PR's CHANGELOG.md merge conflicts. The entry text is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
|
Claude finished @ajitpratap0's task in 26s —— View job Review of PR #960: token-stream fingerprint (v2)I read Overall: The design is sound and the perf result is strong. Findings
SecurityNo concerns found. Input is only tokenized, with no execution or I/O. The frame stack is fixed-size and every index into it is guarded by Test and perf summary
Before merge
|
…5-08] Drop the gosqlx options entry points (no public API growth for internal hooks) and fill the changelog fragments with the amd64 gate numbers. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
|
Claude finished @ajitpratap0's task in 34s —— View job Review of PR #960: token-stream fingerprint (v2)I checked the open items from the earlier reviews against the code. I did not run builds or tests. Resolved
Still open
Perf and security
Before merge: items 1 and 2 are the ones I would fix. The rest can follow. Maintainer decisions M1-M3 are still open in the PR body. |
…fects, symmetric reparse oracle [W5-08] Keep AS in XMLATTRIBUTES/XMLFOREST, stack-free overflow frames beyond depth 32, fix INNER OUTER JOIN, colon-bind, bracket and multi-line quoted-name spacing found by FuzzNormalizeV2 (now run in CI with a dialect argument), substitute placeholders in both texts in the T7 reparse oracle, pooled dialect leak test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
|
Claude finished @ajitpratap0's task in 20s —— View job Review of PR #960: token-stream fingerprint (v2)I re-checked the open items from the earlier reviews against the current branch. I did not run builds or tests. Resolved since the last review
Still open
Perf and security
Before merge: item 1 is the one I would fix. Items 2-4 are small. Maintainer decisions M1-M3 are still open in the PR body. |
…isses [W5-08] Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
|
Claude finished @ajitpratap0's task in 18s —— View job Review of PR #960: token-stream fingerprint (v2)I checked the items left open by the earlier reviews against the current branch with grep. I did not run builds or tests. Resolved
Still open
Performance: The CI perf gate passes.
Security: I found no concerns. Input is only tokenized, with no execution or I/O. Before merge: item 1 is the one to fix. Items 2 and 3 are small. Maintainer decisions M1-M3 are still open in the PR body. |
…ery, fix @@ spacing, honest changelog [W5-08] Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
|
Claude finished @ajitpratap0's task in 30s —— View job Review of PR #960: token-stream fingerprint (v2)I checked the items left open by the earlier reviews with grep against the current branch. I did not run builds or tests, and I did not re-read the full diff. Still open
Resolved earlier
Performance: The CI perf gate passes.
Security: I found no concerns. Input is only tokenized, with no execution or I/O. Before merge: fix item 1. Items 2 and 3 are small. Maintainer decisions M1-M3 are still open in the PR body. |
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
|
Claude finished @ajitpratap0's task in 15s —— View job Review of PR #960: token-stream fingerprint (v2)I checked the items left open by the earlier reviews with grep against the current branch. I did not run builds or tests, and I did not re-read the full diff. Still open
Resolved earlier
Performance: The latest perf-gate run passes.
Security: I found no concerns. Input is only tokenized, with no execution or I/O. Before merge: fix item 1. Items 2 and 3 are small. Maintainer decisions M1-M3 are still open in the PR body. |
|
Claude finished @ajitpratap0's task in 15s —— View job Review of PR #960: token-stream fingerprint (v2)I re-checked the items left open by earlier reviews with grep against the current branch. I did not run builds or tests, and I did not re-read the full diff. Still open
Resolved earlier
Performance: The latest perf-gate run passes.
Security: I found no concerns. Input is only tokenized, with no execution or I/O. Before merge: fix item 1. Items 2 and 3 are small. Maintainer decisions M1-M3 are still open in the PR body. |
Summary
W5-08 (FP-1) with D5(c).
fingerprint.Normalize/Fingerprint(andgosqlx.Normalize/Fingerprint) now build the normal form straight from the token stream: one tokenize, a pooled byte buffer, hash of the buffer. No parse, no formatter. Fingerprints carry av2:prefix. The v1.17 behaviour stays available asfingerprint.ModeLegacy(Options,NormalizeWithOptions,FingerprintWithOptions,VersionOf). The delimited-name predicate moves verbatim tointernal/namequote(D18, no public API) and the parser delegates to it. Legacy golden columns are byte-identical to origin/main (checked with the awk comparison); the new columns are added.Test Plan
task check,task test:cbinding,go test -count=1 ./pkg/compatibility/..., staticcheck on pkg/fingerprint and internal/...: all pass on head 528b38c (the earlier 120stest:racetimeout oncmd/gosqlx/cmdandpkg/gosqlxunder machine load was a re-run with-timeout 900s, reported as a re-run, not a gate result; CI Race Detector shards pass).FuzzNormalizeV2(now with a dialect argument and in the CI Fuzz Regression step) ran clean for 3 consecutive 60s runs after fixing five fuzz-found idempotency defects; 8 seeds are committed.BenchmarkNormalizegeomean -43.3% (simple_select -51.1%, join -35.5%, complex -38.3%);BenchmarkFingerprint-33.5% to -53.4%. Legacy no-regression pair (headBenchmark*Legacyvs baseBenchmarkNormalize/Fingerprint): geomean -1.4%, no regression beyond +3.0% (noise).TiDB bench-compare (workflow run 43271354829, amd64 AMD EPYC 9V74, count=10, head 7463aba; these numbers (0.80/0.93/1.06) predate the 528b38c hot-loop change and were not re-run; that commit changes mysql/mariadb/bigquery double-quote handling and
@@spacing). PLAN section 7 target gosqlx/tidb_digest geomean <= 1.0: MET (C1 was 1.99x Normalize, 2.28x Fingerprint). Geomean time ratio GoSQLX/tidb_digest: gosqlx 0.80, gosqlx_fingerprint 0.93, gosqlx_legacy 1.81, gosqlx_fingerprint_legacy 2.12. Per-query median ns/op:Only
insertfor the fingerprint cell is above 1.0 (1.06).Normalize/complex(arm64, 3s):tokenize~37% cum,appendNormalV2~37% (flat 34%), hash and pool under 5%, runtime alloc/madvise noise ~10%. Thejoinandcomplexmisses of the PLAN <= 0.30x threshold are therefore bounded by the shared tokenizer plus the normal-form writer, not by allocation.Normalize/simple_select-67.80% (0.32x),Normalize/join-53.85% (0.46x),Normalize/complex-53.87% (0.46x). Earlier runs: EPYC 7763 run 43265055242 (head 7463aba) -64.82/-53.62/-56.21% (0.35x/0.46x/0.44x); EPYC 9V45 (head b6cddc9) -74.31/-61.11/-62.49% (0.26x/0.39x/0.375x). PLAN threshold (<= 0.30x): no target reaches it on the final head; simple_select met it only on the 9V45 run, and join and complex miss on every run. The runners differ, so runs are not comparable across CPUs. The final run's Low power list has 5 unrelated benchmarks (L010, LintParallel/rules, ParseE2E/subquery, ParseE2E/delete, Tokenize/scratch/insert); the Normalize targets are not on it.Breaking Changes
See CHANGELOG. Fingerprint format becomes
v2:plus 64 hex (67 bytes;CHAR(64)columns must widen). Normalized text and grouping change (documented classes). Default mode no longer reports syntax errors (usegosqlx.Validate). Normalized text is a grouping key, not guaranteed parseable SQL (the symmetric reparse oracle T7 pins 44 implicit-alias parser-limitation failures plus 2 allowlisted residues, 46 in total; legacy_reparse_skipped=8, eligible 2416, reparsed 2370). Migration:FingerprintWithOptions(sql, fingerprint.Options{Mode: fingerprint.ModeLegacy})reproduces v1.17 exactly;VersionOftells versions apart.Fix round 2 (commit 528b38c): changelog now quotes the final-head 9V74 gate run 43274969051; v2 now redacts double-quoted strings as literals for Dialect mysql/mariadb/bigquery (documented coarsening, default dialects unchanged);
@@version/@@session.sql_modeno longer get a space; the limit/offset T7 skip class is pinned to 0; doc nits (oracle counts removed, deep frames > 32 get full rules, normative).T5 legacy-merge/v2-split pairs (7 allowlisted, each with a reason in
legacyMergeV2SplitAllowlist, equivalence_test.go):Behaviour changes needing maintainer sign-off
Merge-blocking and not yet recorded.
normalizeTokensinpkg/fingerprint/tokens.go(reached fromNormalizeWithOptions/FingerprintWithOptionsinpkg/fingerprint/fingerprint.go, andgosqlx.Normalize/Fingerprintinpkg/gosqlx/gosqlx.go) only tokenizes, so it reports lexical errors but never parse errors (documented inpkg/fingerprint/doc.go; usegosqlx.Validate).dialectTokeninpkg/fingerprint/tokens.go(enabled inappendNormalV2forkeywords.DialectMySQL,DialectMariaDBandDialectBigQuery) treats double-quoted tokens as string literals and redacts them to?, so ANSI_QUOTES table/column names become?; the other grouping coarsenings are the rule set inappendNormalV2.v2:value:normalizeTokensinpkg/fingerprint/tokens.gobuilds the fingerprint fromVersionPrefix("v2:",pkg/fingerprint/options.go) plus 64 hex characters, soCHAR(64)columns must widen;VersionOfinpkg/fingerprint/options.gotells the versions apart.🤖 Generated with Claude Code
https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2