Skip to content

perf(fingerprint): token-stream normal form as default with v2 prefix [W5-08] - #960

Open
ajitpratap0 wants to merge 10 commits into
mainfrom
perf/w5-08-token-stream-fingerprint
Open

ajitpratap0 wants to merge 10 commits into
mainfrom
perf/w5-08-token-stream-fingerprint

Conversation

@ajitpratap0

@ajitpratap0 ajitpratap0 commented Nov 7, 2026 •

Copy link
Copy Markdown
Owner

Summary

W5-08 (FP-1) with D5(c). fingerprint.Normalize/Fingerprint (and gosqlx.Normalize/Fingerprint) now build the normal form straight from the token stream: one tokenize, a pooled byte buffer, hash of the buffer. No parse, no formatter. Fingerprints carry a v2: prefix. The v1.17 behaviour stays available as fingerprint.ModeLegacy (Options, NormalizeWithOptions, FingerprintWithOptions, VersionOf). The delimited-name predicate moves verbatim to internal/namequote (D18, no public API) and the parser delegates to it. Legacy golden columns are byte-identical to origin/main (checked with the awk comparison); the new columns are added.

Test Plan

  • task check, task test:cbinding, go test -count=1 ./pkg/compatibility/..., staticcheck on pkg/fingerprint and internal/...: all pass on head 528b38c (the earlier 120s test:race timeout on cmd/gosqlx/cmd and pkg/gosqlx under machine load was a re-run with -timeout 900s, reported as a re-run, not a gate result; CI Race Detector shards pass). FuzzNormalizeV2 (now with a dialect argument and in the CI Fuzz Regression step) ran clean for 3 consecutive 60s runs after fixing five fuzz-found idempotency defects; 8 seeds are committed.
  • Tests T1-T14 (rules, literal and layout invariance, mutation converses, corpus equivalence vs legacy, retokenize soundness, allocation guards, fuzz, namequote table).
  • Grouping equivalence v2 vs legacy on the RW corpus (T5): realworld-default 2813 statements, 2253 accepted by legacy and 2797 by v2; 2153 legacy groups vs 2145 v2 groups; 1 pair legacy-merged/v2-split, 11 pairs v2-merged/legacy-split. All-default: 2999 statements, 2318 vs 2310 groups. Per-dialect tables and the classified divergence list (cast spelling, reorder, paren, keyword-name, ident-case = 7 pinned, split = 4) are in the test output; every divergence class is pinned in tests.
  • Local benchstat (indicative, shared M4, arm64, count=6, base = origin/main 1f82800): BenchmarkNormalize geomean -43.3% (simple_select -51.1%, join -35.5%, complex -38.3%); BenchmarkFingerprint -33.5% to -53.4%. Legacy no-regression pair (head Benchmark*Legacy vs base BenchmarkNormalize/Fingerprint): geomean -1.4%, no regression beyond +3.0% (noise).

TiDB bench-compare (workflow run 43271354829, amd64 AMD EPYC 9V74, count=10, head 7463aba; these numbers (0.80/0.93/1.06) predate the 528b38c hot-loop change and were not re-run; that commit changes mysql/mariadb/bigquery double-quote handling and @@ spacing). PLAN section 7 target gosqlx/tidb_digest geomean <= 1.0: MET (C1 was 1.99x Normalize, 2.28x Fingerprint). Geomean time ratio GoSQLX/tidb_digest: gosqlx 0.80, gosqlx_fingerprint 0.93, gosqlx_legacy 1.81, gosqlx_fingerprint_legacy 2.12. Per-query median ns/op:

query gosqlx gosqlx_fingerprint gosqlx_legacy gosqlx_fingerprint_legacy tidb_digest gosqlx/tidb fp/tidb
aggregate 1654 1900 3986 4500 1910 0.87 0.99
complex 6636 6856 13552 14500 7551 0.88 0.91
cte 1984 2192 4795 5313 2512 0.79 0.87
delete 694 875 1496 1958 982 0.71 0.89
insert 1367 1544 2462 2970 1460 0.94 1.06
join 2486 2732 5206 5854 2942 0.85 0.93
select_where 1543 1794 3844 4406 2171 0.71 0.83
simple_select 550 742 1529 1958 746 0.74 0.99
subquery 1334 1582 3456 3971 1704 0.78 0.93
update 1024 1228 2412 2932 1298 0.79 0.95
window 1508 1729 3056 3550 1973 0.76 0.88

Only insert for the fingerprint cell is above 1.0 (1.06).

  • CPU profile of Normalize/complex (arm64, 3s): tokenize ~37% cum, appendNormalV2 ~37% (flat 34%), hash and pool under 5%, runtime alloc/madvise noise ~10%. The join and complex misses of the PLAN <= 0.30x threshold are therefore bounded by the shared tokenizer plus the normal-form writer, not by allocation.
  • CI perf gate (layout-randomized, amd64 AMD EPYC 9V74, run 43274969051, final head 528b38c, merge ref 11ae83b, base 1f82800, 20 layouts per arm): PASS. No gated time regression (suite geomean -5.05%, p<0.001), allocs/op 5/21/62 -> 1, B/op -95.92/-94.22/-93.73%, A/A' control 0 of 52 benchmarks moved. Targets: Normalize/simple_select -67.80% (0.32x), Normalize/join -53.85% (0.46x), Normalize/complex -53.87% (0.46x). Earlier runs: EPYC 7763 run 43265055242 (head 7463aba) -64.82/-53.62/-56.21% (0.35x/0.46x/0.44x); EPYC 9V45 (head b6cddc9) -74.31/-61.11/-62.49% (0.26x/0.39x/0.375x). PLAN threshold (<= 0.30x): no target reaches it on the final head; simple_select met it only on the 9V45 run, and join and complex miss on every run. The runners differ, so runs are not comparable across CPUs. The final run's Low power list has 5 unrelated benchmarks (L010, LintParallel/rules, ParseE2E/subquery, ParseE2E/delete, Tokenize/scratch/insert); the Normalize targets are not on it.

Breaking Changes

See CHANGELOG. Fingerprint format becomes v2: plus 64 hex (67 bytes; CHAR(64) columns must widen). Normalized text and grouping change (documented classes). Default mode no longer reports syntax errors (use gosqlx.Validate). Normalized text is a grouping key, not guaranteed parseable SQL (the symmetric reparse oracle T7 pins 44 implicit-alias parser-limitation failures plus 2 allowlisted residues, 46 in total; legacy_reparse_skipped=8, eligible 2416, reparsed 2370). Migration: FingerprintWithOptions(sql, fingerprint.Options{Mode: fingerprint.ModeLegacy}) reproduces v1.17 exactly; VersionOf tells versions apart.

Fix round 2 (commit 528b38c): changelog now quotes the final-head 9V74 gate run 43274969051; v2 now redacts double-quoted strings as literals for Dialect mysql/mariadb/bigquery (documented coarsening, default dialects unchanged); @@version / @@session.sql_mode no longer get a space; the limit/offset T7 skip class is pinned to 0; doc nits (oracle counts removed, deep frames > 32 get full rules, normative).

T5 legacy-merge/v2-split pairs (7 allowlisted, each with a reason in legacyMergeV2SplitAllowlist, equivalence_test.go):

scope pairs
realworld-default 1
clickhouse 1
mysql 4
oracle 1
total 7

Behaviour changes needing maintainer sign-off

Merge-blocking and not yet recorded.

  • M1, no syntax errors reported in token mode: normalizeTokens in pkg/fingerprint/tokens.go (reached from NormalizeWithOptions/FingerprintWithOptions in pkg/fingerprint/fingerprint.go, and gosqlx.Normalize/Fingerprint in pkg/gosqlx/gosqlx.go) only tokenizes, so it reports lexical errors but never parse errors (documented in pkg/fingerprint/doc.go; use gosqlx.Validate).
  • M2, coarsening classes: dialectToken in pkg/fingerprint/tokens.go (enabled in appendNormalV2 for keywords.DialectMySQL, DialectMariaDB and DialectBigQuery) treats double-quoted tokens as string literals and redacts them to ?, so ANSI_QUOTES table/column names become ?; the other grouping coarsenings are the rule set in appendNormalV2.
  • M3, 67-byte v2: value: normalizeTokens in pkg/fingerprint/tokens.go builds the fingerprint from VersionPrefix ("v2:", pkg/fingerprint/options.go) plus 64 hex characters, so CHAR(64) columns must widen; VersionOf in pkg/fingerprint/options.go tells the versions apart.

🤖 Generated with Claude Code

https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2

Test and others added 2 commits November 7, 2026 08:49
… [W5-08]

Normalize/Fingerprint now tokenize once and build the normal form from the
token stream (no parse, no formatter); fingerprints carry a "v2:" prefix.
Legacy v1.17 behaviour is kept behind fingerprint.ModeLegacy. The delimited
name predicate moves verbatim to internal/namequote and the parser delegates.

BREAKING: fingerprint format, grouping and error contract change; see the
CHANGELOG Breaking Changes migration note.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
@vercel

vercel Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
gosqlx-website Ignored Ignored Preview Nov 15, 2026 1:42pm UTC

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 23s —— View job


Review of PR #960: token-stream fingerprint (v2)

I read tokens.go, fingerprint.go and options.go in full. I did not read the tests, the other changed files or the diff, and I did not run any builds or tests. Treat this as a partial review.

Overall: The design is sound. It tokenizes once, uses a pooled buffer, hashes the buffer, and keeps the old behaviour behind ModeLegacy. The API split is clean, and the breaking change is documented and has a migration path.

Issues and suggestions

  1. Hand-rolled state machine is hard to verify (tokens.go:187-484).

    • appendNormalV2 is about 300 lines of interleaved flags. The flags include pendingAS, pendingInto, unaryColon, ddl, with, main, selected, and per-frame modifier, in and row.
    • The fix-ups mutate the buffer retroactively. redactModifierPrefix, finishElement, the row-dedup truncation, and the unaryColon backspace all edit bytes already written.
    • These are the riskiest parts. The tests T1-T14 and the fuzz test are the right safeguard. Please confirm that fuzz checks idempotence (Normalize(Normalize(x)) == Normalize(x)) and that it runs in CI. Several comments (N6/N10/N15/N16) show idempotence bugs were already found by hand.
    • The code would also be easier to maintain if the appendNormalV2 body were split into small methods on a state struct.
  2. Stack depth overflow (tokens.go:188, 249, 439).

    • The frame stack is fixed at 32. Beyond that, f is nil and depth keeps growing, so deeply nested parentheses lose IN-list dedup, cast and modifier handling.
    • The result stays deterministic, so grouping is not corrupted. The effect is that nesting deeper than 32 is normalized less aggressively. Please add a test with more than 32 levels, so a silent index panic cannot slip in. The guards depth <= len(stack) look correct on a read-through.
    • Line 256 sets parentRows.rowPrevious only when depth <= len(stack). At depth == 0 it uses root, which is fine.
  3. Fingerprint stability of the normal form.

    • The hash covers the output bytes. Any future tweak to the rules silently changes every fingerprint.
    • Bumping Version on every rule change is implied but not enforced. The goldens in testdata/goldens/fingerprint.sha256 cover this. Please state in doc.go that any change to the golden file requires a version bump.
  4. Errors versus ErrEmpty.

    • Empty or comment-only input now returns ErrEmpty rather than the old parse error. This is a behaviour change, and the CHANGELOG already notes it.
    • gosqlx.Fingerprint callers who matched on specific error text will break. Please double-check that the CHANGELOG mentions this.
  5. VersionOf (options.go:50).

    • It is correct. Legacy 64-hex values pass as version 1. A v2: prefix with the wrong length falls through to len(fp) != 64 and returns 0.
    • A 64-character string beginning with v2: also returns 0, because v is not a hex digit.
  6. delimited heuristic (tokens.go:67-76).

    • The q == 0 branch infers a quoted identifier from span width versus value length, and from multi-line spans. It is a heuristic coupled to the tokenizer's span semantics. A tokenizer change could break it silently.
    • The comment says it mirrors the parser. Better still, share one function with the parser through internal/namequote. The PR already moved the predicate there, so this is a small remaining duplication.
  7. Minor points.

    • options.go:6-10: the imports are not grouped or sorted the way gofmt/goimports would leave them. gofmt accepts them, so this is cosmetic.
    • normalizeTokens ignores the ok result of the pool assertion only by falling back to a fresh buffer, which is fine. putNormalBuffer drops buffers over 64 KiB, which is a sensible cap.
    • Dialect is applied with tkz.SetDialect only when non-empty. The default dialect when empty is PostgreSQL, as documented. Please make sure PutTokenizer resets the dialect, so a pooled tokenizer does not leak one into another call.
    • FingerprintWithOptions legacy path uses fmt.Sprintf("%x"). hex.EncodeToString would avoid the reflection cost, but that path is legacy, so it matters little.

Test coverage

The coverage described in the PR body is thorough. It includes corpus equivalence against legacy, mutation converses, allocation guards, fuzz tests and golden files. The grouping divergences (1 legacy-merged/v2-split pair, 11 v2-merged/legacy-split pairs) are quantified and pinned. I could not independently verify any of it.

Before merge

  • The amd64 perf-gate verdict is still "PENDING". It is the stated authority for this change and must be filled in.
  • Maintainer decisions M1-M3 are still open.
  • No security concerns found. Input is only tokenized, and there is no execution or I/O.

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown

Perf gate: PASS (shadow mode: report only, this check never fails)

Base 4f8dae1b9859, head 79de86c2da3c. 20 interleaved rounds, 20 randomized code layouts per arm (-randlayout seeds 437204068792..437204068811; A/A' 437204068812..437204068831), benchtime 100ms, -cpu 1, GOMEMLIMIT 64MiB, pinned to CPU 3 (SMT siblings 2-3) with taskset.

Runner CPU: AMD EPYC 7763 64-Core Processor (x86_64). Base and head ran interleaved in this job, so the tables show their ratios only; absolute timings (raw files in the artifact) hold for this CPU and this job alone and are never compared across jobs.

  • 3 benchmark(s) faster by 3% or more

Rules: each arm runs 20 randomized code layouts (-ldflags=-randlayout); time regresses at >= +3% when significant at a false discovery rate of 0.01 over the suite (Benjamini-Hochberg on exact Mann-Whitney U p-values of the per-layout medians, so a regression must hold across layouts, not within one binary), or a suite geomean >= +1.5% (per-layout suite geomeans, same test); allocs/op must not increase; B/op must not rise more than 2%; instructions/op must not rise more than 1.5%. The A/A' control runs the base revision under a disjoint set of layout seeds; when more than 10% of benchmarks move over 3% in it, the run is noisy: re-run once, then inconclusive (neutral).

Gated suite (GOGC=off, GOMAXPROCS=1)

benchmark time p A/A' spread B/op allocs/op status
ParseE2E/simple_select -0.87% 0.029 -0.38% 0.94% +0.00% 3 → 3 ok
ParseE2E/select_where -0.69% 0.234 -0.55% 2.42% +0.00% 16 → 16 ok
ParseE2E/join -0.20% 0.665 -0.55% 1.97% +0.00% 15 → 15 ok
ParseE2E/aggregate -0.76% 0.516 -0.80% 4.14% +0.00% 13 → 13 ok
ParseE2E/subquery -1.37% 0.107 -0.20% 3.68% +0.00% 10 → 10 ok
ParseE2E/cte -0.24% 0.735 -0.47% 2.53% +0.00% 16 → 16 ok
ParseE2E/window -0.35% 0.372 +0.14% 0.99% +0.00% 9 → 9 ok
ParseE2E/complex -0.43% 0.490 -1.01% 1.40% +0.00% 43 → 43 ok
ParseE2E/insert +0.28% 0.626 +0.05% 2.17% +0.00% 17 → 17 ok
ParseE2E/update -0.16% 0.818 +0.42% 1.27% +0.00% 13 → 13 ok
ParseE2E/delete -0.33% 0.425 -0.24% 2.08% +0.00% 5 → 5 ok
ParseRW/postgresql/medium +0.05% 0.935 +0.30% 1.60% -0.00% 583 → 583 ok
ParseRW/mysql/medium +0.41% 0.221 -0.13% 1.18% +0.00% 334 → 334 ok
ParseRW/mariadb/medium -0.20% 0.620 -0.34% 1.01% -0.00% 470 → 470 ok
ParseRW/sqlserver/medium -0.56% 0.149 -0.60% 1.59% -0.00% 853 → 853 ok
ParseRW/oracle/medium -0.21% 0.620 +0.06% 1.20% +0.00% 962 → 962 ok
ParseRW/sqlite/medium -0.77% 0.327 +0.08% 3.16% +0.00% 318 → 318 ok
ParseRW/snowflake/medium -0.51% 0.253 +0.10% 1.36% +0.00% 630 → 630 ok
ParseRW/clickhouse/medium +0.11% 0.925 +0.02% 2.55% +0.00% 336 → 336 ok
ParseFresh/simple_select -0.21% 0.705 +0.05% 1.48% +0.00% 4 → 4 ok
ParseFresh/join -1.00% 0.102 -0.21% 2.10% +0.00% 16 → 16 ok
ParseFresh/complex -0.10% 0.850 +0.10% 1.70% +0.00% 44 → 44 ok
ParseFresh/insert -0.05% 0.839 +0.20% 1.00% +0.00% 18 → 18 ok
ParseFresh/update -0.52% 0.199 -0.23% 1.63% +0.00% 14 → 14 ok
ParseFresh/delete -0.38% 0.579 -0.33% 1.90% +0.00% 6 → 6 ok
PoolGetPut/serial +1.09% 0.027 +0.52% 0.85% +0.00% 0 → 0 ok
Tokenize/scratch/join -0.78% 0.172 -1.43% 1.94% +0.00% 0 → 0 ok
Tokenize/scratch/complex -0.70% 0.137 -2.51% 1.24% +0.00% 0 → 0 ok
Tokenize/scratch/insert -0.73% 0.478 -1.50% 2.56% +0.00% 0 → 0 ok
Normalize/simple_select -62.80% <0.001 +0.31% 1.26% -95.92% 5 → 1 faster, -allocs
Normalize/join -48.57% <0.001 +0.19% 1.12% -94.22% 21 → 1 faster, -allocs
Normalize/complex -51.42% <0.001 +0.40% 1.14% -93.73% 62 → 1 faster, -allocs
TransformParse/simple_select +1.28% 0.088 +0.60% 2.09% +0.00% 3 → 3 ok
TransformParse/join +0.99% 0.096 +0.52% 1.17% +0.00% 15 → 15 ok
TransformParse/complex -0.13% 0.820 -0.98% 1.98% +0.00% 43 → 43 ok
LintParallel/rules -0.53% 0.512 -0.23% 2.85% +0.00% 68934 → 68934 ok
LintParallel/lint +0.20% 0.583 +0.20% 1.18% -0.00% 126680 → 126680 ok
LintRW/postgresql +0.13% 0.495 -0.04% 0.55% +0.00% 14130 → 14130 ok
LintRW/mysql +0.18% 0.659 +0.22% 1.08% +0.00% 17457 → 17457 ok
LintRW/mariadb -0.45% 0.602 +0.56% 1.33% +0.00% 15755 → 15755 ok
LintRW/sqlserver +0.10% 0.738 -0.44% 1.43% +0.00% 13871 → 13871 ok
LintRW/oracle +0.47% 0.341 +0.43% 1.13% -0.00% 16857 → 16857 ok
LintRW/sqlite +0.15% 0.779 +0.51% 1.82% +0.00% 15739 → 15739 ok
LintRW/snowflake +0.16% 0.799 +0.33% 3.14% +0.00% 15204 → 15204 ok
LintRW/clickhouse -0.46% 0.718 -0.78% 3.07% +0.00% 17660 → 17660 ok
LintRule/L007 +0.45% 0.602 -0.93% 2.09% +0.00% 1194 → 1194 ok
LintRule/L010 -0.09% 0.398 -0.10% 0.97% +0.00% 0 → 0 ok
ParseError/syntax_mid/parse -0.23% 0.507 -0.26% 1.01% +0.00% 11 → 11 ok
ParseError/syntax_mid/validate -0.42% 0.234 +0.00% 1.01% +0.00% 11 → 11 ok
ParseError/syntax_end/parse -0.78% 0.148 -0.18% 1.75% +0.00% 14 → 14 ok
ParseError/syntax_end/validate -1.13% 0.148 -0.57% 1.72% +0.00% 14 → 14 ok
ParseError/tokenize/parse +0.24% 0.725 -0.49% 2.82% +0.00% 4 → 4 ok
ParseError/tokenize/validate -0.81% 0.208 +0.07% 1.51% +0.00% 21 → 21 ok
ParseError/long_prefix/parse -0.86% 0.265 -0.47% 1.46% -0.00% 213 → 213 ok
ParseError/long_prefix/validate -0.76% 0.277 -0.49% 1.59% +0.00% 213 → 213 ok
WASMBridge/parse/small +0.28% 0.449 +1.07% 1.23% +0.00% 8 → 8 ok
WASMBridge/parse/medium +0.05% 0.841 +0.35% 0.97% +0.00% 38 → 38 ok
WASMBridge/parse/large +0.61% 0.242 +0.56% 1.30% +0.00% 522 → 522 ok
WASMBridge/lint/small -0.47% 0.330 -0.92% 1.76% +0.00% 33 → 33 ok
WASMBridge/lint/medium +0.34% 0.327 -0.59% 1.01% +0.00% 100 → 100 ok
WASMBridge/lint/large +0.64% 0.495 -0.39% 1.23% -0.00% 1031 → 1031 ok
DetectDialect/empty +0.26% 0.291 -0.08% 16.44% +0.00% 0 → 0 ok
DetectDialect/generic +0.26% 0.213 -0.13% 0.66% +0.00% 0 → 0 ok
DetectDialect/postgresql +0.00% 0.903 +0.15% 0.49% +0.00% 0 → 0 ok
DetectDialect/mysql +0.13% 0.808 +0.20% 2.72% +0.00% 0 → 0 ok
DetectDialect/sqlserver +0.08% 0.818 -0.16% 1.43% +0.00% 0 → 0 ok
DetectDialect/oracle +0.11% 0.499 -0.03% 0.42% +0.00% 0 → 0 ok
DetectDialect/snowflake +0.04% 0.882 -0.19% 0.48% +0.00% 0 → 0 ok
DetectDialect/mariadb +0.14% 0.570 -0.07% 0.76% +0.00% 0 → 0 ok
DetectDialect/mixed +0.02% 0.860 +0.00% 0.28% +0.00% 0 → 0 ok
DetectDialect/large_64KB +0.03% 0.820 -0.03% 0.83% +0.00% 0 → 0 ok
geomean -3.43% <0.001 -0.16%

time and A/A' are Hodges-Lehmann shifts over the per-layout medians. spread: the base arm's robust standard deviation across its layouts (1.4826 x MAD, relative), i.e. how much this benchmark moves between builds that differ only in function placement, plus run-to-run noise; a change much smaller than it cannot be resolved in one run.

A/A' control (base revision, disjoint layout seeds): 0 of 71 benchmarks moved more than 3%; 0 would have been flagged by the time rule; suite geomean -0.16%, p 0.221.

Low power

3 benchmark(s) have under 80% estimated power to be flagged on their own for a +5% slowdown (their spread across layouts is too wide for 20 layouts at the Benjamini-Hochberg level alpha/m = 0.000141). A slowdown below the last column usually goes unflagged there; instructions/op, allocs/op and the suite geomean still guard them.

benchmark spread power at +5% slowdown caught 80% of the time
DetectDialect/empty 16.44% 0% +28.1%
ParseE2E/aggregate 4.14% 43% +6.4%
ParseE2E/subquery 3.68% 61% +5.7%

Power is estimated with the normal approximation for the U test (Noether): power = Phi(ln(1+shift)/s * sqrt(K/2) * sqrt(3/pi) - z(1 - alpha/(2m))), with s the spread and K the layouts per arm.

Instructions per operation (cmd/perfdriver)

case instructions/op status
e2e/aggregate +0.06% ok
e2e/complex -0.03% ok
e2e/cte +0.04% ok
e2e/delete -0.06% ok
e2e/insert -0.04% ok
e2e/join +0.01% ok
e2e/select_where +0.01% ok
e2e/simple_select +0.16% ok
e2e/subquery -0.00% ok
e2e/update +0.02% ok
e2e/window +0.02% ok
rw/postgresql/medium +0.00% ok
Secondary metric: default GC, GOMAXPROCS=1 (informational)
benchmark time p
ParseE2E/simple_select -0.20% 0.791
ParseE2E/select_where -0.19% 0.739
ParseE2E/join +0.04% 0.880
ParseE2E/aggregate -0.70% 0.257
ParseE2E/subquery -0.29% 0.570
ParseE2E/cte -0.04% 0.971
ParseE2E/window -0.84% 0.364
ParseE2E/complex -0.15% 0.739
ParseE2E/insert +1.18% 0.247
ParseE2E/update -0.38% 0.623
ParseE2E/delete -0.35% 0.762
ParseRW/postgresql/medium -0.31% 0.280
ParseRW/mysql/medium +0.28% 0.280
ParseRW/mariadb/medium +0.45% 0.353
ParseRW/sqlserver/medium +0.40% 0.529
ParseRW/oracle/medium -0.94% 0.190
ParseRW/sqlite/medium +0.19% 0.684
ParseRW/snowflake/medium -0.47% 0.684
ParseRW/clickhouse/medium -0.08% 0.912
ParseFresh/simple_select -0.18% 0.545
ParseFresh/join -0.15% 0.734
ParseFresh/complex -1.23% 0.009
ParseFresh/insert +0.32% 0.623
ParseFresh/update +0.26% 0.571
ParseFresh/delete -0.20% 0.739
PoolGetPut/serial +1.58% 0.043
Tokenize/scratch/join -0.54% 0.472
Tokenize/scratch/complex -1.65% 0.257
Tokenize/scratch/insert -1.50% 0.393
Normalize/simple_select -65.58% <0.001
Normalize/join -59.56% <0.001
Normalize/complex -58.78% <0.001
TransformParse/simple_select +0.47% 0.520
TransformParse/join +0.75% 0.393
TransformParse/complex +0.27% 0.912
LintParallel/rules +1.16% 0.105
LintParallel/lint -0.08% 0.971
LintRW/postgresql -0.60% 0.218
LintRW/mysql -0.14% 0.684
LintRW/mariadb +0.05% 0.853
LintRW/sqlserver +0.06% 0.912
LintRW/oracle -0.85% 0.353
LintRW/sqlite -0.30% 0.579
LintRW/snowflake +0.70% 0.353
LintRW/clickhouse -0.12% 0.971
LintRule/L007 +1.55% 0.143
LintRule/L010 +0.13% 0.529
ParseError/syntax_mid/parse -0.57% 0.496
ParseError/syntax_mid/validate +0.77% 0.307
ParseError/syntax_end/parse -1.02% 0.130
ParseError/syntax_end/validate -0.30% 0.705
ParseError/tokenize/parse +0.55% 0.496
ParseError/tokenize/validate -0.28% 0.796
ParseError/long_prefix/parse +1.55% 0.247
ParseError/long_prefix/validate -0.11% 0.912
WASMBridge/parse/small +0.31% 0.739
WASMBridge/parse/medium +1.07% 0.247
WASMBridge/parse/large +0.86% 0.353
WASMBridge/lint/small -0.18% 0.734
WASMBridge/lint/medium -0.13% 0.912
WASMBridge/lint/large +0.16% 0.631
DetectDialect/empty +0.24% 0.910
DetectDialect/generic +0.00% 1.000
DetectDialect/postgresql -0.19% 0.597
DetectDialect/mysql -0.19% 0.677
DetectDialect/sqlserver +0.05% 0.910
DetectDialect/oracle +0.10% 0.622
DetectDialect/snowflake +0.42% 0.273
DetectDialect/mariadb +0.18% 0.820
DetectDialect/mixed -0.23% 0.185
DetectDialect/large_64KB +0.66% 0.105
geomean -3.93%
Environment
date:        2026-11-15T13:45:55Z
base:        4f8dae1b985943b175148867807038c29c33ed2e
head:        working tree of 79de86c2da3cca8cb8df18f97827ff1eb0d8a2f9
cpu model:   AMD EPYC 7763 64-Core Processor (x86_64)
go:          go version go1.26.9 linux/amd64
kernel:      Linux 7.0.0-1017-azure x86_64
layouts:     20 per arm, -ldflags=-randlayout seeds 437204068792..437204068811 (base, head), 437204068812..437204068831 (A/A')
pinning:     pinned to CPU 3 (SMT siblings 2-3) with taskset
gcoff env:   GOGC=off GOMEMLIMIT=64MiB
clocksource: tsc
load:        10.39, 4.17, 1.57
instr count: cachegrind
perf probe:  Error: No supported events found. The instructions:u event is not supported. 

Architecture:                            x86_64
CPU(s):                                  4
Model name:                              AMD EPYC 7763 64-Core Processor
Thread(s) per core:                      2
Core(s) per socket:                      2
Socket(s):                               1
Hypervisor vendor:                       Microsoft
L1d cache:                               64 KiB (2 instances)
L1i cache:                               64 KiB (2 instances)
L2 cache:                                1 MiB (2 instances)
L3 cache:                                32 MiB (1 instance)
benchstat (primary metric, samples pooled across layouts: informational, the verdict uses the per-layout test)
goos: linux
goarch: amd64
pkg: github.com/ajitpratap0/GoSQLX/pkg/fingerprint
cpu: AMD EPYC 7763 64-Core Processor                
                        │     base     │                head                 │
                        │    sec/op    │   sec/op     vs base                │
Normalize/simple_select   2232.0n ± 1%   831.1n ± 2%  -62.76% (p=0.000 n=20)
Normalize/join             6.949µ ± 1%   3.572µ ± 1%  -48.60% (p=0.000 n=20)
Normalize/complex         19.125µ ± 1%   9.301µ ± 1%  -51.37% (p=0.000 n=20)
geomean                    6.669µ        3.022µ       -54.68%

                        │     base     │                head                │
                        │     B/op     │    B/op     vs base                │
Normalize/simple_select    784.00 ± 0%   32.00 ± 0%  -95.92% (p=0.000 n=20)
Normalize/join             2216.0 ± 0%   128.0 ± 0%  -94.22% (p=0.000 n=20)
Normalize/complex          6120.0 ± 0%   384.0 ± 0%  -93.73% (p=0.000 n=20)
geomean                   2.147Ki        116.3       -94.71%

                        │    base     │                head                │
                        │  allocs/op  │ allocs/op   vs base                │
Normalize/simple_select    5.000 ± 0%   1.000 ± 0%  -80.00% (p=0.000 n=20)
Normalize/join            21.000 ± 0%   1.000 ± 0%  -95.24% (p=0.000 n=20)
Normalize/complex         62.000 ± 0%   1.000 ± 0%  -98.39% (p=0.000 n=20)
geomean                    18.67        1.000       -94.64%

pkg: github.com/ajitpratap0/GoSQLX/pkg/gosqlx
                                │    base     │                head                │
                                │   sec/op    │   sec/op     vs base               │
ParseE2E/simple_select            1.569µ ± 1%   1.553µ ± 1%       ~ (p=0.028 n=20)
ParseE2E/select_where             3.434µ ± 6%   3.411µ ± 1%       ~ (p=0.234 n=20)
ParseE2E/join                     4.674µ ± 1%   4.659µ ± 1%       ~ (p=0.664 n=20)
ParseE2E/aggregate                3.420µ ± 3%   3.372µ ± 3%       ~ (p=0.516 n=20)
ParseE2E/subquery                 2.938µ ± 3%   2.889µ ± 3%       ~ (p=0.107 n=20)
ParseE2E/cte                      4.455µ ± 2%   4.430µ ± 1%       ~ (p=0.733 n=20)
ParseE2E/window                   2.840µ ± 1%   2.838µ ± 1%       ~ (p=0.372 n=20)
ParseE2E/complex                  12.01µ ± 1%   11.94µ ± 2%       ~ (p=0.490 n=20)
ParseE2E/insert                   2.155µ ± 2%   2.167µ ± 2%       ~ (p=0.625 n=20)
ParseE2E/update                   1.919µ ± 1%   1.921µ ± 1%       ~ (p=0.815 n=20)
ParseE2E/delete                   1.245µ ± 1%   1.242µ ± 1%       ~ (p=0.425 n=20)
ParseRW/postgresql/medium         168.2µ ± 2%   168.2µ ± 1%       ~ (p=0.931 n=20)
ParseRW/mysql/medium              93.74µ ± 1%   94.09µ ± 1%       ~ (p=0.221 n=20)
ParseRW/mariadb/medium            133.7µ ± 1%   133.7µ ± 1%       ~ (p=0.620 n=20)
ParseRW/sqlserver/medium          271.5µ ± 1%   270.0µ ± 1%       ~ (p=0.149 n=20)
ParseRW/oracle/medium             319.0µ ± 1%   318.3µ ± 1%       ~ (p=0.620 n=20)
ParseRW/sqlite/medium             72.00µ ± 2%   71.15µ ± 1%       ~ (p=0.327 n=20)
ParseRW/snowflake/medium          183.2µ ± 1%   182.9µ ± 1%       ~ (p=0.253 n=20)
ParseRW/clickhouse/medium         83.87µ ± 3%   84.61µ ± 2%       ~ (p=0.925 n=20)
ParseFresh/simple_select          2.862µ ± 1%   2.851µ ± 1%       ~ (p=0.703 n=20)
ParseFresh/join                   6.422µ ± 1%   6.384µ ± 1%       ~ (p=0.101 n=20)
ParseFresh/complex                15.52µ ± 1%   15.47µ ± 1%       ~ (p=0.846 n=20)
ParseFresh/insert                 3.773µ ± 1%   3.768µ ± 1%       ~ (p=0.836 n=20)
ParseFresh/update                 3.543µ ± 2%   3.533µ ± 0%       ~ (p=0.199 n=20)
ParseFresh/delete                 2.776µ ± 2%   2.761µ ± 1%       ~ (p=0.578 n=20)
PoolGetPut/serial                 66.95n ± 1%   67.86n ± 1%       ~ (p=0.025 n=20)
ParseError/syntax_mid/parse       2.349µ ± 1%   2.345µ ± 1%       ~ (p=0.507 n=20)
ParseError/syntax_mid/validate    2.139µ ± 1%   2.128µ ± 1%       ~ (p=0.234 n=20)
ParseError/syntax_end/parse       2.799µ ± 1%   2.787µ ± 1%       ~ (p=0.147 n=20)
ParseError/syntax_end/validate    2.624µ ± 1%   2.592µ ± 2%       ~ (p=0.147 n=20)
ParseError/tokenize/parse         928.0n ± 2%   927.5n ± 2%       ~ (p=0.723 n=20)
ParseError/tokenize/validate      2.797µ ± 1%   2.761µ ± 1%       ~ (p=0.208 n=20)
ParseError/long_prefix/parse      56.80µ ± 1%   56.00µ ± 3%       ~ (p=0.265 n=20)
ParseError/long_prefix/validate   55.02µ ± 1%   54.50µ ± 2%       ~ (p=0.277 n=20)
geomean                           8.122µ        8.089µ       -0.41%

                                │     base     │                head                 │
                                │     B/s      │     B/s       vs base               │
ParseE2E/simple_select            15.80Mi ± 1%   15.97Mi ± 1%       ~ (p=0.029 n=20)
ParseE2E/select_where             28.61Mi ± 5%   28.80Mi ± 1%       ~ (p=0.234 n=20)
ParseE2E/join                     25.71Mi ± 1%   25.79Mi ± 1%       ~ (p=0.673 n=20)
ParseE2E/aggregate                23.43Mi ± 3%   23.77Mi ± 3%       ~ (p=0.516 n=20)
ParseE2E/subquery                 26.62Mi ± 3%   27.07Mi ± 3%       ~ (p=0.107 n=20)
ParseE2E/cte                      26.55Mi ± 2%   26.69Mi ± 1%       ~ (p=0.743 n=20)
ParseE2E/window                   31.57Mi ± 1%   31.60Mi ± 1%       ~ (p=0.387 n=20)
ParseE2E/complex                  30.02Mi ± 1%   30.19Mi ± 2%       ~ (p=0.499 n=20)
ParseE2E/insert                   33.63Mi ± 2%   33.45Mi ± 2%       ~ (p=0.644 n=20)
ParseE2E/update                   27.32Mi ± 1%   27.31Mi ± 1%       ~ (p=0.815 n=20)
ParseE2E/delete                   39.83Mi ± 1%   39.92Mi ± 1%       ~ (p=0.449 n=20)
ParseRW/postgresql/medium         21.33Mi ± 2%   21.33Mi ± 1%       ~ (p=0.931 n=20)
ParseRW/mysql/medium              28.87Mi ± 1%   28.76Mi ± 1%       ~ (p=0.199 n=20)
ParseRW/mariadb/medium            26.52Mi ± 1%   26.52Mi ± 1%       ~ (p=0.588 n=20)
ParseRW/sqlserver/medium          24.51Mi ± 1%   24.64Mi ± 1%       ~ (p=0.136 n=20)
ParseRW/oracle/medium             26.28Mi ± 1%   26.34Mi ± 1%       ~ (p=0.606 n=20)
ParseRW/sqlite/medium             28.69Mi ± 2%   29.03Mi ± 1%       ~ (p=0.324 n=20)
ParseRW/snowflake/medium          27.76Mi ± 1%   27.80Mi ± 1%       ~ (p=0.256 n=20)
ParseRW/clickhouse/medium         26.87Mi ± 3%   26.63Mi ± 2%       ~ (p=0.920 n=20)
ParseError/syntax_mid/parse       19.49Mi ± 1%   19.53Mi ± 1%       ~ (p=0.516 n=20)
ParseError/syntax_mid/validate    21.41Mi ± 1%   21.52Mi ± 1%       ~ (p=0.223 n=20)
ParseError/syntax_end/parse       13.29Mi ± 1%   13.35Mi ± 1%       ~ (p=0.136 n=20)
ParseError/syntax_end/validate    14.17Mi ± 1%   14.35Mi ± 2%       ~ (p=0.143 n=20)
ParseError/tokenize/parse         47.27Mi ± 2%   47.30Mi ± 2%       ~ (p=0.723 n=20)
ParseError/tokenize/validate      15.69Mi ± 2%   15.89Mi ± 1%       ~ (p=0.218 n=20)
ParseError/long_prefix/parse      40.66Mi ± 1%   41.25Mi ± 3%       ~ (p=0.262 n=20)
ParseError/long_prefix/validate   41.99Mi ± 1%   42.38Mi ± 2%       ~ (p=0.268 n=20)
geomean                           25.93Mi        26.06Mi       +0.48%

                                │      base      │                 head                  │
                                │      B/op      │     B/op      vs base                 │
ParseE2E/simple_select              704.0 ± 0%       704.0 ± 0%       ~ (p=1.000 n=20) ¹
ParseE2E/select_where             1.477Ki ± 0%     1.477Ki ± 0%       ~ (p=1.000 n=20) ¹
ParseE2E/join                     1.789Ki ± 0%     1.789Ki ± 0%       ~ (p=1.000 n=20) ¹
ParseE2E/aggregate                1.719Ki ± 0%     1.719Ki ± 0%       ~ (p=1.000 n=20) ¹
ParseE2E/subquery                 1.609Ki ± 0%     1.609Ki ± 0%       ~ (p=1.000 n=20) ¹
ParseE2E/cte                      2.070Ki ± 0%     2.070Ki ± 0%       ~ (p=1.000 n=20) ¹
ParseE2E/window                   1.234Ki ± 0%     1.234Ki ± 0%       ~ (p=1.000 n=20) ¹
ParseE2E/complex                  4.885Ki ± 0%     4.885Ki ± 0%       ~ (p=1.000 n=20) ¹
ParseE2E/insert                     824.0 ± 0%       824.0 ± 0%       ~ (p=1.000 n=20) ¹
ParseE2E/update                     736.0 ± 0%       736.0 ± 0%       ~ (p=1.000 n=20) ¹
ParseE2E/delete                     400.0 ± 0%       400.0 ± 0%       ~ (p=1.000 n=20) ¹
ParseRW/postgresql/medium         64.43Ki ± 0%     64.42Ki ± 0%       ~ (p=0.691 n=20)
ParseRW/mysql/medium              40.70Ki ± 0%     40.70Ki ± 0%       ~ (p=1.000 n=20)
ParseRW/mariadb/medium            58.53Ki ± 0%     58.53Ki ± 0%       ~ (p=0.538 n=20)
ParseRW/sqlserver/medium          81.26Ki ± 0%     81.26Ki ± 0%       ~ (p=0.110 n=20)
ParseRW/oracle/medium             100.9Ki ± 0%     100.9Ki ± 0%       ~ (p=0.914 n=20)
ParseRW/sqlite/medium             37.30Ki ± 0%     37.30Ki ± 0%       ~ (p=1.000 n=20) ¹
ParseRW/snowflake/medium          73.29Ki ± 0%     73.29Ki ± 0%       ~ (p=1.000 n=20)
ParseRW/clickhouse/medium         32.99Ki ± 0%     32.99Ki ± 0%       ~ (p=1.000 n=20) ¹
ParseFresh/simple_select            784.0 ± 0%       784.0 ± 0%       ~ (p=1.000 n=20) ¹
ParseFresh/join                   1.867Ki ± 0%     1.867Ki ± 0%       ~ (p=1.000 n=20) ¹
ParseFresh/complex                4.961Ki ± 0%     4.961Ki ± 0%       ~ (p=1.000 n=20) ¹
ParseFresh/insert                   904.0 ± 0%       904.0 ± 0%       ~ (p=1.000 n=20) ¹
ParseFresh/update                   816.0 ± 0%       816.0 ± 0%       ~ (p=1.000 n=20) ¹
ParseFresh/delete                   480.0 ± 0%       480.0 ± 0%       ~ (p=1.000 n=20) ¹
PoolGetPut/serial                   0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=20) ¹
ParseError/syntax_mid/parse         552.0 ± 0%       552.0 ± 0%       ~ (p=1.000 n=20) ¹
ParseError/syntax_mid/validate      552.0 ± 0%       552.0 ± 0%       ~ (p=1.000 n=20) ¹
ParseError/syntax_end/parse       1.320Ki ± 0%     1.320Ki ± 0%       ~ (p=1.000 n=20) ¹
ParseError/syntax_end/validate    1.320Ki ± 0%     1.320Ki ± 0%       ~ (p=1.000 n=20) ¹
ParseError/tokenize/parse           272.0 ± 0%       272.0 ± 0%       ~ (p=1.000 n=20) ¹
ParseError/tokenize/validate      1.758Ki ± 0%     1.758Ki ± 0%       ~ (p=1.000 n=20) ¹
ParseError/long_prefix/parse      17.14Ki ± 0%     17.14Ki ± 0%       ~ (p=0.083 n=20)
ParseError/long_prefix/validate   17.07Ki ± 0%     17.07Ki ± 0%       ~ (p=1.000 n=20) ¹
geomean                                        ²                 -0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                                │     base     │                head                 │
                                │  allocs/op   │ allocs/op   vs base                 │
ParseE2E/simple_select            3.000 ± 0%     3.000 ± 0%       ~ (p=1.000 n=20) ¹
ParseE2E/select_where             16.00 ± 0%     16.00 ± 0%       ~ (p=1.000 n=20) ¹
ParseE2E/join                     15.00 ± 0%     15.00 ± 0%       ~ (p=1.000 n=20) ¹
ParseE2E/aggregate                13.00 ± 0%     13.00 ± 0%       ~ (p=1.000 n=20) ¹
ParseE2E/subquery                 10.00 ± 0%     10.00 ± 0%       ~ (p=1.000 n=20) ¹
ParseE2E/cte                      16.00 ± 0%     16.00 ± 0%       ~ (p=1.000 n=20) ¹
ParseE2E/window                   9.000 ± 0%     9.000 ± 0%       ~ (p=1.000 n=20) ¹
ParseE2E/complex                  43.00 ± 0%     43.00 ± 0%       ~ (p=1.000 n=20) ¹
ParseE2E/insert                   17.00 ± 0%     17.00 ± 0%       ~ (p=1.000 n=20) ¹
ParseE2E/update                   13.00 ± 0%     13.00 ± 0%       ~ (p=1.000 n=20) ¹
ParseE2E/delete                   5.000 ± 0%     5.000 ± 0%       ~ (p=1.000 n=20) ¹
ParseRW/postgresql/medium         583.0 ± 0%     583.0 ± 0%       ~ (p=1.000 n=20) ¹
ParseRW/mysql/medium              334.0 ± 0%     334.0 ± 0%       ~ (p=1.000 n=20) ¹
ParseRW/mariadb/medium            470.0 ± 0%     470.0 ± 0%       ~ (p=1.000 n=20) ¹
ParseRW/sqlserver/medium          853.0 ± 0%     853.0 ± 0%       ~ (p=1.000 n=20) ¹
ParseRW/oracle/medium             962.0 ± 0%     962.0 ± 0%       ~ (p=1.000 n=20) ¹
ParseRW/sqlite/medium             318.0 ± 0%     318.0 ± 0%       ~ (p=1.000 n=20) ¹
ParseRW/snowflake/medium          630.0 ± 0%     630.0 ± 0%       ~ (p=1.000 n=20) ¹
ParseRW/clickhouse/medium         336.0 ± 0%     336.0 ± 0%       ~ (p=1.000 n=20) ¹
ParseFresh/simple_select          4.000 ± 0%     4.000 ± 0%       ~ (p=1.000 n=20) ¹
ParseFresh/join                   16.00 ± 0%     16.00 ± 0%       ~ (p=1.000 n=20) ¹
ParseFresh/complex                44.00 ± 0%     44.00 ± 0%       ~ (p=1.000 n=20) ¹
ParseFresh/insert                 18.00 ± 0%     18.00 ± 0%       ~ (p=1.000 n=20) ¹
ParseFresh/update                 14.00 ± 0%     14.00 ± 0%       ~ (p=1.000 n=20) ¹
ParseFresh/delete                 6.000 ± 0%     6.000 ± 0%       ~ (p=1.000 n=20) ¹
PoolGetPut/serial                 0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
ParseError/syntax_mid/parse       11.00 ± 0%     11.00 ± 0%       ~ (p=1.000 n=20) ¹
ParseError/syntax_mid/validate    11.00 ± 0%     11.00 ± 0%       ~ (p=1.000 n=20) ¹
ParseError/syntax_end/parse       14.00 ± 0%     14.00 ± 0%       ~ (p=1.000 n=20) ¹
ParseError/syntax_end/validate    14.00 ± 0%     14.00 ± 0%       ~ (p=1.000 n=20) ¹
ParseError/tokenize/parse         4.000 ± 0%     4.000 ± 0%       ~ (p=1.000 n=20) ¹
ParseError/tokenize/validate      21.00 ± 0%     21.00 ± 0%       ~ (p=1.000 n=20) ¹
ParseError/long_prefix/parse      213.0 ± 0%     213.0 ± 0%       ~ (p=1.000 n=20) ¹
ParseError/long_prefix/validate   213.0 ± 0%     213.0 ± 0%       ~ (p=1.000 n=20) ¹
geomean                                      ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                          │    base     │                head                │
                          │  sec/stmt   │  sec/stmt    vs base               │
ParseRW/postgresql/medium   14.01µ ± 2%   14.01µ ± 1%       ~ (p=0.931 n=20)
ParseRW/mysql/medium        11.72µ ± 1%   11.76µ ± 1%       ~ (p=0.218 n=20)
ParseRW/mariadb/medium      14.86µ ± 1%   14.86µ ± 1%       ~ (p=0.615 n=20)
ParseRW/sqlserver/medium    12.93µ ± 1%   12.86µ ± 1%       ~ (p=0.144 n=20)
ParseRW/oracle/medium       13.29µ ± 1%   13.26µ ± 1%       ~ (p=0.620 n=20)
ParseRW/sqlite/medium       12.00µ ± 2%   11.86µ ± 1%       ~ (p=0.317 n=20)
ParseRW/snowflake/medium    14.09µ ± 1%   14.07µ ± 1%       ~ (p=0.256 n=20)
ParseRW/clickhouse/medium   11.98µ ± 3%   12.09µ ± 2%       ~ (p=0.925 n=20)
geomean                     13.07µ        13.05µ       -0.11%

pkg: github.com/ajitpratap0/GoSQLX/pkg/linter
                   │     base     │                head                 │
                   │    sec/op    │    sec/op     vs base               │
LintParallel/rules   19.61m ±  4%   19.57m ±  1%       ~ (p=0.512 n=20)
LintParallel/lint    37.67m ±  1%   37.76m ±  1%       ~ (p=0.583 n=20)
LintRW/postgresql    5.015m ±  1%   5.015m ±  0%       ~ (p=0.495 n=20)
LintRW/mysql         5.388m ±  1%   5.398m ±  1%       ~ (p=0.659 n=20)
LintRW/mariadb       4.900m ±  1%   4.887m ±  2%       ~ (p=0.602 n=20)
LintRW/sqlserver     4.620m ±  1%   4.618m ±  1%       ~ (p=0.738 n=20)
LintRW/oracle        5.522m ±  1%   5.546m ±  1%       ~ (p=0.341 n=20)
LintRW/sqlite        4.431m ±  2%   4.439m ±  1%       ~ (p=0.779 n=20)
LintRW/snowflake     4.699m ±  2%   4.731m ±  3%       ~ (p=0.799 n=20)
LintRW/clickhouse    5.295m ±  3%   5.295m ±  4%       ~ (p=0.718 n=20)
LintRule/L007        2.761m ±  1%   2.764m ±  3%       ~ (p=0.602 n=20)
LintRule/L010        883.3µ ± 21%   698.4µ ± 27%       ~ (p=0.398 n=20)
geomean              5.440m         5.340m        -1.83%

                  │     base     │                head                 │
                  │     B/s      │     B/s       vs base               │
LintRW/postgresql   5.646Mi ± 1%   5.646Mi ± 0%       ~ (p=0.585 n=20)
LintRW/mysql        6.075Mi ± 1%   6.061Mi ± 1%       ~ (p=0.683 n=20)
LintRW/mariadb      6.223Mi ± 1%   6.242Mi ± 2%       ~ (p=0.606 n=20)
LintRW/sqlserver    6.704Mi ± 1%   6.709Mi ± 0%       ~ (p=0.784 n=20)
LintRW/oracle       7.191Mi ± 1%   7.162Mi ± 1%       ~ (p=0.310 n=20)
LintRW/sqlite       6.475Mi ± 1%   6.471Mi ± 1%       ~ (p=0.773 n=20)
LintRW/snowflake    6.638Mi ± 2%   6.595Mi ± 3%       ~ (p=0.804 n=20)
LintRW/clickhouse   6.127Mi ± 3%   6.127Mi ± 4%       ~ (p=0.733 n=20)
geomean             6.370Mi        6.362Mi       -0.12%

                   │      base      │                 head                  │
                   │      B/op      │     B/op      vs base                 │
LintParallel/rules   5.559Mi ± 0%     5.559Mi ± 0%       ~ (p=0.931 n=20)
LintParallel/lint    17.95Mi ± 0%     17.95Mi ± 0%       ~ (p=0.857 n=20)
LintRW/postgresql    1.966Mi ± 0%     1.966Mi ± 0%       ~ (p=0.597 n=20)
LintRW/mysql         2.508Mi ± 0%     2.508Mi ± 0%       ~ (p=0.167 n=20)
LintRW/mariadb       2.280Mi ± 0%     2.280Mi ± 0%       ~ (p=0.273 n=20)
LintRW/sqlserver     1.971Mi ± 0%     1.971Mi ± 0%       ~ (p=0.337 n=20)
LintRW/oracle        2.435Mi ± 0%     2.435Mi ± 0%       ~ (p=0.233 n=20)
LintRW/sqlite        2.173Mi ± 0%     2.173Mi ± 0%       ~ (p=0.524 n=20)
LintRW/snowflake     2.172Mi ± 0%     2.172Mi ± 0%       ~ (p=0.878 n=20)
LintRW/clickhouse    2.440Mi ± 0%     2.440Mi ± 0%       ~ (p=0.693 n=20)
LintRule/L007        92.98Ki ± 0%     92.98Ki ± 0%       ~ (p=1.000 n=20) ¹
LintRule/L010          0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=20) ¹
geomean                           ²                 +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                   │     base      │                 head                 │
                   │   allocs/op   │  allocs/op   vs base                 │
LintParallel/rules   68.93k ± 0%     68.93k ± 0%       ~ (p=0.948 n=20)
LintParallel/lint    126.7k ± 0%     126.7k ± 0%       ~ (p=0.322 n=20)
LintRW/postgresql    14.13k ± 0%     14.13k ± 0%       ~ (p=0.342 n=20)
LintRW/mysql         17.46k ± 0%     17.46k ± 0%       ~ (p=0.716 n=20)
LintRW/mariadb       15.76k ± 0%     15.76k ± 0%       ~ (p=1.000 n=20)
LintRW/sqlserver     13.87k ± 0%     13.87k ± 0%       ~ (p=1.000 n=20)
LintRW/oracle        16.86k ± 0%     16.86k ± 0%       ~ (p=1.000 n=20) ¹
LintRW/sqlite        15.74k ± 0%     15.74k ± 0%       ~ (p=1.000 n=20)
LintRW/snowflake     15.20k ± 0%     15.20k ± 0%       ~ (p=1.000 n=20) ¹
LintRW/clickhouse    17.66k ± 0%     17.66k ± 0%       ~ (p=1.000 n=20)
LintRule/L007        1.194k ± 0%     1.194k ± 0%       ~ (p=1.000 n=20) ¹
LintRule/L010         0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=20) ¹
geomean                          ²                +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                  │     base     │                head                 │
                  │   sec/stmt   │   sec/stmt    vs base               │
LintRW/postgresql   16.44µ ±  1%   16.44µ ±  0%       ~ (p=0.499 n=20)
LintRW/mysql        13.02µ ±  1%   13.04µ ±  1%       ~ (p=0.654 n=20)
LintRW/mariadb      12.89µ ±  1%   12.86µ ±  2%       ~ (p=0.602 n=20)
LintRW/sqlserver    16.68µ ±  1%   16.67µ ±  1%       ~ (p=0.723 n=20)
LintRW/oracle       15.01µ ±  1%   15.07µ ±  1%       ~ (p=0.337 n=20)
LintRW/sqlite       12.73µ ±  2%   12.75µ ±  1%       ~ (p=0.764 n=20)
LintRW/snowflake    14.37µ ±  2%   14.47µ ±  3%       ~ (p=0.794 n=20)
LintRW/clickhouse   13.44µ ±  3%   13.44µ ±  4%       ~ (p=0.718 n=20)
LintRule/L007       981.7n ±  1%   982.7n ±  3%       ~ (p=0.607 n=20)
LintRule/L010       314.0n ± 21%   248.2n ± 27%       ~ (p=0.417 n=20)
geomean             7.446µ         7.281µ        -2.20%

pkg: github.com/ajitpratap0/GoSQLX/pkg/sql/keywords
                         │     base     │                head                 │
                         │    sec/op    │    sec/op     vs base               │
DetectDialect/empty        2.808n ± 22%   2.814n ± 22%       ~ (p=0.291 n=20)
DetectDialect/generic      303.9n ±  0%   304.6n ±  1%       ~ (p=0.213 n=20)
DetectDialect/postgresql   390.1n ±  0%   390.2n ±  0%       ~ (p=0.899 n=20)
DetectDialect/mysql        345.8n ±  2%   347.6n ±  2%       ~ (p=0.805 n=20)
DetectDialect/sqlserver    382.7n ±  1%   384.1n ±  2%       ~ (p=0.815 n=20)
DetectDialect/oracle       355.3n ±  0%   355.3n ±  0%       ~ (p=0.499 n=20)
DetectDialect/snowflake    520.3n ±  0%   519.6n ±  1%       ~ (p=0.878 n=20)
DetectDialect/mariadb      281.2n ±  1%   281.4n ±  1%       ~ (p=0.569 n=20)
DetectDialect/mixed        442.8n ±  0%   443.1n ±  0%       ~ (p=0.857 n=20)
DetectDialect/large_64KB   466.6µ ±  1%   469.2µ ±  1%       ~ (p=0.820 n=20)
geomean                    465.0n         465.9n        +0.19%

                         │     base     │                head                 │
                         │     B/s      │     B/s       vs base               │
DetectDialect/generic      166.3Mi ± 0%   165.9Mi ± 1%       ~ (p=0.204 n=20)
DetectDialect/postgresql   163.8Mi ± 0%   163.8Mi ± 0%       ~ (p=0.899 n=20)
DetectDialect/mysql        160.0Mi ± 2%   159.1Mi ± 2%       ~ (p=0.784 n=20)
DetectDialect/sqlserver    162.0Mi ± 1%   161.4Mi ± 2%       ~ (p=0.815 n=20)
DetectDialect/oracle       163.7Mi ± 0%   163.7Mi ± 0%       ~ (p=0.525 n=20)
DetectDialect/snowflake    163.1Mi ± 0%   163.4Mi ± 1%       ~ (p=0.867 n=20)
DetectDialect/mariadb      159.4Mi ± 1%   159.3Mi ± 1%       ~ (p=0.597 n=20)
DetectDialect/mixed        159.4Mi ± 0%   159.3Mi ± 0%       ~ (p=0.899 n=20)
DetectDialect/large_64KB   134.1Mi ± 1%   133.4Mi ± 1%       ~ (p=0.784 n=20)
geomean                    158.8Mi        158.5Mi       -0.19%

                         │     base     │                head                 │
                         │     B/op     │    B/op     vs base                 │
DetectDialect/empty        0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
DetectDialect/generic      0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
DetectDialect/postgresql   0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
DetectDialect/mysql        0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
DetectDialect/sqlserver    0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
DetectDialect/oracle       0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
DetectDialect/snowflake    0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
DetectDialect/mariadb      0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
DetectDialect/mixed        0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
DetectDialect/large_64KB   0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
geomean                               ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                         │     base     │                head                 │
                         │  allocs/op   │ allocs/op   vs base                 │
DetectDialect/empty        0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
DetectDialect/generic      0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
DetectDialect/postgresql   0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
DetectDialect/mysql        0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
DetectDialect/sqlserver    0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
DetectDialect/oracle       0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
DetectDialect/snowflake    0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
DetectDialect/mariadb      0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
DetectDialect/mixed        0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
DetectDialect/large_64KB   0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
geomean                               ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

pkg: github.com/ajitpratap0/GoSQLX/pkg/sql/tokenizer
                         │    base     │                head                │
                         │   sec/op    │   sec/op     vs base               │
Tokenize/scratch/join      1.489µ ± 1%   1.478µ ± 1%       ~ (p=0.172 n=20)
Tokenize/scratch/complex   3.956µ ± 1%   3.930µ ± 3%       ~ (p=0.136 n=20)
Tokenize/scratch/insert    744.8n ± 2%   741.2n ± 2%       ~ (p=0.478 n=20)
geomean                    1.637µ        1.627µ       -0.61%

                         │     base     │                head                 │
                         │     B/s      │     B/s       vs base               │
Tokenize/scratch/join      80.71Mi ± 1%   81.28Mi ± 1%       ~ (p=0.155 n=20)
Tokenize/scratch/complex   91.13Mi ± 1%   91.73Mi ± 3%       ~ (p=0.142 n=20)
Tokenize/scratch/insert    97.32Mi ± 2%   97.78Mi ± 2%       ~ (p=0.478 n=20)
geomean                    89.45Mi        90.00Mi       +0.62%

                         │     base     │                head                 │
                         │     B/op     │    B/op     vs base                 │
Tokenize/scratch/join      0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
Tokenize/scratch/complex   0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
Tokenize/scratch/insert    0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
geomean                               ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                         │     base     │                head                 │
                         │  allocs/op   │ allocs/op   vs base                 │
Tokenize/scratch/join      0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
Tokenize/scratch/complex   0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
Tokenize/scratch/insert    0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=20) ¹
geomean                               ²               +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

pkg: github.com/ajitpratap0/GoSQLX/pkg/transform
                             │    base     │                head                │
                             │   sec/op    │   sec/op     vs base               │
TransformParse/simple_select   1.558µ ± 2%   1.580µ ± 1%       ~ (p=0.087 n=20)
TransformParse/join            4.940µ ± 1%   4.953µ ± 3%       ~ (p=0.096 n=20)
TransformParse/complex         13.20µ ± 1%   13.29µ ± 2%       ~ (p=0.820 n=20)
geomean                        4.667µ        4.702µ       +0.76%

                             │     base     │                 head                  │
                             │     B/op     │     B/op      vs base                 │
TransformParse/simple_select     704.0 ± 0%     704.0 ± 0%       ~ (p=1.000 n=20) ¹
TransformParse/join            1.789Ki ± 0%   1.789Ki ± 0%       ~ (p=1.000 n=20) ¹
TransformParse/complex         4.883Ki ± 0%   4.883Ki ± 0%       ~ (p=1.000 n=20) ¹
geomean                        1.818Ki        1.818Ki       +0.00%
¹ all samples are equal

                             │    base    │                head                 │
                             │ allocs/op  │ allocs/op   vs base                 │
TransformParse/simple_select   3.000 ± 0%   3.000 ± 0%       ~ (p=1.000 n=20) ¹
TransformParse/join            15.00 ± 0%   15.00 ± 0%       ~ (p=1.000 n=20) ¹
TransformParse/complex         43.00 ± 0%   43.00 ± 0%       ~ (p=1.000 n=20) ¹
geomean                        12.46        12.46       +0.00%
¹ all samples are equal

pkg: github.com/ajitpratap0/GoSQLX/wasm/internal/bridge
                        │    base     │                head                │
                        │   sec/op    │   sec/op     vs base               │
WASMBridge/parse/small    4.746µ ± 1%   4.769µ ± 1%       ~ (p=0.449 n=20)
WASMBridge/parse/medium   21.13µ ± 1%   21.17µ ± 1%       ~ (p=0.841 n=20)
WASMBridge/parse/large    192.6µ ± 1%   193.1µ ± 1%       ~ (p=0.242 n=20)
WASMBridge/lint/small     6.671µ ± 1%   6.645µ ± 1%       ~ (p=0.331 n=20)
WASMBridge/lint/medium    30.88µ ± 1%   30.98µ ± 1%       ~ (p=0.327 n=20)
WASMBridge/lint/large     158.3µ ± 3%   159.8µ ± 1%       ~ (p=0.495 n=20)
geomean                   29.28µ        29.37µ       +0.30%

                        │     base     │                head                 │
                        │     B/s      │     B/s       vs base               │
WASMBridge/parse/small    7.834Mi ± 1%   7.801Mi ± 1%       ~ (p=0.456 n=20)
WASMBridge/parse/medium   11.78Mi ± 1%   11.76Mi ± 1%       ~ (p=0.825 n=20)
WASMBridge/parse/large    5.813Mi ± 1%   5.798Mi ± 1%       ~ (p=0.218 n=20)
WASMBridge/lint/small     5.579Mi ± 1%   5.598Mi ± 1%       ~ (p=0.330 n=20)
WASMBridge/lint/medium    8.059Mi ± 1%   8.035Mi ± 1%       ~ (p=0.317 n=20)
WASMBridge/lint/large     7.076Mi ± 3%   7.005Mi ± 1%       ~ (p=0.473 n=20)
geomean                   7.448Mi        7.425Mi       -0.30%

                        │     base     │                 head                  │
                        │     B/op     │     B/op      vs base                 │
WASMBridge/parse/small    1.531Ki ± 0%   1.531Ki ± 0%       ~ (p=1.000 n=20) ¹
WASMBridge/parse/medium   5.961Ki ± 0%   5.961Ki ± 0%       ~ (p=1.000 n=20) ¹
WASMBridge/parse/large    74.97Ki ± 0%   74.97Ki ± 0%       ~ (p=1.000 n=20) ¹
WASMBridge/lint/small     4.492Ki ± 0%   4.492Ki ± 0%       ~ (p=1.000 n=20) ¹
WASMBridge/lint/medium    16.02Ki ± 0%   16.02Ki ± 0%       ~ (p=1.000 n=20) ¹
WASMBridge/lint/large     123.5Ki ± 0%   123.5Ki ± 0%       ~ (p=0.751 n=20)
geomean                   13.51Ki        13.51Ki       -0.00%
¹ all samples are equal

                        │    base     │                 head                 │
                        │  allocs/op  │  allocs/op   vs base                 │
WASMBridge/parse/small     8.000 ± 0%    8.000 ± 0%       ~ (p=1.000 n=20) ¹
WASMBridge/parse/medium    38.00 ± 0%    38.00 ± 0%       ~ (p=1.000 n=20) ¹
WASMBridge/parse/large     522.0 ± 0%    522.0 ± 0%       ~ (p=1.000 n=20) ¹
WASMBridge/lint/small      33.00 ± 0%    33.00 ± 0%       ~ (p=1.000 n=20) ¹
WASMBridge/lint/medium     100.0 ± 0%    100.0 ± 0%       ~ (p=1.000 n=20) ¹
WASMBridge/lint/large     1.031k ± 0%   1.031k ± 0%       ~ (p=1.000 n=20) ¹
geomean                    90.24         90.24       +0.00%
¹ all samples are equal

                        │    base     │                 head                 │
                        │   outB/op   │   outB/op    vs base                 │
WASMBridge/parse/small     635.0 ± 0%    635.0 ± 0%       ~ (p=1.000 n=20) ¹
WASMBridge/parse/medium   2.661k ± 0%   2.661k ± 0%       ~ (p=1.000 n=20) ¹
WASMBridge/parse/large    33.16k ± 0%   33.16k ± 0%       ~ (p=1.000 n=20) ¹
WASMBridge/lint/small      22.00 ± 0%    22.00 ± 0%       ~ (p=1.000 n=20) ¹
WASMBridge/lint/medium     293.0 ± 0%    293.0 ± 0%       ~ (p=1.000 n=20) ¹
WASMBridge/lint/large      294.0 ± 0%    294.0 ± 0%       ~ (p=1.000 n=20) ¹
geomean                    688.2         688.2       +0.00%
¹ all samples are equal

Run, raw data and benchstat

#961 replaced direct CHANGELOG.md edits with per-PR changelog.d/
fragments, which removes this PR's CHANGELOG.md merge conflicts. The
entry text is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 26s —— View job


Review of PR #960: token-stream fingerprint (v2)

I read tokens.go and fuzz_test.go in full, and I used the earlier review and the perf-gate report for the rest. I did not run builds or tests, and I did not read the other test files or the remaining changed files, so this is a partial review.

Overall: The design is sound and the perf result is strong. Normalize/simple_select is -73%, allocs go from 7 to 1, and no gated benchmark regressed. Legacy behaviour stays reachable through ModeLegacy.

Findings

  1. Idempotence is fuzzed, which answers the earlier question.

    • FuzzNormalizeV2 (fuzz_test.go:40-61) checks Normalize(Normalize(x)) == Normalize(x).
    • It also checks that Fingerprint equals "v2:" + sha256(Normalize).
    • Please confirm CI runs it with -fuzz for a short time, not only the seed corpus. A plain go test replays seeds only.
    • The fuzz target only runs with the default Options. Dialect-specific quoting goes through namequote.MustStayQuoted(value, opts.Dialect), so consider a variant that fuzzes the dialect too.
  2. Frames deeper than 32 levels (tokens.go:188,249,439).

    • Past depth 32, f is nil and depth keeps growing. That output is deterministic, but a ) at depth > 32 then reads parent = &root (line 451-454 guards this, so it is safe).
    • IN-list dedup, cast and modifier handling are silently dropped at that depth. That can break idempotence: the first pass and the second pass might disagree about frames.
    • Please add a seed with more than 32 nested parentheses, for example IN (1,1) at depth 40, to the fuzz and unit tests.
  3. delimited is a span-width heuristic (tokens.go:75).

    • When Quote == 0, it infers a quoted identifier from End.Column-Start.Column > len(Value). That depends on tokenizer span semantics.
    • Escaped quotes inside an identifier can make the span wider than the value. A tokenizer change would break this silently.
    • The doc comment says it mirrors the parser. Moving the whole predicate into internal/namequote and having both callers use it would remove the duplication. A test that pins the span semantics would also help.
  4. Readability and maintainability of appendNormalV2 (about 300 lines).

    • It mixes many flags and edits already written bytes: redactModifierPrefix, finishElement, the row-dedup truncation and the unaryColon backspace.
    • Move the state into a struct with small methods, one per rule (N2, N10, N11, N12...). The test suite is strong enough to make that refactor safe, and it would make later rule changes much easier to review.
    • Several one-line helpers (delimited, nameToken, literalString) sit on single lines with no blank lines between them. gofmt accepts this, but it reads poorly.
  5. Hash stability policy.

    • The hash covers the output bytes, so any rule tweak changes every fingerprint.
    • State in doc.go that any change to testdata/goldens/fingerprint.sha256 requires a Version bump.
  6. Behaviour changes to call out in the migration notes.

    • Default mode no longer returns syntax errors. Callers who relied on Fingerprint to validate need gosqlx.Validate.
    • Empty or comment-only input now returns ErrEmpty. Please confirm the breaking-changes fragment lists this, since callers may match on error text.
    • The value grows from 64 to 67 bytes, which breaks CHAR(64) columns. This is already documented.
  7. Pooled tokenizer dialect (tokens.go:33-35).

    • SetDialect is only called when the dialect is non-empty. Please confirm PutTokenizer resets the dialect. Otherwise a prior call's dialect leaks into a later call that uses the default.
    • A test that alternates dialects on one goroutine would catch this.
  8. Minor points.

    • string(b.data) and string(encoded[:]) each allocate once. That matches the 1 alloc/op in the benchmark.
    • The legacy hash path uses fmt.Sprintf("%x"). hex.EncodeToString would be cheaper, but this is the legacy path.
    • putNormalBuffer drops buffers over 64 KiB. That is a sensible cap.

Security

No concerns found. Input is only tokenized, with no execution or I/O. The frame stack is fixed-size and every index into it is guarded by depth <= len(stack).

Test and perf summary

  • Test coverage looks thorough: corpus equivalence against legacy, mutation converses, allocation guards, goldens and fuzz. The grouping divergences are quantified and pinned.
  • The perf gate passes. There is no time regression, allocs/op and B/op are unchanged on the parse suites, and LintRule/L010 (+6.5%, p=0.028) is flagged as low power. It is unrelated to this change and is likely noise.
  • Open maintainer decisions M1-M3 are still listed in the PR body.

Before merge

  • Add the >32-depth nesting test (item 2).
  • Check the dialect leak in the pooled tokenizer (item 7).
  • Confirm the changelog covers the ErrEmpty change (item 6).

Test and others added 2 commits November 12, 2026 19:39
…5-08]

Drop the gosqlx options entry points (no public API growth for internal
hooks) and fill the changelog fragments with the amd64 gate numbers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 34s —— View job


Review of PR #960: token-stream fingerprint (v2)

I checked the open items from the earlier reviews against the code. I did not run builds or tests.

Resolved

  • Pooled tokenizer dialect leak (earlier item 7). PutTokenizer restores the default dialect and string-escape mode (pkg/sql/tokenizer/pool.go:129-139), so a dialect set by one call cannot reach the next. A test that alternates dialects on one goroutine would still be cheap insurance.
  • Idempotence. FuzzNormalizeV2 covers Normalize(Normalize(x)) == Normalize(x). Please confirm CI runs it with a short -fuzz budget, because a plain go test only replays the seeds.

Still open

  1. No test for nesting deeper than 32 levels.
    • The frame stack in tokens.go has a fixed size of 32. Beyond that, IN-list dedup, cast and modifier handling are silently skipped.
    • I found no unit test or fuzz seed for it. The deep_nesting entries I found are corpus allow-list names in equivalence_test.go. Please add a case such as WHERE a IN (1,1) wrapped in 40 parentheses. It should assert determinism, idempotence and no panic, and the doc should state that dedup stops at depth 32.
  2. The changelog does not list the ErrEmpty change.
    • changelog.d/w5-08-token-stream-fingerprint.added.md mentions ErrEmpty. A grep of the three w5-08 fragments for "empty" or ErrEmpty found only that file, so the breaking-changes fragment does not appear to list it.
    • Empty or comment-only input used to return a parse error and now returns ErrEmpty. Please add that to the breaking-changes fragment, next to the note that syntax errors are no longer reported.
  3. The delimited span-width heuristic still depends on tokenizer span semantics. The predicate has moved to internal/namequote, but the Quote == 0 fallback is still inferred from span width and line count. Please add a test that pins this, for example an identifier with escaped quotes and a multi-line quoted identifier. A tokenizer span change would then fail a test instead of silently changing fingerprints.
  4. Fuzzing only uses the default Options. Dialect-specific quoting goes through namequote.MustStayQuoted(value, dialect). Fuzzing a dialect chosen from the input would cover it.
  5. Hash stability policy. Any change to the rules changes every fingerprint. Please add a line to doc.go saying that any edit to testdata/goldens/fingerprint.sha256 requires a Version bump.
  6. Maintainability (non-blocking).
    • appendNormalV2 is about 300 lines, and several rules edit bytes already written (redactModifierPrefix, finishElement, row-dedup truncation, the unaryColon backspace).
    • Moving the state into a struct with one method per rule would make later changes easier to review. The test suite looks strong enough to support that as a follow-up.

Perf and security

  • The perf gate passes. Normalize/simple_select is -73.6%, allocs/op go from 7 to 1, and no gated benchmark regressed. LintRule/L010 is flagged low power and is unrelated to this change.
  • I found no security concerns. The input is only tokenized, and every access to the frame stack is guarded by a depth check.

Before merge: items 1 and 2 are the ones I would fix. The rest can follow. Maintainer decisions M1-M3 are still open in the PR body.

…fects, symmetric reparse oracle [W5-08]

Keep AS in XMLATTRIBUTES/XMLFOREST, stack-free overflow frames beyond depth 32,
fix INNER OUTER JOIN, colon-bind, bracket and multi-line quoted-name spacing
found by FuzzNormalizeV2 (now run in CI with a dialect argument), substitute
placeholders in both texts in the T7 reparse oracle, pooled dialect leak test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
@github-actions github-actions Bot added the ci-cd label Nov 12, 2026
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 20s —— View job


Review of PR #960: token-stream fingerprint (v2)

I re-checked the open items from the earlier reviews against the current branch. I did not run builds or tests.

Resolved since the last review

  • Nesting deeper than 32. The frame stack in tokens.go is no longer fixed at 32. It grows with append (lines 512-514), so nothing is silently skipped at depth. tokens_test.go:262 tests depths 33, 34 and 65 for the expected output and idempotence. My earlier concern about this no longer applies.
  • Pooled tokenizer dialect leak. PutTokenizer restores the default dialect, as noted in the previous review.

Still open

  1. ErrEmpty is not in the breaking-changes fragment.
    • Empty or comment-only input used to return a parse error and now returns ErrEmpty.
    • A grep of changelog.d/w5-08-* for "empty" matches only the .added.md fragment.
    • Please add a line to w5-08-token-stream-fingerprint.breaking-changes.md, next to the note that syntax errors are no longer reported. Callers that matched on the old error will break.
  2. The delimited fallback depends on tokenizer span semantics.
    • When Quote == 0, a quoted identifier is inferred from span width and line count.
    • I did not find a test that pins this with an identifier containing escaped quotes or a multi-line quoted identifier. A tokenizer span change would otherwise alter fingerprints silently.
  3. The fuzz target only uses the default Options. Dialect-specific quoting goes through namequote.MustStayQuoted(value, dialect). Choosing the dialect from the fuzz input would cover it. Please also confirm CI runs -fuzz for a short budget, because a plain go test only replays the seeds.
  4. Hash stability policy. Please state in pkg/fingerprint/doc.go that any edit to testdata/goldens/fingerprint.sha256 requires a Version bump.
  5. Maintainability (non-blocking). appendNormalV2 is about 300 lines, and several rules edit bytes already written (redactModifierPrefix, finishElement, row-dedup truncation, the unaryColon backspace). Splitting it into a state struct with one method per rule would be easier to review. The tests look strong enough to support that as a follow-up.

Perf and security

  • The CI perf gate passes. Normalize/simple_select is -74.3% and allocs/op go from 5 to 1. No gated parse or tokenize benchmark moved significantly, and the A/A' control is clean.
  • I found no security concerns. The input is only tokenized, with no execution or I/O, and stack accesses are guarded by depth checks.

Before merge: item 1 is the one I would fix. Items 2-4 are small. Maintainer decisions M1-M3 are still open in the PR body.

…isses [W5-08]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 18s —— View job


Review of PR #960: token-stream fingerprint (v2)

I checked the items left open by the earlier reviews against the current branch with grep. I did not run builds or tests.

Resolved

  • Fuzzing covers dialects. FuzzNormalizeV2 picks the dialect from the input (fuzz_test.go:18,49-50).
  • CI runs the fuzz target. .github/workflows/test.yml:224 runs -fuzz=FuzzNormalizeV2 -fuzztime=10s.
  • Nesting deeper than 32 levels. The frame stack now grows with append, and tokens_test.go tests depths 33, 34 and 65. Earlier reviews flagged this and it is no longer an issue.
  • Pooled tokenizer dialect leak. PutTokenizer resets the dialect, so one call cannot affect the next.

Still open

  1. ErrEmpty is missing from the breaking-changes fragment.
    • Of the changelog.d/w5-08-* fragments, only .added.md mentions "empty".
    • Empty or comment-only input used to return a parse error and now returns ErrEmpty. Callers who matched on the old error will break.
    • Please add a line to w5-08-token-stream-fingerprint.breaking-changes.md, next to the note that syntax errors are no longer reported.
  2. Hash stability policy is not in doc.go. It describes the format and VersionOf, but it does not say that any edit to testdata/goldens/fingerprint.sha256 requires a normal-form version bump. Please add one sentence.
  3. The delimited fallback depends on tokenizer span semantics. When Quote == 0, a quoted identifier is inferred from span width and line count. I did not look for a test that pins this. A small test for an identifier with escaped quotes and one for a multi-line quoted identifier would make a tokenizer span change fail loudly.
  4. Maintainability (non-blocking). appendNormalV2 is long, and several rules edit bytes already written (redactModifierPrefix, finishElement, row-dedup truncation, the unaryColon backspace). Splitting it into a state struct with one method per rule would be easier to review. The fuzz and equivalence tests make that a safe follow-up.

Performance: The CI perf gate passes.

  • Normalize/simple_select is -74%, join -61% and complex -62%. Allocs/op drop to 1.
  • No gated parse, tokenize or lint benchmark moved significantly, and the A/A' control is clean.

Security: I found no concerns. Input is only tokenized, with no execution or I/O.

Before merge: item 1 is the one to fix. Items 2 and 3 are small. Maintainer decisions M1-M3 are still open in the PR body.

…ery, fix @@ spacing, honest changelog [W5-08]

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
@github-actions

github-actions Bot commented Nov 12, 2026 •

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 30s —— View job


Review of PR #960: token-stream fingerprint (v2)

I checked the items left open by the earlier reviews with grep against the current branch. I did not run builds or tests, and I did not re-read the full diff.

Still open

  1. ErrEmpty is missing from the breaking-changes fragment.
    • In changelog.d/w5-08-*, only .added.md mentions empty input.
    • Empty or comment-only input used to return a parse error and now returns ErrEmpty. Callers that matched on the old error will break.
    • Please add a line to w5-08-token-stream-fingerprint.breaking-changes.md, next to the note that syntax errors are no longer reported.
  2. Hash stability policy is not in pkg/fingerprint/doc.go.
    • It describes the format and VersionOf, but it does not say that any edit to testdata/goldens/fingerprint.sha256 requires a normal-form version bump.
    • Please add one sentence.
  3. The delimited fallback depends on tokenizer span semantics (tokens.go:82).
    • I found a multi-line backtick case, N3_backtick_multiline, in tokens_test.go:137.
    • I did not find a test for the Quote == 0 span-width path with escaped quotes. A tokenizer span change could alter fingerprints without any test failing.
  4. Maintainability (non-blocking).
    • appendNormalV2 is long, and several rules edit bytes already written (redactModifierPrefix, finishElement, row-dedup truncation, the unaryColon backspace).
    • Splitting it into a state struct with one method per rule would be easier to review. The fuzz and equivalence tests make that a safe follow-up.
    • Long single-line predicates such as tokens.go:124 are hard to read. A package-level set of names would be simpler.

Resolved earlier

  • Fuzzing covers dialects, and CI runs FuzzNormalizeV2.
  • Nesting deeper than 32 levels is handled and tested.
  • The pooled tokenizer resets its dialect.

Performance: The CI perf gate passes.

  • Normalize/simple_select is -64.8%, join -53.6% and complex -56.2%, and allocs/op drop to 1.
  • No gated parse, tokenize or lint benchmark regressed, and the A/A' control is clean.
  • The PR body correctly says the PLAN ≤0.30x threshold is missed.

Security: I found no concerns. Input is only tokenized, with no execution or I/O.

Before merge: fix item 1. Items 2 and 3 are small. Maintainer decisions M1-M3 are still open in the PR body.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DnbRnTFErhoMJdAjVEeCg2
@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 15s —— View job


Review of PR #960: token-stream fingerprint (v2)

I checked the items left open by the earlier reviews with grep against the current branch. I did not run builds or tests, and I did not re-read the full diff.

Still open

  1. ErrEmpty is still missing from the breaking-changes fragment.
    • In changelog.d/w5-08-*, only .added.md mentions empty input (line 4-5).
    • Empty or comment-only input used to return a parse error and now returns ErrEmpty. Callers that matched on the old error will break.
    • Please add a line to w5-08-token-stream-fingerprint.breaking-changes.md, next to the note that syntax errors are no longer reported.
  2. Hash stability policy is still not in pkg/fingerprint/doc.go.
    • The only "empty" hit in doc.go is the legacy migration note on line 34. There is no statement that editing testdata/goldens/fingerprint.sha256 requires a normal-form version bump.
    • Please add one sentence.
  3. The delimited span-width fallback is only partly pinned.
    • tokens_test.go has N3_backtick_multiline (line 137), N3_backtick_escape (line 211) and N3_bracket_escape (line 212), so escaped quotes and multi-line names are covered for those dialects.
    • I did not check whether these cases reach the Quote == 0 path. If they don't, a tokenizer span change could still alter fingerprints. A test that asserts the tokenizer reports Quote == 0 for these inputs would pin the dependency.
  4. Maintainability (non-blocking). appendNormalV2 is long and several rules edit bytes already written (redactModifierPrefix, finishElement, row-dedup truncation, the unaryColon backspace). Splitting it into a state struct with one method per rule would be easier to review. The fuzz and equivalence tests make that a safe follow-up.

Resolved earlier

  • Fuzzing covers dialects, and CI runs FuzzNormalizeV2 with -fuzztime=10s.
  • Nesting deeper than 32 levels is handled and tested (depths 33, 34, 65).
  • The pooled tokenizer resets its dialect on PutTokenizer.

Performance: The latest perf-gate run passes.

  • Normalize/simple_select is -67.8%, join -53.9% and complex -53.9%, and allocs/op drop to 1.
  • No gated parse, tokenize or lint benchmark regressed, and the A/A' control is clean (0 of 52 moved).
  • The PR body states honestly that the PLAN ≤0.30x threshold is missed.

Security: I found no concerns. Input is only tokenized, with no execution or I/O.

Before merge: fix item 1. Items 2 and 3 are small. Maintainer decisions M1-M3 are still open in the PR body.

@github-actions

github-actions Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Claude finished @ajitpratap0's task in 15s —— View job


Review of PR #960: token-stream fingerprint (v2)

I re-checked the items left open by earlier reviews with grep against the current branch. I did not run builds or tests, and I did not re-read the full diff.

Still open

  1. ErrEmpty is not in the breaking-changes fragment.
    • In changelog.d/w5-08-*, only token-stream-fingerprint.added.md (lines 4-5) mentions empty input.
    • Empty or comment-only input used to return a parse error and now returns ErrEmpty. Callers that matched on the old error will break.
    • Please add a line to w5-08-token-stream-fingerprint.breaking-changes.md, next to the note that syntax errors are no longer reported.
  2. Hash stability policy is not in pkg/fingerprint/doc.go. A grep for "golden" or "bump" finds nothing. Please add one sentence saying that any edit to testdata/goldens/fingerprint.sha256 requires a normal-form version bump.
  3. The delimited span-width fallback is not directly pinned.
    • tokens.go:89 reads Token.Quote, and the fallback infers a quoted name from span width and line count.
    • I found no test in pkg/fingerprint that asserts which tokenizer cases report Quote == 0. The only copy of the span predicate is in equivalence_test.go:98.
    • Please add a small test that pins the tokenizer's Quote and span output for escaped-quote and multi-line quoted identifiers. A tokenizer change would then fail a test instead of silently changing fingerprints.
  4. Maintainability (non-blocking). appendNormalV2 is long, and several rules edit bytes already written (redactModifierPrefix, finishElement, row-dedup truncation, the unaryColon backspace). Splitting it into a state struct with one method per rule would be easier to review. The fuzz and equivalence tests make that a safe follow-up.

Resolved earlier

  • Fuzzing covers dialects, and CI runs FuzzNormalizeV2 with -fuzztime=10s.
  • Nesting deeper than 32 levels is handled and tested (depths 33, 34, 65).
  • The pooled tokenizer resets its dialect in PutTokenizer.

Performance: The latest perf-gate run passes.

  • Normalize/simple_select is -63%, join -49% and complex -52%. Allocs/op drop to 1.
  • No gated parse, tokenize or lint benchmark regressed, and the A/A' control is clean (0 of 52 moved).
  • This run is on a different CPU from earlier ones, so the numbers are not comparable across runs.
  • The PR body states that the PLAN ≤0.30x threshold is missed.

Security: I found no concerns. Input is only tokenized, with no execution or I/O.

Before merge: fix item 1. Items 2 and 3 are small. Maintainer decisions M1-M3 are still open in the PR body.

This branch was successfully deployed

1 active (outdated) deployment
Preview — b6cddc9f Deployed Nov 12, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci-cd documentation Improvements or additions to documentation parser performance

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant