Skip to content

Phase 32 slice A: tests in folders that set their tier, type-checked - #255

Merged
vicmaster merged 3 commits into
masterfrom
phase-32-slice-a
Nov 20, 2026
Merged

vicmaster merged 3 commits into
masterfrom
phase-32-slice-a

Conversation

@vicmaster

@vicmaster vicmaster commented Nov 20, 2026 •

Copy link
Copy Markdown
Owner

Phase 32 slice A (docs/specs/PHASE-32-SPEC.md, FR-A1 to FR-A7). The tests move out of the repo root into folders that set their tier, the runner proves the unit tier never launches Chrome, and the tests are type-checked. No test changes what it asserts, except where noted below.

What changed

  • Folders set the tier. test/unit (Node only), test/browser (launches Chrome), test/e2e (spawns the MCP server or the CLI) and test/manual (needs a human or a running viewer). test-env.ts is now test/env.ts, and test-fixtures/ is now test/fixtures/. Files keep their test- names.
  • The runner reads folders. test:fast runs test/unit, and test:full adds test/browser and test/e2e. test/manual never runs. The text-match tiering (CHROME_MARKERS, CHROME_ANYWAY, INTERACTIVE) is deleted. The runner switches to the repo root itself, so it works from any directory. Filters match a test's path, so unit/ selects a folder. --root <dir> points the runner at another folder, which its own tests use.
  • The tier is proved at run time. Every test/unit file runs with FRAMESMITH_CHROME_PATH and PUPPETEER_EXECUTABLE_PATH set to a path that does not exist. A unit test that launches Chrome therefore fails, and the runner adds a line telling you to move it to test/browser. The new test/unit/test-runner-guard.ts proves this for a launch through src/screenshot.ts and for a direct puppeteer.launch. Each file was classified by running it under that guard: 88 passed and 40 failed with a launch error.
  • No test skips on a missing Chrome. test-evaluate.ts used to catch a failed launch in its detailed-mode block and count it as a pass. That block now lives in test/browser/test-evaluate-detailed.ts, without the catch.
  • Tests are type-checked. tsconfig.test.json extends the main config with strict and noEmit over test/**. npm run typecheck:tests runs it, and so does CI's fast job, after the build. The 54 existing errors are fixed by typing the tests, not with any or @ts-ignore. In practice that meant using the exported EvaluationIssue type, adding the projectId a Canvas requires, and writing 'end' and 'start' where tests had 'flex-end' and 'flex-start' (both render the same CSS). Feedback and critique fixtures now match their real types, and StressRecord fixtures gained the required type.
  • Files are named by subject. test-phase2.ts and test-phase3.ts are split by subject; the table below maps every assertion. test-correctness-fixes.ts is now test-contrast-rounding-preset-inherit.ts, and test-gradient-shadow-move-fixes.ts is now test-raw-css-styles-repo-move.ts. Hardcoded /tmp outputs now use os.tmpdir().
  • Paths that break on the move are fixed. These are the imports, the fixture reads, the benchmark import, and test-guidelines.ts, which now derives the repo root from its new depth. Every "Usage" header, every test-to-test comment, the README scripts table, three VISION lines and four src/ comments name the new paths. Closed specs keep their old paths. The spec's stale-path check now returns nothing.

Counts

There were 138 active tests before. There are 143 now: the 136 that moved, plus 5 from splitting the phase files, the detailed-evaluate file, and the new runner-guard test, minus the 2 phase files the splits replace. By folder, that is 90 unit, 43 browser and 10 e2e, with 2 more in manual.

Verification

  • npx tsc -p tsconfig.test.json exits 0.
  • npm run build is clean.
  • The fast tier, run from test/ (npx tsx ../scripts/run-tests.ts), passes 90 of 90.
  • The full tier, run from test/ against the installed Chrome, passes 142 of 143. The one failure, unit/test-viewer-watch.ts ("global canvas write triggers a reload"), is a timing flake: its file watcher waits a fixed time and missed the event while the full run was under load. It passed in the fast tier and in 5 of 5 runs on its own, and this change only touched its import lines.
  • Every test/browser file, the five new ones included, fails under the guard. Every test/unit file passes under it.
  • test/unit/test-discoverability.ts passes 160 of 160.
  • CI catches a type error in a test. Commit 5e9b4cf added one on purpose, and CI's fast job failed at npm run typecheck:tests on Node 22 and Node 24 (run 44760993199). Commit 0cbc9dd reverts it. That probe commit also picked up an unrelated local .framesmith/workspace.json edit, which the revert removes, so the branch's net diff touches no .framesmith/ file.
  • Of the unit files with a Chrome marker, test-cliche.ts, test-stress.ts and test-evaluate.ts were read to confirm they never launch. The first two only import shutdown or a type from src/screenshot.ts. The third launched only in the block that moved out.

Assertions from the split files

Old file Assertions (function) Now in
test-phase2.ts presets: 4 presets, dark/light/material/minimal present, dark has colors and typography, preset applied to canvas (testPresets, 8) test/unit/test-presets-icons.ts
test-phase2.ts Lucide: icon count, search/heart/home, SVG tag/size/color, unknown → null, icon nodes insert and render (testIcons, 12) test/unit/test-presets-icons.ts
test-phase2.ts PNG, PDF and JPEG export (testExport, 3) test/browser/test-export.ts
test-phase2.ts component created and registered, instances with overrides, nested override, default kept, screenshot captured (testComponents, 9) test/browser/test-component-instances.ts
test-phase2.ts componentId by binding resolves (testComponentBinding, 2) test/browser/test-component-instances.ts
test-phase3.ts linear and radial gradients, stops, fill fallback, background: property, screenshot captured (testGradients, 8) test/browser/test-gradients-shadows.ts
test-phase3.ts structured, multi and legacy shadows, blur, backdrop blur (testShadowsAndBlur, 7) test/browser/test-gradients-shadows.ts
test-phase3.ts 3 responsive shots, labels and order, mobile width 390, all have data, mobile smaller than desktop (testResponsiveBreakpoints, 7) test/browser/test-responsive.ts (breakpointShots)
test-phase3.ts identical canvases diff under 1%, diff image, different canvases diff above 0%, changed pixels (testDiffMode, 4) test/browser/test-diff.ts
test-evaluate.ts detailed mode runs and scores (testDetailedMode, 2; the catch that counted a pass without Chrome is removed) test/browser/test-evaluate-detailed.ts

Where every file went

Old path New path Why
test-animations.ts test/browser/test-animations.ts fails with Chrome unlaunchable
test-auto-relative.ts test/browser/test-auto-relative.ts fails with Chrome unlaunchable
test-backdrop-filter.ts test/browser/test-backdrop-filter.ts fails with Chrome unlaunchable
test-check-drift-url.ts test/browser/test-check-drift-url.ts fails with Chrome unlaunchable
test-color-system-readback.ts test/browser/test-color-system-readback.ts fails with Chrome unlaunchable
test-custom-fonts.ts test/browser/test-custom-fonts.ts fails with Chrome unlaunchable
test-default-font.ts test/browser/test-default-font.ts fails with Chrome unlaunchable
test-desktop-app-overlay-detailed.ts test/browser/test-desktop-app-overlay-detailed.ts fails with Chrome unlaunchable
test-desktop-app-rules-detailed.ts test/browser/test-desktop-app-rules-detailed.ts fails with Chrome unlaunchable
test-desktop-primitives-render.ts test/browser/test-desktop-primitives-render.ts fails with Chrome unlaunchable
test-element-states-render.ts test/browser/test-element-states-render.ts fails with Chrome unlaunchable
test-fluid-widths.ts test/browser/test-fluid-widths.ts fails with Chrome unlaunchable
test-grid.ts test/browser/test-grid.ts fails with Chrome unlaunchable
test-import-html.ts test/browser/test-import-html.ts fails with Chrome unlaunchable
test-import-url.ts test/browser/test-import-url.ts fails with Chrome unlaunchable
test-material-icons.ts test/browser/test-material-icons.ts fails with Chrome unlaunchable
test-overlays-render.ts test/browser/test-overlays-render.ts fails with Chrome unlaunchable
test-patterns-desktop-window.ts test/browser/test-patterns-desktop-window.ts fails with Chrome unlaunchable
test-patterns-desktop.ts test/browser/test-patterns-desktop.ts fails with Chrome unlaunchable
test-patterns-mobile.ts test/browser/test-patterns-mobile.ts fails with Chrome unlaunchable
test-pointer-targets.ts test/browser/test-pointer-targets.ts fails with Chrome unlaunchable
test-positioning.ts test/browser/test-positioning.ts fails with Chrome unlaunchable
test-primitives.ts test/browser/test-primitives.ts fails with Chrome unlaunchable
test-render-capture.ts test/browser/test-render-capture.ts fails with Chrome unlaunchable
test-responsive-reflow.ts test/browser/test-responsive-reflow.ts fails with Chrome unlaunchable
test-responsive.ts test/browser/test-responsive.ts fails with Chrome unlaunchable
test-root-centering.ts test/browser/test-root-centering.ts fails with Chrome unlaunchable
test-scaffold-system.ts test/browser/test-scaffold-system.ts fails with Chrome unlaunchable
test-state-model-render.ts test/browser/test-state-model-render.ts fails with Chrome unlaunchable
test-stress-render.ts test/browser/test-stress-render.ts fails with Chrome unlaunchable
test-style-attr-escape.ts test/browser/test-style-attr-escape.ts fails with Chrome unlaunchable
test-svg-paths.ts test/browser/test-svg-paths.ts fails with Chrome unlaunchable
test-sync-drift.ts test/browser/test-sync-drift.ts fails with Chrome unlaunchable
test-text-props.ts test/browser/test-text-props.ts fails with Chrome unlaunchable
test-usability.ts test/browser/test-usability.ts fails with Chrome unlaunchable
test-viewer-compare.ts test/browser/test-viewer-compare.ts fails with Chrome unlaunchable
test-viewer-surface-browser.ts test/browser/test-viewer-surface-browser.ts fails with Chrome unlaunchable
test-visual.ts test/browser/test-visual.ts fails with Chrome unlaunchable
test-cli.ts test/e2e/test-cli.ts spawns the MCP server or the CLI
test-evaluate-fonts.ts test/e2e/test-evaluate-fonts.ts spawns the MCP server or the CLI
test-genre-mcp.ts test/e2e/test-genre-mcp.ts spawns the MCP server or the CLI
test-mobile-rules-detailed.ts test/e2e/test-mobile-rules-detailed.ts spawns the MCP server or the CLI
test-mobile-rules.ts test/e2e/test-mobile-rules.ts spawns the MCP server or the CLI
test-ridgeline.ts test/e2e/test-ridgeline.ts spawns the MCP server or the CLI
test-structures-mcp.ts test/e2e/test-structures-mcp.ts spawns the MCP server or the CLI
test-surface.ts test/e2e/test-surface.ts spawns the MCP server or the CLI
test-verify-inherited-tokens.ts test/e2e/test-verify-inherited-tokens.ts spawns the MCP server or the CLI
test-viewer-rebuild.ts test/e2e/test-viewer-rebuild.ts spawns the MCP server or the CLI
test-env.ts test/env.ts shared test setup
test-viewer-navbar.ts test/manual/test-viewer-navbar.ts needs a human or a running viewer
test-viewer.ts test/manual/test-viewer.ts needs a human or a running viewer
test-advisory-honesty.ts test/unit/test-advisory-honesty.ts passes with Chrome unlaunchable; Chrome-capable import
test-apply-structure.ts test/unit/test-apply-structure.ts passes with Chrome unlaunchable
test-archive-view.ts test/unit/test-archive-view.ts passes with Chrome unlaunchable; Chrome-capable import
test-autofix.ts test/unit/test-autofix.ts passes with Chrome unlaunchable; Chrome-capable import
test-batch-design-idmap.ts test/unit/test-batch-design-idmap.ts passes with Chrome unlaunchable
test-borders-dasharray.ts test/unit/test-borders-dasharray.ts passes with Chrome unlaunchable
test-build-log.ts test/unit/test-build-log.ts passes with Chrome unlaunchable
test-calibration.ts test/unit/test-calibration.ts passes with Chrome unlaunchable; Chrome-capable import
test-chart.ts test/unit/test-chart.ts passes with Chrome unlaunchable
test-charts.ts test/unit/test-charts.ts passes with Chrome unlaunchable
test-cliche.ts test/unit/test-cliche.ts passes with Chrome unlaunchable; has a Chrome marker, read: never launches
test-color-system.ts test/unit/test-color-system.ts passes with Chrome unlaunchable; Chrome-capable import
test-component-structures.ts test/unit/test-component-structures.ts passes with Chrome unlaunchable
test-components.ts test/unit/test-components.ts passes with Chrome unlaunchable
test-contrast-alpha.ts test/unit/test-contrast-alpha.ts passes with Chrome unlaunchable; Chrome-capable import
test-correctness-fixes.ts test/unit/test-contrast-rounding-preset-inherit.ts passes with Chrome unlaunchable; Chrome-capable import
test-coverage.ts test/unit/test-coverage.ts passes with Chrome unlaunchable; Chrome-capable import
test-critique.ts test/unit/test-critique.ts passes with Chrome unlaunchable; Chrome-capable import
test-design-language.ts test/unit/test-design-language.ts passes with Chrome unlaunchable; Chrome-capable import
test-design-md-tolerant.ts test/unit/test-design-md-tolerant.ts passes with Chrome unlaunchable
test-design-md.ts test/unit/test-design-md.ts passes with Chrome unlaunchable
test-design-system-elevation.ts test/unit/test-design-system-elevation.ts passes with Chrome unlaunchable
test-design-system-inheritance.ts test/unit/test-design-system-inheritance.ts passes with Chrome unlaunchable
test-desktop-app-overlay.ts test/unit/test-desktop-app-overlay.ts passes with Chrome unlaunchable; Chrome-capable import
test-desktop-app-rules.ts test/unit/test-desktop-app-rules.ts passes with Chrome unlaunchable; Chrome-capable import
test-desktop-app.ts test/unit/test-desktop-app.ts passes with Chrome unlaunchable; Chrome-capable import
test-desktop-primitives.ts test/unit/test-desktop-primitives.ts passes with Chrome unlaunchable; Chrome-capable import
test-desktop-structures.ts test/unit/test-desktop-structures.ts passes with Chrome unlaunchable; Chrome-capable import
test-discoverability.ts test/unit/test-discoverability.ts passes with Chrome unlaunchable; Chrome-capable import
test-diversification.ts test/unit/test-diversification.ts passes with Chrome unlaunchable
test-drift.ts test/unit/test-drift.ts passes with Chrome unlaunchable; Chrome-capable import
test-dual-theme.ts test/unit/test-dual-theme.ts passes with Chrome unlaunchable; Chrome-capable import
test-element-states.ts test/unit/test-element-states.ts passes with Chrome unlaunchable; Chrome-capable import
test-empty-thumbnail.ts test/unit/test-empty-thumbnail.ts passes with Chrome unlaunchable; Chrome-capable import
test-eval-cache-key.ts test/unit/test-eval-cache-key.ts passes with Chrome unlaunchable; Chrome-capable import
test-evaluate.ts test/unit/test-evaluate.ts passes with Chrome unlaunchable; has a Chrome marker, read: never launches
test-feedback.ts test/unit/test-feedback.ts passes with Chrome unlaunchable
test-find-nodes.ts test/unit/test-find-nodes.ts passes with Chrome unlaunchable
test-flow-critique.ts test/unit/test-flow-critique.ts passes with Chrome unlaunchable
test-fluid-type.ts test/unit/test-fluid-type.ts passes with Chrome unlaunchable
test-font-aliases.ts test/unit/test-font-aliases.ts passes with Chrome unlaunchable; Chrome-capable import
test-fonts-by-name.ts test/unit/test-fonts-by-name.ts passes with Chrome unlaunchable; Chrome-capable import
test-genre-report.ts test/unit/test-genre-report.ts passes with Chrome unlaunchable; Chrome-capable import
test-guidelines.ts test/unit/test-guidelines.ts passes with Chrome unlaunchable
test-import-core.ts test/unit/test-import-core.ts passes with Chrome unlaunchable; Chrome-capable import
test-import-structure.ts test/unit/test-import-structure.ts passes with Chrome unlaunchable; Chrome-capable import
test-init.ts test/unit/test-init.ts passes with Chrome unlaunchable
test-llm-judge.ts test/unit/test-llm-judge.ts passes with Chrome unlaunchable
test-mobile-primitives.ts test/unit/test-mobile-primitives.ts passes with Chrome unlaunchable
test-motion-tokens.ts test/unit/test-motion-tokens.ts passes with Chrome unlaunchable; Chrome-capable import
test-ops-parser.ts test/unit/test-ops-parser.ts passes with Chrome unlaunchable
test-overlays.ts test/unit/test-overlays.ts passes with Chrome unlaunchable; Chrome-capable import
test-patterns.ts test/unit/test-patterns.ts passes with Chrome unlaunchable; Chrome-capable import
test-preservation.ts test/unit/test-preservation.ts passes with Chrome unlaunchable
test-project-evaluate.ts test/unit/test-project-evaluate.ts passes with Chrome unlaunchable
test-gradient-shadow-move-fixes.ts test/unit/test-raw-css-styles-repo-move.ts passes with Chrome unlaunchable
test-replace-matching.ts test/unit/test-replace-matching.ts passes with Chrome unlaunchable
test-repo-assets.ts test/unit/test-repo-assets.ts passes with Chrome unlaunchable
test-repo-canvases.ts test/unit/test-repo-canvases.ts passes with Chrome unlaunchable
test-repo-external-change.ts test/unit/test-repo-external-change.ts passes with Chrome unlaunchable
test-repo-workspace-external-change.ts test/unit/test-repo-workspace-external-change.ts passes with Chrome unlaunchable
test-runner-timeout.ts test/unit/test-runner-timeout.ts passes with Chrome unlaunchable
test-scales.ts test/unit/test-scales.ts passes with Chrome unlaunchable; Chrome-capable import
test-set-genre.ts test/unit/test-set-genre.ts passes with Chrome unlaunchable; Chrome-capable import
test-shadow-warning.ts test/unit/test-shadow-warning.ts passes with Chrome unlaunchable
test-silent-drops.ts test/unit/test-silent-drops.ts passes with Chrome unlaunchable; Chrome-capable import
test-skeleton.ts test/unit/test-skeleton.ts passes with Chrome unlaunchable; Chrome-capable import
test-state-checks.ts test/unit/test-state-checks.ts passes with Chrome unlaunchable; Chrome-capable import
test-stress.ts test/unit/test-stress.ts passes with Chrome unlaunchable; has a Chrome marker, read: never launches
test-structures.ts test/unit/test-structures.ts passes with Chrome unlaunchable
test-tailwind-map.ts test/unit/test-tailwind-map.ts passes with Chrome unlaunchable; Chrome-capable import
test-token-lint.ts test/unit/test-token-lint.ts passes with Chrome unlaunchable; Chrome-capable import
test-typography-depth.ts test/unit/test-typography-depth.ts passes with Chrome unlaunchable; Chrome-capable import
test-unknown-icons.ts test/unit/test-unknown-icons.ts passes with Chrome unlaunchable; Chrome-capable import
test-variants.ts test/unit/test-variants.ts passes with Chrome unlaunchable; Chrome-capable import
test-version-hash.ts test/unit/test-version-hash.ts passes with Chrome unlaunchable
test-viewer-feedback.ts test/unit/test-viewer-feedback.ts passes with Chrome unlaunchable; Chrome-capable import
test-viewer-logo-favicon.ts test/unit/test-viewer-logo-favicon.ts passes with Chrome unlaunchable; Chrome-capable import
test-viewer-panels.ts test/unit/test-viewer-panels.ts passes with Chrome unlaunchable; Chrome-capable import
test-viewer-perf.ts test/unit/test-viewer-perf.ts passes with Chrome unlaunchable; Chrome-capable import
test-viewer-repo-aggregation.ts test/unit/test-viewer-repo-aggregation.ts passes with Chrome unlaunchable
test-viewer-repo-lifecycle.ts test/unit/test-viewer-repo-lifecycle.ts passes with Chrome unlaunchable
test-viewer-resolves-workspace-tokens.ts test/unit/test-viewer-resolves-workspace-tokens.ts passes with Chrome unlaunchable
test-viewer-sidebar.ts test/unit/test-viewer-sidebar.ts passes with Chrome unlaunchable; Chrome-capable import
test-viewer-surface.ts test/unit/test-viewer-surface.ts passes with Chrome unlaunchable; Chrome-capable import
test-viewer-watch.ts test/unit/test-viewer-watch.ts passes with Chrome unlaunchable
test-workspace-mcp-tools.ts test/unit/test-workspace-mcp-tools.ts passes with Chrome unlaunchable
test-workspace-migration.ts test/unit/test-workspace-migration.ts passes with Chrome unlaunchable

Tests sat loose in the repo root, tiered by a text match that guessed which ones launch Chrome,
and were never type-checked.

- test/unit, test/browser, test/e2e and test/manual; the runner reads the folders, and runs
  test/unit with Chrome unlaunchable, naming any unit test that launches it
- tsconfig.test.json (strict, noEmit) and `npm run typecheck:tests`, run in CI's fast job;
  the 54 existing errors fixed by typing the tests
- test-phase2/3 split by subject; the two issue-named tests renamed by what they check;
  evaluate's detailed block moved to test/browser without its skip-on-no-Chrome catch
@vicmaster
vicmaster merged commit dafa0e1 into master Nov 20, 2026
3 checks passed
@vicmaster
vicmaster deleted the phase-32-slice-a branch November 20, 2026 16:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant