Skip to content

Stop fetching patch views during hosted scans - #2021

Open
Mikola Lysenko (mikolalysenko) wants to merge 2 commits into
refactor/vendor-refusals-into-corefrom
refactor/hosted-scan-no-patch-view
Open

Mikola Lysenko (mikolalysenko) wants to merge 2 commits into
refactor/vendor-refusals-into-corefrom
refactor/hosted-scan-no-patch-view

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

LLM Description written by Claude Code:claude-opus-5-5

Stacked on #2015 (which is on #2013).

Summary

Hosted scans no longer call /patches/view. A view carries every patched file's blob contents, and v5 keeps no hosted ledger, so the scan was fetching full patch bodies only to feed checks that run within the same scan. Each of those checks now uses data the scan already has, or fetches its own:

Consumer Before Now
In-run --vex records from the scan's view fetches fetches the records it attests itself via vex_sources::fetch_records, the same path standalone vex uses; output unchanged
Gem stale-install probe patched file hashes from the view sha256 of the archive RubyGems installed from (<home>/cache/<leaf>.gem, and committed vendor/cache) vs. the grant's artifact sha256
Python stale-install probe patched file hashes from the view the patched wheel's own RECORD, downloaded through its grant URL (sha256-checked) only when an installed copy exists and its WHEEL tags match; never under --offline
vlt warm-tree heal record byte checks vlt's hidden lock alone: the pinned integrity means healthy, anything else means stale, and no readable hidden lock means undeterminable
In-memory engine / Node binding fetchPatch provider, presence-checked only not called; the binding still accepts a fetchPatch method and ignores it, so existing hosts (depscan's autopatch provider) keep working

When an agent-mode manifest holds a record for the same patch, the gem and Python probes use that local record instead, so an agent→hosted migration stays quiet.

Vendored mode, agent mode, get <uuid>, vendor eject and standalone vex are unchanged and still use the view.

Behavior changes

  • record_fetch_failed no longer exists. A view failure during in-run VEX is reported as vex_record_fetch_failed.
  • A hosted scan with --vex where every record fetch fails ends with a record_unavailable skip instead of manifest_not_found.
  • Review this one: the Python probe now downloads a patched wheel, which a hosted scan never did before, but only for a confirmed patch with an installed copy whose tags match, at most once per patch. An artifact-free alternative would be the installed dist's direct_url.json archive hash, but not every installer writes one, so I kept the wheel RECORD.
  • vlt: a FIFO or symlinked hidden lock used to get its store entry invalidated when a record proved the bytes stale. Now the entry is kept and the advisory asks for vlt ci, as it does when there is no hidden lock (new test). The advisory names the missing or unreadable hidden lock.

CLI_CONTRACT.md, index.d.ts and docs/testing/poetry-compatibility.md are updated.

Validation

Check Result
cargo fmt --check pass
cargo clippy --workspace --all-features -- -D warnings pass
core --lib (isolated HOME, see #2015 for the NuGet cache test) 7137 passed
CLI lib 860 passed
e2e_redirect_gem_stale_install 42 passed
e2e_vex_lockfile 354 passed
hosted_memory_engine 49 passed
in_process_redirect 134 passed
pdm / pipenv / poetry redirect suites 6 / 11 / 9 passed
scan 248 passed
Node smoke test 14/14 passed

An earlier run also passed cli, get, cli_parse_scan, cli_parse_get, hosted_memory_parity, hosted_memory_rollout, in_process_rollback_hosted, in_process_scan, in_process_vendor_bun_takeover, mode_migration_vlt and mode_migration_bun.

Tests that used a view request as proof that a redirect was confirmed now assert zero view calls, or confirm the redirect through a re-scan's envelope. Fixtures that lay down a gem install by hand now write the cache archive RubyGems would. The pdm, pipenv and poetry stale-install tests now serve a real patched wheel from the mock server.

Not run: the hatch and uv real-toolchain e2e suites, which already serve real wheels.

🤖 Generated with Claude Code


Note

Medium Risk
Changes hosted post-rewrite behavior (new wheel downloads, vlt heal rules, and in-run VEX fetch timing) without altering vendored/agent modes or standalone vex’s overall model.

Overview
Hosted scan / get --mode hosted no longer call /patches/view after a rewrite. v5 does not persist hosted records, and views were only feeding same-run checks that now use data the scan already has or fetch narrowly on their own.

In-run --vex takes confirmed pin UUIDs and loads records through vex_sources::fetch_records (like standalone vex), instead of a pre-built map from the scan. record_fetch_failed is gone; view failures show up as vex_record_fetch_failed / record_unavailable.

Stale-install probes no longer depend on fetched patch bodies: gems compare the installed cache/<leaf>.gem (and committed vendor cache) to grant artifact shas, with agent manifest.json afterHash still used when present; Python downloads the pinned wheel once ( RECORD + matching WHEEL tags) via the grant URL when a copy is installed—not under --offline; vlt heal/stale checks use node_modules/.vlt-lock.json integrity only, so missing/unreadable hidden lock keeps store entries and advises vlt ci instead of record-driven invalidation.

Contract docs and tests are updated (including zero-view assertions and gem cache fixtures).

Reviewed by Cursor Bugbot for commit fb3d418. Configure here.


Generated by Claude Code

A hosted scan fetched the full patch view (every patched file's blob
content) for each confirmed redirect, only to judge installed copies
and feed the same run's VEX. It now makes no view call at all:

- In-run `--vex` fetches the records it attests from itself, through
  the same record path the standalone `vex` uses, so the scan pipeline
  never asks for them.
- The gem stale-install probe judges an installed gem by the archive
  RubyGems installed it from (`<home>/cache/<leaf>.gem`) against the
  patched artifact's sha256.
- The Python stale-install probe downloads the patched wheel the lock
  now pins, through its grant URL and verified against the pinned
  sha256, and compares the installed copy with the wheel's own RECORD
  when the installed WHEEL tags match. It downloads only when a copy
  is installed.
- Both probes still treat an agent-mode manifest record of the same
  patch as proof of an in-place patch.
- The vlt warm-tree heal judges a store entry by vlt's hidden lock
  alone: the pinned integrity there is healthy, another is stale, and
  without a readable hidden lock the copy is kept and reported as
  unchecked.
- The in-memory hosted engine no longer calls the provider's
  `fetchPatch` or warns `record_fetch_failed`. The Node binding still
  accepts a `fetchPatch` provider method and ignores it, so existing
  providers keep working.

Assisted-by: Claude Code:claude-opus-5-5

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

…c) e2e

judge_installed_wheel read the installed WHEEL and every RECORD path with
bare std::fs reads. Site-packages is user-writable, so a FIFO there would
block the hosted scan's blocking pool. Read through
read_regular_*_sync so a FIFO or device counts as unreadable, and add a
regression test for both the WHEEL and a RECORD path.

The real-vlt e2e for heal rule (c) without a hidden lock still expected
the old record-based heal (copies invalidated). Hosted scans no longer
fetch patch records, so that case is now undeterminable: the copies are
kept and the advisory says to run `vlt ci`. Match the in-process test and
check that `vlt ci` then installs the patched package.

Co-Authored-By: Claude <noreply@anthropic.com>
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

bugbot run


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit fb3d418. Configure here.

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Labeled Ready for review by the burn-down agent.

  • Head: fb3d4183852150c00139b671240b93019ba88655
  • CI: 49/49 check runs green (30 success, 19 skipped by path filters)
  • Fixed in fb3d418:
    • e2e_redirect_vlt_build rule (c) without a hidden lock still expected the old record-based heal. This PR makes that case undeterminable (copies kept, advisory says run vlt ci), so the e2e now matches the in-process test and checks that vlt ci installs the patched package.
    • Bugbot (medium): the wheel probe read site-packages files with bare std::fs reads, so a FIFO could block the scan. It now uses the regular-file guard, with a FIFO regression test.
  • Bugbot: re-reviewed fb3d418, no new issues.
  • Reviewer note: stacked on Refuse vendored patches the same way everywhere #2015, which is stacked on Stop downloading vlt artifacts during hosted scans #2013. heal_rule_c_no_record now covers the same scenario as heal_rule_c_no_hidden_lock because patch records are never fetched; consider dropping one of them.
  • Slack announcement not sent this run (Slack send tool unavailable); next run retries.

Generated by Claude Code

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Nov 4, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Final review brief (for the human approver)

What it does: hosted scans stop calling /patches/view, which returned the full patched file contents just to feed checks inside the same scan. In-run --vex now fetches its own records, the same way standalone vex does. The gem stale-install probe compares the sha256 of the cached .gem with the grant's. The Python probe downloads the patched wheel, verifies its sha256 and reads its RECORD. The vlt heal trusts only vlt's hidden lock. Stacked on #2015, so it merges into that branch, not main.

Risk: medium. I found no security problem and nothing that makes the probes warn wrongly. The risk is in the Python probe: it now judges an installed copy only when its WHEEL tags and dist-info name exactly match the first patched wheel for that patch. Where they don't, a stale venv goes unwarned with no signal (see the open questions).

Look here

  • wheel_record.rs:40-50: checks the sha256 before it parses anything. Downloads are capped at 256MB and RECORD/WHEEL reads at 16MB. Nothing is extracted. RECORD paths go through is_safe_relative_subpath.
  • wheel_record.rs:134-171:`` the tag gate.
  • python.rs:91-110:``` how the override is chosen and the --offline skip.
  • hosted.rs:387-407:```` any cached .gem whose sha isn't a patched one now warns as stale.
  • vlt_heal.rs:258-286:````` with no readable hidden lock the result is now Undeterminable (it used to be `Healthy`), and the in-run VEX drops it. That errs on the safe side.

Verified: I read the PR's own diff (42843ef..fb3d4183852150c00139b671240b93019ba88655, 31 files). No view calls remain on the scan, memory-engine or Node paths. --offline and --dry-run are honoured. CRLF in RECORD is handled. CHANGELOG.md is untouched and there is no debug leftover. Ran locally: wheel_record unit tests (5), CLI stale (23), in_process_redirect_poetry and e2e_redirect_gem_stale_install, all passing. The poetry test depends on the served wheel, so it exercises the new probe. CI: 30 passed and 19 skipped by path filter (49 checks). Bugbot: passing, and its one thread is resolved.

Changes I made: none.

Open questions / nits (none block merging)

  1. When a patch ships several platform wheels, or the patched wheel changes its dist-info name or tags (PyYAML vs pyyaml, py2.py3 → py3), the probe never judges the installed copy. Try every matching .whl override? This could be a follow-up.
  2. Every re-scan downloads the whole patched wheel again for each pypi candidate that has an installed copy and no local record. That costs about what the old view fetch did.
  3. No test covers the --offline skip of the wheel probe, or a gem with no .gem under <home>/cache (git or path sources).
  4. Nit: hosted.rs:974-976 still says "patch records (fetched below)", but that fetch is gone.

Auto-merge is not armed. This PR targets #2015's branch, which has no review rule, so arming it would merge it immediately. Approve and merge it after #2015, or retarget it to main once #2013 and #2015 land.


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants