You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Old stored headers may not decode under the new strict header decoder (deploy blocker) #376
From the external review of #374. This one gates the deploy, not the merge.
Stored headers are now read back with a strict decoder that depends on activation
heights (HeaderForm::Stored). Nothing establishes that every mainnet header the old code wrote still decodes under it and re-hashes to the key it is stored at.
If any stored header fails to decode or re-hashes differently, sync stalls on the
existing /var/lib/rustock immediately after deploy. The production node is at #9,422,658, so this is a live store with the full history behind it, not a
fresh sync.
Why it is not caught by anything we have
mainnet_block_rules replays blocks from the store in canonical order; it does
not assert that every header already written round-trips through the new
decoder. CI builds a fresh database. So the failure mode is specific to an
upgrade-in-place, which is exactly how this will be deployed.
What would settle it
A read-only scan of CF_HEADERS and the freezer that, for every entry, decodes
the stored bytes with the new decoder and asserts hash(decoded) == key.
Read-only is enough, and paseo already has r-x only on /var/lib/rustock
(enforced by ACL), so the scan cannot damage the store.
This should run and pass before the new binary is installed. A rollback after
the fact may not be sufficient on its own: if the new binary has already written
state under the new encoding, swapping the binary back could require a resync.
Status
Reported by the external reviewer. Not verified here — verifying it is the
work this issue asks for.
Update: the PR already ships this scan.crates/cli/examples/verify_stored_headers.rs (357 lines, new in #374) does what this issue asks for, and more.
Its own header states the relevant part:
it hashes each header as rustock writes it back, which is how rskj hashes one, against the hash it is stored under, which the header above it names: a header read in a form rustock can't write back the same would hash as another block.
Concretely:
BlockStore::open_read_only and Freezer::open_read_only — both opened read-only, so it cannot touch the store
config.activation_heights.clone().install() runs before any header is read, which is the thing this issue was worried about: the strict decoder depends on activation heights
a rehash closure computes header.hash() and compares it against the key the header is stored under, recording a Rehashed { stored, computed } failure on mismatch
genesis is deliberately excluded, with a correct reason: rskj's GenesisHeader writes its difficulty with a leading zero no decoder keeps, so its hash is known separately
it also runs the full HeaderVerifier, fork detection (the 449-block window) and merged-mining checks, multi-threaded, over a selectable range
So this issue is not "write a scan" — it is "run the scan that exists, against this node's store, and record the result."
One real gap in coverage
The tool walks the canonical chain: canonical_hash(number) downward, following parent_hash, counting missing for canonical numbers with no header. It does not iterate every entry in CF_HEADERS, so stored non-canonical and uncle headers are not rehash-checked.
For the deploy risk this issue is about — sync stalling on the existing store — the canonical chain plus the freezer is the part that matters, so this is sufficient to clear the blocker. Full CF_HEADERS coverage would be a strictly stronger check and is worth a follow-up if we want it.
Status
Building verify_stored_headers --release in a bwrap sandbox now (network off, no credentials, no /srv/rustock). Will run it against /var/lib/rustock bound read-only, with the node stopped, and post the output here.
Activity
SDL-A1 commentedon Nov 18, 2026
Update: the PR already ships this scan.
crates/cli/examples/verify_stored_headers.rs(357 lines, new in #374) does what this issue asks for, and more.Its own header states the relevant part:
Concretely:
BlockStore::open_read_onlyandFreezer::open_read_only— both opened read-only, so it cannot touch the storeconfig.activation_heights.clone().install()runs before any header is read, which is the thing this issue was worried about: the strict decoder depends on activation heightsrehashclosure computesheader.hash()and compares it against the key the header is stored under, recording aRehashed { stored, computed }failure on mismatchGenesisHeaderwrites its difficulty with a leading zero no decoder keeps, so its hash is known separatelyHeaderVerifier, fork detection (the 449-block window) and merged-mining checks, multi-threaded, over a selectable rangeSo this issue is not "write a scan" — it is "run the scan that exists, against this node's store, and record the result."
One real gap in coverage
The tool walks the canonical chain:
canonical_hash(number)downward, followingparent_hash, countingmissingfor canonical numbers with no header. It does not iterate every entry inCF_HEADERS, so stored non-canonical and uncle headers are not rehash-checked.For the deploy risk this issue is about — sync stalling on the existing store — the canonical chain plus the freezer is the part that matters, so this is sufficient to clear the blocker. Full
CF_HEADERScoverage would be a strictly stronger check and is worth a follow-up if we want it.Status
Building
verify_stored_headers --releasein a bwrap sandbox now (network off, no credentials, no/srv/rustock). Will run it against/var/lib/rustockbound read-only, with the node stopped, and post the output here.