Skip to content

Old stored headers may not decode under the new strict header decoder (deploy blocker) #376

Open

Description

@SDL-A1

From the external review of #374. This one gates the deploy, not the merge.

Stored headers are now read back with a strict decoder that depends on activation
heights (HeaderForm::Stored). Nothing establishes that every mainnet header the
old code wrote still decodes under it and re-hashes to the key it is stored at.

If any stored header fails to decode or re-hashes differently, sync stalls on the
existing /var/lib/rustock immediately after deploy. The production node is at
#9,422,658, so this is a live store with the full history behind it, not a
fresh sync.

Why it is not caught by anything we have

mainnet_block_rules replays blocks from the store in canonical order; it does
not assert that every header already written round-trips through the new
decoder. CI builds a fresh database. So the failure mode is specific to an
upgrade-in-place, which is exactly how this will be deployed.

What would settle it

A read-only scan of CF_HEADERS and the freezer that, for every entry, decodes
the stored bytes with the new decoder and asserts hash(decoded) == key.

Read-only is enough, and paseo already has r-x only on /var/lib/rustock
(enforced by ACL), so the scan cannot damage the store.

This should run and pass before the new binary is installed. A rollback after
the fact may not be sufficient on its own: if the new binary has already written
state under the new encoding, swapping the binary back could require a resync.

Status

Reported by the external reviewer. Not verified here — verifying it is the
work this issue asks for.

Activity

  1. SDL-A1 commented on Nov 18, 2026

    @SDL-A1
    CollaboratorAuthor

    Update: the PR already ships this scan. crates/cli/examples/verify_stored_headers.rs (357 lines, new in #374) does what this issue asks for, and more.

    Its own header states the relevant part:

    it hashes each header as rustock writes it back, which is how rskj hashes one, against the hash it is stored under, which the header above it names: a header read in a form rustock can't write back the same would hash as another block.

    Concretely:

    • BlockStore::open_read_only and Freezer::open_read_only — both opened read-only, so it cannot touch the store
    • config.activation_heights.clone().install() runs before any header is read, which is the thing this issue was worried about: the strict decoder depends on activation heights
    • a rehash closure computes header.hash() and compares it against the key the header is stored under, recording a Rehashed { stored, computed } failure on mismatch
    • genesis is deliberately excluded, with a correct reason: rskj's GenesisHeader writes its difficulty with a leading zero no decoder keeps, so its hash is known separately
    • it also runs the full HeaderVerifier, fork detection (the 449-block window) and merged-mining checks, multi-threaded, over a selectable range

    So this issue is not "write a scan" — it is "run the scan that exists, against this node's store, and record the result."

    One real gap in coverage

    The tool walks the canonical chain: canonical_hash(number) downward, following parent_hash, counting missing for canonical numbers with no header. It does not iterate every entry in CF_HEADERS, so stored non-canonical and uncle headers are not rehash-checked.

    For the deploy risk this issue is about — sync stalling on the existing store — the canonical chain plus the freezer is the part that matters, so this is sufficient to clear the blocker. Full CF_HEADERS coverage would be a strictly stronger check and is worth a follow-up if we want it.

    Status

    Building verify_stored_headers --release in a bwrap sandbox now (network off, no credentials, no /srv/rustock). Will run it against /var/lib/rustock bound read-only, with the node stopped, and post the output here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions