Skip to content

Let the chunk and metadata servers lay their WAL out zero-filled, opt-in - #641

Merged
matrixarkai merged 7 commits into
mainfrom
perf/a-chunk-or-metadata-server-can-lay-its-wal-out-zero-filled
Nov 16, 2026
Merged

matrixarkai merged 7 commits into
mainfrom
perf/a-chunk-or-metadata-server-can-lay-its-wal-out-zero-filled

Conversation

@matrixarkai

Copy link
Copy Markdown
Owner

What

An opt-in zero-filled layout for the write-ahead logs of the chunk and metadata servers,
selected by one environment variable, plus the matrixraft pin bump that provides it.
Off by default: with the variable unset nothing changes.

  • Pin: matrixraft moves from 70ff96f to 6ecf5a2 (adds
    PeerTuning::wal_zero_fill_bytes and the journal scrub; Cargo.lock moves only the
    matrixraft entry).
  • Replica journal: ReplicaJournal::open_with_options takes a zero-fill chunk. When
    nonzero, a journal this call creates, or the replacement a checkpoint writes, is a
    zero-filled file whose outcome frames (magic MOZ2) end in one nonzero terminator
    byte, so appends overwrite in place and the file does not grow. The log length is kept
    apart from the file length. The original layout and its recovery are unchanged, and an
    existing journal keeps its layout.
  • Servers: MATRIXOBJECT_WAL_ZERO_FILL_BYTES (unset or empty = the original layout;
    otherwise 4096 through 536870912, anything else stops the process at startup with
    status 2) selects the layout for the Raft WAL of a chunk voter, the Raft WAL of a
    metadata voter and the standalone chunk journal. Documented in docs/configuring.md.
  • Scrub: ReplicaJournal::scrub_step validates the journal from a frame boundary until
    a byte budget is read, never into the zero fill; it applies nothing and does not move
    the append cursor. Nothing calls it yet.

Recovery rule

Recovery stops at an all-zero header and requires zeros after it. A bad header is dropped
only when nothing but zeros follows; a bad payload only when its last byte is zero and
nothing but zeros follows. A frame whose terminator is present, or that has anything after
it, refuses the open, so an acknowledged record is never dropped as if it were a cut
write. What a cut write left past the log is zeroed before the next append.

Tests

  • Three chunk voters and three metadata voters with the setting on keep every acknowledged
    page or placement across leader loss and a restart of all voters, with the layout
    asserted on disk. With it off the voters keep the original layout.
  • An out-of-range value stops each server at startup.
  • A 512 MiB fill, as large as the peer WAL limit, opens, writes and reopens without
    tripping that limit.
  • A counting test pins one sync for a batch of deferred outcome appends in both layouts.
  • An ignored test times the chunk group with the setting on and off, interleaved in both
    orders.

Measured

On a quiet host a single-voter peer write is about 2.4 times faster with the setting on.
The three-process chunk group, which puts three voters on one disk, showed no difference
at 4 MiB or 32 MiB chunks. An earlier run on a busy host was too noisy to read a gain from.
The setting is therefore worth turning on where each voter has its own device; it is not
expected to help three voters sharing one.

🤖 Generated with Claude Code

superhaiou and others added 7 commits November 16, 2026 09:13
ReplicaJournal::open_with_options takes a zero-fill chunk. When nonzero, a
journal this call creates, or the replacement a checkpoint writes, is laid
out as a zero-filled file whose outcome frames (magic MOZ2) end in one
nonzero terminator byte, so appends overwrite in place and the file does not
grow. The log length is kept apart from the file length.

Recovery stops at an all-zero header and requires zeros after it. A bad
header is dropped only when nothing but zeros follows; a bad payload only
when its last byte is zero and nothing but zeros follows. A frame whose
terminator is present, or that has anything after it, refuses the open, so an
acknowledged record is never dropped as if it were a cut write. What a cut
write left past the log is zeroed before the next append. The original layout
and its recovery are unchanged and an existing journal keeps its layout. The
server does not pass a chunk yet, so nothing changes by default.

Tests assert what was acknowledged: a crash at every byte of the last write
followed by further writes, a flipped byte in every byte of the log, stray
bytes past the end, clean reopen without file growth and one barrier per
write, checkpoint replacement, and the original layout kept.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ReplicaJournal::scrub_step validates the journal from a frame boundary until
a byte budget is read: from zero the checkpoint image and its digest, then
each frame's header, payload digest and terminator, up to the log end and
never into the zero fill. It reads with positioned I/O, applies nothing and
does not move the append cursor. Nothing calls it yet.

A counting test pins one sync for a batch of deferred outcome appends in the
original and the zero-tail layout, and none before the batch syncs.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The first chunk of a zero-tail replica journal is filled while the journal
opens, so the fill is paid before the node serves. A refill, once a chunk of
log has been written, is still paid by the write that needs it.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Take the durable peer as it is on main, which adds the opt-in zero-filled WAL
layout (PeerTuning::wal_zero_fill_bytes) and the journal scrub. Cargo.lock
moves only the matrixraft entry.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
MATRIXOBJECT_WAL_ZERO_FILL_BYTES (unset or empty = the original layout;
otherwise 4096 through 536870912, anything else stops the process at startup
with status 2) selects the zero-filled layout for the Raft WAL of a chunk
voter, the Raft WAL of a metadata voter and the standalone chunk journal.
ChunkRaftOptions and MetaRaftOptions gain wal_zero_fill_bytes, and
ReplicaNodeOptions gains it with a default of None; the peers open with
PeerTuning, and the journal with open_with_options, which now checks the
bound itself. Documented in docs/configuring.md.

Tests: three chunk voters and three metadata voters with the setting on keep
every acknowledged page or placement across leader loss and a restart of all
voters, with the layout asserted on disk; with it off the voters keep the
original layout; an out-of-range value stops each server at startup; a 512 MiB
fill, as large as the peer WAL limit, opens, writes and reopens without
tripping that limit. An ignored test times the chunk group with the setting on
and off, interleaved in both orders.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A single-voter peer write is about 2.4 times faster with the setting on; the
three-process chunk group, which puts three voters on one disk, showed no
difference at 4 MiB or 32 MiB chunks. Say so, and say that an earlier busy-host
run was too noisy to read a gain from.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The metaserver validates MATRIXOBJECT_WAL_ZERO_FILL_BYTES for every build so a
bad value stops it at startup, but only the consensus build hands the value to a
peer, so a default-features build with warnings denied rejected it as unused.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@matrixarkai
matrixarkai force-pushed the perf/a-chunk-or-metadata-server-can-lay-its-wal-out-zero-filled branch from 1fa09e9 to 70d6c8a Compare November 16, 2026 17:17
@matrixarkai
matrixarkai merged commit 1c790cb into main Nov 16, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants