Repository navigation
Let the chunk and metadata servers lay their WAL out zero-filled, opt-in - #641
Merged
matrixarkai merged 7 commits intoNov 16, 2026
Conversation
ReplicaJournal::open_with_options takes a zero-fill chunk. When nonzero, a journal this call creates, or the replacement a checkpoint writes, is laid out as a zero-filled file whose outcome frames (magic MOZ2) end in one nonzero terminator byte, so appends overwrite in place and the file does not grow. The log length is kept apart from the file length. Recovery stops at an all-zero header and requires zeros after it. A bad header is dropped only when nothing but zeros follows; a bad payload only when its last byte is zero and nothing but zeros follows. A frame whose terminator is present, or that has anything after it, refuses the open, so an acknowledged record is never dropped as if it were a cut write. What a cut write left past the log is zeroed before the next append. The original layout and its recovery are unchanged and an existing journal keeps its layout. The server does not pass a chunk yet, so nothing changes by default. Tests assert what was acknowledged: a crash at every byte of the last write followed by further writes, a flipped byte in every byte of the log, stray bytes past the end, clean reopen without file growth and one barrier per write, checkpoint replacement, and the original layout kept. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
ReplicaJournal::scrub_step validates the journal from a frame boundary until a byte budget is read: from zero the checkpoint image and its digest, then each frame's header, payload digest and terminator, up to the log end and never into the zero fill. It reads with positioned I/O, applies nothing and does not move the append cursor. Nothing calls it yet. A counting test pins one sync for a batch of deferred outcome appends in the original and the zero-tail layout, and none before the batch syncs. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The first chunk of a zero-tail replica journal is filled while the journal opens, so the fill is paid before the node serves. A refill, once a chunk of log has been written, is still paid by the write that needs it. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Take the durable peer as it is on main, which adds the opt-in zero-filled WAL layout (PeerTuning::wal_zero_fill_bytes) and the journal scrub. Cargo.lock moves only the matrixraft entry. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
MATRIXOBJECT_WAL_ZERO_FILL_BYTES (unset or empty = the original layout; otherwise 4096 through 536870912, anything else stops the process at startup with status 2) selects the zero-filled layout for the Raft WAL of a chunk voter, the Raft WAL of a metadata voter and the standalone chunk journal. ChunkRaftOptions and MetaRaftOptions gain wal_zero_fill_bytes, and ReplicaNodeOptions gains it with a default of None; the peers open with PeerTuning, and the journal with open_with_options, which now checks the bound itself. Documented in docs/configuring.md. Tests: three chunk voters and three metadata voters with the setting on keep every acknowledged page or placement across leader loss and a restart of all voters, with the layout asserted on disk; with it off the voters keep the original layout; an out-of-range value stops each server at startup; a 512 MiB fill, as large as the peer WAL limit, opens, writes and reopens without tripping that limit. An ignored test times the chunk group with the setting on and off, interleaved in both orders. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A single-voter peer write is about 2.4 times faster with the setting on; the three-process chunk group, which puts three voters on one disk, showed no difference at 4 MiB or 32 MiB chunks. Say so, and say that an earlier busy-host run was too noisy to read a gain from. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The metaserver validates MATRIXOBJECT_WAL_ZERO_FILL_BYTES for every build so a bad value stops it at startup, but only the consensus build hands the value to a peer, so a default-features build with warnings denied rejected it as unused. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
matrixarkai
force-pushed
the
perf/a-chunk-or-metadata-server-can-lay-its-wal-out-zero-filled
branch
from
November 16, 2026 17:17
1fa09e9 to
70d6c8a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
An opt-in zero-filled layout for the write-ahead logs of the chunk and metadata servers,
selected by one environment variable, plus the matrixraft pin bump that provides it.
Off by default: with the variable unset nothing changes.
70ff96fto6ecf5a2(addsPeerTuning::wal_zero_fill_bytesand the journal scrub;Cargo.lockmoves only thematrixraft entry).
ReplicaJournal::open_with_optionstakes a zero-fill chunk. Whennonzero, a journal this call creates, or the replacement a checkpoint writes, is a
zero-filled file whose outcome frames (magic
MOZ2) end in one nonzero terminatorbyte, so appends overwrite in place and the file does not grow. The log length is kept
apart from the file length. The original layout and its recovery are unchanged, and an
existing journal keeps its layout.
MATRIXOBJECT_WAL_ZERO_FILL_BYTES(unset or empty = the original layout;otherwise 4096 through 536870912, anything else stops the process at startup with
status 2) selects the layout for the Raft WAL of a chunk voter, the Raft WAL of a
metadata voter and the standalone chunk journal. Documented in
docs/configuring.md.ReplicaJournal::scrub_stepvalidates the journal from a frame boundary untila byte budget is read, never into the zero fill; it applies nothing and does not move
the append cursor. Nothing calls it yet.
Recovery rule
Recovery stops at an all-zero header and requires zeros after it. A bad header is dropped
only when nothing but zeros follows; a bad payload only when its last byte is zero and
nothing but zeros follows. A frame whose terminator is present, or that has anything after
it, refuses the open, so an acknowledged record is never dropped as if it were a cut
write. What a cut write left past the log is zeroed before the next append.
Tests
page or placement across leader loss and a restart of all voters, with the layout
asserted on disk. With it off the voters keep the original layout.
tripping that limit.
orders.
Measured
On a quiet host a single-voter peer write is about 2.4 times faster with the setting on.
The three-process chunk group, which puts three voters on one disk, showed no difference
at 4 MiB or 32 MiB chunks. An earlier run on a busy host was too noisy to read a gain from.
The setting is therefore worth turning on where each voter has its own device; it is not
expected to help three voters sharing one.
🤖 Generated with Claude Code