Skip to content

Warn when the metadata Raft WAL has no checkpoint interval, and test the compacting append under failure - #640

Merged
matrixarkai merged 3 commits into
mainfrom
log-append-crash-and-checkpoint-warning
Nov 16, 2026
Merged

matrixarkai merged 3 commits into
mainfrom
log-append-crash-and-checkpoint-warning

Conversation

@matrixarkai

Copy link
Copy Markdown
Owner

What

Three small changes that make the compacting log append safe to operate.

  1. A startup warning. With MATRIXOBJECT_METASERVER_RAFT_CHECKPOINT_BYTES unset
    (its default, 0), the metadata Raft WAL is never truncated and a voter refuses
    appends once it reaches 512 MiB. The metaserver now says so at startup when Raft is
    configured and the variable is unset.
  2. A configuration-guide section that documents the variable as required for any
    workload that publishes many objects or appends to log objects.
  3. Two process-level crash tests for append_object_compacting (merged in Keep a log object appendable past its extent bound #639).

Why

Measured on a real three-metaserver, three-chunkserver star group with 300-byte
appends and full-manifest publication: with checkpointing off, the three voters wrote
about 2 GB of WAL in total and all exited after about 2,770 appends. The cause is
visible only in the voter's own log, so an operator has no way to find it before the
cluster stops. The warning and the guide section put the setting where it is needed.

Crash tests

  • With a star member dead, the compaction is refused, and the manifest and the bytes
    are unchanged.
  • With the metadata leader dead, the object is exactly before or exactly after: an
    acknowledged append is published, and an unpublished one retries cleanly.

They run in the existing node_regime test group, one process-level fixture at a time.

Not changed

No behaviour changes when the variable is set. The warning is printed to standard
error and does not alter how the metaserver starts.

🤖 Generated with Claude Code

superhaiou and others added 3 commits November 16, 2026 07:33
With MATRIXOBJECT_METASERVER_RAFT_CHECKPOINT_BYTES unset the metadata WAL is
never truncated and refuses appends at 512 MiB; a log workload reached that
after about 2,770 appends. The metaserver now says so at startup and the
configuration guide documents the variable as required for such workloads.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…tadata leader

Process-level: with a star member dead the compaction is refused and the
manifest and bytes are unchanged; with the metadata leader dead the object is
exactly before or exactly after, an acknowledged append is published, and an
unpublished one retries cleanly.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The message literal carried runs of spaces from a lost line continuation.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@matrixarkai
matrixarkai merged commit 8303112 into main Nov 16, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants