Skip to content

Appends survive one stopped data node: availability rule, default heartbeat expiry, switch on failure - #642

Merged
matrixarkai merged 2 commits into
mainfrom
feat/appends-survive-one-stopped-data-node
Nov 16, 2026
Merged

matrixarkai merged 2 commits into
mainfrom
feat/appends-survive-one-stopped-data-node

Conversation

@matrixarkai

Copy link
Copy Markdown
Owner

Behaviour change to read first

MATRIXOBJECT_METASERVER_HEARTBEAT_TTL_MS now defaults to 15000 ms (it was 0, off). A
chunkserver silent for longer than that leaves the placement roster and returns on its next
report; 0 is the explicit off switch, and the value in effect is printed on the startup line
as heartbeat_ttl_ms=. A fixed Star group is now considered available for placement only while
all of its members report online and not draining (Raft groups keep the majority rule).

Why

A routed Star group needs every member to acknowledge a write (N=3, W=3, R=1). Measured with
one metaserver and two Star groups, one chunkserver stopped with SIGKILL after 15 acknowledged
appends:

  • Metadata counted a three-member Star group as available while two members were online, so
    placement kept handing out the broken group. About every second append failed in 5 to 22 ms
    with short of a write quorum of 3, and an immediate retry succeeded on the other group.
  • With a single group every append failed for the whole 8 s observation window and nothing
    recovered.
  • Heartbeat expiry was off, so a stopped node stayed online in the roster indefinitely.
  • In every case the manifest was unchanged after a failure and every acknowledged byte read
    back exactly.

Changes

  1. Placement availability of a Star group (project_star_health, chunk_groups.rs).
  2. Heartbeat expiry on by default (above).
  3. Publication no longer requires every group an object already names to be writable; only a
    group the publication adds to or newly names must be.
  4. RemoteMetaClient::append_object_with_failover wraps append_object. After an attempt is
    refused before publication and the manifest is still the one the caller passed, it probes the
    members of the groups the next attempt would use, reports unreachable members offline, waits
    and plans again, so placement chooses another group and the whole attempt is rewritten there.
    It reports success only when the full write quorum of that group acknowledged it and the
    conditional publication succeeded. The wait before the first switch is 1 s and doubles to 64 s
    (configurable; tests shorten it). PublicationUnconfirmed is never retried. A manifest that
    changed since the caller read it is a conflict and is returned unchanged.
  5. Pages of refused attempts are returned in AbandonedAttempt and are never deleted; physical
    reclamation is still disabled.

A retry cannot duplicate or lose an acknowledged append: publication is conditional on the exact
manifest the caller supplied, a refused attempt has not published, each attempt takes fresh page
identities, and at most one attempt can publish.

Measured (real processes, one machine)

40 of 40 appends acknowledged through a SIGKILL of one chunkserver, one switch, slowest append
77 ms. Before: about half failed with two groups and all failed with one.

Tests

tests/member_failure_append.rs (process level, run one test at a time): 30 appends through a
SIGKILL with every one acknowledged once and the object equal to the acknowledged bytes; a single
group refuses cleanly before publication and heals when the node returns; a stopped node leaves
the roster within the TTL and returns on its next report; the default and the off switch; a
returning member with no data never serves wrong bytes and is resynced by the repair worker; a
conflicting append is returned to the caller without a switch. Three planted defects (majority
rule for Star groups, publication requiring every named group, retrying through a conflict) each
turned a test red and were restored.

Not done

  • A distinct read-only or draining mode is not carried by the chunkserver's own report.
  • A returning node is not held out of placement until it has caught up; repair order by remaining
    replica count is not implemented (the worker sweeps groups in catalog order).
  • Abandoned pages are returned to the caller, not recorded durably in metadata.
  • Everything above was exercised against a single metadata server, not a metadata quorum.

🤖 Generated with Claude Code

superhaiou and others added 2 commits November 16, 2026 09:44
…iry, switch on failure

A fixed three-member Star group needs every member to acknowledge a write, but
placement counted it available while two members were online, so one stopped
member kept attracting writes that then failed. A Star group is now available
only while all members report online and not draining.

Heartbeat expiry is on by default (15000 ms, 0 turns it off) and the metaserver
prints the value in effect.

Publication no longer requires every group already named by an object to be
writable; only a group the publication adds to or newly names must be.

RemoteMetaClient::append_object_with_failover re-plans after a refusal before
publication, reports unreachable members offline, waits 1 s doubling to 64 s
(configurable), and returns the pages of refused attempts without deleting them.

Measured with real processes: 40 of 40 appends acknowledged through a SIGKILL
of one chunkserver, one switch, slowest append 77 ms; before, about half failed
with two groups and all failed with one.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… is not retried

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@matrixarkai
matrixarkai merged commit 2a2d7ad into main Nov 16, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants