Repository navigation
Appends survive one stopped data node: availability rule, default heartbeat expiry, switch on failure - #642
Merged
Conversation
…iry, switch on failure A fixed three-member Star group needs every member to acknowledge a write, but placement counted it available while two members were online, so one stopped member kept attracting writes that then failed. A Star group is now available only while all members report online and not draining. Heartbeat expiry is on by default (15000 ms, 0 turns it off) and the metaserver prints the value in effect. Publication no longer requires every group already named by an object to be writable; only a group the publication adds to or newly names must be. RemoteMetaClient::append_object_with_failover re-plans after a refusal before publication, reports unreachable members offline, waits 1 s doubling to 64 s (configurable), and returns the pages of refused attempts without deleting them. Measured with real processes: 40 of 40 appends acknowledged through a SIGKILL of one chunkserver, one switch, slowest append 77 ms; before, about half failed with two groups and all failed with one. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… is not retried Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Behaviour change to read first
MATRIXOBJECT_METASERVER_HEARTBEAT_TTL_MSnow defaults to 15000 ms (it was0, off). Achunkserver silent for longer than that leaves the placement roster and returns on its next
report;
0is the explicit off switch, and the value in effect is printed on the startup lineas
heartbeat_ttl_ms=. A fixed Star group is now considered available for placement only whileall of its members report online and not draining (Raft groups keep the majority rule).
Why
A routed Star group needs every member to acknowledge a write (N=3, W=3, R=1). Measured with
one metaserver and two Star groups, one chunkserver stopped with SIGKILL after 15 acknowledged
appends:
placement kept handing out the broken group. About every second append failed in 5 to 22 ms
with
short of a write quorum of 3, and an immediate retry succeeded on the other group.recovered.
back exactly.
Changes
project_star_health,chunk_groups.rs).group the publication adds to or newly names must be.
RemoteMetaClient::append_object_with_failoverwrapsappend_object. After an attempt isrefused before publication and the manifest is still the one the caller passed, it probes the
members of the groups the next attempt would use, reports unreachable members offline, waits
and plans again, so placement chooses another group and the whole attempt is rewritten there.
It reports success only when the full write quorum of that group acknowledged it and the
conditional publication succeeded. The wait before the first switch is 1 s and doubles to 64 s
(configurable; tests shorten it).
PublicationUnconfirmedis never retried. A manifest thatchanged since the caller read it is a conflict and is returned unchanged.
AbandonedAttemptand are never deleted; physicalreclamation is still disabled.
A retry cannot duplicate or lose an acknowledged append: publication is conditional on the exact
manifest the caller supplied, a refused attempt has not published, each attempt takes fresh page
identities, and at most one attempt can publish.
Measured (real processes, one machine)
40 of 40 appends acknowledged through a SIGKILL of one chunkserver, one switch, slowest append
77 ms. Before: about half failed with two groups and all failed with one.
Tests
tests/member_failure_append.rs(process level, run one test at a time): 30 appends through aSIGKILL with every one acknowledged once and the object equal to the acknowledged bytes; a single
group refuses cleanly before publication and heals when the node returns; a stopped node leaves
the roster within the TTL and returns on its next report; the default and the off switch; a
returning member with no data never serves wrong bytes and is resynced by the repair worker; a
conflicting append is returned to the caller without a switch. Three planted defects (majority
rule for Star groups, publication requiring every named group, retrying through a conflict) each
turned a test red and were restored.
Not done
replica count is not implemented (the worker sweeps groups in catalog order).
🤖 Generated with Claude Code