Repository navigation
Serve the replicated object plane through an opt-in S3 gateway mode - #643
Merged
matrixarkai merged 1 commit intoNov 16, 2026
Merged
Conversation
object_server normally serves one in-process store. With MATRIXOBJECT_CLUSTER_META set it serves the replicated object plane instead: a metadata quorum plus chunk groups running as separate processes, reached through the cluster SDK. The in-process path is untouched; the listener now dispatches through a small RequestHandler trait that the in-process service implements, so either backend sits behind the same accept loop. The gateway answers the S3 verbs a shared-log client needs (PUT, append by write offset, GET with ranges, HEAD, DELETE as a tombstone, ListObjectsV2) and answers every other verb with NotImplemented, so a probing client falls back and nothing returns a plausible wrong answer. An append is an offset condition plus a compare-and-set on the manifest generation; the receipt reports the length the published manifest reports, so it always says where the bytes landed. An unconfirmed publication is not retried and is not reported as success. An object past the 4096-extent limit is refused with a clear 400 before any page is written. scripts/cluster_gateway_bringup.sh starts and stops a local metaserver, three chunkservers and the gateway, touching only its own processes. The harness driver and probe used to measure it are included. docs/cluster-gateway.md lists the verbs, the mapping to the SDK and the limits. Measured on one host with a release build, one metaserver and three chunkservers as one replicated group (every write acknowledged by all three), the TemporalStore scale harness at 2000 operations per phase, interleaved with the in-process door: synchronous write p50 about 43 to 59 ms against about 4.5 to 4.9 ms for the door; asynchronous write and replica read are the same order. After 4000 writes the three chunk journals hold 441 bytes per write while the metadata outcomes hold about 134 KB per write, because every append publishes the full manifest outcome; that metadata cost, not the data, is what a synchronous write pays for. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
An opt-in mode of
object_serverthat serves the replicated object plane instead of anin-process store. With
MATRIXOBJECT_CLUSTER_METAunset (the default) nothing changes.With it set,
object_serverfronts a metadata quorum plus chunk groups running as separateprocesses, reached through the cluster SDK. A shared-log client such as TemporalStore can then
run as its own cluster on top of a MatrixObject cluster, without linking the SDK.
Why
Until now the S3 front door never reached the replicated plane:
src/netdid not usesrc/clusterat all, so a client talking S3 always got one in-process store. This door is deliberatelysmall. It answers the verbs a shared-log client needs and answers everything else with an S3
NotImplemented, so a probing client falls back and nothing returns a plausible wrong answer.Served verbs
PUT; append by write offset; GET and ranged GET; HEAD; DELETE (a tombstone generation);
ListObjectsV2 (
prefix,start-after,continuation-token,max-keys,delimiter). Themapping to the SDK and every limit is in
docs/cluster-gateway.md. Refused withNotImplemented: multipart, copy, tagging, versioning, CORS, bucket delete,If-Match.An append is an offset condition plus a compare-and-set on the manifest generation, and the
reply reports the length the published manifest reports, so a receipt always says where the
bytes landed. An unconfirmed publication is not retried and is not reported as success (503).
An object past the 4096-extent limit is refused with a clear
400before any page is written.Changes outside the new module
ObjectHttpServercan now serve anyRequestHandler; the in-processObjectServiceimplements it and remains the default. The existing append tests in
src/net/mod.rsarebyte-identical to main.
Tooling
scripts/cluster_gateway_bringup.sh up|down|statusstarts and stops a local metaserver, threechunkservers and the gateway, touching only its own processes. The harness driver and a verb
probe used for the measurements are included.
Measured
One host, release build, one metaserver and three chunkservers as one replicated group (every
write acknowledged by all three), the TemporalStore scale harness at 2000 operations per phase,
interleaved with the in-process door, no errors:
After 4000 writes the three chunk journals together hold 441 bytes per write, while the
metadata outcomes hold about 134 KB per write: every append publishes the full manifest outcome.
That metadata cost, not the data, is what a synchronous write pays for. The compact
publication that removes most of it (about 92 times fewer metadata bytes) and the tail merge
that keeps a log appendable past 4096 extents are measured separately and follow in a later
change with their own tests.
Not done
start (the SDK takes an exclusive
afterkey, not a prefix).🤖 Generated with Claude Code