Skip to content

Serve the replicated object plane through an opt-in S3 gateway mode - #643

Merged
matrixarkai merged 1 commit into
mainfrom
feat/an-opt-in-s3-gateway-over-the-replicated-object-plane
Nov 16, 2026
Merged

matrixarkai merged 1 commit into
mainfrom
feat/an-opt-in-s3-gateway-over-the-replicated-object-plane

Conversation

@matrixarkai

@matrixarkai matrixarkai commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

What

An opt-in mode of object_server that serves the replicated object plane instead of an
in-process store. With MATRIXOBJECT_CLUSTER_META unset (the default) nothing changes.

With it set, object_server fronts a metadata quorum plus chunk groups running as separate
processes, reached through the cluster SDK. A shared-log client such as TemporalStore can then
run as its own cluster on top of a MatrixObject cluster, without linking the SDK.

Why

Until now the S3 front door never reached the replicated plane: src/net did not use
src/cluster at all, so a client talking S3 always got one in-process store. This door is deliberately
small. It answers the verbs a shared-log client needs and answers everything else with an S3
NotImplemented, so a probing client falls back and nothing returns a plausible wrong answer.

Served verbs

PUT; append by write offset; GET and ranged GET; HEAD; DELETE (a tombstone generation);
ListObjectsV2 (prefix, start-after, continuation-token, max-keys, delimiter). The
mapping to the SDK and every limit is in docs/cluster-gateway.md. Refused with
NotImplemented: multipart, copy, tagging, versioning, CORS, bucket delete, If-Match.

An append is an offset condition plus a compare-and-set on the manifest generation, and the
reply reports the length the published manifest reports, so a receipt always says where the
bytes landed. An unconfirmed publication is not retried and is not reported as success (503).
An object past the 4096-extent limit is refused with a clear 400 before any page is written.

Changes outside the new module

ObjectHttpServer can now serve any RequestHandler; the in-process ObjectService
implements it and remains the default. The existing append tests in src/net/mod.rs are
byte-identical to main.

Tooling

scripts/cluster_gateway_bringup.sh up|down|status starts and stops a local metaserver, three
chunkservers and the gateway, touching only its own processes. The harness driver and a verb
probe used for the measurements are included.

Measured

One host, release build, one metaserver and three chunkservers as one replicated group (every
write acknowledged by all three), the TemporalStore scale harness at 2000 operations per phase,
interleaved with the in-process door, no errors:

gateway in-process door
sync write p50 / p99 59.1 / 167 ms and 42.5 / 150 ms 4.9 / 8.4 ms and 4.5 / 8.5 ms
async write p50 0.9 to 1.1 ms 0.8 ms
replica read p50 144 to 176 µs 108 to 132 µs

After 4000 writes the three chunk journals together hold 441 bytes per write, while the
metadata outcomes hold about 134 KB per write: every append publishes the full manifest outcome.
That metadata cost, not the data, is what a synchronous write pays for. The compact
publication that removes most of it (about 92 times fewer metadata bytes) and the tail merge
that keeps a log appendable past 4096 extents are measured separately and follow in a later
change with their own tests.

Not done

  • Raft metadata (three voters) and failure injection are not exercised by this change's tests.
  • Whole-object GET reads the whole object into the gateway; listing walks the catalog from the
    start (the SDK takes an exclusive after key, not a prefix).
  • The gateway is unauthenticated, like the in-process door.

🤖 Generated with Claude Code

object_server normally serves one in-process store. With MATRIXOBJECT_CLUSTER_META set
it serves the replicated object plane instead: a metadata quorum plus chunk groups
running as separate processes, reached through the cluster SDK. The in-process path is
untouched; the listener now dispatches through a small RequestHandler trait that the
in-process service implements, so either backend sits behind the same accept loop.

The gateway answers the S3 verbs a shared-log client needs (PUT, append by write offset,
GET with ranges, HEAD, DELETE as a tombstone, ListObjectsV2) and answers every other verb
with NotImplemented, so a probing client falls back and nothing returns a plausible wrong
answer. An append is an offset condition plus a compare-and-set on the manifest
generation; the receipt reports the length the published manifest reports, so it always
says where the bytes landed. An unconfirmed publication is not retried and is not
reported as success. An object past the 4096-extent limit is refused with a clear 400
before any page is written.

scripts/cluster_gateway_bringup.sh starts and stops a local metaserver, three chunkservers
and the gateway, touching only its own processes. The harness driver and probe used to
measure it are included. docs/cluster-gateway.md lists the verbs, the mapping to the SDK
and the limits.

Measured on one host with a release build, one metaserver and three chunkservers as one
replicated group (every write acknowledged by all three), the TemporalStore scale harness
at 2000 operations per phase, interleaved with the in-process door: synchronous write p50
about 43 to 59 ms against about 4.5 to 4.9 ms for the door; asynchronous write and replica
read are the same order. After 4000 writes the three chunk journals hold 441 bytes per
write while the metadata outcomes hold about 134 KB per write, because every append
publishes the full manifest outcome; that metadata cost, not the data, is what a
synchronous write pays for.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@matrixarkai
matrixarkai merged commit 15221b8 into main Nov 16, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants