[yeya24]

What this PR does:

Adds experimental, opt-in push coalescing to the ingester: concurrent write requests for the same tenant are merged into a single TSDB commit.

Problem

Every Push takes its own appender and calls Commit() on it. Commit() writes the WAL through wlog.(*WL).Log, which holds the tenant's WL.mtx for the whole record loop. When a distributor shards one large tenant's remote-write requests by series across many ingesters, every ingester sees a very high rate of requests that each carry only a few samples, and takes that one mutex once per request. Once any waiter has waited longer than 1ms, sync.Mutex switches to starvation mode and hands the lock over in strict FIFO order; if woken waiters are then slow to get back on a CPU, each handoff costs far more than the work done under the lock, throughput drops below the arrival rate, and in-flight pushes pile up.

Approach

One merge point per tenant TSDB (userTSDB.pushCoalescer), so a batch never spans tenants and the WAL mutex it relieves is that tenant's.

• A request that arrives while no commit is in flight commits immediately on its own: no added latency on an idle ingester.
• A request that arrives while a commit is in flight joins the next batch. When the in-flight commit ends, the first queued request becomes the leader of that batch: it appends every request in the batch into one appender, calls Commit() once, and hands each request its own result.
• No timer and no "wait for N ms". A request only ever waits for a commit that was already in flight.
• Joining a batch is a single compare-and-swap on a lock-free (Treiber) stack, so the merge point cannot itself fall into mutex starvation mode.
• A batch is truncated, and the rest becomes the next batch, at -ingester.push-coalesce.max-batch-requests (default 32), -ingester.push-coalesce.max-batch-samples (default 2000) or -ingester.push-coalesce.max-batch-bytes (default off). Requests that disagree on DiscardOutOfOrder never share a batch, because it is an appender-level option. Truncation keeps arrival order, so a request too large to join anyone else's batch still leads its own.
• A request whose context is already done is rejected before it joins, and dropped if it is canceled while queued.
• Commit() failure fails every request in the batch. An append error that requires discarding the appender rolls the whole batch back; the offending request gets its error and the others get a retryable error. Label-set order is validated before a request joins a batch, so the one deterministic error on that path cannot take a whole batch down.

The append phase of Push is unchanged: it is wrapped in a closure over the request's own locals, so per-request limits, active series, errors and discarded-sample counts stay attributed to the request they belong to, and metrics are still emitted by each request's own goroutine after the commit. A follower never returns before the leader has finished appending its samples, which keeps the pooled write request alive for as long as the leader reads it.

Disabled by default (-ingester.push-coalesce.enabled=false); with it off, Push follows exactly the existing path.

Semantic differences when enabled

A batch shares one appender, and Append only checks against committed state while Commit re-checks against what earlier samples in the same batch left behind, counting rather than returning what it then rejects. When two requests in one batch write the same series:

Case	Separate commits	Coalesced
Same timestamp, different value	second request gets 400 duplicate sample	dropped without an error; same stored data
Out of order, beyond the window	second request gets 400 too old	dropped without an error; same stored data
Out of order, the batch's own newest sample pushes it past the window	rejected as too old	stored in the OOO head

The third case can only ever store more, never less, and the affected timestamps lie in [H − W, t_X − W) where H is the head max time before the batch and t_X the batch's newest sample: a band as wide as the batch's advance of the head max (one scrape interval in steady state, against an OOO window of typically tens of minutes), and only for a series silent for longer than the window.

For reference, Cortex already behaves this way within a single request: two samples for one series at the same timestamp in one request return 200 and keep the first value, while the same two samples sent as two requests return 400 (pinned by TestIngesterPush_DuplicateIsAlreadySilentWithinOneRequest). Coalescing widens that from one request to one batch. While enabled, the new-value-for-timestamp and sample-too-old reasons of cortex_discarded_samples_total can under-report accordingly.

Tests

• Coalescer unit tests (fake appender, deterministic batch composition): a lone request commits alone; queued requests share one commit; commit failure fails the whole batch; a fatal append rolls back the batch with correct per-request errors; context canceled before joining, while queued, and while queued in a batch that is then rolled back; every truncation limit, including that an oversized request still leads its own batch; 64×50 concurrent pushes each appended exactly once.
• Push-level tests against a real TSDB, comparing with separate commits:
  • TestIngesterPushCoalescing_DivergenceFromSeparateCommits: 8 scenarios, each comparing every request's error and the full head contents including the OOO head. Each scenario declares whether it expects errors or data to differ, and both directions are asserted, so a TSDB change either way fails the test.
  • TestIngesterPushCoalescing_OutOfOrderEscapeIsBoundedByInBatchHeadAdvance: pins the band above with samples just inside, below and above it.
  • TestIngesterPushCoalescing_ErrorsStayAttributedToTheirRequest and ..._PerUserMetricTotalsMatchSeparateCommits: per-request errors and the ingested, failed and discarded-sample counters match separate commits, with the expected values also written out.
  • Native histograms and exemplars through a leader match separate commits.
  • 32 goroutines pushing concurrently lose, duplicate or corrupt no sample.
• All of ./pkg/ingester/... passes, also with -race.

Benchmarks

BenchmarkIngesterPushCoalescing* in pkg/ingester/push_coalesce_bench_test.go. Single tenant, 5 samples per request, OOO window on, every series created before the clock starts. AMD EPYC 7R13, local NVMe, Go 1.27, GOMAXPROCS=8, median of 3 runs.

Pushers	Off pushes/s	On pushes/s	Throughput	Off p99	On p99	Requests per commit
1	74,152	70,916	0.96×	0.02ms	0.02ms	1.0
64	78,556	273,926	3.49×	4.91ms	0.37ms	31.5
1,024	75,639	150,741	1.99×	64.7ms	10.9ms	32.0
4,096	68,111	98,101	1.44×	295ms	54.7ms	32.0
16,384	62,303	64,053	1.03×	1,086ms	324ms	32.0

When concurrent requests often write the same series (512 shared series): 100k → 204k pushes/s at 1,024 pushers, 78k → 159k at 4,096, still 32 requests per commit.

Mutex profile at 4,096 pushers: aggregate mutex delay drops from 12,817 s to 0.06 s; wlog.(*WL).Log accounts for 9,807 s of the former and does not appear in the latter.

The cost is p50: a queued request waits for the in-flight commit (at 4,096 pushers, 0.03ms → 41ms), while p99 improves 3–13×. At high concurrency throughput levels off, because the leader appends a batch serially. In an earlier revision of this change, -ingester.push-coalesce.max-batch-requests=128 measured 15–20% faster than the default 32 at 1,024–4,096 pushers; the default is kept conservative.

Which issue(s) this PR fixes:
None filed.

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
• docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags