Skip to content

feat(relay): start-first update path with health-gated cutover and rollback - #77

Merged
Mearman merged 3 commits into
mainfrom
feat/blue-green-relay-update
Nov 16, 2026
Merged

Mearman merged 3 commits into
mainfrom
feat/blue-green-relay-update

Conversation

@Mearman

@Mearman Mearman commented Nov 16, 2026

Copy link
Copy Markdown
Member

The relay had no side-by-side update: replacing it refused connections while the new process started and ended every open session.

This adds a systemd socket handoff. Two slots (blue, green) accept on one systemd-owned socket, so the new version starts beside the old one on the same port with no proxy. cascade-relay-update.sh NEW_BINARY starts the idle slot, waits for /health to return 200 and stay healthy, then stops the old slot, which drains its open sessions. If the new slot exits, never becomes healthy or dies in the settle window, it is stopped and the old slot keeps serving (exit status 1).

Relay changes: SIGTERM drains (/health returns 503 draining, new connections_open and draining metrics, --drain-timeout-seconds), and listenfd support for the inherited socket. systemd flips an inherited socket to blocking mode on every instance start, which broke the first version (the old slot's accept hung after a failed start), so accepting on an inherited socket runs on a thread with a 100 ms accept timeout.

SO_REUSEPORT was rejected because closing a listener resets its queued connections. Docker start-first does not apply: a published port can be bound by one container, and docker compose ignores the setting. The docs say so and give the stop_grace_period and proxy options for Docker.

test/relay-update/test_update.py runs the shipped units under systemctl --user with a probe on the shared port: a clean update, the new slot SIGKILLed mid-update, and a binary that cannot start. It passes on a Linux host; the new relay-update CI job is unverified until this PR's checks run, as it needs a user systemd manager on the runner.

…llback

The relay now drains on SIGTERM: it stops accepting, /health turns to 503
"draining", and it exits once its open connections end or the drain timeout
passes. It can serve on a listening socket inherited from systemd, accepting on
a dedicated thread because systemd flips an inherited socket to blocking mode
each time it starts an instance.

Two systemd slots share one socket unit, so the new version starts beside the
old one on the same port without a proxy. cascade-relay-update.sh starts the
idle slot, health-gates it, then stops the old slot so it drains. A new slot
that exits, never becomes healthy or dies during the settle period is stopped
and the old slot keeps serving.

A test under test/relay-update drives the shipped units with systemctl --user:
a clean update, the new slot killed mid-update, and a binary that cannot start,
each with a probe on the shared port. CI runs it in a relay-update job.
Copilot AI balanced review requested due to automatic review settings November 16, 2026 08:41

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

…nt simulator

The accept timeout is built with TimeValLike::microseconds so it compiles where time_t and
suseconds_t are narrower than i64 (macOS). The inherited-listener path and its types are gated
to Unix, so Windows builds carry no dead code. A public doc comment no longer links to a private
item.

The iOS logic tests target iPhone 17: the runner image no longer provides an iPhone 16
simulator.
Copilot AI balanced review requested due to automatic review settings November 16, 2026 11:07

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

Copilot AI balanced review requested due to automatic review settings November 16, 2026 11:19

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota. Try requesting a review at a lower effort level.

@Mearman
Mearman marked this pull request as ready for review November 16, 2026 12:31
@Mearman
Mearman merged commit d823cfe into main Nov 16, 2026
55 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants