Skip to content

docs(postgresql): hot standby — peer config on both sides, safe scale-out after switchover - #868

Open
SuJinpei wants to merge 1 commit into
mainfrom
docs/pg-xcr-v434-switchover-safety
Open

SuJinpei wants to merge 1 commit into
mainfrom
docs/pg-xcr-v434-switchover-safety

Conversation

@SuJinpei

@SuJinpei SuJinpei commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

What

Updates the PostgreSQL hot standby guide after a customer DR switchover on Operator v4.3.4 left both clusters as Standby Leader.

  • Configure the peer connection on the primary cluster as well. Adds peerHost / peerPort / bootstrapSecret on the primary, pointing at the standby. With Operator v4.3.4, a standby without peerHost cannot resolve its peer after an operator restart (including the restart from upgrading the operator). Its promotion is then never applied.
  • Switchover: scale out safely. Phase 2 now promotes at one instance. A new Phase 2b removes the leftover pgdata-<cluster>-1 volume before scaling out. Scaling out onto that stale volume can let it take over as leader during the post-promotion rolling update, which discards transactions committed since it went stale.
  • Troubleshooting: adds "Promotion never takes effect after an operator restart" with the error signature (secret name starting with .), the cause, and a recovery procedure: an LSN check on both sides, then a peer secret and a single patch.

Validation

  • Reproduced on an ACP 4.4 cross-cluster pair running Operator v4.3.4. A restart of the operator on a standby without peerHost produced the exact error, and both sides stayed Standby Leader after the switchover.
  • The recovery procedure in Troubleshooting (with numberOfInstances: 1) promoted the stuck cluster in about 25 s with no data loss. It also resolved the customer incident.
  • The stale-volume takeover was reproduced twice on the same pair; the operator log shows cannot perform switch over before re-creating the pod: no replicas, followed by the stale member acquiring the leader lock.
  • Phase 2b (delete the leftover volume, then scale out) is how the customer restored two instances. It was not separately re-run on the test pair.

EN only; zh is regenerated by the translation pipeline. Tracking: ECO-1852 (internal).

🤖 Generated with Claude Code

…-out after switchover

- Configure peerHost/peerPort/bootstrapSecret on the primary cluster too. With
  Operator v4.3.4 a standby without peerHost cannot resolve its peer after an
  operator restart (incl. upgrade); its promotion is never applied and both
  clusters stay Standby Leader.
- Switchover Phase 2 promotes at one instance; new Phase 2b removes the
  leftover pgdata-<cluster>-1 volume before scaling out. Scaling out onto that
  stale volume lets it take over as leader during the post-promotion rolling
  update and discards transactions committed since it went stale.
- Troubleshooting: "Promotion never takes effect after an operator restart"
  with the validated recovery (LSN check, peer secret, single patch).

Refs ECO-1852.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch had an error being deployed

1 failed deployment
translate — 1f017dc5 Deployed Nov 7, 2026 by SuJinpei via build #2780
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant