Failover

Failover moves service responsibility from an unhealthy primary to a qualified standby. The safe sequence is detection, decision, fencing, catch-up, promotion, traffic shift, client recovery, and later failback.

RTO and RPO

• Recovery Time Objective (RTO) is the business target for acceptable service interruption.
• Recovery Point Objective (RPO) is the business target for acceptable data loss measured in time or transactions.

They are objectives, not measured guarantees. Heartbeat interval is only one component of detection time, and replication mode is only one component of data-loss behavior.

Asynchronous streaming replication can acknowledge a commit before a standby has it, so a primary/storage loss can lose acknowledged data. Synchronous replication waits for the configured standby state, adding network/durability latency and potentially reducing write availability. State exactly what the acknowledgement means: received, written, flushed, or applied.

Safe state machine

1. Combine several health signals and debounce transient failures.
2. Obtain a quorum/coordination decision.
3. Fence the old writer using STONITH, storage ownership, or a monotonically checked token.
4. Verify the standby's timeline, WAL position, and recovery state.
5. Promote and run readiness checks.
6. Switch routing.
7. Retry ambiguous client operations with idempotency keys.
8. Rebuild the old primary as a replica before any failback.

A DNS change alone is not fencing. A paused or partitioned old primary can continue writing even when the controller cannot see it.

Split brain and external effects

Only the current ownership epoch may mutate durable state. Every side-effecting dependency that matters must reject stale epochs, or the old leader can still corrupt data outside the database.

Failback

Failback is a planned migration. Require a stability window, resynchronization, new fencing decision, and explicit routing change. Immediate automatic failback can flap traffic and multiply recovery work.

Diagram scenarios

The animation covers async acknowledged-data loss, synchronous commit trade-offs, the complete failover state machine, stale-primary rejection, and deliberate failback.

Related concept

Interactive scenarios