System Design Cases
On-call rotation & burnout prevention
On-call rotation & burnout prevention concept page. Architecture: Production tier (app + db) emits metrics to Prometheus, which fires alerts via Alertmanager into PagerDuty. PagerDuty pages region-specific primary on-call (US/EU/APAC for follow-the-sun) with escalation chain to secondary then manager. Primary uses Slack #incident channel, Runbook wiki, and Statuspage for comms. Includes ADR on PagerDuty node about on-call pay model (base stipend + per-page premium vs flat salary uplift). Five scenarios: weekly rotation handoff Monday morning, alert fatigue tuning to reduce noise 80%, follow-the-sun handoff between US/EU/APAC timezones, escalation chain when primary doesn't ack, and burnout prevention loop with pages-per-person metric tracking and toil reduction.
Sustainable on-call: actionable pages, staffing assumptions, handoffs, and fatigue controls
On-call is scheduled operational work that requires staffing, compensation, training, escalation, rest, and continuous reduction of avoidable pages. A page should require urgent human action because a service objective or imminent failure is at risk. Non-urgent work belongs in tickets or business-hours queues.
Google’s single-site example derives eight engineers from two simultaneous week-long primary/secondary rotations and a 25% on-call-time design goal. It is a documented organizational model, not a universal legal minimum. Google separately aims for at least 50% engineering project work and no more than 50% total operational work; that is not a claim that all organizations must have exactly 25% toil.
Модель и предпосылки
- A primary acknowledges and triages; a secondary is a defined fallback/partner; escalation reaches an empowered incident structure. Every role has a coverage and no-ack timer appropriate to the service.
- Follow-the-sun reduces night work only when UTC schedules, daylight-saving changes, local holidays, overlap, language, permissions, and acknowledged handoffs are designed. 17 Pacific is 01 or 02 in Central Europe depending on seasonal offsets, not “zero night work.”
- Fatigue can slow reaction, reduce attention and impair judgment. Teams measure page/incident load, after-hours disruption, recovery, staffing, toil and psychological safety, then remove noisy alerts and provide rest/rotation relief.
Проверяемые утверждения
- C1. Google’s eight-person single-site minimum follows its assumptions of primary plus secondary coverage, week-long shifts and at most 25% on-call time; it is not a universal team-size law.
- C2. Google targets at least 50% engineering work and caps total operational work at 50%; average or example toil percentages are not universal guarantees.
- C3. Paging alerts should be actionable and aligned with service-impact symptoms; frequent low-priority alerts create fatigue and mask serious signals.
- C4. Fatigue is associated with nonstandard/extended schedules and can reduce attention, memory, reaction and judgment.
- C5. Multi-site handoff reduces night work only with explicit schedules and acknowledged transfer; timezone arithmetic changes with daylight saving.
Исполняемые сценарии
Actionable page and fallback. A service-impacting condition pages the primary, then uses a defined secondary and escalation path if acknowledgement or mitigation fails.
Runbook-guided mitigation. The responder uses safe access and a reversible action, then verifies the actual service outcome.
Timezone-aware acknowledged handoff. A UTC schedule and seasonal local-time calculation precede a two-way state transfer; follow-the-sun is not assumed to erase night work.
Fatigue and toil control. Rotation governance measures operational load, removes noisy pages, funds engineering reduction, and provides relief after disruption.
Ошибки проектирования
- Не считайте headcount универсальным без primary/secondary, shift length, incident load, leave и redundancy assumptions.
- Не page-ите неactionable warning; создайте ticket и устраните alert noise.
- Не передавайте смену сообщением без acknowledgement, current state, risks, permissions и next checkpoint.
- Не лечите burnout только resilience training; уменьшайте system load, ночные вызовы и staffing gaps.
Границы гарантии
- Employment law, compensation, accommodations, health and safety requirements vary by jurisdiction and organization.
- Follow-the-sun adds coordination overhead and cannot eliminate every local night, holiday or incident overrun.
- Aggregate page counts can hide unequal burden; review per-person distribution without exposing private health data.