Incident management: command, operations, communication, planning, and handoff

Incident management creates a temporary command structure for uncertain, time-sensitive work. Google SRE separates command, operations, communications, and planning. The incident commander coordinates objectives and priorities; Operations changes the system; Communications maintains stakeholder updates; Planning tracks future work and handoffs. Smaller incidents may combine roles deliberately, but responsibilities must remain explicit.

Mitigation restores acceptable service and can precede root-cause analysis, yet it is not “act without evidence.” Changes remain scoped, reviewed when feasible, logged, reversible, and verified against user impact. A live incident document records times, decisions, owners and evidence so that status and later learning do not depend on memory.

Модель и предпосылки

• Example timeline is internally consistent: detect 12
  UTC, declare 12
  , choose rollback 12
  , complete mitigation 12
  , verify through 12
  , resolve 12
  .
• External updates follow a declared cadence and contain impact, scope, current action and next-update time without asserting an unverified root cause.
• A handoff is acknowledged by the receiving commander/operations lead and transfers current status, outstanding risks, permissions and next checkpoints; “follow the sun” does not guarantee nobody works at night.

Проверяемые утверждения

• C1. Incident response benefits from explicit command, operations, communications and planning responsibilities plus a shared incident state.
• C2. The incident commander coordinates rather than becoming the default system operator; role combinations are explicit when staffing is small.
• C3. Communications provide accurate, regular updates and avoid speculation about root cause while investigation is incomplete.
• C4. Handoffs require an explicit transfer and acknowledgement rather than an assumed timezone boundary; multi-site operation has coordination trade-offs even when it reduces night shifts.
• C5. Incident preparation, detection, response and recovery are part of a broader risk-management lifecycle.

Исполняемые сценарии

▶Walkthrough: declare-and-assign-roles (not found)

Detect, declare, and assign roles. At 12

monitoring detects impact; at 12 the commander declares the incident and assigns explicit responsibilities.

▶Walkthrough: mitigate-and-verify (not found)

Mitigate and verify. Operations chooses rollback at 12

, completes it at 12, and resolves only after user-level verification at 12.

▶Walkthrough: customer-communication (not found)

Evidence-based customer communication. Communications reports confirmed impact, mitigation and the next update without speculating about root cause.

▶Walkthrough: acknowledged-handoff (not found)

Acknowledged handoff. The outgoing planning lead transfers current state and receives explicit acknowledgement from the next shift.

Ошибки проектирования

• Не назначайте IC одновременно главным оператором без явного решения о совмещении ролей.
• Не публикуйте root cause во время mitigation как установленный факт.
• Не объявляйте resolved до user-level verification и фиксации residual risk.
• Не обещайте фиксированный MTTR multiplier или универсальные SEV deadlines без собственной статистики/договора.

Границы гарантии

• Severity definitions, update cadence and escalation timers are organization-specific.
• Security, privacy, safety and regulatory incidents can require specialized command and notification paths.
• Timeline timestamps are observations with sources; clock correction and late evidence must remain auditable.

Связанные материалы

[CONCEPT]observability-pillars [CONCEPT]oncall-rotation-burnout [CONCEPT]post-mortems

Официальные и первичные источники

1. Google SRE Book: Managing Incidents
2. Google SRE Workbook: Incident Response
3. Google SRE Incident Management Guide
4. NIST SP 800-61 Rev. 3: Incident Response Recommendations
5. Google SRE Book: Being On-Call