Postmortems: evidence, blameless analysis, and verified action items

A postmortem is a durable learning record after an incident or near miss. It reconstructs impact and timeline from evidence, explains contributing technical and organizational conditions without hindsight blame, and creates concrete follow-up work. A template can help, but Google does not prescribe one universal number of required sections for every organization.

Blameless does not mean consequence-free or evidence-free. It means the analysis avoids treating a person’s action as a sufficient cause and asks why the action made sense in context and which controls were absent or ineffective. Accountability appears through clear owners, priorities, tracked work, measurable endpoints, and review of whether risk actually fell.

Модель и предпосылки

• Facts, hypotheses and decisions are marked separately. Times include source and timezone; later corrections remain visible.
• A database example is precise: adding a NOT NULL column without a suitable default to a table that already contains rows fails while checking existing rows. A safe staged migration can add nullable, backfill in batches, validate, then enforce the constraint.
• Cloudflare’s July 2, 2019 outage was triggered by a WAF managed-rule regular expression that exhausted CPUs globally; it was not a BGP root cause.

Проверяемые утверждения

• C1. A useful postmortem records impact, timeline, root/contributing causes and follow-up work, while templates and review practice are adapted to the organization.
• C2. Blameless analysis examines system conditions and supports open reporting; action items still need owners, priority, tracking and measurable completion.
• C3. The Cloudflare July 2, 2019 outage came from a WAF regex that caused CPU exhaustion, not a BGP configuration event.
• C4. A NOT NULL constraint requires existing rows to satisfy it; staged backfill/validation avoids pretending the invalid schema would install and fail only later at runtime.
• C5. Closing an action ticket is not evidence of effectiveness; the postmortem process should verify the intended control and monitor recurrence/risk.

Исполняемые сценарии

Evidence to reviewed record. The author separates observed facts, hypotheses and decisions, then peers review the system conditions without personal blame.

Correct NOT NULL migration analysis. Existing rows make an immediate NOT NULL check relevant; the corrective action stages backfill and validation before enforcement.

Accountable action lifecycle. A material finding becomes owned work and is closed only after the expected control is measured.

Historical claim discipline. Named incidents are attributed to their primary RCA and are not rewritten to fit a favorite failure category.

Ошибки проектирования

• Не используйте “human error” как terminal root cause и не скрывайте decision context.
• Не подменяйте evidence красивой линейной историей: инциденты обычно имеют несколько contributing factors.
• Не создавайте action item без owner, priority, due/review point и measurable endpoint.
• Не заявляйте, что NOT NULL с существующими NULL успешно установлен и сломал только последующий request.

Границы гарантии

• Postmortem can contain sensitive security, personal or legal data; sharing follows access and redaction policy.
• Causal language reflects evidence confidence; counterfactuals and “five whys” are prompts, not proofs.
• Some actions reduce likelihood, others reduce impact or detection time; success criteria should name which risk dimension changes.

Связанные материалы

Официальные и первичные источники

1. Google SRE Workbook: Postmortem Culture
2. Google SRE Book: Postmortem Culture
3. Cloudflare: Details of the outage on July 2, 2019
4. PostgreSQL: Constraints
5. PostgreSQL: ALTER TABLE