Chaos engineering: hypotheses, real topology, abort gates, and blast radius

Chaos engineering is disciplined experimentation on a system to build confidence in its ability to withstand turbulent conditions. A useful experiment defines observable steady state, a falsifiable hypothesis, realistic variables, scope, approval, monitoring, stop conditions, recovery, and learning. Random destruction without these controls is not the method.

Production can expose behavior staging misses, but “run in production” is not a command to begin there. Teams start with analysis and lower-risk environments, validate observability and rollback, then graduate to the smallest justified production blast radius. Safety, customer impact, data integrity, legal constraints, and human readiness can veto an experiment.

Модель и предпосылки

• The topology has exactly three application instances behind a load balancer and a teaching three-voter Raft datastore with one elected leader and two followers. A commit requires a majority including the leader; read mode, lease and stale-read semantics remain outside this scenario unless separately specified.
• The experiment controller applies one declared fault. Independent steady-state metrics drive an alarm that stops further actions and invokes recovery; a stop condition is not a promise of zero impact.
• The hypothesis is evaluated from user-visible outcomes and invariants, not merely instance health.

Проверяемые утверждения

• C1. Chaos experiments begin with measurable steady state and a hypothesis tested with realistic events while minimizing blast radius.
• C2. AWS FIS stop conditions are alarms that stop a running experiment; they require alarm configuration and cannot guarantee prevention of all harm.
• C3. An experiment template declares actions, targets, stop conditions and permissions; least privilege and target scope are safety controls.
• C4. Teams can begin in non-production and graduate toward production after confidence grows, using a small blast radius.
• C5. The teaching datastore is explicitly a three-voter Raft model: an elected leader commits after replication to a majority, so isolating one follower leaves a two-voter majority while other partitions/elections require separate tests.

Исполняемые сценарии

Single-instance hypothesis. One of exactly three application instances is terminated while user success and latency remain within the declared steady-state bounds.

Replica partition with explicit protocol. The experiment isolates one follower and verifies replication lag and leader quorum behavior without inventing split-brain semantics.

Abort on steady-state breach. A user-impact alarm stops new fault actions and the owner verifies recovery; the stop path has measurable latency.

Graduate blast radius. A lower-risk rehearsal precedes a narrowly scoped production experiment, and inconclusive telemetry blocks expansion.

Ошибки проектирования

• Не запускайте fault без hypothesis, owner, stop condition, recovery plan и evidence window.
• Не называйте выключение stateless pod проверкой quorum или split-brain.
• Не полагайтесь на один control plane: потеря telemetry или IAM ошибка должны остановить дальнейшие действия.
• Не увеличивайте blast radius после неопределённого результата; сначала восстановите измеримость.

Границы гарантии

• A successful experiment raises confidence for the tested scope and time; it does not prove resilience to every failure.
• Abort automation has detection and actuation latency, so customer/data limits remain explicit.
• Destructive, irreversible, safety-critical, or regulated experiments may be unacceptable even with technical controls.

Связанные материалы

Официальные и первичные источники

1. Principles of Chaos Engineering
2. AWS Fault Injection Service: Stop conditions
3. AWS Fault Injection Service: Experiment templates
4. Google Cloud: Getting started with chaos engineering
5. Raft consensus paper