Availability numbers: define the event before counting nines

Availability is not one universal uptime percentage. Define the service level indicator (SLI), eligible population, “good” outcome and measurement window. For an online API, a useful SLI is often:

good eligible requests / all eligible requests

A good event can require a correct response within a latency threshold. Document exclusions narrowly; maintenance, client errors and dependency failures are not automatically excluded just because a contract could be written that way.

Nines and error budgets

For target S over one window, the error-budget fraction is 1 − S.

With 1,000,000 eligible requests and a 99.9% good-event SLO, the budget is 1,000 bad events. A time-based 99.9% target over a 30-day window corresponds to about 43.2 minutes only when “up time” is actually the chosen SLI. Request-based and time-based budgets answer different questions.

An SLA is a contractual promise with remedies and precise measurement language. It may differ from an internal SLO; there is no rule that it must be “one nine lower”. Choose the internal margin from measurement error, response time, business risk and change policy.

Series dependencies

If a request requires components with availabilities A1, A2, ... and their failures are independent under the same window and event definition, a simplified model is:

A_series = A1 × A2 × ...

This is an assumption-heavy model. Shared networks, credentials, deployments, regions, control planes and overload create correlated failure. Measure the end-to-end SLI and use fault testing; do not multiply vendor percentages and call the result a guarantee.

Parallel redundancy

For independent alternatives with perfect detection and failover:

A_parallel = 1 − (1 − A1)(1 − A2)...

This is an upper-bound model when the router, dependency, state, software defect or operator action is shared. Redundant copies in one failure domain do not remove that domain. Recovery time, false-positive health checks and in-flight request loss also consume the budget.

Static stability means the system can keep serving a target load after a failure without first depending on the failed control plane to add capacity.

Headroom arithmetic

For N equally loaded active instances, losing one redistributes the same load across N − 1. Remaining utilization becomes:

U_after = U_before × N / (N − 1)

If the desired post-failure ceiling is H, then:

U_before ≤ H × (N − 1) / N

For two instances and H = 80%, each should run at no more than 40% before the failure. For three, the bound is about 53.3%. Unequal traffic, warm-up, connection limits and stateful shards require a more detailed model and load test.

Scenarios

▶Walkthrough: request-sli (not found)

Good and eligible request semantics are explicit.

▶Walkthrough: series-risk (not found)

Critical-path multiplication is presented with its independence assumptions.

▶Walkthrough: parallel-failover (not found)

Redundancy plus the shared failure domains that limit it.

▶Walkthrough: capacity-headroom (not found)

Correct N/(N−1) redistribution and a two-instance example.

▶Walkthrough: error-budget (not found)

A request-based SLO becomes a countable error budget.

Operational review

• Track success and latency from the user boundary, segmented by region and operation.
• Define rolling-window burn rates and alert on fast and slow budget consumption.
• Test one-instance/zone/dependency loss at expected peak, including connection redistribution.
• Separate availability, durability, freshness and correctness SLIs.
• Record unknown outcomes and retries; duplicated side effects are not “available success”.
• Choose targets from user and business risk, not from a default table of nines.

Primary sources

• Google SRE, Embracing Risk: https://sre.google/sre-book/embracing-risk/
• Google SRE Workbook, Implementing SLOs: https://sre.google/workbook/implementing-slos/
• Google SRE availability table: https://sre.google/sre-book/availability-table/
• AWS Builders' Library, Static stability using Availability Zones: https://aws.amazon.com/builders-library/static-stability-using-availability-zones/