"It should never go down" is not a specification. Real systems fail; the useful questions are how often, for how long, and how gracefully. This lesson gives you the vocabulary and the math to reason about staying up.
Availability and the "nines"
Availability is the fraction of time a system is usable, usually quoted in "nines." Because a year is about 8,760 hours, the allowed downtime shrinks quickly:
- 90% (one nine): ~36.5 days/year of downtime
- 99% (two nines): ~3.65 days/year
- 99.9% (three nines): ~8.76 hours/year
- 99.99% (four nines): ~52 minutes/year
- 99.999% (five nines): ~5 minutes/year
Each extra nine is ~10x less downtime — and usually far more than 10x the cost. Match the target to the business, not to ego.
Availability vs reliability vs durability
These three get conflated constantly:
- Availability — is the system up and responding right now? (uptime %)
- Reliability — does it do the right thing consistently over time, without errors? A system can be available but unreliable: up, yet returning wrong answers.
- Durability — once data is stored, does it survive? ("11 nines of durability" means practically no stored object is ever lost.) Durability is about not losing data, even during an outage.
A cache can be low-durability (fine to lose) yet high-availability; a ledger must be high-durability above all else.
Redundancy: no single point of failure
Availability comes from redundancy — having a spare ready when something dies.
- Active-active — multiple nodes serve traffic at the same time. If one dies, the others absorb its load with no failover delay. Higher utilization, but you must handle shared or replicated state.
- Active-passive — one node serves while a standby waits and takes over on failure. Simpler, but there is a failover gap and the standby sits idle.
The goal is to eliminate single points of failure (SPOFs): the one load balancer, the one primary database, the one region.
Failure domains & blast radius
A failure domain is the scope a single failure can take out — a machine, a rack, an availability zone, or a whole region. Blast radius is how much breaks when one thing breaks.
Good design contains the blast radius: spread replicas across zones so a zone outage degrades but does not kill the service; partition users into cells so a bad deploy hits 1/N of them rather than everyone.
Health checks & graceful degradation
- Health checks — the load balancer periodically pings each backend (for example
/healthz); an unhealthy instance is pulled out of rotation so traffic only reaches good nodes. - Graceful degradation — when a dependency fails, serve a reduced experience instead of an error. If the recommendation service is down, show a generic list rather than a blank page. Degrade, do not collapse.
MTBF and MTTR
Two numbers describe failure behavior:
- MTBF (Mean Time Between Failures) — how long, on average, between outages. Higher is better.
- MTTR (Mean Time To Recovery) — how long, on average, to recover once you fail. Lower is better.
Availability rises roughly as MTBF / (MTBF + MTTR). You improve uptime both by failing less and by recovering faster — and fast recovery (good monitoring, automated failover, quick rollbacks) is often the cheaper lever.
SLA vs SLO vs SLI
- SLI (Indicator) — the measurement: for example, the percentage of requests served under 300 ms, or the success rate.
- SLO (Objective) — the internal target for an SLI: for example, "99.9% of requests succeed each month."
- SLA (Agreement) — the contractual promise to customers, with penalties if missed. It is usually looser than the SLO so you keep a safety margin.
You measure the SLI, aim at the tighter SLO, and promise the looser SLA.
In the interview
- Translate a nines target into downtime per year to show it is a real constraint.
- Kill SPOFs explicitly and place redundancy across failure domains (multi-AZ).
- Mention health checks and graceful degradation — they signal operational maturity.