High Availability

A complete guide to high availability in production engineering — covering availability math, failure modes, redundancy patterns, load balancing, health checks, auto-scaling, database HA, multi-region architecture, and the readiness checklist every production system needs.

Introduction

High availability means the system is operational and serving requests when users need it.

Not sometimes. Not most of the time. Reliably, measurably, and provably close to all of the time.

A system that is down 8 hours a month is not production-ready, regardless of how fast it runs when it is up. Availability is a first-class system property that must be designed for deliberately — it does not emerge from writing good code alone.

High Availability (HA) is the discipline of eliminating or tolerating every category of failure that would cause the system to stop serving users. This means redundancy at every layer, fast failure detection, automatic recovery, and load distribution that prevents any single component from being a bottleneck or a single point of failure.

Availability Math

Availability is expressed as a percentage of time the system is operational over a measurement period.

Availability = (Uptime / (Uptime + Downtime)) × 100

The industry shorthand is "nines":

Availability	Downtime per Year	Downtime per Month	Downtime per Week
99%	3.65 days	7.2 hours	1.68 hours
99.9%	8.76 hours	43.8 minutes	10.1 minutes
99.95%	4.38 hours	21.9 minutes	5.04 minutes
99.99%	52.6 minutes	4.38 minutes	1.01 minutes
99.999%	5.26 minutes	26.3 seconds	6.05 seconds

Moving from 99.9% to 99.99% reduces allowable downtime from 43 minutes per month to 4 minutes per month. That is a 10× improvement, and it requires a fundamentally different architecture — not just better code.

Composite Availability

When multiple components sit in series (each must be up for the system to work), availability multiplies:

System Availability = Component_A × Component_B × Component_C

A chain of three components each at 99.9% availability gives:

0.999 × 0.999 × 0.999 = 0.997 = 99.7%

Three 99.9% components in series produce a system at only 99.7%. This is why eliminating single points of failure and adding redundancy is essential — the math punishes long chains of unredundant dependencies.

Categories of Failure

High availability design starts by enumerating every failure mode. You cannot design against failures you have not named.

mindmap
  root((Failure Modes))
    Hardware
      Server crash
      Disk failure
      Network card failure
      Power supply failure
    Software
      Application crash
      Memory leak
      Deadlock
      Unhandled exception
    Infrastructure
      Cloud availability zone outage
      Load balancer failure
      DNS failure
      Network partition
    Dependency
      Database unavailable
      Cache miss or failure
      Downstream API timeout
      Message queue full
    Operational
      Bad deployment
      Configuration error
      Capacity exhaustion
      Certificate expiry
    Human
      Accidental deletion
      Misconfigured firewall
      Failed migration
      Operator error during incident

Each failure mode requires a specific mitigation. No single technique covers all categories.

The Pillars of High Availability

High availability is built on five foundational practices, applied at every layer of the system.

flowchart TB
    HA["High Availability"]
    HA --> R["Redundancy\nEliminate single points\nof failure"]
    HA --> FD["Failure Detection\nKnow when something\nhas failed"]
    HA --> FR["Failover\nRoute around\nfailed components"]
    HA --> LB["Load Distribution\nSpread load, prevent\nbottlenecks"]
    HA --> Recovery["Self-Healing\nRecover automatically\nwithout human intervention"]

Redundancy Patterns

N+1 Redundancy

The minimum viable HA pattern. For every component you need N of to serve capacity, provision N+1. The extra instance absorbs traffic if any one instance fails.

flowchart LR
    LB["Load Balancer"]
    A["App Instance 1\n(Active)"]
    B["App Instance 2\n(Active)"]
    C["App Instance 3\n(Standby / N+1)"]

    LB --> A
    LB --> B
    LB -.->|"Takes over on failure"| C

N+1 is the floor. It handles single-instance failure but does not protect against zone-level outages or correlated failures.

2N Redundancy (Active-Passive)

Two complete copies of the system. One active, one idle. On failure of the active, the passive becomes active.

Property	Value
Redundancy	Full duplicate capacity
Cost	2× infrastructure cost
Failover time	Seconds to minutes (depends on automation)
Best for	Databases, stateful components

2N+1 Redundancy (Quorum)

Used when a decision must be made between distributed nodes (leader election, consensus). An odd number of nodes ensures a majority is always achievable.

flowchart TB
    subgraph Quorum["3 Nodes — Quorum = 2"]
        N1["Node 1\n(Leader)"]
        N2["Node 2\n(Follower)"]
        N3["Node 3\n(Follower)"]
    end

    N1 <--> N2
    N2 <--> N3
    N1 <--> N3

    Failure["Node 1 fails"] --> Election["Nodes 2 and 3\nelect new leader\n(Quorum achieved: 2/3)"]

With 3 nodes, one failure is tolerated. With 5 nodes, two failures are tolerated.

Geographic Redundancy

Redundancy distributed across multiple availability zones or regions. Protects against infrastructure-level failures that affect an entire datacenter.

flowchart TB
    DNS["Global Load Balancer"]

    subgraph AZ_A["Availability Zone A"]
        App_A1["App Instance 1"]
        App_A2["App Instance 2"]
    end

    subgraph AZ_B["Availability Zone B"]
        App_B1["App Instance 3"]
        App_B2["App Instance 4"]
    end

    subgraph AZ_C["Availability Zone C"]
        App_C1["App Instance 5"]
        App_C2["App Instance 6"]
    end

    DNS --> AZ_A
    DNS --> AZ_B
    DNS --> AZ_C

The rule: always deploy across at least 2 availability zones. Deploy across 3 for mission-critical systems.

Load Balancing

A load balancer distributes incoming requests across multiple backend instances. It is both a redundancy mechanism and a failure detection mechanism.

Load Balancing Algorithms

Algorithm	How It Works	Best For
Round Robin	Requests distributed in rotation	Homogeneous instances, stateless
Least Connections	Routes to instance with fewest active connections	Long-lived connections, WebSockets
IP Hash	Consistent routing based on client IP	Session affinity requirements
Weighted Round Robin	Round robin with capacity-based weights	Heterogeneous instance sizes
Random	Randomly chosen backend	Simple stateless services

Layer 4 vs Layer 7 Load Balancing

flowchart TB
    subgraph L4["Layer 4 (TCP/UDP)"]
        L4_Client["Client"]
        L4_LB["L4 Load Balancer\n(Routes by IP + port)"]
        L4_B1["Backend 1"]
        L4_B2["Backend 2"]
        L4_Client --> L4_LB --> L4_B1
        L4_LB --> L4_B2
    end

    subgraph L7["Layer 7 (HTTP/HTTPS)"]
        L7_Client["Client"]
        L7_LB["L7 Load Balancer\n(Routes by URL, headers,\ncookies, content type)"]
        L7_API["API Service"]
        L7_Static["Static Assets"]
        L7_WS["WebSocket Service"]
        L7_Client --> L7_LB
        L7_LB -->|"/api/*"| L7_API
        L7_LB -->|"/static/*"| L7_Static
        L7_LB -->|"/ws/*"| L7_WS
    end

Property	Layer 4	Layer 7
Routing basis	IP, port, TCP/UDP	URL, headers, cookies, body
SSL termination	Pass-through	Terminates SSL, inspects content
Content routing	Not possible	Route by path, host, header
Performance	Faster — less processing	Slightly slower — more processing
Health checks	TCP connection	HTTP endpoint check

Most production systems use a Layer 7 load balancer at the edge and Layer 4 internally.

Health Checks and Instance Removal

A load balancer continuously probes each backend. When a backend fails its health check, the load balancer removes it from rotation automatically.

sequenceDiagram
    participant LB as Load Balancer
    participant Healthy as Backend 1 (Healthy)
    participant Failed as Backend 2 (Failed)

    loop Every 10 seconds
        LB->>Healthy: GET /health
        Healthy-->>LB: 200 OK
        LB->>Failed: GET /health
        Failed-->>LB: Connection timeout
    end

    Note over LB: Backend 2 fails 3 consecutive checks
    LB->>LB: Remove Backend 2 from pool
    Note over LB: All traffic now routes to Backend 1
    LB-->>Failed: Still probing (for recovery detection)

    Failed-->>LB: 200 OK (recovered)
    LB->>LB: Re-add Backend 2 to pool

Key parameters:

Check interval — how frequently the load balancer probes (commonly 10–30 seconds)
Healthy threshold — consecutive successes before marking healthy (typically 2–3)
Unhealthy threshold — consecutive failures before marking unhealthy (typically 2–3)
Timeout — how long to wait for a response (commonly 5 seconds)

Health Checks

Health checks are the mechanism by which infrastructure determines whether a component is ready to serve traffic. A component can be running but not healthy — a misconfigured database connection, an exhausted thread pool, or a broken dependency can all make an instance unable to serve requests correctly.

Liveness vs Readiness

Check Type	Question It Answers	Action on Failure
Liveness	Is this instance alive at all?	Restart the instance
Readiness	Is this instance ready to accept traffic?	Remove from load balancer pool
Startup	Has the instance finished initializing?	Don't route traffic until startup passes

A liveness check that is too aggressive will restart healthy instances. A readiness check that is too permissive routes traffic to broken instances. Both endpoints must be implemented correctly.

Deep Health Check

A shallow health check (200 OK on any request) is nearly useless. A deep health check validates that the instance's dependencies are functional:

# Example deep health check response
GET /health

{
  "status": "healthy",
  "checks": {
    "database": {
      "status": "healthy",
      "latency_ms": 3
    },
    "redis": {
      "status": "healthy",
      "latency_ms": 1
    },
    "downstream_api": {
      "status": "degraded",
      "latency_ms": 450,
      "message": "Elevated latency"
    }
  },
  "uptime_seconds": 86400
}

Return 200 when all critical dependencies are healthy. Return 503 when a critical dependency is down. Return 200 with a degraded status for non-critical issues — so traffic is not removed but operators are alerted.

Auto-Scaling

Static capacity sizing is an HA anti-pattern. A fixed fleet that is sized for average load will be overwhelmed by peak load and over-provisioned during off-peak hours. Auto-scaling adjusts capacity dynamically based on demand.

Horizontal vs Vertical Scaling

flowchart LR
    subgraph Horizontal["Horizontal Scaling (Scale Out/In)"]
        H1["Add more instances"]
        H2["Each instance handles\na fraction of total load"]
        H3["Preferred for HA —\nno single point of failure"]
    end

    subgraph Vertical["Vertical Scaling (Scale Up/Down)"]
        V1["Larger instances\n(more CPU, RAM)"]
        V2["One instance handles\nmore load"]
        V3["Single point of failure\nremains"]
    end

Horizontal scaling is preferred for high availability because it distributes failure surface. Vertical scaling has a ceiling and still leaves a single point of failure.

Auto-Scaling Triggers

Trigger	Metric	Example Threshold
CPU utilization	Average CPU % across fleet	Scale out if CPU > 70% for 3 min
Memory utilization	Average memory % across fleet	Scale out if memory > 80%
Request rate	Requests per second	Scale out if RPS > 1000
Queue depth	Messages waiting in queue	Scale out if queue > 500
Response latency	p99 latency exceeds threshold	Scale out if p99 > 500ms
Custom metric	Application-defined business metric	Scale on orders/second

Scale-Out vs Scale-In Asymmetry

Scale out aggressively. Scale in conservatively.

A system that scales out too slowly leaves users experiencing degraded performance. A system that scales in too aggressively causes thrashing (constant add/remove cycles) and can leave the fleet under-provisioned if load rebounds.

flowchart LR
    subgraph ScaleOut["Scale-Out Policy"]
        SO1["Threshold breached\n(CPU > 70%)"]
        SO2["Wait 1 minute\n(short cooldown)"]
        SO3["Add 2 instances"]
    end

    subgraph ScaleIn["Scale-In Policy"]
        SI1["Threshold sustained low\n(CPU < 30%)"]
        SI2["Wait 15 minutes\n(long cooldown)"]
        SI3["Remove 1 instance"]
    end

Common practice: scale out after 1–3 minutes of elevated load, scale in after 10–15 minutes of sustained low load.

Predictive Scaling

For predictable traffic patterns (daily peaks, scheduled batch jobs, planned events), pre-scale the fleet before load arrives rather than waiting for auto-scaling to react.

Database High Availability

Databases are the most common single point of failure in production systems. A stateless application tier can be replaced in seconds. A database failure that causes data loss or corruption is catastrophic.

Primary-Replica Architecture

The foundation of database HA. Writes go to the primary. Reads can be served by replicas. If the primary fails, a replica is promoted.

flowchart TB
    App["Application"]
    Primary["Primary DB\n(Accepts reads + writes)"]
    Replica1["Read Replica 1\n(Read-only)"]
    Replica2["Read Replica 2\n(Read-only)"]
    Replica3["Standby Replica\n(Failover candidate)"]

    App -->|"Writes"| Primary
    App -->|"Reads"| Replica1
    App -->|"Reads"| Replica2

    Primary -->|"Async replication"| Replica1
    Primary -->|"Async replication"| Replica2
    Primary -->|"Sync replication"| Replica3

    Failure["Primary fails"] --> Promote["Promote Standby Replica\nto Primary"]

Replication Type	Consistency	Latency impact	Data loss risk
Synchronous	Strong — no lag	Higher write latency	None (zero RPO)
Asynchronous	Eventual — some lag	No write latency overhead	Seconds of data loss on failure

Connection Pooling

Direct database connections are expensive. Connection pools maintain a set of pre-established connections that the application reuses. This prevents connection exhaustion under load and reduces connection setup latency.

flowchart LR
    App1["App Instance 1"]
    App2["App Instance 2"]
    App3["App Instance 3"]

    Pool["Connection Pool\n(PgBouncer / ProxySQL)\nMax 20 DB connections"]

    DB["Database\n(Max 100 connections)"]

    App1 -->|"Up to 100 connections"| Pool
    App2 -->|"Up to 100 connections"| Pool
    App3 -->|"Up to 100 connections"| Pool
    Pool -->|"20 pooled connections"| DB

Without a connection pool, 100 application instances each opening 10 connections would saturate a database at its connection limit.

Multi-AZ Database Deployment

Cloud providers offer managed multi-AZ database deployments (AWS RDS Multi-AZ, Google Cloud SQL HA, Azure SQL Zone-Redundant). The database is automatically replicated to a standby in a different availability zone. Failover to the standby is automatic and typically completes in 60–120 seconds.

flowchart TB
    subgraph AZ_A["Availability Zone A"]
        Primary_DB["Primary DB\n(Active — accepts all traffic)"]
    end

    subgraph AZ_B["Availability Zone B"]
        Standby_DB["Standby DB\n(Passive — synchronous replica)"]
    end

    Primary_DB -->|"Synchronous replication"| Standby_DB

    Failure["AZ-A failure"] --> Failover["Automatic failover\nStandby promoted to primary\nDNS endpoint updated\n~60–120 seconds"]
    Failover --> Standby_DB

Circuit Breaker Pattern

A circuit breaker prevents cascading failures by stopping requests to a failing dependency before those failures accumulate and overwhelm the system.

Without a circuit breaker, requests to a slow or failing dependency pile up. Threads are consumed waiting for responses. The application's own thread pool is exhausted. The application itself becomes unavailable — not because of its own failure, but because of a dependency's.

Circuit Breaker State Machine

stateDiagram-v2
    [*] --> Closed

    Closed --> Open: Failure rate exceeds threshold\n(e.g., 50% of last 10 requests failed)
    Open --> HalfOpen: Timeout expires\n(e.g., 30 seconds)
    HalfOpen --> Closed: Test request succeeds
    HalfOpen --> Open: Test request fails

State	Behavior
Closed	Requests pass through normally. Failures are counted.
Open	Requests fail immediately without contacting the dependency.
Half-Open	One test request is allowed through to check if the dependency has recovered.

The circuit breaker fails fast when the dependency is down — returning an error in microseconds instead of blocking for 30 seconds waiting for a timeout.

Graceful Degradation

When a dependency fails, the system should degrade gracefully rather than fail completely.

Degradation Strategy	Description	Example
Fallback response	Return a cached, default, or reduced response	Return cached product list when DB is slow
Feature flag off	Disable the feature that depends on the failing dependency	Hide recommendation engine when ML service is down
Queue and retry	Accept the request, process it later when the dependency recovers	Queue writes when downstream API is unavailable
Read-only mode	Accept reads, reject writes until dependency recovers	Display existing data, disable mutations
Static fallback	Serve pre-rendered static content	Show cached homepage instead of dynamic content

The goal is to keep the system partially useful rather than completely broken.

Multi-Region High Availability

Single-region deployments are subject to region-level outages. For systems requiring 99.99% availability or higher, multi-region deployment is necessary.

Active-Passive Multi-Region

One region serves all production traffic. The second region is a hot standby, ready to receive traffic on failover.

flowchart TB
    DNS["Global Load Balancer / DNS"]

    subgraph Primary["Primary Region — us-east-1 (100% traffic)"]
        P_LB["Load Balancer"]
        P_App["Application Fleet"]
        P_DB["Primary Database"]
    end

    subgraph DR["DR Region — us-west-2 (0% traffic)"]
        D_LB["Load Balancer"]
        D_App["Application Fleet (scaled down)"]
        D_DB["Replica Database"]
    end

    DNS -->|"100%"| P_LB
    DNS -.->|"0% — ready"| D_LB

    P_DB -->|"Continuous replication"| D_DB

    Failure["us-east-1 outage"] --> Cutover["DNS failover\n~1–5 minutes"]
    Cutover --> D_LB

Active-Active Multi-Region

All regions simultaneously serve live traffic. No manual failover required. If a region fails, the global load balancer routes its traffic to the remaining healthy regions.

flowchart TB
    DNS["Global Load Balancer\n(Anycast / Route 53 / Cloudflare)"]

    subgraph US["us-east-1 (35%)"]
        US_App["Application"]
        US_DB["Database (Primary)"]
    end

    subgraph EU["eu-west-1 (35%)"]
        EU_App["Application"]
        EU_DB["Database (Primary)"]
    end

    subgraph APAC["ap-southeast-1 (30%)"]
        APAC_App["Application"]
        APAC_DB["Database (Primary)"]
    end

    DNS --> US
    DNS --> EU
    DNS --> APAC

    US_DB <-->|"Bidirectional replication"| EU_DB
    EU_DB <-->|"Bidirectional replication"| APAC_DB
    US_DB <-->|"Bidirectional replication"| APAC_DB

Active-active eliminates the need for failover but introduces complexity:

Write conflicts — two regions writing the same record simultaneously require conflict resolution (last-write-wins, CRDTs, or application-level resolution)
Replication lag — users may temporarily see stale data immediately after a cross-region write
Cost — full production capacity in every region

Active-active is the right choice when downtime cost exceeds the infrastructure cost, or when global latency requirements demand regional proximity.

Zero-Downtime Deployments

A deployment that takes down the application is a planned availability event. Every deployment strategy must be evaluated against its availability impact.

Strategy	Downtime	Rollback Speed	Traffic during deploy
Recreate	Yes — full	Deploy old version	Zero — service interrupted
Rolling	No	Slow — roll back all instances	Mixed versions in flight
Blue-Green	No	Instant — flip traffic back	100% on one color
Canary	No	Instant — set canary weight to 0	Split by percentage

For a highly available system, any deployment strategy that requires downtime is unacceptable.

Observability for High Availability

A system that is unavailable but doesn't know it is available in name only. Observability is the prerequisite for high availability.

The Four Golden Signals for Availability

Signal	What to Measure	Availability Implication
Latency	p50, p95, p99 response time	High latency is often the first sign of failure
Traffic	Requests per second	Unexpected drops in traffic may indicate an unreachable service
Errors	Error rate (5xx / total requests)	Rising error rate precedes full unavailability
Saturation	CPU, memory, queue depth, connection pool usage	Saturation predicts imminent unavailability

Alerting Thresholds

Metric	Warning	Critical
Error rate	> 1%	> 5%
p99 latency	> 500ms	> 2000ms
CPU utilization	> 70% sustained	> 90% sustained
Memory utilization	> 80%	> 95%
DB connection pool usage	> 75%	> 90%
Health check failures	Any failure	2+ consecutive failures

SLA, SLO, and SLI

Availability commitments require precise definitions.

Term	Definition	Example
SLA	Service Level Agreement — contractual commitment with consequences	99.9% monthly uptime or credits issued
SLO	Service Level Objective — internal target, more ambitious than SLA	99.95% monthly uptime target (internal)
SLI	Service Level Indicator — the actual measured metric	99.97% uptime measured last 30 days

The relationship:

SLI (measured) ≥ SLO (internal target) ≥ SLA (customer commitment)

Set internal SLOs above the SLA commitment to create a buffer. If the SLO is breached, you have time to fix the problem before the SLA is breached. If the SLA is breached first, it is already a customer-impacting failure.

Error Budget

An error budget is the allowable downtime derived from an SLO. It makes availability concrete and tradeable.

Error Budget = 100% - SLO
For SLO of 99.9%: Error Budget = 0.1% = 43.8 minutes/month

When the error budget is consumed, the engineering team stops new feature deployments and focuses exclusively on reliability until the budget is replenished at the start of the next period.