Full Stack • Java • System Design • Cloud • AI Engineering

Disaster Recovery for Production Systems

A complete guide to disaster recovery in production engineering — covering RTO and RPO, backup strategies, DR architecture patterns, multi-region failover, database recovery, runbooks, and the readiness checklist every production system needs.

Introduction

Every production system will experience a disaster at some point.

Not a hypothetical disaster — a real one. A cloud region becomes unavailable. A database is accidentally deleted. A ransomware attack encrypts production data. A bad migration corrupts millions of rows. A critical dependency shuts down with no warning.

The question is not whether a disaster will happen. The question is: how fast can you recover, and how much data can you afford to lose?

Disaster Recovery (DR) is the set of policies, procedures, and infrastructure that enables a system to continue operating — or rapidly resume operating — after a catastrophic failure.

A system without a DR plan is not production-ready. A DR plan that has never been tested is not a DR plan.

Two Fundamental Metrics

All disaster recovery design starts with two numbers. Every architectural decision flows from them.

Recovery Time Objective (RTO)

RTO is the maximum acceptable time the system can be unavailable after a disaster.

"We must be back online within 4 hours."

RTO drives decisions about:

  • Whether you need a hot standby (minutes) or can restore from backup (hours)
  • How much automation is required in the recovery procedure
  • Whether manual failover is acceptable or failover must be automatic

Recovery Point Objective (RPO)

RPO is the maximum acceptable amount of data loss measured in time.

"We can tolerate losing at most 15 minutes of data."

RPO drives decisions about:

  • How frequently data must be backed up or replicated
  • Whether synchronous or asynchronous replication is required
  • The cost of storage and replication infrastructure

RTO and RPO Relationship

flowchart LR
    Disaster["Disaster Occurs\n(T=0)"]
    RPO["RPO Window\n(Max data loss)"]
    RTO["RTO Window\n(Max downtime)"]
    Recovery["System Recovered\n(T=RTO)"]
    LastBackup["Last Good Backup\n(T = -RPO)"]

    LastBackup --> Disaster --> Recovery
    LastBackup -. "RPO: data in this window may be lost" .-> Disaster
    Disaster -. "RTO: system must recover within this window" .-> Recovery
RTO / RPO Meaning Typical Architecture
RTO = 0 Zero downtime — system never stops serving Active-Active multi-region
RTO < 5m Near-zero — automatic failover within minutes Hot standby with auto-failover
RTO < 1h Warm recovery Warm standby, pre-provisioned
RTO < 4h Cold recovery — restore from backup Backup and restore
RPO = 0 Zero data loss — synchronous replication Synchronous multi-region writes
RPO < 15m Near-zero — frequent snapshots or async replication Continuous replication
RPO < 24h Daily backups acceptable Nightly backup to cold storage

Types of Disasters

Disaster recovery must address multiple failure categories. Each requires different mitigations.

mindmap
  root((Disaster Types))
    Infrastructure
      Cloud region outage
      Availability zone failure
      Network partition
      Hardware failure
    Data
      Accidental deletion
      Data corruption
      Failed migration
      Ransomware
    Application
      Bad deployment
      Logic bug causing data loss
      Certificate expiry
      Configuration error
    External
      Third-party dependency outage
      DNS provider failure
      CDN outage
      DDoS attack

DR Architecture Patterns

There are four standard DR architecture patterns. They differ in cost, recovery time, and complexity.

flowchart TB
    Patterns["DR Architecture Patterns"]
    Patterns --> BaR["Backup and Restore\nRPO: hours\nRTO: hours\nCost: low"]
    Patterns --> Pilot["Pilot Light\nRPO: minutes\nRTO: 30–60 min\nCost: low-medium"]
    Patterns --> Warm["Warm Standby\nRPO: seconds-minutes\nRTO: 5–30 min\nCost: medium"]
    Patterns --> Active["Active-Active\nRPO: 0\nRTO: 0\nCost: high"]

Pattern 1: Backup and Restore

The simplest DR strategy. Regular backups are stored in a separate region. On disaster, the system is rebuilt from backup.

flowchart LR
    subgraph Primary["Primary Region — us-east-1"]
        App["Application"]
        DB["Database"]
    end

    subgraph DR["DR Region — us-west-2"]
        S3["S3 / Cold Storage\n(Backups)"]
        Restore["Restore Process\n(Manual or scripted)"]
    end

    DB -->|"Nightly backup"| S3
    S3 --> Restore

    Disaster["Disaster in us-east-1"] --> Restore
    Restore --> NewDB["Rebuilt Database"]
    Restore --> NewApp["Rebuilt Application"]
Property Value
RTO 1–8 hours (depends on data size)
RPO Time since last backup (hours)
Cost Lowest — pay only for storage
Automation Partially automated with scripts
Best for Non-critical systems, batch workloads

Key requirements:

  • Backups must be tested regularly — an untested backup is worthless
  • Backup must be stored in a geographically separate location
  • Recovery scripts must be documented, versioned, and runnable without production access

Pattern 2: Pilot Light

A minimal version of the production environment runs in the DR region at all times. Core data is continuously replicated. On disaster, the DR infrastructure is scaled up to full capacity.

flowchart TB
    subgraph Primary["Primary Region — LIVE"]
        PApp["Full Application\n(All pods, full capacity)"]
        PDB["Primary Database\n(Writes accepted)"]
    end

    subgraph DR["DR Region — PILOT LIGHT"]
        DApp["Minimal App\n(0 pods — scaled to 0)"]
        DDB["Replica Database\n(Read-only, synced)"]
    end

    PDB -->|"Continuous async replication"| DDB

    Disaster --> ScaleUp["Scale DR app to full capacity\nRedirect DNS to DR region"]
    ScaleUp --> DApp
    DApp --> DDB
Property Value
RTO 30–60 minutes (scale-up time + DNS propagation)
RPO Seconds to minutes (async replication lag)
Cost Low — pay only for replicated data and minimal infra
Best for Systems that can tolerate 30–60 minute recovery

Pilot light requires:

  • All infrastructure defined as code (Terraform / CloudFormation) — scale-up is a script execution
  • Database replication running continuously
  • DNS or load balancer cutover pre-configured and tested
  • Regular DR drills to verify scale-up time is within RTO

Pattern 3: Warm Standby

A scaled-down but fully functional version of production runs in the DR region continuously. On disaster, it is scaled to full production capacity and receives live traffic.

flowchart TB
    subgraph Primary["Primary Region — LIVE (100%)"]
        P_LB["Load Balancer"]
        P_App["Application\n(10 pods, full capacity)"]
        P_DB["Primary Database"]
    end

    subgraph DR["DR Region — WARM (25% capacity)"]
        D_LB["Load Balancer"]
        D_App["Application\n(2–3 pods, reduced capacity)"]
        D_DB["Replica Database\n(Promoted to primary on failover)"]
    end

    DNS["Global DNS\n/ Traffic Manager"]

    DNS -->|"100% traffic"| P_LB
    DNS -.->|"0% traffic (ready)"| D_LB

    P_DB -->|"Continuous replication"| D_DB

    Disaster --> Promote["Promote DR DB to primary\nScale DR app to full capacity"]
    Promote --> D_App
    DNS -->|"100% traffic → DR"| D_LB
Property Value
RTO 5–30 minutes
RPO Seconds (continuous replication)
Cost Medium — 25–50% of full production running cost
Best for Business-critical systems with moderate recovery budget

Failover steps for warm standby:

  1. Detect failure in primary region (automated monitoring)
  2. Promote replica database to primary (automated or one command)
  3. Scale DR application to full production capacity
  4. Update DNS or load balancer to route traffic to DR region
  5. Verify health checks pass
  6. Communicate status to stakeholders

Pattern 4: Active-Active

Multiple regions simultaneously serve live production traffic. No failover is required — if one region fails, the others absorb the load automatically.

flowchart TB
    DNS["Global Load Balancer\n(Route 53 / Cloudflare)"]

    subgraph RegionA["Region A — us-east-1\n(50% traffic)"]
        A_LB["Load Balancer"]
        A_App["Application\n(Full capacity)"]
        A_DB["Database\n(Primary — accepts writes)"]
    end

    subgraph RegionB["Region B — eu-west-1\n(50% traffic)"]
        B_LB["Load Balancer"]
        B_App["Application\n(Full capacity)"]
        B_DB["Database\n(Primary — accepts writes)"]
    end

    DNS --> A_LB
    DNS --> B_LB

    A_DB <-->|"Bidirectional replication"| B_DB
Property Value
RTO 0 — no failover needed, traffic auto-reroutes
RPO 0 (synchronous) or near-zero (async)
Cost Highest — full infrastructure in every region
Best for Global platforms requiring 99.999% availability

Active-active challenges:

  • Write conflicts — two regions accepting writes to the same data simultaneously requires conflict resolution
  • Replication lag — with async replication, a user in Region A may not immediately see data written in Region B
  • Global consistency — achieving strong consistency across regions requires synchronous replication, which increases write latency
  • Cost — full infrastructure duplicated across regions

Backup Strategy

Backups are the last line of defence. Every production system needs a documented, tested backup strategy.

The 3-2-1 Backup Rule

flowchart LR
    Data["Production Data"]
    Data --> C1["Copy 1\n(Primary database)"]
    Data --> C2["Copy 2\n(Same-region replica\nor snapshot)"]
    Data --> C3["Copy 3\n(Different region\nor offline storage)"]

    C1 --> T1["Storage Type: Live"]
    C2 --> T2["Storage Type: Regional snapshot"]
    C3 --> T3["Storage Type: Cross-region / cold"]
  • 3 copies of the data
  • 2 different storage media or locations
  • 1 copy stored offsite (different region or air-gapped)

Backup Types

Type Description RPO RTO
Full backup Complete snapshot of all data Hours Hours
Incremental Only changes since the last backup Minutes–hours Hours
Differential All changes since the last full backup Hours Medium
Continuous (WAL) Database write-ahead log streamed in real time Seconds Minutes
Point-in-time Restore to any specific timestamp Seconds Minutes

Backup Schedule by Data Criticality

Data Type Backup Frequency Retention Method
Financial transactions Continuous (WAL) 7 years WAL + daily full
User account data Hourly snapshot 90 days Incremental
Application config On every change Indefinitely Git + secrets vault
Audit logs Continuous 7 years Append-only store
Session data Not backed up N/A Ephemeral

Database Disaster Recovery

The database is usually the most critical and most difficult component to recover.

Database Replication Architecture

flowchart TB
    PrimaryDB["Primary Database\n(Accepts reads + writes)"]

    subgraph SameRegion["Same Region"]
        ReadReplica1["Read Replica\n(AZ-b)"]
        ReadReplica2["Read Replica\n(AZ-c)"]
    end

    subgraph CrossRegion["Cross-Region DR"]
        DRReplica["DR Replica\n(us-west-2)\n(Promoted on disaster)"]
    end

    WALBackup["Continuous WAL\nBackup to S3"]

    PrimaryDB --> ReadReplica1
    PrimaryDB --> ReadReplica2
    PrimaryDB --> DRReplica
    PrimaryDB --> WALBackup

Point-in-Time Recovery (PITR)

PITR allows restoring a database to any specific timestamp — not just the last backup.

sequenceDiagram
    participant Engineer
    participant RDS
    participant S3

    Note over RDS: 14:30 — Bad migration runs
    Note over RDS: Data corrupted

    Engineer->>RDS: Identify corruption timestamp (14:28)
    Engineer->>RDS: Initiate PITR to 14:27:59
    RDS->>S3: Retrieve last full backup
    RDS->>S3: Retrieve WAL logs up to 14:27:59
    RDS->>RDS: Replay WAL to target timestamp
    RDS-->>Engineer: New instance ready at 14:27:59 state

    Note over RDS: Data restored — 1 minute of data loss

PITR is the most important database DR capability for protecting against accidental deletions and failed migrations.

Requirements for PITR:

  • Continuous WAL (Write-Ahead Log) archiving to S3 or equivalent
  • Retention of WAL files for the RPO window (minimum)
  • Regular PITR tests — verify you can actually restore to a specific timestamp

Database Failover Sequence

sequenceDiagram
    participant Monitor
    participant Primary
    participant Replica
    participant App
    participant DNS

    Monitor->>Primary: Health check fails (3 consecutive)
    Monitor->>Replica: Initiate promotion
    Replica->>Replica: Replay remaining WAL
    Replica->>Replica: Accept write connections
    Monitor->>DNS: Update connection string to replica
    DNS-->>App: New primary endpoint propagated
    App->>Replica: Reconnect — now primary
    Note over Replica: Serving as primary

Multi-Region Failover Architecture

For systems with an RTO under 15 minutes, failover must be partially or fully automated.

flowchart TB
    Monitor["Monitoring\n(CloudWatch / Prometheus)"]
    Monitor -->|"Region health check fails"| AutoFailover["Auto-Failover\nOrchestrator"]

    AutoFailover --> Step1["1. Promote DR DB\nto primary"]
    AutoFailover --> Step2["2. Scale DR application\nto full capacity"]
    AutoFailover --> Step3["3. Update DNS / Route 53\nhealth-based routing"]
    AutoFailover --> Step4["4. Notify on-call team\nvia PagerDuty"]
    AutoFailover --> Step5["5. Update status page"]

    Step1 --> Step2 --> Step3 --> Step4 --> Step5

    Step3 --> DR["DR Region\nnow serving\n100% traffic"]

Route 53 Health-Based Routing

AWS Route 53 can automatically reroute traffic to a healthy region when health checks fail.

flowchart LR
    Client --> Route53["Route 53\n(Failover routing policy)"]
    Route53 -->|"Primary healthy"| Primary["Primary Region\n(us-east-1)"]
    Route53 -->|"Primary unhealthy\n(3 health check failures)"| DR["DR Region\n(us-west-2)"]

    Route53 --> HC1["Health Check\n/health endpoint\nevery 10s"]
    HC1 --> Primary

Configuration:

  • Health check interval: 10 seconds
  • Failure threshold: 3 consecutive failures
  • Failover time: ~30–60 seconds from failure to DNS propagation

Disaster Recovery Runbook

A runbook is a documented, step-by-step procedure for executing recovery. It must be:

  • Written before a disaster (not during)
  • Tested regularly
  • Accessible without production credentials (in case credentials are the failure)
  • Updated after every incident or infrastructure change

Runbook Template

INCIDENT: [Type of failure — e.g., Primary database unavailable]

SEVERITY: P0 / P1

OWNER: [On-call engineer]

---
PRE-CONDITIONS
- [ ] Confirm failure is genuine (check monitoring, not just one signal)
- [ ] Notify stakeholders and open incident bridge
- [ ] Check status of DR environment before starting failover

RECOVERY STEPS
1. [ ] Promote DR database to primary
   Command: aws rds promote-read-replica --db-instance-identifier prod-dr-replica
   Expected output: DB status changes from "read-only" to "available"
   Rollback: N/A (promotion is one-way)

2. [ ] Update application connection string
   Command: kubectl set env deployment/payment-service DB_HOST=<dr-endpoint>
   Expected: Pods restart and connect successfully
   Verify: Check /actuator/health/readiness — db: UP

3. [ ] Update Route 53 DNS
   Command: aws route53 change-resource-record-sets ...
   Expected: Traffic begins routing to DR region within 60 seconds

4. [ ] Scale DR application to full production capacity
   Command: kubectl scale deployment payment-service --replicas=10
   Expected: All 10 pods Running and Ready

5. [ ] Verify end-to-end health
   - [ ] /actuator/health returns UP
   - [ ] Payment test transaction succeeds
   - [ ] Error rate on Grafana is < 0.1%

POST-RECOVERY
- [ ] Notify stakeholders — service restored
- [ ] Update status page
- [ ] Begin incident post-mortem
- [ ] Plan primary region recovery

DR Testing

A DR plan that has never been tested is not a DR plan. It is a hypothesis.

Types of DR Tests

Test Type Description Frequency
Tabletop exercise Walk through the runbook verbally with the team Quarterly
Component failover Fail one component (e.g., database) and verify recovery Monthly
Full DR drill Simulate complete region failure, execute full failover Semi-annually
Chaos engineering Inject random failures in production (with safeguards) Continuously
Backup restoration Restore a backup to a test environment and verify integrity Monthly

Chaos Engineering

flowchart LR
    ChaosEngine["Chaos Engineering\n(Chaos Monkey / LitmusChaos)"]
    ChaosEngine --> K1["Kill random pod\n(verify auto-restart)"]
    ChaosEngine --> K2["Simulate AZ failure\n(verify traffic shifts)"]
    ChaosEngine --> K3["Inject network latency\n(verify circuit breakers)"]
    ChaosEngine --> K4["Corrupt DB connection\n(verify reconnect)"]
    ChaosEngine --> K5["Kill service dependency\n(verify fallback)"]

Chaos engineering validates that the DR mechanisms actually work under realistic conditions — not just on paper.

Principle: Run chaos experiments in production with safeguards. A DR system never tested in production cannot be trusted to work in production.

RTO and RPO by System Tier

Different parts of a system have different recovery requirements and budgets.

System Tier Example RTO Target RPO Target DR Pattern
Tier 1 — Mission Critical Payment processing < 5 minutes 0–15 seconds Active-Active
Tier 2 — Business Critical Customer portal < 30 minutes < 5 minutes Warm Standby
Tier 3 — Important Reporting dashboard < 4 hours < 1 hour Pilot Light
Tier 4 — Non-Critical Admin tools, dev tools < 24 hours < 24 hours Backup + Restore

Classify every service into a tier. Only Tier 1 justifies the cost of Active-Active. Over-engineering DR for Tier 4 systems wastes budget that should go to Tier 1.

DR Cost vs Recovery Time Trade-off

The faster the required recovery time, the higher the infrastructure cost.

flowchart LR
    subgraph Cost["Increasing Cost →"]
        BaR["Backup + Restore\nLowest cost\nRTO: hours"]
        PL["Pilot Light\nLow cost\nRTO: 30–60 min"]
        WS["Warm Standby\nMedium cost\nRTO: 5–30 min"]
        AA["Active-Active\nHighest cost\nRTO: 0"]
    end

    BaR --> PL --> WS --> AA

The right pattern is determined by the business impact per minute of downtime:

  • If 1 hour of downtime costs $100 → Backup + Restore is sufficient
  • If 1 hour of downtime costs $1,000,000 → Active-Active is justified

Production DR Readiness Checklist

Backup

  • Automated backups are configured for all databases
  • Backups are stored in a geographically separate region
  • Backup restoration has been tested in the last 30 days
  • Point-in-time recovery is enabled and tested
  • Backup retention period meets compliance requirements

Replication

  • Database replication is configured to DR region
  • Replication lag is monitored and alerted
  • Replica promotion has been tested (not just configured)
  • Application can reconnect automatically after DB failover

Infrastructure

  • All infrastructure is defined as code (Terraform / CDK)
  • DR environment can be provisioned from code in < 30 minutes
  • DR environment capacity matches production capacity (warm/hot)
  • DNS failover is configured and tested

Runbooks

  • Recovery runbooks exist for every P0/P1 failure scenario
  • Runbooks are accessible without production credentials
  • Runbooks have been walked through in the last 90 days
  • Commands in runbooks are tested and verified

Testing

  • Full DR drill performed in the last 6 months
  • Backup restoration tested in the last 30 days
  • Chaos engineering experiments run regularly
  • RTO and RPO targets verified in last DR test

Communication

  • Status page exists and is updated during incidents
  • Stakeholder notification list is current
  • On-call escalation path is defined and tested
  • Post-mortem process is defined

Summary

Disaster recovery is not a feature — it is a system design requirement that must be defined, architected, and tested before a disaster occurs.

flowchart TB
    Define["Define RTO and RPO\n(Business requirements)"]
    Classify["Classify services\nby tier"]
    Choose["Choose DR pattern\nper tier"]
    Implement["Implement backups,\nreplication, failover"]
    Test["Test regularly\n(drills, chaos, restore)"]
    Document["Write and maintain\nrunbooks"]

    Define --> Classify --> Choose --> Implement --> Test --> Document
    Document -->|"After each incident"| Test

Key principles:

  • RTO and RPO must be defined first — every DR decision flows from these numbers
  • Match the pattern to the tier — not every service needs Active-Active
  • Backups are worthless unless tested — restore to a test environment monthly
  • Runbooks must be written before a disaster — not improvised during one
  • Test in production — a DR system never tested against real infrastructure cannot be trusted
  • Automate failover — if RTO < 15 minutes, human-triggered failover will miss the window