Full Stack • Java • System Design • Cloud • AI Engineering

Deployment Strategies for Production Systems

A complete guide to deployment strategies in production engineering — covering rolling updates, blue-green, canary, feature flags, and database migration patterns with architecture diagrams and decision frameworks.

Introduction

Deploying software to production is one of the highest-risk operations an engineering team performs.

A poorly executed deployment can cause:

  • Complete service outages
  • Data corruption
  • Revenue loss
  • Customer data exposure
  • Hours of emergency rollback effort

A well-designed deployment strategy eliminates or minimises all of these risks. It allows teams to ship changes frequently, safely, and with confidence — even across systems serving millions of users.

The choice of deployment strategy is a system design decision. It depends on the risk tolerance of the system, the ability to run multiple versions simultaneously, and the speed of rollback required.

What Makes a Deployment Strategy Production-Ready

A production-grade deployment strategy must satisfy:

Property What It Means
Zero downtime Users experience no service interruption during a release
Fast rollback A bad release can be reversed in minutes, not hours
Incremental exposure Changes reach a small subset of users before full rollout
Observable Metrics and errors are monitored throughout the rollout
Database compatible Schema changes are backward compatible with the running version
Automated Deployment steps are executed by a pipeline, not manually

Deployment Strategy Overview

flowchart TB
    Strategies["Deployment Strategies"]
    Strategies --> Recreate["Recreate\n(Stop all → Deploy all)"]
    Strategies --> Rolling["Rolling Update\n(Instance by instance)"]
    Strategies --> BlueGreen["Blue-Green\n(Full parallel environment)"]
    Strategies --> Canary["Canary\n(Gradual traffic shift)"]
    Strategies --> Feature["Feature Flags\n(Code deployed, feature toggled)"]

    Recreate --> R1["❌ Downtime\n✅ Simple"]
    Rolling --> R2["✅ No downtime\n✅ Resource efficient"]
    BlueGreen --> R3["✅ Instant rollback\n⚠️ Double resources"]
    Canary --> R4["✅ Risk controlled\n⚠️ Complex routing"]
    Feature --> R5["✅ Deploy and release decoupled\n⚠️ Flag management overhead"]

Strategy 1: Recreate Deployment

How It Works

All running instances are stopped first. Then the new version is deployed.

flowchart LR
    subgraph Before["Before Deployment"]
        V1A["Instance: v1"]
        V1B["Instance: v1"]
        V1C["Instance: v1"]
    end

    subgraph During["During Deployment"]
        Down["⛔ All Instances Stopped\n(Service Unavailable)"]
    end

    subgraph After["After Deployment"]
        V2A["Instance: v2"]
        V2B["Instance: v2"]
        V2C["Instance: v2"]
    end

    Before --> During --> After

When to Use

  • Development and staging environments
  • Applications that cannot run two versions simultaneously (tight DB schema coupling)
  • Systems with a maintenance window (batch jobs, scheduled processors)
  • Very early-stage products with no SLA requirements

Trade-offs

Advantage Disadvantage
Simplest to implement Guaranteed downtime during deployment
No version conflicts Users are impacted on every release
Clean state guaranteed Rollback also causes downtime

Not acceptable for production systems with availability requirements.

Strategy 2: Rolling Update

How It Works

Instances are updated one at a time (or in small batches). At any point during the deployment, both the old and new versions are running simultaneously.

flowchart LR
    LB["Load Balancer"]

    subgraph Step1["Step 1 — Start"]
        A1["v1"] 
        A2["v1"] 
        A3["v1"]
        A4["v1"]
    end

    subgraph Step2["Step 2 — Rolling"]
        B1["v2"] 
        B2["v2"] 
        B3["v1"]
        B4["v1"]
    end

    subgraph Step3["Step 3 — Complete"]
        C1["v2"] 
        C2["v2"] 
        C3["v2"]
        C4["v2"]
    end

    Step1 --> Step2 --> Step3
    LB --> Step1

Traffic is only sent to healthy instances at each step. An instance is only updated after the previous one passes its readiness probe.

Rolling Update in Kubernetes

strategy:
  type: RollingUpdate
  rollingUpdate:
    maxUnavailable: 1
    maxSurge: 1
Parameter Meaning
maxUnavailable Maximum number of pods that can be unavailable at once
maxSurge Maximum extra pods that can be created above desired count

With maxUnavailable: 1 and maxSurge: 1 on a 4-pod deployment:

  • A fifth pod (v2) starts before any v1 pod is removed
  • One v1 pod is removed only after the new v2 pod passes readiness
  • This ensures there are always at least 3 healthy pods throughout

Version Compatibility Requirement

Because v1 and v2 run simultaneously during a rolling update:

  • The API must be backward compatible — v2 must accept v1-format requests
  • The database schema must be backward compatible — v2 schema additions must not break v1 queries
  • Message formats must remain compatible — old consumers must still process new message shapes
flowchart LR
    Client --> LB["Load Balancer"]
    LB --> Pod1["Pod: v1"]
    LB --> Pod2["Pod: v1"]
    LB --> Pod3["Pod: v2"]
    LB --> Pod4["Pod: v2"]
    Pod1 --> DB["Shared Database\n(must support both versions)"]
    Pod2 --> DB
    Pod3 --> DB
    Pod4 --> DB

When to Use

  • Standard production deployments with no strict rollback time requirement
  • Services with backward-compatible API and database changes
  • Kubernetes-native workloads

Trade-offs

Advantage Disadvantage
No downtime Two versions run simultaneously
Resource efficient (no extra infra) Rollback is another rolling update (takes time)
Simple Kubernetes configuration API and schema must be backward compatible

Strategy 3: Blue-Green Deployment

How It Works

Two identical environments — Blue (current live) and Green (new version) — exist simultaneously. All traffic runs on Blue. Green is deployed and tested in isolation. When ready, all traffic is switched from Blue to Green in a single step.

flowchart TB
    DNS["DNS / Load Balancer\n(Traffic Switch)"]

    subgraph Blue["Blue Environment (Live — v1)"]
        B1["Pod: v1"]
        B2["Pod: v1"]
        B3["Pod: v1"]
    end

    subgraph Green["Green Environment (Staging — v2)"]
        G1["Pod: v2"]
        G2["Pod: v2"]
        G3["Pod: v2"]
    end

    DNS -->|"100% traffic"| Blue
    DNS -.->|"0% traffic\n(ready to switch)"| Green

After the switch:

flowchart TB
    DNS["DNS / Load Balancer\n(After Switch)"]

    subgraph Blue["Blue Environment (Idle — v1)"]
        B1["Pod: v1"]
        B2["Pod: v1"]
    end

    subgraph Green["Green Environment (Now Live — v2)"]
        G1["Pod: v2"]
        G2["Pod: v2"]
        G3["Pod: v2"]
    end

    DNS -.->|"0% traffic\n(kept for rollback)"| Blue
    DNS -->|"100% traffic"| Green

Rollback

If Green has a problem, rollback is a single DNS or load balancer switch back to Blue — typically completing in under 60 seconds.

sequenceDiagram
    participant Team
    participant LoadBalancer
    participant Green
    participant Blue

    Team->>LoadBalancer: Switch traffic to Green
    LoadBalancer-->>Green: 100% traffic
    Note over Green: Error rate spikes

    Team->>LoadBalancer: Rollback — switch to Blue
    LoadBalancer-->>Blue: 100% traffic
    Note over Blue: Service fully restored in < 60s

Database Considerations

Blue-Green requires both versions to share the same database during the cutover window, which means the same backward compatibility constraints as rolling updates apply.

flowchart LR
    Blue["Blue Pods (v1)"] --> DB["Shared Database"]
    Green["Green Pods (v2)"] --> DB

After the switch and a stabilization period, the old Blue environment can be scaled down and the database migrated to the v2 schema.

When to Use

  • High-value releases where instant rollback is required
  • Major version changes that are hard to test incrementally
  • Regulated systems (financial, healthcare) requiring a clean cutover
  • Systems where testing the full environment before go-live is mandatory

Trade-offs

Advantage Disadvantage
Instant rollback (< 60 seconds) Doubles infrastructure cost during deployment
Zero user impact on cutover Database changes require backward compatibility
Full environment tested before go-live Traffic switch is all-or-nothing
Clean version separation More complex infrastructure management

Strategy 4: Canary Deployment

How It Works

A small percentage of traffic is routed to the new version. The majority continues on the current version. The new version is monitored closely. If metrics are healthy, traffic is gradually shifted until the canary reaches 100%.

flowchart LR
    Users["All Users\n(1000 req/s)"]
    LB["Load Balancer\n(Traffic Splitting)"]
    V1["Stable: v1\n90% traffic → 900 req/s"]
    V2["Canary: v2\n10% traffic → 100 req/s"]

    Users --> LB
    LB -->|"90%"| V1
    LB -->|"10%"| V2

Canary Rollout Progression

flowchart LR
    S1["Stage 1\nCanary: 5%\nStable: 95%"]
    S2["Stage 2\nCanary: 20%\nStable: 80%"]
    S3["Stage 3\nCanary: 50%\nStable: 50%"]
    S4["Stage 4\nCanary: 100%\nOld: 0%"]

    S1 -->|"Metrics healthy"| S2
    S2 -->|"Metrics healthy"| S3
    S3 -->|"Metrics healthy"| S4
    S1 -->|"Error spike"| Rollback["Rollback to 0%"]
    S2 -->|"Error spike"| Rollback
    S3 -->|"Error spike"| Rollback

Each stage is held for a defined soak time (e.g., 15–30 minutes) while metrics are observed. Automation can promote or rollback based on SLO thresholds.

What to Monitor During a Canary

Signal Threshold to Watch
Error rate Canary error rate vs stable
p99 latency Canary latency vs stable
Business metrics Order completion, payment success
CPU / Memory Resource consumption per pod

If the canary's error rate exceeds the stable baseline by more than a defined threshold, the deployment is automatically rolled back to 0%.

Canary in Kubernetes with Argo Rollouts

strategy:
  canary:
    steps:
    - setWeight: 5
    - pause:
        duration: 15m
    - setWeight: 20
    - pause:
        duration: 15m
    - setWeight: 50
    - pause:
        duration: 15m
    - setWeight: 100
    analysis:
      templates:
      - templateName: error-rate-check
      args:
      - name: service-name
        value: payment-service

When to Use

  • High-traffic services where a bad deployment affects many users
  • Changes to core business logic (payments, orders, authentication)
  • Releases that need real traffic validation before full rollout
  • Systems with mature monitoring and SLO dashboards in place

Trade-offs

Advantage Disadvantage
Limits blast radius of bad releases Two versions run simultaneously
Detects real-world issues early Requires traffic splitting infrastructure
Automated rollback on SLO breach Users may experience inconsistent behaviour
Gradual confidence building Canary analysis setup requires investment

Strategy 5: Feature Flags

How It Works

Code is deployed to production but the feature is switched off. Features are enabled independently from deployment — by toggling a flag in a configuration store, without any new deployment.

flowchart LR
    Deploy["Deploy v2\n(Feature code present\nbut flag OFF)"]
    Deploy --> Prod["Production\n(v2 running, feature hidden)"]
    Prod --> Toggle["Enable flag\nfor 1% of users"]
    Toggle --> Rollout["Gradually increase\nto 100%"]
    Toggle -->|"Problems"| Disable["Disable flag instantly\n(no deployment needed)"]

Feature Flag Architecture

flowchart TB
    App["Application"] --> FlagClient["Feature Flag Client\n(LaunchDarkly / Unleash / Flipt)"]
    FlagClient --> FlagStore["Flag Store\n(Remote config)"]
    FlagStore --> Rules["Targeting Rules\n(user ID, region, % rollout)"]
    Rules --> Decision{"Flag ON\nor OFF?"}
    Decision -->|"ON"| NewCode["Execute new feature code"]
    Decision -->|"OFF"| OldCode["Execute existing code path"]

Targeting Strategies

Feature flags enable precise control over who sees a feature:

Strategy Example
User segment Enable for internal employees only
Percentage rollout Enable for 5% of users randomly
Region Enable in US before EU
Account tier Enable for premium users first
A/B testing Split traffic to measure conversion impact

Deployment vs Release

Feature flags decouple the deployment from the release:

flowchart LR
    Code["Code merged\nto main"] --> Deploy["Deploy to production\n(flag OFF)"]
    Deploy --> Verify["Verify system\nstability"]
    Verify --> Release["Enable flag\nfor target users"]
    Release --> Monitor["Monitor metrics"]
    Monitor -->|"Issue"| Disable["Disable flag\n(instant rollback)"]
    Monitor -->|"Healthy"| Expand["Expand to 100%"]

This means a deployment is a low-risk infrastructure operation. The release is a separate, controlled business decision.

When to Use

  • Long-running feature development that merges incrementally to main
  • A/B testing and experimentation
  • Kill switches for third-party integrations
  • Gradual rollout to user segments without infrastructure routing changes
  • Emergency disable capability for new features

Trade-offs

Advantage Disadvantage
Instant rollback without a deployment Flag debt accumulates if not cleaned up
Deploy and release are independent Code has conditional branches long-term
Precise targeting (user, region, %) Requires flag management infrastructure
Enables safe trunk-based development Testing all flag combinations adds complexity

Deployment Strategy Comparison

Strategy Downtime Rollback Speed Resource Cost Version Overlap Complexity
Recreate Yes Fast (redeploy) Low None Low
Rolling Update No Slow (re-roll) Low Yes Low
Blue-Green No Instant (< 60s) 2x during deploy Short window Medium
Canary No Instant (0%) Low overhead Yes High
Feature Flags No Instant (toggle) None Yes (in code) Medium

Database Migration Patterns

Every deployment strategy must address database schema changes. Schema migrations are the most common cause of deployment failures.

The Core Challenge

flowchart LR
    OldApp["v1 App\n(running during migration)"]
    NewApp["v2 App\n(needs new schema)"]
    DB["Database\n(shared)"]

    OldApp --> DB
    NewApp --> DB

If v1 and v2 run simultaneously, the database must be compatible with both versions.

Expand and Contract Pattern

The safest approach to schema changes in zero-downtime deployments.

flowchart LR
    Phase1["Phase 1: Expand\nAdd new column\n(nullable, no default constraint)\nBoth v1 and v2 work"]
    Phase2["Phase 2: Migrate\nBackfill existing rows\nv2 writes to both old and new column"]
    Phase3["Phase 3: Contract\nRemove old column\nOnly after v1 is fully retired"]

    Phase1 --> Phase2 --> Phase3

Example — renaming a column:

Phase Schema State App Versions Supported
Expand Both customer_name and full_name exist v1 (reads old), v2 (reads new)
Migrate Backfill full_name from customer_name v1 and v2 both work
Contract Remove customer_name v2 only

Rules for Safe Schema Migrations

Operation Safe in Zero-Downtime? Notes
Add nullable column ✅ Yes Old app ignores new column
Add column with default ✅ Yes Old app ignores new column
Add index (concurrent) ✅ Yes Use CREATE INDEX CONCURRENTLY
Add new table ✅ Yes Old app is unaffected
Rename column ❌ No (directly) Use expand and contract
Drop column ❌ No (directly) Only after all app versions stop using it
Change column type ❌ No (directly) Use expand and contract with a new column
Add NOT NULL constraint ❌ No (directly) Backfill nulls first, then add constraint

CI/CD Pipeline Integration

A deployment strategy is only as good as the pipeline that executes it. Every production deployment should flow through an automated pipeline.

flowchart LR
    Code["Code Merged\nto Main"]
    Code --> Build["Build\n(Compile + Test)"]
    Build --> UnitTests["Unit Tests\n+ Integration Tests"]
    UnitTests --> ContainerBuild["Container Image Build\n(Docker)"]
    ContainerBuild --> SecurityScan["Security Scan\n(Image vulnerabilities)"]
    SecurityScan --> PushRegistry["Push to\nContainer Registry"]
    PushRegistry --> DeployStaging["Deploy to Staging\n(Rolling / Blue-Green)"]
    DeployStaging --> SmokeTests["Smoke Tests\n(Automated)"]
    SmokeTests --> DeployProd["Deploy to Production\n(Canary / Blue-Green)"]
    DeployProd --> Monitor["Monitor SLOs\n(Auto-rollback if breach)"]

Pipeline Gates

Each stage is a gate. A failure at any stage stops the pipeline and prevents the change from reaching production.

Gate What It Checks
Unit tests Individual component correctness
Integration tests Component interaction and API contracts
Security scan Known CVEs in dependencies and base images
Staging deploy Change deploys successfully in a production-like environment
Smoke tests Critical user journeys work end-to-end
SLO monitoring Production metrics stay within acceptable bounds

Choosing the Right Strategy

flowchart TD
    Q1{"Is downtime acceptable?"}
    Q1 -- Yes --> Recreate["Recreate\n(Simple, full restart)"]
    Q1 -- No --> Q2{"Need instant rollback?"}
    Q2 -- Yes --> Q3{"Can afford double infra?"}
    Q3 -- Yes --> BlueGreen["Blue-Green\n(Instant switch, full env)"]
    Q3 -- No --> Q4{"High traffic, need risk control?"}
    Q4 -- Yes --> Canary["Canary\n(Gradual traffic shift)"]
    Q4 -- No --> Rolling["Rolling Update\n(Default for most services)"]
    Q2 -- No --> Q5{"Decouple deploy from release?"}
    Q5 -- Yes --> FeatureFlags["Feature Flags\n(Toggle without deployment)"]
    Q5 -- No --> Rolling
Scenario Recommended Strategy
Standard microservice update Rolling Update
Major version with complex rollback requirements Blue-Green
Core payment or auth service change Canary
Large feature developed over multiple sprints Feature Flags
Batch job or scheduled processor Recreate
Experiment or A/B test Feature Flags

Production Deployment Checklist

Before executing any production deployment:

Pre-Deployment

  • All tests pass in CI (unit, integration, smoke)
  • Container image scanned for vulnerabilities
  • Database migrations reviewed for backward compatibility
  • Rollback plan documented and tested in staging
  • Monitoring dashboards open and baseline captured
  • On-call engineer notified and available

During Deployment

  • Error rate monitored continuously
  • p99 latency compared to pre-deployment baseline
  • Business metrics (orders, payments) tracked in real time
  • Readiness probes passing on new instances before traffic shift

Post-Deployment

  • SLO compliance confirmed for 15–30 minutes post-deploy
  • Logs checked for new error patterns
  • Database migration completed and verified
  • Old environment (blue) retained for rollback window (minimum 1 hour)
  • Deployment documented in change log

Summary

Deployment strategy is a production system design decision — not just a DevOps concern. The right strategy depends on availability requirements, rollback speed, resource constraints, and the nature of the change being shipped.

Strategy Best For Key Requirement
Recreate Non-production, maintenance windows Acceptable downtime
Rolling Standard service updates Backward-compatible changes
Blue-Green High-stakes releases, regulated systems Double infrastructure
Canary High-traffic critical services Traffic splitting + monitoring
Feature Flags Long-lived features, experiments, kill switches Flag management tooling

The most mature engineering teams combine strategies — rolling updates for infrastructure changes, canary for logic changes, and feature flags for product rollouts. Every critical service should have a documented deployment strategy before going to production.