Full Stack • Java • System Design • Cloud • AI Engineering

Health Checks in Production Systems

A complete guide to health checks in production engineering — covering liveness, readiness, and startup probes, deep health check design, load balancer integration, Kubernetes configuration, and operational best practices.

Introduction

A service that is running is not necessarily a service that is healthy.

A pod can be alive — consuming CPU, responding to the OS — while being completely unable to serve requests. Its database connection pool may be exhausted, its downstream dependency timed out, or its message queue consumer stalled.

Health checks are the mechanism by which a production system continuously verifies that each component is not just running but actually capable of doing useful work.

Without health checks:

  • Load balancers route traffic to broken instances
  • Kubernetes keeps pods running that cannot serve any requests
  • Deployments complete successfully while silently serving errors
  • Engineers discover failures from customer support tickets

With properly configured health checks:

  • Traffic is automatically removed from unhealthy instances
  • Broken pods are restarted before customers notice
  • Deployments are gated — new instances only receive traffic when they are genuinely ready
  • Cascading failures are detected and contained early

The Three Types of Health Checks

Production systems use three distinct health check types, each serving a different purpose.

flowchart TB
    Kubernetes["Kubernetes / Orchestrator"]
    Kubernetes --> Liveness["Liveness Probe\nIs the process alive?\nRestart if failing"]
    Kubernetes --> Readiness["Readiness Probe\nCan this instance serve traffic?\nRemove from load balancer if failing"]
    Kubernetes --> Startup["Startup Probe\nHas the app finished initializing?\nBlock liveness/readiness until passing"]
Probe Question It Answers Action on Failure
Liveness Is the process in a broken state? Restart the container
Readiness Is the instance ready to accept traffic? Remove from load balancer pool
Startup Has the app finished starting up? Delay other probes

These three probes work together. Each is configured independently and checked on its own schedule.

Liveness Probe

Purpose

The liveness probe answers: "Is this process alive and functional, or is it deadlocked/corrupted and needs a restart?"

A liveness failure triggers a container restart. Use it only for conditions that are unrecoverable without a restart.

When to Fail Liveness

  • The application has entered a deadlock
  • The event loop is blocked and will never recover
  • A critical goroutine or thread has panicked
  • The application is out of memory and cannot recover

When NOT to Fail Liveness

  • A downstream dependency (database, cache) is temporarily unavailable
  • A queue is temporarily unresponsive
  • The service is under high load

Failing liveness for dependency issues causes a restart loop — the restarted pod will immediately fail again because the dependency is still down.

sequenceDiagram
    participant Kubernetes
    participant Pod
    participant LivenessEndpoint

    Kubernetes->>LivenessEndpoint: GET /actuator/health/liveness
    LivenessEndpoint-->>Kubernetes: 200 UP

    Note over Pod: Deadlock occurs

    Kubernetes->>LivenessEndpoint: GET /actuator/health/liveness
    LivenessEndpoint-->>Kubernetes: 503 DOWN

    Kubernetes->>Pod: Restart container
    Note over Pod: Fresh start — deadlock cleared

Liveness Endpoint Response

GET /actuator/health/liveness

{
  "status": "UP"
}

Readiness Probe

Purpose

The readiness probe answers: "Is this instance currently able to serve user traffic?"

A readiness failure removes the pod from the load balancer's active pool without restarting it. Traffic stops flowing to the instance until it recovers.

When to Fail Readiness

  • The database connection pool is exhausted
  • A required cache connection is unavailable
  • The service is still warming up (e.g., loading large in-memory data)
  • The service is intentionally draining before a graceful shutdown
  • A downstream dependency is degraded enough to affect responses

Readiness Flow

sequenceDiagram
    participant LoadBalancer
    participant Kubernetes
    participant Pod
    participant Database

    LoadBalancer->>Pod: User Request
    Pod->>Database: Query
    Database-->>Pod: Connection refused (pool exhausted)

    Kubernetes->>Pod: GET /actuator/health/readiness
    Pod-->>Kubernetes: 503 OUT_OF_SERVICE

    Kubernetes->>LoadBalancer: Remove Pod from pool
    Note over LoadBalancer: No more traffic sent to this pod

    Note over Database: Pool recovers

    Kubernetes->>Pod: GET /actuator/health/readiness
    Pod-->>Kubernetes: 200 UP
    Kubernetes->>LoadBalancer: Re-add Pod to pool

Readiness Endpoint Response

GET /actuator/health/readiness

{
  "status": "UP",
  "components": {
    "db": { "status": "UP" },
    "redis": { "status": "UP" }
  }
}

When any component returns DOWN, the overall status becomes OUT_OF_SERVICE and the pod stops receiving traffic.

Startup Probe

Purpose

The startup probe answers: "Has the application finished its initialization sequence?"

Until the startup probe passes, Kubernetes suppresses both liveness and readiness checks. This prevents premature restarts for slow-starting applications (e.g., those loading large datasets, running database migrations, or JVM applications with slow warm-up).

flowchart LR
    Start["Container Starts"]
    Start --> StartupCheck{"Startup Probe\nPassing?"}
    StartupCheck -- No --> Wait["Wait & Retry\n(up to failureThreshold)"]
    Wait --> StartupCheck
    StartupCheck -- Yes --> Normal["Enable Liveness\n& Readiness Probes"]
    StartupCheck -- Timeout --> Kill["Kill & Restart Container"]

When to Use a Startup Probe

  • Applications that run database migrations on startup
  • JVM services with long classloading or Spring context initialization
  • Services that pre-load large caches or datasets into memory
  • Any service where startup takes more than 10–15 seconds

Without a startup probe, you must set very generous initialDelaySeconds on liveness — which means slow detection of actual deadlocks in production. The startup probe is a cleaner solution.

Shallow vs Deep Health Checks

Health check depth is a critical design decision. Not all health check endpoints should check the same things.

flowchart TB
    Types["Health Check Depth"]
    Types --> Shallow["Shallow Check\n(Ping)\nIs the HTTP server alive?"]
    Types --> Medium["Medium Check\nAre critical dependencies reachable?"]
    Types --> Deep["Deep Check\nAre all components functional?"]

Shallow Health Check

Checks only that the HTTP server is accepting connections.

GET /health → 200 OK
Use case Detail
Purpose Confirm the process is not crashed
Suitable for Liveness probe
Checks HTTP server only
Latency < 5ms

Medium Health Check (Dependency Check)

Checks that critical dependencies are reachable and responding.

GET /health/ready

{
  "status": "UP",
  "components": {
    "database": { "status": "UP", "responseTimeMs": 3 },
    "redis":    { "status": "UP", "responseTimeMs": 1 }
  }
}
Use case Detail
Purpose Confirm the instance can handle real requests
Suitable for Readiness probe
Checks DB connectivity, cache reachability
Latency 5–50ms

Deep Health Check (Functional Check)

Verifies actual business functionality end-to-end — a read query, a cache set/get cycle, a message queue ping.

GET /health/deep

{
  "status": "UP",
  "components": {
    "database":    { "status": "UP", "query": "SELECT 1", "responseTimeMs": 4 },
    "redis":       { "status": "UP", "operation": "PING", "responseTimeMs": 1 },
    "kafka":       { "status": "UP", "lag": 0, "responseTimeMs": 12 },
    "paymentApi":  { "status": "UP", "responseTimeMs": 45 }
  },
  "checkedAt": "2026-07-05T14:22:10Z"
}
Use case Detail
Purpose Full operational verification
Suitable for Internal monitoring dashboards, alerting only
Checks All dependencies with functional operations
Latency 50–500ms
Warning Do NOT use as a liveness or readiness probe — too slow and too risky

Health Check Endpoints — Standard Layout

Every production service should expose a consistent set of endpoints:

Endpoint Type Used By
/actuator/health/liveness Shallow Kubernetes liveness probe
/actuator/health/readiness Medium Kubernetes readiness probe
/actuator/health Medium Load balancer health check
/actuator/health/deep Deep Internal monitoring only

The deep check endpoint must never be exposed publicly or used by automated probes — only consumed by internal monitoring systems.

Health Checks in Load Balancers

Before Kubernetes, and still used alongside it, load balancers perform their own health checking independent of container orchestration.

flowchart TB
    Internet --> LB["Load Balancer\n(AWS ALB / Nginx / HAProxy)"]
    LB --> App1["Instance 1\n✅ Healthy"]
    LB --> App2["Instance 2\n✅ Healthy"]
    LB --> App3["Instance 3\n❌ Failing /health"]

    LB -. "Stops routing to Instance 3" .-> App3
    LB -. "All traffic to Instances 1 & 2" .-> App1
    LB -. "All traffic to Instances 1 & 2" .-> App2

How load balancer health checking works:

  1. The load balancer polls /health on each instance every N seconds
  2. If an instance returns a non-2xx response for M consecutive checks, it is marked unhealthy
  3. No new requests are routed to the unhealthy instance
  4. The load balancer continues polling — when the instance recovers, it is re-added to the pool

AWS ALB example configuration:

Setting Recommended Value Reason
Health check path /actuator/health Checks DB + cache connectivity
Healthy threshold 2 successes Avoid flapping — confirm recovery is stable
Unhealthy threshold 3 failures Balance speed vs false positives
Interval 15 seconds Frequent enough without excessive overhead
Timeout 5 seconds Fail fast on hung instances

Health Checks in Kubernetes

Kubernetes health probes are configured per container in the pod spec.

flowchart TB
    Pod["Pod Lifecycle"]
    Pod --> S["1. Container starts"]
    S --> SP["2. Startup probe runs\n(liveness & readiness suppressed)"]
    SP --> LP["3. Liveness probe enabled\n(checks every N seconds)"]
    SP --> RP["4. Readiness probe enabled\n(gates traffic routing)"]
    LP --> Restart["Restart container\non failure"]
    RP --> Remove["Remove from\nService endpoints on failure"]

Kubernetes Probe Configuration

livenessProbe:
  httpGet:
    path: /actuator/health/liveness
    port: 8080
  initialDelaySeconds: 0
  periodSeconds: 10
  failureThreshold: 3
  timeoutSeconds: 3

readinessProbe:
  httpGet:
    path: /actuator/health/readiness
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 10
  failureThreshold: 3
  timeoutSeconds: 5

startupProbe:
  httpGet:
    path: /actuator/health/liveness
    port: 8080
  failureThreshold: 30
  periodSeconds: 5

Configuration explained:

Field Liveness Readiness Startup
initialDelaySeconds 0 (startup probe handles delay) 5s N/A
periodSeconds 10s 10s 5s
failureThreshold 3 3 30 (allows 150s)
timeoutSeconds 3s 5s —
On failure Restart pod Remove from LB Kill & restart

Graceful Shutdown and Health Checks

Health checks play a critical role in zero-downtime deployments and graceful shutdown.

sequenceDiagram
    participant Kubernetes
    participant Pod
    participant LoadBalancer

    Note over Kubernetes: Rolling update begins

    Kubernetes->>Pod: SIGTERM signal
    Pod->>Pod: Set readiness to DOWN
    Pod-->>LoadBalancer: /readiness → 503

    Note over LoadBalancer: Stop routing new requests to this pod

    Pod->>Pod: Drain in-flight requests
    Note over Pod: Wait preStop hook (e.g., 15 seconds)

    Pod->>Pod: Finish remaining work
    Pod->>Kubernetes: Process exits cleanly

    Note over Kubernetes: Start new pod with updated version

Steps for correct graceful shutdown:

  1. Kubernetes sends SIGTERM to the container
  2. Application immediately fails its readiness probe → removed from load balancer pool
  3. Application finishes processing all in-flight requests
  4. Application closes connections to databases and queues
  5. Application exits with code 0
  6. Kubernetes registers the pod as terminated

A preStop hook with a short sleep (5–15s) ensures the load balancer has time to propagate the readiness failure before the process stops accepting connections.

lifecycle:
  preStop:
    exec:
      command: ["sleep", "10"]

terminationGracePeriodSeconds: 60

Health Check Anti-Patterns

Anti-Pattern 1: Failing Liveness on Dependency Issues

flowchart LR
    DB["Database\n(Temporarily Down)"]
    Pod["Pod\nFails /liveness\nbecause DB is down"]
    Kubernetes["Kubernetes\nRestarts Pod"]
    Pod2["New Pod\nFails immediately —\nDB still down"]

    DB --> Pod
    Pod --> Kubernetes
    Kubernetes --> Pod2
    Pod2 --> DB

Result: Restart storm. Every pod keeps restarting while the database is down, making recovery harder.

Fix: Dependency failures belong in the readiness probe, not liveness.

Anti-Pattern 2: Deep Check as Liveness Probe

If a deep health check calls a slow external API and that API is slow (200ms+), every liveness probe call takes 200ms. Under load, liveness probes time out, pods get restarted unnecessarily.

Fix: Liveness = shallow check only. Deep checks = internal monitoring only.

Anti-Pattern 3: No Health Check Timeout

A health endpoint that waits indefinitely for a database response will block the probe thread. If the database is hung, the health check hangs too, and Kubernetes never gets a response — eventually timing out and restarting the pod unnecessarily.

Fix: Always set a short timeout on every dependency call inside a health check (< 2 seconds).

Anti-Pattern 4: Skipping the Startup Probe for Slow Services

Without a startup probe, a JVM service that takes 45 seconds to initialize will have its liveness probe fail and be restarted before it finishes starting. The service never successfully starts.

Fix: Always configure a startup probe for services with non-trivial startup times.

Anti-Pattern 5: Exposing Deep Health Checks Publicly

A deep health check may reveal internal topology — database hosts, service names, response times. This is valuable operational data that should not be exposed to the public internet.

Fix: Protect deep health check endpoints with network policy or authentication.

Health Check Architecture — Full Picture

flowchart TB
    Internet --> LB["Load Balancer\n(Polls /health every 15s)"]
    LB --> GW["API Gateway"]
    GW --> S1["Service A"]
    GW --> S2["Service B"]
    GW --> S3["Service C"]

    K8s["Kubernetes\nControl Plane"] -. "Liveness probe\n(restart on fail)" .-> S1
    K8s -. "Readiness probe\n(remove from LB)" .-> S1
    K8s -. "Startup probe\n(suppress others)" .-> S1

    S1 --> DB["PostgreSQL"]
    S1 --> Cache["Redis"]
    S1 --> MQ["Kafka"]

    S1 -- "/readiness checks" --> DB
    S1 -- "/readiness checks" --> Cache

    Monitor["Internal Monitoring\n(Prometheus)"] -. "Polls /health/deep\nevery 60s" .-> S1

Every layer has its own health verification:

  • Load balancer — routes traffic only to healthy instances
  • Kubernetes — restarts broken pods, gates traffic via readiness
  • Internal monitoring — deep checks feed dashboards and alerts

Health Check Readiness Checklist

Before deploying a service to production:

Probe Configuration

  • Liveness probe configured — shallow check only (/liveness)
  • Readiness probe configured — checks DB and cache connectivity (/readiness)
  • Startup probe configured for services with startup time > 15 seconds
  • Probe timeouts set (timeoutSeconds < 5s for liveness)
  • failureThreshold tuned to avoid false restarts

Endpoint Design

  • Liveness endpoint returns quickly (< 10ms) — no dependency calls
  • Readiness endpoint checks all required dependencies with a short timeout
  • Deep health endpoint exists for internal monitoring
  • Deep health endpoint is not publicly accessible

Graceful Shutdown

  • Application fails readiness on SIGTERM before draining
  • preStop hook adds a short delay for LB propagation
  • terminationGracePeriodSeconds is long enough for in-flight requests to complete
  • Application exits with code 0 on clean shutdown

Load Balancer

  • Load balancer health check path is configured
  • Unhealthy threshold is tuned (3 failures before removal)
  • Healthy threshold requires 2+ successes before re-admission

Summary

Health checks are not optional in production. They are the mechanism that makes high availability, zero-downtime deployments, and automatic recovery possible.

Probe Checks Action on Failure Depth
Liveness Process state Restart container Shallow
Readiness Dependencies Remove from load balancer Medium
Startup Init complete Delay other probes Shallow

Key principles:

  • Liveness must be shallow — never fail it for dependency issues
  • Readiness reflects actual capability — check every dependency the service needs
  • Startup probe prevents false restarts — use it for any service with non-trivial startup
  • Graceful shutdown requires readiness cooperation — fail readiness on SIGTERM before draining
  • Deep checks belong in monitoring — not in Kubernetes probes