Full Stack • Java • System Design • Cloud • AI Engineering

Observability: Logging, Metrics, and Tracing

Learn how production applications should implement observability through structured logging, metrics collection, and distributed tracing — with architecture diagrams, tooling choices, and best practices for every layer.

Introduction

You cannot fix what you cannot see.

In production, applications fail in unexpected ways — slow database queries, memory leaks, cascading timeouts, third-party API degradation, and silent business logic errors. Without observability, engineers are guessing in the dark.

Observability is the ability to understand the internal state of a system by examining its external outputs.

It answers three fundamental questions:

Question Answer Comes From
What happened and when? Logs
How is the system performing over time? Metrics
Why did this specific request fail? Distributed Traces

A production system is not complete until all three are implemented, correlated, and actionable.

The Three Pillars of Observability

mindmap
  root((Observability))
    Logs
      Structured JSON
      Centralized Storage
      Searchable
      Alertable
    Metrics
      Latency
      Error Rate
      Throughput
      Saturation
      Dashboards
    Traces
      Request Journey
      Span Correlation
      Latency Breakdown
      Service Map

Each pillar serves a distinct purpose. Together they provide complete system visibility.

Why Observability Matters in System Design

Without observability:

  • An engineer sees a 500 error but cannot determine which service caused it
  • A slow API is reported by customers before the engineering team detects it
  • A memory leak causes weekly restarts with no root cause ever found
  • A business transaction silently fails — no error, no alert, no log

With observability:

  • Failures are detected automatically within seconds
  • The exact service, method, and database query that caused a failure is known
  • Latency regressions are caught during deployment, not after
  • SLA compliance is measurable and provable

Observability Architecture

A complete observability stack has a consistent flow: applications emit telemetry data, agents collect it, a pipeline processes it, storage persists it, and dashboards visualise it.

flowchart TB
    Services["Application Services\n(Spring Boot / Node / Python)"]
    OTel["OpenTelemetry Agent\n(Auto-instrumentation)"]
    Collector["OpenTelemetry Collector\n(Process & Route)"]
    Prometheus["Prometheus\n(Metrics Store)"]
    Loki["Loki / Elasticsearch\n(Log Store)"]
    Tempo["Tempo / Jaeger\n(Trace Store)"]
    Grafana["Grafana\n(Unified Dashboards & Alerts)"]

    Services --> OTel
    OTel --> Collector
    Collector --> Prometheus
    Collector --> Loki
    Collector --> Tempo
    Prometheus --> Grafana
    Loki --> Grafana
    Tempo --> Grafana

This architecture:

  • Uses OpenTelemetry as the single vendor-neutral instrumentation standard
  • Routes all telemetry through a central Collector for processing and fan-out
  • Stores logs, metrics, and traces in purpose-built backends
  • Provides a unified view in Grafana — one tool to query across all three pillars

Pillar 1 — Logging

What Logs Are

Logs are timestamped, immutable records of discrete events that occurred inside an application.

A log entry answers: What happened, on which service, at what time, in which context?

Structured vs Unstructured Logs

Most applications start with unstructured (plain text) logs. Production systems must use structured JSON logs.

Unstructured — avoid in production:

2026-07-05 12:30:01 ERROR Payment failed for customer 1234

Structured JSON — required for production:

{
  "timestamp": "2026-07-05T12:30:01.452Z",
  "level": "ERROR",
  "service": "payment-service",
  "traceId": "4bf92f3577b34da6",
  "spanId": "00f067aa0ba902b7",
  "customerId": "CUST-1234",
  "paymentId": "PAY-9876",
  "event": "payment_processing_failed",
  "errorCode": "INSUFFICIENT_FUNDS",
  "durationMs": 234,
  "environment": "production"
}

Structured logs can be:

  • Queried — filter by customerId, errorCode, or service
  • Aggregated — count error rates by service
  • Correlated — join with traces using traceId
  • Alerted — trigger alerts when error patterns appear

Log Levels

Use log levels consistently across every service.

Level When to Use Volume
ERROR Operation failed and requires attention Low
WARN Degraded behaviour, recoverable — monitor closely Low
INFO Key business events (payment initiated, user logged in) Medium
DEBUG Detailed flow for troubleshooting — disabled in prod Never in prod
TRACE Extremely verbose — never enabled in production Never in prod

Rule: INFO and above in production. DEBUG only in development or on-demand via dynamic log level changes.

What to Log

Always log:

  • Application startup and shutdown
  • Authentication events (login, logout, token refresh)
  • Authorization failures
  • Business transactions (created, updated, completed, failed)
  • External API calls (outbound HTTP, messaging publish)
  • Database errors and slow query warnings
  • Circuit breaker state transitions (open, half-open, closed)
  • Configuration changes

Never log:

  • Passwords or credentials
  • Full credit card numbers
  • PII in plain text (mask where needed)
  • Full request bodies containing sensitive fields
  • JWT token payloads

Centralized Log Architecture

Each service writes to its local log stream. A log shipper collects and forwards to a central store.

flowchart LR
    App1["Service A\n(Pod)"] --> Fluentd
    App2["Service B\n(Pod)"] --> Fluentd
    App3["Service C\n(Pod)"] --> Fluentd
    Fluentd["Fluentd / Fluent Bit\n(Log Shipper)"] --> ES["Elasticsearch\n(Log Store)"]
    ES --> Kibana["Kibana\n(Search & Dashboards)"]
    ES --> Alerts["Alert Rules\n(Error Spikes)"]

Common stacks:

Stack Components Best For
ELK Stack Elasticsearch + Logstash + Kibana Large enterprise
EFK Stack Elasticsearch + Fluentd + Kibana Kubernetes-native
Grafana Loki Loki + Promtail + Grafana Lightweight, GitOps
AWS CloudWatch Logs + Insights AWS-native workloads

Log Retention Policy

Environment Retention Reason
Production 30–90 days Incident investigation and compliance
Staging 7–14 days Test debugging
Development 3–7 days Local troubleshooting
Audit Logs 1–7 years Regulatory and compliance requirement

Pillar 2 — Metrics

What Metrics Are

Metrics are numeric measurements collected over time that describe system behaviour at a point in time.

A metric answers: How is this system performing right now, and how has that changed?

The Four Golden Signals

Google's Site Reliability Engineering book defines four metrics every production service must track:

Signal What It Measures Example
Latency Time to serve a request p50, p95, p99 response time
Traffic Demand on the system Requests per second
Errors Rate of failed requests 5xx errors / total requests
Saturation How full the system is CPU %, memory %, queue depth

Track these four signals for every service. When any golden signal degrades, an alert fires.

RED Method (For Services)

For request-driven services, use the RED method:

Signal Description
Rate Requests per second
Errors Failed requests per second
Duration Time per request (latency histogram)

USE Method (For Infrastructure)

For infrastructure resources (CPU, memory, disk), use the USE method:

Signal Description
Utilization Percentage of time the resource is busy
Saturation Amount of work queued waiting to be processed
Errors Count of error events

Metrics Architecture

flowchart LR
    Services["Application Services"] --> Endpoints["/actuator/prometheus"]
    Endpoints --> Prometheus["Prometheus\n(Pull-based scrape)"]
    Prometheus --> Grafana["Grafana\n(Dashboards)"]
    Prometheus --> AlertManager["AlertManager\n(Routing & Silencing)"]
    AlertManager --> PagerDuty["PagerDuty / Slack / Email"]

How it works:

  1. Each service exposes a /metrics endpoint in Prometheus format
  2. Prometheus scrapes the endpoint at a regular interval (e.g., every 15 seconds)
  3. Metrics are stored in Prometheus's time-series database
  4. Grafana queries Prometheus to render dashboards
  5. AlertManager evaluates alert rules and routes notifications

Essential Metrics to Instrument

API / HTTP Layer:

  • http_requests_total — request count by status code and endpoint
  • http_request_duration_seconds — latency histogram
  • http_requests_in_flight — concurrent active requests

JVM / Runtime (Spring Boot):

  • jvm_memory_used_bytes — heap and non-heap usage
  • jvm_gc_pause_seconds — garbage collection duration
  • jvm_threads_live — active thread count

Database:

  • db_connection_pool_active — active connections
  • db_connection_pool_pending — waiting connections
  • db_query_duration_seconds — slow query detection

Messaging (Kafka):

  • kafka_consumer_lag — how far behind the consumer is
  • kafka_producer_record_error_total — failed publishes

Business Metrics (custom):

  • payments_processed_total — by status (SUCCESS, FAILED)
  • orders_created_total — new orders per minute
  • login_attempts_total — by result (SUCCESS, INVALID_CREDENTIALS)

SLI, SLO, and SLA

Term Definition Example
SLI Service Level Indicator — the actual measured metric p99 latency = 180ms
SLO Service Level Objective — the engineering target p99 latency < 300ms
SLA Service Level Agreement — contractual commitment 99.9% availability per month

SLIs are measured from metrics. SLOs drive alert thresholds. SLAs are the external commitment.

flowchart LR
    Metrics --> SLI["SLI\n(Measured)"]
    SLI --> SLO["SLO\n(Target)"]
    SLO --> Alert["Alert if SLO at risk"]
    SLO --> SLA["SLA\n(Customer Commitment)"]

Alerting Best Practices

Alert on symptoms, not causes.

❌ Cause-based Alert ✅ Symptom-based Alert
CPU usage > 80% p99 latency > 1 second
JVM heap usage > 70% Error rate > 1% over 5 minutes
Disk usage > 85% Payment success rate < 99%

Rules for good alerts:

  • Every alert must be actionable — someone must be able to do something about it
  • Include a runbook link in the alert body
  • Avoid alert fatigue — tune thresholds so alerts are rare and meaningful
  • Use multi-window alerting to avoid false positives from short spikes

Pillar 3 — Distributed Tracing

What Tracing Is

In a microservices architecture, a single user request can pass through many services. Logs from each service are disconnected. Metrics show degradation but not which call is slow.

Distributed tracing follows a request across every service it touches, recording the time spent at each step.

A trace answers: Exactly where did this request spend its time, and where did it fail?

Trace Concepts

flowchart LR
    Trace["Trace\n(Single Request Journey)"]
    Trace --> SpanA["Span: API Gateway\n(0–12ms)"]
    SpanA --> SpanB["Span: Customer Service\n(12–45ms)"]
    SpanB --> SpanC["Span: PostgreSQL Query\n(25–42ms)"]
    SpanA --> SpanD["Span: Payment Service\n(45–180ms)"]
    SpanD --> SpanE["Span: External Bank API\n(50–175ms)"]
Concept Description
Trace The full end-to-end journey of one request
Span A single unit of work within the trace (one service call)
Trace ID A unique identifier that flows through every service in the chain
Span ID Identifies a specific span within a trace
Parent Span The span that initiated the current span

How Trace Propagation Works

sequenceDiagram
    participant Client
    participant Gateway
    participant OrderService
    participant PaymentService
    participant Database

    Client->>Gateway: HTTP Request
    Note over Gateway: Generate traceId: abc123

    Gateway->>OrderService: Forward (traceId: abc123, spanId: s1)
    OrderService->>PaymentService: Call (traceId: abc123, spanId: s2, parentSpanId: s1)
    PaymentService->>Database: Query (traceId: abc123, spanId: s3, parentSpanId: s2)
    Database-->>PaymentService: Result
    PaymentService-->>OrderService: Response
    OrderService-->>Gateway: Response
    Gateway-->>Client: HTTP Response

The traceId is injected into every HTTP header and message envelope as the request propagates. Each service reads it, creates a child span, and forwards it downstream.

OpenTelemetry — The Standard

OpenTelemetry (OTel) is the open standard for collecting logs, metrics, and traces. It replaces vendor-specific SDKs (Zipkin, Jaeger, Datadog) with a single unified instrumentation layer.

flowchart LR
    App["Application Code"] --> OTelSDK["OTel SDK\n(Auto + Manual Instrumentation)"]
    OTelSDK --> Collector["OTel Collector\n(Filter, Batch, Route)"]
    Collector --> Jaeger["Jaeger\n(Trace Backend)"]
    Collector --> Tempo["Grafana Tempo\n(Trace Backend)"]
    Collector --> Datadog["Datadog / Dynatrace\n(Commercial)"]

Why OTel matters:

  • Instrument once — switch backends without code changes
  • Auto-instrumentation for HTTP, JDBC, Kafka, Redis out of the box
  • Supported by every major cloud provider and APM vendor
  • CNCF graduated project — stable and widely adopted

Trace Sampling

Tracing every request at full volume can be expensive. Use sampling strategies:

Strategy Description Best For
Head sampling Decision made at request entry point (e.g., 10%) Low-cost baseline
Tail sampling Decision made after trace completes — keep errors High-value coverage
Adaptive Rate adjusts based on traffic volume automatically High-throughput systems

Rule: Always sample 100% of errors and slow traces (> 1s). Sample a percentage of healthy fast traces.

Tracing Backends

Tool Type Best For
Jaeger Open source Self-hosted, Kubernetes-native
Grafana Tempo Open source Integrated with Grafana stack
Zipkin Open source Lightweight, simple setup
AWS X-Ray Managed AWS-native workloads
Datadog APM Commercial Full-stack APM with ML anomaly detection

Correlating Logs, Metrics, and Traces

The real power of observability comes from correlation — using a traceId to jump between tools.

flowchart LR
    Alert["Alert fires:\nPayment error rate > 1%"] --> Grafana["Grafana Dashboard\n(Error rate spike at 14:32)"]
    Grafana --> Kibana["Kibana / Loki\n(Filter logs: level=ERROR, service=payment-service, 14:32)"]
    Kibana --> TraceLink["traceId: 4bf92f3577b34da6"]
    TraceLink --> Jaeger["Jaeger / Tempo\n(Full trace: Payment timed out at Bank API call, 175ms)"]

Investigation workflow:

  1. Alert fires — error rate on payment service exceeded threshold
  2. Grafana — shows exactly when the spike started and which endpoint
  3. Logs — filtered by service=payment-service and level=ERROR reveal the error message and traceId
  4. Trace — the traceId shows the full call chain and which downstream dependency is slow

Without the traceId linking all three pillars, this investigation would take hours instead of minutes.

Configuring Observability Per Environment

Concern Development Staging Production
Log level DEBUG INFO INFO (WARN for libraries)
Log format Human-readable text JSON JSON
Metrics collection Optional Enabled Enabled
Trace sampling 100% 100% 5–10% + 100% of errors
Alerting None Slack only PagerDuty + Slack
Log retention 3–7 days 7–14 days 30–90 days
Dashboards None Basic Full SLO dashboards

Observability for Kubernetes

In a Kubernetes environment, the observability stack must account for ephemeral pods and dynamic scaling.

flowchart TB
    Pod1["Pod: payment-service-7d9f"] --> Sidecar1["Fluent Bit Sidecar"]
    Pod2["Pod: payment-service-3c1a"] --> Sidecar2["Fluent Bit Sidecar"]
    Sidecar1 --> Collector["OTel Collector\n(DaemonSet)"]
    Sidecar2 --> Collector
    Collector --> Loki["Loki (Logs)"]
    Collector --> Prometheus["Prometheus (Metrics)"]
    Collector --> Tempo["Tempo (Traces)"]
    Prometheus --> Grafana
    Loki --> Grafana
    Tempo --> Grafana

Key Kubernetes observability concerns:

  • Pod labels — include app, version, env labels on all pods so logs and metrics can be filtered by deployment version
  • Resource limits — always set CPU and memory limits; Prometheus can alert when pods approach limits
  • Readiness and liveness probes — feed directly into high-availability decisions (covered in Health Checks)
  • Namespace isolation — separate observability stack per namespace for multi-tenant environments

Dashboards — What to Build

A production Grafana setup should have at minimum:

Service Overview Dashboard (per service)

  • Request rate (RPS)
  • Error rate (%)
  • p50 / p95 / p99 latency
  • Active pods / instances
  • JVM memory and GC (if Java)

Infrastructure Dashboard

  • Node CPU and memory utilisation
  • Disk I/O and network throughput
  • Kubernetes pod restart count

Database Dashboard

  • Connection pool usage
  • Active queries
  • Query latency p95
  • Replication lag

Business Dashboard

  • Orders created per minute
  • Payments processed (success vs failed)
  • Active users
  • Revenue-correlated metrics

SLO Dashboard

  • Current SLO compliance (e.g., 99.94% this month)
  • Error budget remaining
  • Time to burn through remaining error budget at current rate

Observability Readiness Checklist

Before deploying a service to production, verify:

Logging

  • Logs are structured JSON with timestamp, level, service, traceId
  • Log level is INFO in production — no DEBUG or TRACE
  • No sensitive data (passwords, card numbers, PII) in logs
  • Logs are shipped to a central store and searchable
  • Log retention policy is defined and enforced

Metrics

  • Service exposes a /metrics (Prometheus) endpoint
  • Four Golden Signals are instrumented (latency, traffic, errors, saturation)
  • Custom business metrics are collected
  • Grafana dashboard exists for the service
  • Alert rules defined for SLO thresholds

Tracing

  • OpenTelemetry SDK is configured
  • traceId appears in all log entries
  • traceId is propagated in outbound HTTP headers and Kafka messages
  • Traces are visible in the trace backend (Jaeger / Tempo)
  • Sampling is configured (100% errors, percentage of healthy requests)

Alerting

  • Alert fires when error rate exceeds threshold
  • Alert fires when p99 latency exceeds SLO
  • Alerts route to the correct team channel
  • Each alert has a runbook link

Summary

Observability is not an afterthought — it is a design requirement for every production service.

Pillar Primary Tool Answers
Logging ELK / Loki What happened and in what context?
Metrics Prometheus + Grafana How is the system performing?
Tracing Jaeger / Tempo Where did this specific request fail?

A well-observable system means:

  • Detection — failures are found by your monitoring, not your customers
  • Diagnosis — root cause is identified in minutes, not hours
  • Recovery — corrective action is taken with confidence
  • Prevention — patterns in data catch problems before they become incidents

Build observability in from day one. Retrofitting it onto a running production system is significantly harder and more expensive.