Backend interview prep

Start Learning

Distributed Tracing on OpenShift

Learn distributed tracing in OpenShift using Spring Boot, OpenTelemetry, Jaeger, and Tempo. Understand Trace IDs, Span IDs, request flow, context propagation, and enterprise microservices observability.

Introduction

In a monolithic application, debugging is straightforward because a request is processed within a single application.

In a microservices architecture, a single user request may travel through multiple services.

Example:

  • API Gateway
  • Authentication Service
  • Customer Service
  • Payment Service
  • Inventory Service
  • Notification Service
  • Database
  • Kafka

If one service becomes slow, how do you identify it?

Logs from one service are not enough.

Metrics show system health but cannot follow an individual request.

This is where Distributed Tracing becomes essential.

Distributed tracing follows a request across every microservice, allowing developers to understand where time is spent and where failures occur.

Learning Objectives

By the end of this article, you will understand:

  • What is Distributed Tracing?
  • Trace ID and Span ID
  • OpenTelemetry Architecture
  • Jaeger Architecture
  • Spring Boot Integration
  • Context Propagation
  • Distributed Tracing in OpenShift
  • Enterprise Best Practices

Why Distributed Tracing?

Consider an online payment.

The request travels through multiple services.

flowchart LR
    A[Client]
    B[API Gateway]
    C[Payment Service]
    D[Fraud Service]
    E[Notification Service]
    F[(Oracle Database)]

    A --> B
    B --> C
    C --> D
    C --> E
    C --> F

If the payment takes 6 seconds, which service caused the delay?

Logs alone cannot answer this.

What is a Trace?

A Trace represents the complete lifecycle of one request.

Example

Trace ID

8a3f12bc45de78

Every microservice handling the request shares the same Trace ID.

What is a Span?

A Span represents one unit of work.

Example

Client Request

↓

API Gateway

↓

Payment Service

↓

Database Query

Each operation creates its own Span.

Trace Architecture

flowchart LR
    A[Client]
    B[API Gateway]
    C[Payment Service]
    D[Fraud Service]
    E[Oracle Database]

    A --> B
    B --> C
    C --> D
    C --> E

One Trace contains multiple Spans.

Trace vs Span

Component Description
Trace Complete request
Span Single operation
Parent Span Calls another Span
Child Span Sub-operation
Trace ID Shared by all Spans
Span ID Unique per Span

OpenTelemetry Architecture

flowchart LR
    A[Spring Boot]
    B[OpenTelemetry SDK]
    C[OpenTelemetry Collector]
    D[Jaeger]
    E[Grafana]

    A --> B
    B --> C
    C --> D
    D --> E

OpenTelemetry generates traces and exports them.

Request Flow

sequenceDiagram
    participant Client
    participant Gateway
    participant Payment
    participant Database

    Client->>Gateway: Request

    Gateway->>Payment: Forward Request

    Payment->>Database: Query

    Database-->>Payment: Response

    Payment-->>Gateway: Success

    Gateway-->>Client: Response

Each call becomes a Span.

Trace Context Propagation

Every request carries tracing headers.

traceparent

00-4bf92f3577b34da6a3ce929d0e0e4736

b7ad6b7169203331

01

These headers ensure every downstream service continues the same Trace.

Spring Boot Architecture

flowchart LR
    A[REST Controller]
    B[Service Layer]
    C[Repository]
    D[(Database)]

    A --> B
    B --> C
    C --> D

Each layer contributes a Span.

Add Dependencies

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>

<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>

Configure Spring Boot

management.tracing.enabled=true

management.tracing.sampling.probability=1.0

management.otlp.tracing.endpoint=http://otel-collector:4318/v1/traces

Trace Flow

flowchart LR
    A[Spring Boot]
    B[OpenTelemetry]
    C[Collector]
    D[Jaeger]

    A --> B
    B --> C
    C --> D

Jaeger Architecture

flowchart LR
    A[Spring Boot Pods]
    B[OpenTelemetry Collector]
    C[Jaeger Collector]
    D[Jaeger Query]
    E[Jaeger UI]

    A --> B
    B --> C
    C --> D
    D --> E

Viewing Traces

Jaeger displays:

  • Trace ID
  • Span Duration
  • Service Name
  • Errors
  • Timeline
  • Parent/Child Relationships

Banking Example

A payment request travels through:

flowchart LR
    A[Customer]
    B[API Gateway]
    C[Payment Service]
    D[Fraud Service]
    E[Kafka]
    F[Notification Service]
    G[(Oracle Database)]

    A --> B
    B --> C
    C --> D
    C --> G
    D --> E
    E --> F

Distributed tracing shows the exact time spent in each component.

Correlation Between Logs and Traces

Every log should include:

  • Trace ID
  • Span ID

Example

INFO

TraceID=8a3f12bc45de78

SpanID=bf1298aa

Payment Created

This allows engineers to correlate logs with traces.

OpenShift Architecture

flowchart TD
    A[Spring Boot Pods]
    B[OpenTelemetry Collector]
    C[Jaeger]
    D[Grafana]
    E[Operations Team]

    A --> B
    B --> C
    C --> D
    D --> E

Useful Commands

View Pods

oc get pods

View Collector

oc get deployment opentelemetry-collector

View Jaeger

oc get route jaeger

Describe Pod

oc describe pod payment-service

Common Problems

No Traces

Verify:

  • OpenTelemetry dependency
  • Collector endpoint
  • Sampling enabled

Missing Trace IDs

Verify context propagation between services.

Empty Jaeger Dashboard

Check:

  • Collector availability
  • Network connectivity
  • Exporter configuration

Partial Traces

Possible causes:

  • One service not instrumented
  • Missing tracing headers
  • Incorrect OpenTelemetry configuration

Production Best Practices

  • Instrument every microservice.
  • Enable OpenTelemetry auto-instrumentation where possible.
  • Include Trace ID in application logs.
  • Propagate trace context across HTTP and messaging.
  • Monitor collector health.
  • Sample intelligently in production.
  • Protect tracing endpoints.
  • Retain traces according to business requirements.
  • Integrate traces with logs and metrics.
  • Use dashboards for root-cause analysis.

Common Mistakes

❌ Instrumenting only one microservice.

❌ Disabling context propagation.

❌ Logging without Trace IDs.

❌ Sending every trace in high-volume production without sampling.

❌ Ignoring asynchronous tracing for Kafka events.

Advantages

  • End-to-end request visibility
  • Faster root-cause analysis
  • Better performance optimization
  • Reduced Mean Time to Resolution (MTTR)
  • Improved microservices observability
  • Better production debugging
  • Enterprise-ready monitoring
  • Complete request lifecycle tracking

Summary

Distributed tracing enables developers to follow a request across every microservice involved in processing it.

Key takeaways:

  • A Trace represents the complete request lifecycle.
  • A Span represents an individual operation within that request.
  • OpenTelemetry generates and exports traces.
  • Jaeger visualizes request execution across services.
  • Trace IDs should be included in application logs for complete observability.
  • Combining logs, metrics, and traces provides a comprehensive observability solution for Spring Boot applications running on OpenShift.

Interview Questions

  1. What is Distributed Tracing?
  2. What is the difference between a Trace and a Span?
  3. What is a Trace ID?
  4. What is a Span ID?
  5. Why is OpenTelemetry preferred over vendor-specific libraries?
  6. How does context propagation work?
  7. What role does Jaeger play?
  8. Why should logs include Trace IDs?
  9. How do traces complement logs and metrics?
  10. What are the best practices for distributed tracing in OpenShift?