Tracing with OpenTelemetry
Follow a request across services with distributed tracing, export spans over OTLP, and put trace IDs in your logs.
spring-boot-starter-opentelemetry
management.opentelemetry.tracing.export.otlp.endpoint=http://collector:4318/v1/traces
management.tracing.sampling.probability=1.0
Observation.createNotStarted("name", registry).observe(() -> work());Metrics tell you that checkout got slow. A trace tells you why: this request spent 40 ms in the controller, 30 ms in the database and 2.1 seconds waiting for the stock service. A trace is the whole journey of one request; each step is a span with a start, a duration and attributes. Trace context travels between services in the W3C traceparent HTTP header, so spans from different applications join into one tree.
The pieces in Spring Boot
Spring instruments code through the Micrometer Observation API, and Micrometer Tracing bridges observations to OpenTelemetry. Spring Boot 4 has a starter for this: spring-boot-starter-opentelemetry. It brings the OpenTelemetry bridge and the OTLP exporters, so spans can be sent to any collector or backend that speaks OTLP, such as Jaeger, Grafana Tempo or a vendor agent. Incoming requests and calls made with RestClient and WebClient built from the auto-configured builders are traced without any code. Kafka and RabbitMQ observations are off by default: turn them on with spring.kafka.listener.observation-enabled, spring.kafka.template.observation-enabled and spring.rabbitmq.listener.simple.observation-enabled. Spring Boot does not create spans for JDBC queries on its own; that needs an extra library such as datasource-micrometer.
management.opentelemetry.tracing.export.otlp.endpoint=http://otel-collector:4318/v1/traces
management.tracing.sampling.probability=0.1Sampling
The default sampling probability is 0.1: one trace in ten is recorded. That surprises people testing locally who see only some requests. Set 1.0 in development. In production, tracing every request costs CPU, network and storage, so most teams keep a low rate or sample in the collector. The decision is made once at the first service and passed downstream, so a trace is either complete or absent.
Propagation needs the injected builder
Outgoing calls carry the trace header only when the client is instrumented. Build RestClient from the auto-configured RestClient.Builder bean; RestClient.create() creates a client Spring Boot never sees, and the trace silently stops at your service. The same applies to WebClient.Builder.
Async work
Trace context lives in thread-local state. Work handed to another thread through @Async loses it unless the executor propagates context. Set spring.task.execution.propagate-context=true and Spring Boot decorates its task executor so the current observation follows the task.
Logs and traces together
With tracing active Spring Boot adds the trace and span IDs to every log line, and structured log formats include them as fields. During an incident you copy a trace ID from a slow trace and find every log line from every service that handled that request. This is the main day-to-day payoff of tracing, so check it works before you need it.
Custom spans come from the same Observation API used for metrics: wrap the work in Observation.createNotStarted(...).observe(...). Put IDs in high cardinality key values so they land on the span and not in metric tags.
Example
import io.micrometer.observation.Observation;
import io.micrometer.observation.ObservationRegistry;
import org.springframework.stereotype.Service;
import org.springframework.web.client.RestClient;
record StockLevel(String sku, int available) {
}
public class StockClient {
private final RestClient http;
private final ObservationRegistry observations;
// The injected builder is instrumented, so the traceparent header is added to every call.
StockClient(RestClient.Builder builder, ObservationRegistry observations) {
this.http = builder.baseUrl("http://stock-service").build();
this.observations = observations;
}
public StockLevel level(String sku) {
return Observation.createNotStarted("stock.lookup", observations)
.highCardinalityKeyValue("stock.sku", sku)
.observe(() -> http.get()
.uri("/stock/{sku}", sku)
.retrieve()
.body(StockLevel.class));
}
}
When to use it
- Find which of five downstream services adds two seconds to a checkout request.
- Return a trace ID in error responses so support can locate the exact request in logs and traces.
- Keep one trace across an HTTP request that hands work to an @Async method and a Kafka listener.
More examples
Tracing configuration
<code>spring.application.name</code> becomes the service name shown in the tracing UI, so set it in every service. The endpoint uses port 4318, the conventional OTLP over HTTP port; check what your collector actually listens on. The two Kafka properties are off by default; without them the trace ends when a message is sent and a new, unrelated trace starts in the listener.
spring.application.name=orders
# Send spans to an OpenTelemetry collector over OTLP/HTTP.
management.opentelemetry.tracing.export.otlp.endpoint=http://otel-collector:4318/v1/traces
# Record every trace while developing; lower this in production.
management.tracing.sampling.probability=1.0
# Keep the trace when work moves to @Async threads.
spring.task.execution.propagate-context=true
# Carry the trace through Kafka: producer side and listener side.
spring.kafka.template.observation-enabled=true
spring.kafka.listener.observation-enabled=true
# Extra resource attributes attached to every span.
management.opentelemetry.resource-attributes.deployment.environment=staging
Returning the trace ID to clients
When a customer reports an error, the trace ID from the response header takes support straight to the trace and the matching log lines. The ID reveals nothing about your system, so it is safe to return. The header is set before the chain runs because headers cannot be added once the response is committed.
import io.micrometer.tracing.Span;
import io.micrometer.tracing.Tracer;
import jakarta.servlet.FilterChain;
import jakarta.servlet.ServletException;
import jakarta.servlet.http.HttpServletRequest;
import jakarta.servlet.http.HttpServletResponse;
import java.io.IOException;
import org.springframework.stereotype.Component;
import org.springframework.web.filter.OncePerRequestFilter;
@Component
public class TraceIdHeaderFilter extends OncePerRequestFilter {
private final Tracer tracer;
TraceIdHeaderFilter(Tracer tracer) {
this.tracer = tracer;
}
@Override
protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain chain)
throws ServletException, IOException {
Span span = tracer.currentSpan();
if (span != null) {
response.setHeader("X-Trace-Id", span.context().traceId());
}
chain.doFilter(request, response);
}
}
A collector for local development
Jaeger accepts OTLP directly, so a single container is enough to see traces on a laptop: run it with Docker Compose, point the OTLP endpoint at <code>http://localhost:4318/v1/traces</code> and open the UI on port 16686. In production the application usually sends to an OpenTelemetry collector that forwards to the real backend.
services:
jaeger:
image: jaegertracing/jaeger:2.5.0
ports:
- "16686:16686"
- "4318:4318"
Discussion