Propagating Context Across Services

Write the traceparent header on every outbound call so the next service continues your trace instead of starting its own.

Syntaxotel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(req.Header))

Two services each start their own trace unless the first one tells the second what it was doing. Propagation is that telling: the caller writes its trace ID and current span ID into the outbound request, and the receiver reads them back out. This single step is the difference between a pile of separate per-service traces and one tree for the whole request.

The traceparent header

The standard carrier for HTTP is the W3C Trace Context header traceparent. A value looks like 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01. That is a version, then the 32 hex digit trace ID, then the 16 hex digit span ID of the caller's span, then flags, where 01 means the trace was sampled. A companion header, tracestate, carries extra vendor data.

Injecting and extracting

A propagator reads and writes those headers. Inject takes a context and writes the headers out. Extract takes the headers and returns a context holding the caller's span context, and a span started from that context becomes a child of the caller's span. A carrier is the thing being written to: propagation.HeaderCarrier wraps an http.Header, and propagation.MapCarrier wraps a map[string]string, which is how a trace travels through a queue message.

What trips people up

Set the global propagator once at startup. The Go SDK installs none, so Inject writes nothing at all until you do. Build outbound requests with http.NewRequestWithContext: http.NewRequest gives the request an empty context, so there is no span to inject and the header is left off.

Example

Example · go
package main

import (
	"context"
	"fmt"
	"net/http"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/propagation"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	"go.opentelemetry.io/otel/trace"
)

func main() {
	// Without this line the global propagator does nothing at all.
	otel.SetTextMapPropagator(propagation.TraceContext{})
	otel.SetTracerProvider(sdktrace.NewTracerProvider())

	// Service A starts a span and builds the outbound request.
	ctx, span := otel.Tracer("a").Start(context.Background(), "call b")
	defer span.End()
	req, err := http.NewRequestWithContext(ctx, http.MethodGet, "http://b/items", nil)
	if err != nil {
		panic(err)
	}
	otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(req.Header))
	fmt.Println("traceparent sent:", req.Header.Get("traceparent") != "")

	// Service B reads the header back out of the request it received.
	bCtx := otel.GetTextMapPropagator().Extract(context.Background(), propagation.HeaderCarrier(req.Header))
	same := trace.SpanContextFromContext(bCtx).TraceID() == span.SpanContext().TraceID()
	fmt.Println("same trace:", same)
}

// Output:
// traceparent sent: true
// same trace: true

When to use it

  • Make a checkout request show as one trace across the API, the payment service and the worker.
  • Keep a trace alive across a queue so the job that runs later still belongs to the request that queued it.
  • Work out which upstream caller is responsible for a burst of failing requests.
  • Pass a trace through a third-party gateway that only forwards known headers.

More examples

An HTTP client that propagates on its own

This is the shorthand for the whole lesson: wrap the transport once and every outbound call is propagated. Use it for all service-to-service HTTP. The comment on getItem is the part worth remembering, because a missing context produces a trace that silently ends at the caller with no error anywhere.

Example · go
// One client for the whole service. NewTransport injects the traceparent
// header on every request it sends, and creates a span for the call.
var client = &http.Client{
	Transport: otelhttp.NewTransport(http.DefaultTransport),
}

// getItem must use NewRequestWithContext. The transport reads the span out of
// the request's context, and http.NewRequest leaves that context empty, so
// the header would be missing and the trace would stop here.
func getItem(ctx context.Context, id string) (*http.Response, error) {
	req, err := http.NewRequestWithContext(ctx, http.MethodGet, "http://inventory/items/"+id, nil)
	if err != nil {
		return nil, err
	}
	return client.Do(req)
}

Carrying a trace through a queue message

Propagation is not an HTTP feature. MapCarrier is a plain map[string]string, so any transport with string headers works: SQS message attributes, Kafka headers, a JSON field. Reach for this when a background worker's spans appear as orphan traces with no sign of the request that queued the job.

Example · go
// Producer: copy the trace into the message's headers before sending.
func publish(ctx context.Context, q Queue, body []byte) error {
	ctx, span := otel.Tracer("orders").Start(ctx, "publish order.created",
		trace.WithSpanKind(trace.SpanKindProducer))
	defer span.End()

	headers := propagation.MapCarrier{}
	otel.GetTextMapPropagator().Inject(ctx, headers)
	return q.Send(ctx, Message{Headers: headers, Body: body})
}

// Consumer: rebuild the context from those headers, then nest under it.
func handle(ctx context.Context, m Message) error {
	ctx = otel.GetTextMapPropagator().Extract(ctx, propagation.MapCarrier(m.Headers))
	ctx, span := otel.Tracer("worker").Start(ctx, "handle order.created",
		trace.WithSpanKind(trace.SpanKindConsumer))
	defer span.End()
	return process(ctx, m.Body)
}

Server-side extraction, shorthand and longhand

These two do the same job, and seeing the long form explains why the short one works. Use the wrapper in real services; use the long form when you need a span name that otelhttp cannot produce, or when the incoming request came from somewhere else entirely. Naming spans by raw URL path, as here, creates one span name per ID, so prefer the route pattern in production.

Example · go
// Shorthand, one line where you build the server:
//
//	srv.Handler = otelhttp.NewHandler(mux, "orders")
//
// Longhand: the same two steps written out. Worth reading once, and worth
// writing when a request did not arrive over net/http.
func tracing(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		// Step one: read traceparent into a context.
		ctx := otel.GetTextMapPropagator().Extract(r.Context(), propagation.HeaderCarrier(r.Header))
		// Step two: start a server span as a child of what Extract found.
		ctx, span := otel.Tracer("orders").Start(ctx, r.Method+" "+r.URL.Path,
			trace.WithSpanKind(trace.SpanKindServer))
		defer span.End()
		// Handlers downstream must see the new context.
		next.ServeHTTP(w, r.WithContext(ctx))
	})
}

Related

go tracing conceptsgo otel setupgo context httpgo messaging basicsgo grpc interceptors

Discussion

  • Be the first to comment on this lesson.