Sampling and Cost

Head sampling decides before the request runs; tail sampling decides in the collector, once the errors are visible.

Syntaxsdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1)))

A busy service produces far more spans than anyone will look at, and storing traces costs money per span. Sampling means keeping some traces and throwing the rest away. The whole question is which ones.

Head sampling

Head sampling decides at the start, when the root span is created, before anything is known about how the request went. TraceIDRatioBased(0.1) keeps a tenth of traces by looking at the trace ID itself, so every service that sees the same trace ID reaches the same answer. Wrap it in ParentBased so that a span which arrived with a parent follows the decision carried in the traceparent flags instead of deciding again. Without that wrapper, half of a trace's spans can go missing.

A dropped span is cheap but not free. The code still calls Start, and the span returned is non-recording. span.IsRecording() reports false for it, so guard any expensive attribute building with that check.

Tail sampling

Head sampling cannot keep all the errors, because at the start no error has happened yet. Tail sampling moves the decision to the OpenTelemetry Collector, which buffers a trace's spans for a few seconds and then applies policies: keep anything with an error status, keep anything slower than 500ms, keep a small percentage of the rest.

What trips people up

Tail sampling needs every span of a trace to reach the same collector instance. Behind a load balancer that means a first collector layer that routes by trace ID. Otherwise each collector sees fragments and the policies keep the wrong things.

Example

Example · go
package main

import (
	"context"
	"fmt"

	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	"go.opentelemetry.io/otel/trace"
)

func main() {
	// Drop traces that begin in this service; keep any trace whose caller
	// already decided to sample it.
	tp := sdktrace.NewTracerProvider(
		sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.NeverSample())),
	)
	tracer := tp.Tracer("orders")

	// No parent, so the root sampler decides: NeverSample.
	_, local := tracer.Start(context.Background(), "cron tick")
	fmt.Println("local root recorded:", local.IsRecording())
	local.End()

	// Any non-zero IDs stand in for what a real traceparent header carries.
	// FlagsSampled is the "-01" on the end of that header.
	remote := trace.NewSpanContext(trace.SpanContextConfig{
		TraceID:    trace.TraceID{1},
		SpanID:     trace.SpanID{1},
		TraceFlags: trace.FlagsSampled,
		Remote:     true,
	})
	ctx := trace.ContextWithRemoteSpanContext(context.Background(), remote)
	_, child := tracer.Start(ctx, "handle request")
	fmt.Println("remote child recorded:", child.IsRecording())
	child.End()
}

// Output:
// local root recorded: false
// remote child recorded: true

When to use it

  • Cut a tracing bill by keeping 5% of normal traffic and every failed request.
  • Make sure a trace that crosses six services is complete rather than half sampled.
  • Keep every trace for a newly launched endpoint for its first week in production.
  • Stop a traffic spike from overwhelming the collector and the trace backend.

More examples

A ratio sampler, in code and as environment variables

The two forms are equivalent, and the environment form is the one to deploy, because the ratio you want on a laptop is 1.0 and the ratio you want under production traffic is not. One catch: WithSampler overrides OTEL_TRACES_SAMPLER, so a hardcoded sampler makes the variables look broken.

Example · go
// Longhand: keep 10% of the traces that start in this service, decided here.
tp := sdktrace.NewTracerProvider(
	sdktrace.WithBatcher(exp),
	sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1))),
)

// Shorthand: leave WithSampler out and set two environment variables. The SDK
// builds the same sampler from them.
//
//	OTEL_TRACES_SAMPLER=parentbased_traceidratio
//	OTEL_TRACES_SAMPLER_ARG=0.1
//
// With neither the option nor the variables, the default is
// ParentBased(AlwaysSample), which keeps everything.
tp = sdktrace.NewTracerProvider(sdktrace.WithBatcher(exp))

Tail sampling policies in the collector

Policies are an OR: a trace is kept if any policy says keep. This is the configuration most teams end up with, because it keeps every failure and every slow request while paying for only a twentieth of the healthy traffic. Note that tail_sampling must come before batch in the pipeline, and that it needs memory for num_traces buffered traces.

Example · yaml
# The processors section of an OpenTelemetry Collector config.
processors:
  tail_sampling:
    # Hold a trace's spans this long before deciding. Any span arriving later
    # is judged on its own and usually dropped.
    decision_wait: 10s
    # How many in-flight traces to keep buffered in memory.
    num_traces: 50000
    policies:
      - name: errors
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow
        type: latency
        latency:
          threshold_ms: 500
      - name: the-rest
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

service:
  pipelines:
    traces:
      processors: [tail_sampling, batch]

A custom sampler that always keeps one endpoint

A Sampler is an interface with two methods, so you can write your own and wrap an existing one. Reach for this when one endpoint matters more than the rest: a payment path, or a new feature you want full traces for during its first week. The decision is still made at the head, so it can only use what is known at Start: the span name, kind, initial attributes and parent.

Example · go
// keepPayments samples every trace that starts with a payment request, and
// delegates every other decision to the sampler it wraps.
type keepPayments struct {
	rest sdktrace.Sampler
}

func (k keepPayments) ShouldSample(p sdktrace.SamplingParameters) sdktrace.SamplingResult {
	if strings.HasPrefix(p.Name, "POST /payments") {
		return sdktrace.SamplingResult{
			Decision: sdktrace.RecordAndSample,
			// Pass the caller's tracestate through unchanged.
			Tracestate: trace.SpanContextFromContext(p.ParentContext).TraceState(),
		}
	}
	return k.rest.ShouldSample(p)
}

func (k keepPayments) Description() string {
	return "keepPayments{" + k.rest.Description() + "}"
}

// Used like any built-in sampler.
var sampler sdktrace.Sampler = keepPayments{
	rest: sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.05)),
}

Related

go tracing conceptsgo otel setupgo trace propagationgo metricsgo load shedding

Discussion

  • Be the first to comment on this lesson.