Logs, Metrics and Traces

Metrics say when, logs say what, traces say where — and a propagated trace id is what turns the three into one dataset.

Syntaxtraceparent: 00-<32 hex trace-id>-<16 hex span-id>-<2 hex flags>

Three signals exist because each has a different cost curve. A metric is a number aggregated over a time window: fixed size regardless of traffic, so a year of it is affordable. A log is one record per event, so cost grows with requests. A trace is one request's path through the system, each hop a span carrying a start, a duration and a parent id.

What each answers

Signal	Answers	Cannot answer
Metrics	Is it broken, since when, how badly	Which request, and why
Logs	What this process decided, with the arguments	Where the time went across nine services
Traces	Which hop was slow and what it called	How often — traces are sampled

None is sufficient alone. The four golden signals in the Google SRE book — latency, traffic, errors, saturation — are metrics, and they are what pages you; the other two are what you open afterwards.

Correlation

The join key is a trace id propagated on every outbound call. W3C Trace Context, a Recommendation since 23 November 2021, standardises it as a traceparent request header: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01 — version, 32-hex trace id, 16-hex span id, flags. Put that id in every log line and attach it to metric samples as an exemplar (Prometheus stores exemplars behind --enable-feature=exemplar-storage, added in 2.26) and a p99 spike becomes a path to one trace, then to that request's logs.

What trips people up

A service that mints a fresh trace id instead of continuing the incoming one breaks the chain silently. Every service's traces look correct in isolation, and no request is ever visible end to end.

Example

// Express middleware: one trace id ties this request's span, logs and metric
// sample together. Needs @opentelemetry/api plus a configured SDK exporter.
import { trace, context, propagation, metrics, SpanStatusCode } from '@opentelemetry/api';
import pino from 'pino';

const tracer = trace.getTracer('checkout-api', '1.4.0');
const meter = metrics.getMeter('checkout-api');
// A histogram, not a gauge of the mean: buckets are what make p99 computable.
const duration = meter.createHistogram('http.server.request.duration', { unit: 'ms' });
const log = pino();

export function observability(req, res, next) {
  // Continue the caller's trace if it sent a traceparent header, start a new
  // one if it did not. propagation.extract reads W3C Trace Context for you.
  const parent = propagation.extract(context.active(), req.headers);

  // SHORTHAND: startActiveSpan creates the span AND makes it the active
  // context for the duration of the callback.
  tracer.startActiveSpan('http.server', {}, parent, (span) => {
    // LONGHAND - exactly the same thing, written out:
    //   const span = tracer.startSpan('http.server', {}, parent);
    //   context.with(trace.setSpan(parent, span), () => { /* ... */ span.end(); });

    const { traceId, spanId } = span.spanContext();
    // The join key. Every log line in this request now carries it.
    req.log = log.child({ trace_id: traceId, span_id: spanId });
    // Returned so support can quote it from a customer's failed request.
    res.setHeader('traceparent', `00-${traceId}-${spanId}-01`);

    const started = process.hrtime.bigint();
    res.on('finish', () => {
      const ms = Number(process.hrtime.bigint() - started) / 1e6;
      // Low-cardinality attributes only: the route TEMPLATE, never req.url.
      const attrs = {
        'http.route': req.route?.path ?? 'unmatched',
        'http.response.status_code': res.statusCode,
      };
      duration.record(ms, attrs);
      span.setAttributes(attrs);
      if (res.statusCode >= 500) span.setStatus({ code: SpanStatusCode.ERROR });
      req.log.info({ ms, status: res.statusCode }, 'request complete');
      span.end();
    });

    next();
  });
}

When to use it

A customer reports a checkout failure at 14:12 with an order id, and you need the exact path that one request took.
Log ingest cost is growing faster than traffic and you have to decide what to stop emitting.
Adding a new service and wiring it so its spans join existing traces instead of starting fresh ones.
A latency panel spikes with no matching error rate, and you need to find which hop absorbed the time.

More examples

Pivoting from a metric to a trace to the logs

The three signals are only worth their cost if you can move between them in one direction: alert on the metric, narrow the window and the route, jump to a trace through an exemplar, then read that request's logs. Reach for this sequence when a latency panel moves and you have no idea which dependency caused it — it replaces the usual approach of grepping logs by timestamp and hoping.

# 1. METRIC - when and where. Error ratio per route, five-minute windows.
sum by (route) (rate(http_server_requests_total{status=~"5.."}[5m]))
  / sum by (route) (rate(http_server_requests_total[5m]))

# 2. METRIC - latency as a distribution for the same route, not an average.
#    Aggregate the buckets first, then take the quantile exactly once.
histogram_quantile(
  0.99,
  sum by (le, route) (rate(http_server_request_duration_seconds_bucket[5m]))
)

# 3. TRACE - ask for the exemplars attached to those buckets. Each exemplar
#    carries the trace_id of one real request that landed in that bucket.
#    GET /api/v1/query_exemplars
#        ?query=http_server_request_duration_seconds_bucket{route="/checkout"}
#        &start=2026-10-01T14:10:00Z&end=2026-10-01T14:20:00Z
#    Requires --enable-feature=exemplar-storage on the Prometheus server.

# 4. LOGS - take that trace_id to the log store (LogQL here). One request,
#    every service it touched, in order.
#    {app=~"checkout|pricing|tax"} | json | trace_id = "4bf92f3577b34da6a3ce929d0e0e4736"

One collector pipeline per signal

Pipelines are per-signal, which is the point: the same attribute can be dropped from metrics and kept on spans, and each backend can be swapped without touching application code. Use this shape when the application already emits OTLP and you want cardinality control, batching and vendor choice to live in infrastructure rather than in a redeploy.

# otel-collector-config.yaml - the app speaks OTLP once; the collector fans
# the three signals out to three backends, so the trace id attached in code
# survives into all of them.
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 5s
  # Strip the attributes that would explode metric cardinality BEFORE export.
  # Spans keep them; only the metrics pipeline uses this processor.
  attributes/scrub:
    actions:
      - key: user.id
        action: delete
      - key: session.id
        action: delete

exporters:
  prometheusremotewrite:
    # Prometheus needs --web.enable-remote-write-receiver for this endpoint.
    endpoint: http://prometheus:9090/api/v1/write
  otlp/traces:
    endpoint: tempo:4317
    tls:
      insecure: true
  otlphttp/logs:
    endpoint: http://loki:3100/otlp

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [attributes/scrub, batch]
      exporters: [prometheusremotewrite]
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/traces]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/logs]

Deciding what belongs in a log line

Shows the two decisions that set your log bill: structure (fields, not prose) and volume (tie verbose logging to the trace sampling decision, so a sampled request has full detail and the rest cost nothing). Reach for it when log ingest is growing faster than traffic, or when an incident needed a field nobody thought to emit.

import { trace, TraceFlags } from '@opentelemetry/api';

// One object per line, not a sentence: the log store indexes fields, and
// "user 41 failed" cannot be queried, filtered or counted.
export function logContext(req) {
  const span = trace.getActiveSpan();
  const ctx = span?.spanContext();
  return {
    trace_id: ctx?.traceId,          // the join key, in every single line
    span_id: ctx?.spanId,
    route: req.route?.path,
    tenant: req.tenant.id,           // fine in a log, fatal as a metric label
    // No card numbers, tokens, auth headers or full request bodies: logs are
    // replicated, retained for weeks, and readable by more people than prod.
  };
}

// Debug-level detail only for requests the tracer already decided to sample.
// Keeps the volume proportional to the sampling rate instead of to traffic.
export function debugIfSampled(req, payload, message) {
  const ctx = trace.getActiveSpan()?.spanContext();
  const sampled = ctx ? (ctx.traceFlags & TraceFlags.SAMPLED) !== 0 : false;
  if (sampled) req.log.debug({ ...payload, ...logContext(req) }, message);
}

Discussion

Be the first to comment on this lesson.