Instrumenting a Go Service

Wire up a TracerProvider, an OTLP exporter and a resource, then let otelhttp and otelsql create the spans for you.

Syntaxtp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exp), sdktrace.WithResource(res))

Instrumenting a service means wiring three things together: a TracerProvider, which creates tracers; an exporter, which sends finished spans somewhere; and instrumentation, which creates spans for work you did not write by hand. Until a provider is installed, every tracing call still compiles and does nothing. That is deliberate, so a library can be instrumented without forcing an exporter on anyone who imports it.

The pieces

The exporter here speaks OTLP, the OpenTelemetry wire protocol. otlptracehttp.New posts spans over HTTP, by default to localhost:4318, which is where a collector or an agent listens. A resource describes the process the spans came from. service.name is the one attribute you must set, because it is what a trace UI groups spans by. WithBatcher wraps the exporter in a batch processor, so spans queue up and go out in groups instead of one request per span. otel.SetTracerProvider(tp) installs the provider globally, which is what makes otel.Tracer("name") work anywhere in the program.

Instrumenting the edges

Two wrappers cover most of a web service. otelhttp.NewHandler(mux, "orders") starts a span for every inbound request. For the database, otelsql.Open returns an ordinary *sql.DB whose queries create spans, but only when you call the context-taking methods such as QueryContext, because the span needs the request's context to find its parent. otelhttp is an OpenTelemetry contrib module; otelsql is a community package, github.com/XSAM/otelsql. The API modules are at v1 and the instrumentation modules are still at v0, so pin both in go.mod.

Example

Example · go
package main

import (
	"context"
	"fmt"
	"net/http"

	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.37.0"
)

func run(ctx context.Context) error {
	// WithInsecure means plain HTTP to a collector on localhost:4318.
	exp, err := otlptracehttp.New(ctx, otlptracehttp.WithInsecure())
	if err != nil {
		return err
	}
	// service.name is the name you will search for in the trace UI.
	res := resource.NewWithAttributes(semconv.SchemaURL, semconv.ServiceName("orders"))
	tp := sdktrace.NewTracerProvider(sdktrace.WithBatcher(exp), sdktrace.WithResource(res))
	otel.SetTracerProvider(tp)
	// Shutdown flushes the spans still sitting in the batch queue.
	defer func() { _ = tp.Shutdown(ctx) }()

	mux := http.NewServeMux()
	// "GET /healthz" as a pattern needs Go 1.22 or later.
	mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusNoContent)
	})
	fmt.Println("listening on :8080")
	// NewHandler starts one span per request before mux sees it.
	return http.ListenAndServe(":8080", otelhttp.NewHandler(mux, "orders"))
}

func main() {
	if err := run(context.Background()); err != nil {
		fmt.Println("exit:", err)
	}
}

// Output:
// listening on :8080

When to use it

  • Get a timeline for every HTTP request a service handles, without editing each handler.
  • Find out whether a slow endpoint is spending its time in your code or in Postgres.
  • Give every service in a cluster a service.name that the trace UI can group by.
  • Point a service at a local collector on your laptop and at a cluster collector in production.

More examples

Tracing database/sql queries

This is how a trace gains a row per SQL statement without touching any query. The important half is the second function: wrapping the driver is useless if your queries do not carry the request context. Reach for this the first time a handler is slow and you cannot tell whether the time went to your code or the database.

Example · go
// store.go
package store

import (
	"context"
	"database/sql"

	"github.com/XSAM/otelsql"
	_ "github.com/jackc/pgx/v5/stdlib" // registers the "pgx" driver
)

// Open has the same shape as sql.Open and returns a plain *sql.DB, so the
// rest of the code does not change.
func Open(dsn string) (*sql.DB, error) {
	return otelsql.Open("pgx", dsn)
}

// ProductName must take a context. The query span finds its parent through
// ctx; db.QueryRow, which takes no context, would produce a stray root span.
func ProductName(ctx context.Context, db *sql.DB, id int) (string, error) {
	var name string
	err := db.QueryRowContext(ctx, "SELECT name FROM products WHERE id = $1", id).Scan(&name)
	return name, err
}

A hand-written span inside a handler

Automatic instrumentation gives you one span for the whole request, which tells you it took 900ms but not where. Add spans by hand around the few phases you would want to see separately. Note that spans are ended explicitly here when the work is finished rather than at the end of the function, so the timeline shows the real duration of each phase.

Example · go
func (h *handlers) createOrder(w http.ResponseWriter, r *http.Request) {
	// otelhttp already started a span for this request. This one nests inside
	// it, because r.Context() carries that span.
	ctx, span := otel.Tracer("orders").Start(r.Context(), "validate order")
	err := h.validate(ctx, r)
	span.End()
	if err != nil {
		http.Error(w, "invalid order", http.StatusBadRequest)
		return
	}

	// Start from r.Context() again, not from the context of the span that just
	// ended: the two phases are siblings under the request span, and a child of
	// an already-ended span would outlive its parent in the timeline.
	ctx, span = otel.Tracer("orders").Start(r.Context(), "insert order")
	defer span.End()
	if err := h.store.Insert(ctx, r.Body); err != nil {
		span.RecordError(err)
		http.Error(w, "could not save order", http.StatusInternalServerError)
		return
	}
	w.WriteHeader(http.StatusCreated)
}

The same configuration from environment variables

The two forms produce the same provider. Use the environment form for anything you deploy, because the collector address differs between your laptop and a cluster and you do not want a rebuild to change it. Note that resource.NewWithAttributes reads no environment variables at all, which is why OTEL_SERVICE_NAME looks ignored until you switch to resource.New.

Example · go
// Longhand: the endpoint and the service name written in code.
exp, err := otlptracehttp.New(ctx,
	otlptracehttp.WithEndpointURL("http://collector:4318/v1/traces"),
)
if err != nil {
	return err
}
res := resource.NewWithAttributes(semconv.SchemaURL, semconv.ServiceName("orders"))

// Shorthand: the same two values read from the environment. The exporter
// reads OTEL_EXPORTER_OTLP_TRACES_ENDPOINT, falling back to
// OTEL_EXPORTER_OTLP_ENDPOINT. resource.WithFromEnv reads OTEL_SERVICE_NAME
// and OTEL_RESOURCE_ATTRIBUTES.
exp, err = otlptracehttp.New(ctx)
if err != nil {
	return err
}
res, err = resource.New(ctx, resource.WithFromEnv(), resource.WithTelemetrySDK())
if err != nil {
	return err
}

Related

go tracing conceptsgo trace propagationgo middlewarego sql poolgo config env

Discussion

  • Be the first to comment on this lesson.