Making It Observable

Give the orders API traces, metrics and structured logs that share one trace id, plus liveness and readiness health checks.

Syntax

using var activity = telemetry.Source.StartActivity("PlaceOrder");
telemetry.OrdersPlaced.Add(1, new KeyValuePair<string, object?>("channel", "web"));
app.MapHealthChecks("/health/ready", new() { Predicate = c => c.Tags.Contains("ready") });

When a customer reports that order placement hung at 14:02, the on-call engineer needs three things: a trace showing where the time went, a metric showing whether it was one request or everyone, and logs for that request that can be found by trace id. .NET emits all three through platform APIs (ActivitySource, Meter, ILogger); OpenTelemetry collects them and exports them over OTLP to whatever backend the company runs.

Traces

ASP.NET Core starts an Activity for every request on the source Microsoft.AspNetCore, HttpClient does the same for outgoing calls, and Npgsql emits one per command on its Npgsql source. The project adds one source, Orders.Api, for business operations such as PlaceOrder, created once as a singleton. StartActivity returns null when nothing listens, so always use ?. on it. Child activities inherit the current trace id through Activity.Current, and HttpClient forwards it in the traceparent header, which is how one trace spans the API, the payment service and the database.

Metrics

Built-in meters already give request duration (http.server.request.duration), active requests and Kestrel connections. Add business counters: orders.placed, orders.payment_declined. Tags must have bounded values: channel=web is fine, customer_id creates one time series per customer and can take down the metrics backend.

Structured logs

Log with message templates through the [LoggerMessage] source generator: placeholders become fields (OrderId, CustomerId), no string is built when the level is disabled, and the event id is stable for alerts. The host adds TraceId and SpanId to the logging scope, so with scopes included every log line carries the trace it belongs to. In containers, write JSON to stdout or export logs over OTLP; never write log files inside the container.

Health checks

Endpoint	Checks	Kubernetes uses it to	Failure means
`/health/live`	The process answers; no dependencies	Restart the container	Deadlocked or wedged process
`/health/ready`	PostgreSQL reachable, migrations applied	Remove the pod from the Service	Cannot serve right now; do not restart

The classic outage is a database check on the liveness probe: PostgreSQL has a two-minute failover, every pod fails liveness, Kubernetes restarts all of them, and the API comes back slower than the database did. Tag checks and filter each endpoint with Predicate. Give the readiness check a short timeout, exclude both endpoints from tracing so probes do not drown real traffic, and keep them anonymous but free of version numbers or connection details.

Choices the team makes

Export OTLP to a collector rather than to a vendor directly, so changing vendors is a collector config change. Sample traces at the collector (tail sampling keeps every error and a fraction of successes) rather than dropping them in the app. Keep ILogger in the code even if Serilog is chosen as the provider.

Example

Loading editor…

Press Run to execute the code.

When to use it

Find every log line for the order placement that took 9 seconds by searching the trace id shown in the tracing UI.
Take a pod out of the Service during a PostgreSQL failover through /health/ready without Kubernetes restarting it through /health/live.
Alert when the orders.placed counter drops to zero for ten minutes during business hours, a symptom HTTP error rates can miss.

More examples

The production OpenTelemetry wiring

<code>UseOtlpExporter()</code> sends traces, metrics and logs to one OTLP endpoint, normally a collector next to the pod; the endpoint comes from <code>OTEL_EXPORTER_OTLP_ENDPOINT</code>, so the same image works under Aspire locally and in the cluster. The filter keeps probe requests out of traces: two probes every ten seconds on twelve pods otherwise outnumber real orders overnight. The <code>Npgsql</code> source adds one span per SQL command without an extra package.

using OpenTelemetry;
using OpenTelemetry.Metrics;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource.AddService(
        serviceName: "orders-api",
        serviceVersion: typeof(Program).Assembly.GetName().Version?.ToString()))
    .WithTracing(tracing => tracing
        .AddSource("Orders.Api")
        .AddSource("Npgsql")
        .AddAspNetCoreInstrumentation(o => o.Filter = http => !http.Request.Path.StartsWithSegments("/health"))
        .AddHttpClientInstrumentation())
    .WithMetrics(metrics => metrics
        .AddMeter("Orders.Api")
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation())
    .UseOtlpExporter();   // reads OTEL_EXPORTER_OTLP_ENDPOINT, defaults to localhost:4317

builder.Logging.AddOpenTelemetry(logging =>
{
    logging.IncludeScopes = true;
    logging.IncludeFormattedMessage = true;
});

var app = builder.Build();
app.Run();

public partial class Program;

A real PostgreSQL readiness check with a JSON body

<code>AddNpgSql</code> from AspNetCore.HealthChecks.NpgSql runs <code>SELECT 1;</code> on a pooled connection. Keep the check timeout below the probe's <code>timeoutSeconds</code> (1 second by default; the Deployment in the shipping lesson sets 3), so the check reports Unhealthy before the kubelet gives up on the request, instead of waiting on a dead TCP connection for thirty seconds. The writer reports names, statuses and durations only, never exception messages, which can include host names. <code>AllowAnonymous()</code> exempts both endpoints from the fallback policy.

using System.Text.Json;
using Microsoft.AspNetCore.Diagnostics.HealthChecks;
using Microsoft.Extensions.Diagnostics.HealthChecks;

var builder = WebApplication.CreateBuilder(args);
var connectionString = builder.Configuration.GetConnectionString("Orders")!;

builder.Services.AddHealthChecks()
    .AddCheck("self", () => HealthCheckResult.Healthy(), tags: ["live"])
    .AddNpgSql(connectionString, name: "postgres", tags: ["ready"], timeout: TimeSpan.FromSeconds(2));

var app = builder.Build();

app.MapHealthChecks("/health/live", new HealthCheckOptions { Predicate = c => c.Tags.Contains("live") }).AllowAnonymous();
app.MapHealthChecks("/health/ready", new HealthCheckOptions
{
    Predicate = c => c.Tags.Contains("ready"),
    ResponseWriter = async (http, report) =>
    {
        http.Response.ContentType = "application/json";
        await JsonSerializer.SerializeAsync(http.Response.Body, new
        {
            status = report.Status.ToString(),
            checks = report.Entries.Select(e => new { name = e.Key, status = e.Value.Status.ToString(), ms = (int)e.Value.Duration.TotalMilliseconds }),
        });
    },
}).AllowAnonymous();

app.Run();

Log levels for production

<code>Microsoft.AspNetCore</code> at Warning removes the per-request Information lines that tracing already covers, often the largest share of log volume. EF Core's command category logs SQL text at Information; keep it at Warning in production and raise it with an environment variable such as <code>Logging__LogLevel__Microsoft.EntityFrameworkCore.Database.Command=Information</code> while investigating. Scopes must be on for the trace id to appear in each line.

// appsettings.Production.json
{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning",
      "Microsoft.EntityFrameworkCore.Database.Command": "Warning",
      "Orders.Api": "Information"
    },
    "Console": {
      "FormatterName": "json",
      "FormatterOptions": { "IncludeScopes": true, "UseUtcTimestamp": true, "TimestampFormat": "yyyy-MM-ddTHH:mm:ss.fffZ" }
    }
  }
}

Discussion

Be the first to comment on this lesson.