Alerting on the Right Thing

Alert on failed and slow requests rather than CPU, state the action in the alert, and point it at a runbook.

Syntaxburn rate = (failures / requests) / (1 - objective)

An alert is a rule that interrupts a human. That makes it expensive, and it means the bar is high: an alert should fire when something a user would notice is wrong, and not otherwise. Every alert that fires and needs no action teaches the person on call to ignore the next one.

Symptoms, not causes

CPU at 90% is a cause. It might mean trouble, or it might be a nightly import doing exactly what it should. The symptoms are what users feel: requests failing, requests slow, work not getting done. Four rules cover most applications - error ratio, latency at a high percentile, consumer lag on a Messenger queue, and staleness of a job that must run. Resource graphs still belong on the dashboard; they explain an incident, they do not define one.

Error budget and burn rate

Pick a target: 99.9% of requests succeed over thirty days. The 0.1% left is the error budget. Burn rate is how fast you are spending it: a burn rate of 1 spends the budget exactly over the window, 14.4 spends 2% of a thirty-day budget in an hour. Alerting on burn rate instead of a raw percentage gives you one rule that pages for a sudden outage and stays quiet during a slow trickle you can fix on Monday.

What trips people up

404s. Scanners generate thousands, and an alert built on log volume will page you for them. Symfony's prod configuration already excludes 404 and 405 from triggering the fingers_crossed handler; keep them out of the alerting path too.

Example

Example · php
// src/Observability/ErrorBudget.php
declare(strict_types=1);

namespace App\Observability;

final class ErrorBudget
{
    /**
     * @param float $objective fraction of requests that must succeed, e.g. 0.999
     */
    public function __construct(private readonly float $objective)
    {
    }

    public function burnRate(int $requests, int $failures): float
    {
        if ($requests < 1) {
            return 0.0; // no traffic is not the same as no failures
        }

        $allowed = 1.0 - $this->objective; // 0.001 of requests may fail

        return ($failures / $requests) / $allowed;
    }

    public function pages(int $requests, int $failures): bool
    {
        // 14.4x over an hour spends 2% of a 30-day budget: the usual
        // threshold for "wake someone up" rather than "open a ticket".
        return $this->burnRate($requests, $failures) >= 14.4;
    }
}

// Usage, from a test or a console command:
$budget = new ErrorBudget(0.999);

echo $budget->burnRate(20_000, 40), "\n";
var_dump($budget->pages(20_000, 40));

echo $budget->burnRate(20_000, 300), "\n";
var_dump($budget->pages(20_000, 300));

// Output:
// 2
// bool(false)
// 15
// bool(true)

When to use it

  • Replace a CPU alert that pages during every nightly import with one that pages when requests fail.
  • Decide whether a rising error rate is worth waking someone for, or worth a ticket.
  • Catch a Messenger backlog before customers notice their confirmation emails never arrived.
  • Give the person on call a first command to run instead of a graph to stare at.

More examples

A rule that names the action

Two things make these usable at three in the morning. The ratio is computed from the same counter in both halves, so a traffic drop does not change it. And each alert carries an action and a link, which is why the rules live in the repository next to the manifests rather than being typed into a web form: they are reviewed like code. The second rule has severity ticket, not page, because a backlog is urgent in hours, not minutes.

Example · yaml
# deploy/prometheus/rules/storefront.yaml
groups:
  - name: storefront-availability
    rules:
      - alert: StorefrontHighErrorRate
        # app_http_requests_total is the counter from the metrics lesson:
        # namespace 'app' plus name 'http_requests_total'.
        expr: |
          sum(rate(app_http_requests_total{status=~"5.."}[5m]))
            / sum(rate(app_http_requests_total[5m])) > 0.02
        for: 10m
        labels:
          severity: page
        annotations:
          summary: 'Over 2% of storefront requests are failing'
          action: 'Roll back the newest release, then work through the runbook'
          runbook: 'https://runbooks.example.com/storefront/high-error-rate'

      - alert: StorefrontQueueBacklog
        expr: app_messenger_queue_depth{transport="async"} > 5000
        for: 15m
        labels:
          severity: ticket
        annotations:
          summary: 'Async work is arriving faster than consumers drain it'
          action: 'Scale the consumer Deployment, then find the slow handler'
          runbook: 'https://runbooks.example.com/storefront/queue-backlog'

Keeping noise out of the signal

If anything in your alerting path counts error-level log lines, this configuration decides what it sees. fingers_crossed keeps debug records in memory and flushes the whole buffer only when an error-level record arrives, which gives you the lead-up to a failure without logging every healthy request. Routing deprecations to their own channel keeps them out of that trigger: a deprecation is work to schedule, not an incident. Note that excluded_http_codes applies to the HTTP status; an exception your own code logs at error level still flushes the buffer.

Example · yaml
# config/packages/prod/monolog.yaml
monolog:
    handlers:
        main:
            type: fingers_crossed
            # Buffer everything; write it only when an error actually happens.
            action_level: error
            handler: nested
            # A 404 is a client mistake, not an outage, and scanners
            # produce thousands. Same for 405.
            excluded_http_codes: [404, 405]
            buffer_size: 50
        nested:
            type: stream
            path: php://stderr
            level: debug
            formatter: monolog.formatter.json
        deprecation:
            type: stream
            channels: [deprecation]
            path: php://stderr
            formatter: monolog.formatter.json

The runbook the alert points at

Write this before the alert is enabled, not after the first incident. The test of a runbook is whether someone who did not build the feature can follow it at 3am: concrete commands, in order, with the decision points spelled out. Keep it in the repository so a change to the deployment can change the runbook in the same review, and keep it short - a runbook nobody can finish reading during an outage is a document, not a tool.

Example · markdown
<!-- docs/runbooks/high-error-rate.md -->
# Runbook: StorefrontHighErrorRate

**Fires when** over 2% of requests return 5xx for 10 minutes.
**User impact** checkout and article pages fail for roughly 1 request in 50.

## First two minutes
1. Open the storefront dashboard. Is the error rate one route or all routes?
2. `kubectl rollout history deployment/storefront -n storefront`
   Did a release land within 15 minutes of the alert? If so, roll back:
   `kubectl rollout undo deployment/storefront -n storefront`
3. Check the queue panel. A backlog alongside errors points at the database.

## If no release is involved
- One route only: open a trace for that route and read the widest span.
- All routes, database spans slow: check connection count and slow queries.
- All routes, 503 at the edge: check pod readiness, not the application.

## Escalate
If the error rate is still above 2% after 20 minutes, page the on-call
database engineer. Say what you have already ruled out.

## Afterwards
Add what you learned to this file. A runbook that was not enough is a bug.

Related

symfony metricssymfony slosymfony health checkssymfony prod errorssymfony log channels

Discussion

  • Be the first to comment on this lesson.