Knowing It Works

What to check after a deploy, which numbers to read first when it is slow, and a dashboard people will open.

Syntaxphp bin/console app:smoke

"It works" has to be something you can check in a minute, from outside the application, without asking anyone. Three habits get you there: a smoke check you run after every deploy, an agreed order for reading the numbers when something is slow, and one small dashboard.

Service level indicators

An SLI is a measured ratio: successful requests divided by all requests, or requests under 500ms divided by all requests. An SLO is the target for that ratio over a window - 99.9% over thirty days. The value of writing them down is not the number; it is that "is it working?" stops being a matter of opinion.

After a deploy

Check in this order, because each step rules out the step after it. Are all replicas ready, with the old ones gone? Did the migration apply? Is the error rate the same as ten minutes ago? Is p95 latency flat? Is the queue draining rather than growing? Did a new exception class appear in the logs? A smoke check that calls a few real URLs and compares status codes covers the first question quickly, and it runs the same way in CI and against production.

When it is slow

Error rate first, because errors can look like slowness. Then p95 by route, to find out whether it is one route or everything. Then a trace of that route to see which span grew. Then database time against application time. Guessing at this stage is what turns a ten-minute incident into an hour.

Example

Example · php
// src/Command/SmokeCheckCommand.php
declare(strict_types=1);

namespace App\Command;

use Symfony\Component\Console\Attribute\AsCommand;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
use Symfony\Component\Console\Style\SymfonyStyle;
use Symfony\Component\DependencyInjection\Attribute\Autowire;
use Symfony\Contracts\HttpClient\Exception\TransportExceptionInterface;
use Symfony\Contracts\HttpClient\HttpClientInterface;

#[AsCommand(name: 'app:smoke', description: 'Checks a deployed instance answers the routes that matter')]
final class SmokeCheckCommand extends Command
{
    /** @var array<string, int> path => expected status */
    private const CHECKS = [
        '/health/ready' => 200,
        '/' => 200,
        '/articles/7' => 200,
        '/articles/does-not-exist' => 404,
    ];

    public function __construct(
        private readonly HttpClientInterface $client,
        #[Autowire(env: 'SMOKE_BASE_URL')] private readonly string $baseUrl,
    ) {
        parent::__construct();
    }

    protected function execute(InputInterface $input, OutputInterface $output): int
    {
        $io = new SymfonyStyle($input, $output);
        $failures = 0;

        foreach (self::CHECKS as $path => $expected) {
            try {
                $status = $this->client->request('GET', rtrim($this->baseUrl, '/').$path)
                    ->getStatusCode();
            } catch (TransportExceptionInterface) {
                $status = 0; // nothing answered at all
            }

            $ok = $status === $expected;
            $failures += $ok ? 0 : 1;
            $io->writeln(sprintf('%s %-28s expected %d, got %d', $ok ? 'ok  ' : 'FAIL', $path, $expected, $status));
        }

        return 0 === $failures ? Command::SUCCESS : Command::FAILURE;
    }
}

// $ SMOKE_BASE_URL=https://storefront.example.com php bin/console app:smoke
// ok   /health/ready                expected 200, got 200
// ok   /                            expected 200, got 200
// FAIL /articles/7                  expected 200, got 500
// ok   /articles/does-not-exist     expected 404, got 404
// (exit code 1)

When to use it

  • Decide in under a minute whether a release that just went out needs rolling back.
  • Run the same smoke check against a staging URL in CI and against production after deploy.
  • Give a new team member one dashboard and one order to read it in.
  • Settle an argument about whether the site is slow with a p95 figure per route.

More examples

The longhand of that injected environment variable

Worth knowing in both forms, because you will read both. The attribute keeps the wiring next to the parameter it describes, which suits a value used in one class. The YAML form is what you need when the value differs per environment file, or when the class comes from a package you cannot annotate. Note the named argument $baseUrl: Symfony matches it to the parameter name, so the other constructor argument keeps being autowired.

Example · yaml
# config/services.yaml
services:
    _defaults:
        autowire: true
        autoconfigure: true

    App\:
        resource: '../src/'

    # Equivalent to #[Autowire(env: 'SMOKE_BASE_URL')] on the constructor
    # parameter. Same result; pick one place to look.
    App\Command\SmokeCheckCommand:
        arguments:
            $baseUrl: '%env(SMOKE_BASE_URL)%'

The four panels, as queries

Four panels, one screen, no scrolling - that is the dashboard people actually open during an incident. The second query is why request duration was recorded as a histogram rather than an average: the _bucket series it produces is what lets a percentile be computed per route after the fact. Averages hide the slow tail completely. Keep resource graphs on a second dashboard; they explain an incident but they do not tell you whether there is one.

Example · yaml
# docs/dashboards/storefront.md (the panels, written out)
# Order matters: the top-left panel is the one people read first.

panels:
  - title: 'Availability (SLI) - target 99.9%'
    expr: |
      1 - (
        sum(rate(app_http_requests_total{status=~"5.."}[5m]))
          / sum(rate(app_http_requests_total[5m]))
      )

  - title: 'Latency p95 by route'
    expr: |
      histogram_quantile(0.95, sum by (route, le) (
        rate(app_http_request_duration_seconds_bucket[5m])
      ))

  - title: 'Orders per minute (is the product working?)'
    expr: 'sum(rate(app_orders_placed_total[5m])) * 60'

  - title: 'Async backlog'
    expr: 'sum(app_messenger_queue_depth) by (transport)'

The first minute after a rolling deploy

Keep this sequence somewhere you can paste from, because the order is the useful part: infrastructure, then schema, then routes, then logs, then workers. Each step rules out the next, so you stop at the first failure instead of debugging the application when the problem is a pod that never became ready. The grep works because prod logs are JSON on stderr; with the default line formatter you would be matching on free text.

Example · bash
# Did the new pods become ready, and did the old ones go away?
kubectl rollout status deployment/storefront -n storefront --timeout=120s
kubectl get pods -n storefront -l app=storefront

# Did the migration run, and is the schema what the entities expect?
kubectl exec -n storefront deploy/storefront -- \
  php bin/console doctrine:migrations:status

# Do the real routes answer? Run the smoke check from inside the cluster,
# so a broken ingress does not mask a working application (or the reverse).
kubectl exec -n storefront deploy/storefront -- \
  env SMOKE_BASE_URL=http://storefront.storefront.svc:8000 \
  php bin/console app:smoke

# Any new exception class in the last two minutes?
kubectl logs -n storefront deploy/storefront --since=2m \
  | grep '"level_name":"ERROR"'

# Are the workers consuming, or stuck restarting?
kubectl get pods -n storefront -l app=storefront-worker

Related

symfony alertingsymfony metricssymfony health checkssymfony k8s probessymfony prod checklist

Discussion

  • Be the first to comment on this lesson.