Retries, Failures and Idempotency

Why a handler will run twice, what the retry strategy and failure transport do, and how to make repeat runs harmless.

Syntaxretry_strategy: { max_retries: 3, delay: 1000, multiplier: 2, max_delay: 0, jitter: 0.1 }

A worker is a process talking over a network. It gets killed mid-job, databases time out, remote APIs answer 503. Messenger's response is to retry, which is also why a handler must be safe to run more than once.

The retry strategy

When a handler throws, the message goes back to the transport with a delay. Each transport has a retry strategy: max_retries (default 3), delay in milliseconds (1000), multiplier (2), max_delay (0, meaning no ceiling) and jitter (0.1, a random variation of plus or minus ten per cent so a thousand failed messages do not retry in lockstep). With the defaults, delays are roughly 1s, 2s and 4s.

Two exceptions change that. UnrecoverableMessageHandlingException stops retrying immediately: throw it when the input is wrong and waiting cannot help. RecoverableMessageHandlingException forces a retry: by default it ignores max_retries and the message is retried for ever, so pass forceRetry: false (added in Symfony 8.1) when the transport's limit should still apply. Its retryDelay argument overrides the computed delay.

The failure transport

After the last retry the message moves to the failure transport, a queue of dead messages, instead of vanishing. Read it with messenger:failed:show, replay with messenger:failed:retry, discard with messenger:failed:remove.

Idempotency

Idempotent means a second run has the same effect as the first. A worker can be killed after charging a card and before acknowledging the message, and the message will come back. Make each effect conditional on state you can check first, or give the work a key the database or the remote API will refuse twice.

Example

Example · php
// src/MessageHandler/ChargeOrderHandler.php
declare(strict_types=1);

namespace App\MessageHandler;

use App\Entity\Order;
use App\Message\ChargeOrder;
use App\Payment\PaymentGatewayInterface;
use Doctrine\ORM\EntityManagerInterface;
use Psr\Log\LoggerInterface;
use Symfony\Component\Messenger\Attribute\AsMessageHandler;
use Symfony\Component\Messenger\Exception\UnrecoverableMessageHandlingException;

#[AsMessageHandler]
final class ChargeOrderHandler
{
    public function __construct(
        private readonly EntityManagerInterface $entityManager,
        private readonly PaymentGatewayInterface $gateway,
        private readonly LoggerInterface $logger,
    ) {
    }

    public function __invoke(ChargeOrder $message): void
    {
        $order = $this->entityManager->getRepository(Order::class)->find($message->orderId);

        if (null === $order) {
            // Retrying cannot bring back a deleted order, so stop now.
            throw new UnrecoverableMessageHandlingException(
                sprintf('Order %d no longer exists.', $message->orderId)
            );
        }

        if (null !== $order->getChargeId()) {
            // This message has already been handled once. Do nothing.
            $this->logger->info('Order {id} already charged', ['id' => $message->orderId]);

            return;
        }

        // The key is derived from the order, so a repeat attempt reaches the
        // same charge at the gateway rather than creating a second one.
        $charge = $this->gateway->charge($order->getTotal(), 'order-'.$message->orderId);
        $order->setChargeId($charge->id);

        // $order is already managed, so persist() is not needed. flush() is:
        // without it nothing is written and the next retry charges again.
        $this->entityManager->flush();
    }
}

When to use it

  • Stop retrying a message whose row was deleted, instead of filling the log with the same error.
  • Back off for a minute when a supplier API rate-limits the worker.
  • Replay yesterday's failed payment messages after the gateway outage is over.
  • Make a charge handler safe to run twice after a worker is killed mid-job.

More examples

Retry and failure configuration

Reach for a longer backoff when the usual failure is a third party being down, and for fewer retries when a failure means bad data. The failure transport is a normal transport; nothing consumes it, which is the point.

Example · yaml
# config/packages/messenger.yaml
framework:
    messenger:
        # Default destination for messages that exhausted their retries.
        failure_transport: failed

        transports:
            async:
                dsn: '%env(MESSENGER_TRANSPORT_DSN)%'
                retry_strategy:
                    max_retries: 5
                    delay: 2000        # milliseconds before the first retry
                    multiplier: 3      # 2s, 6s, 18s, 54s, ...
                    max_delay: 600000  # never wait more than 10 minutes
                # A per-transport override of failure_transport above.
                failure_transport: failed

            # Shorthand: options as DSN query parameters.
            failed: 'doctrine://default?queue_name=failed'

Inspecting and replaying failures

The usual sequence after an outage: show to see the scale, show with an id to read the exception, then retry once the cause is fixed. Replaying runs the handler again from the start, so it only ends well if the handler is idempotent.

Example · bash
# List what is waiting in the failure transport.
php bin/console messenger:failed:show

# Only one message class, with a count per class.
php bin/console messenger:failed:show --class-filter='App\Message\ChargeOrder' --stats

# Show one message in full, including the exception that killed it.
php bin/console messenger:failed:show 1234 -vv

# Replay it. Without --force the command asks first.
php bin/console messenger:failed:retry 1234 --force

# Give up on it.
php bin/console messenger:failed:remove 1234

Asking for a later retry

Use this when the failure carries its own hint about when to come back, such as a rate limit window. The constructor signature is (string $message = '', int $code = 0, ?Throwable $previous = null, ?int $retryDelay = null, bool $forceRetry = true), so pass retryDelay by name. Mind the last argument: because forceRetry defaults to true, this exception otherwise retries for ever and ignores max_retries, so add forceRetry: false, introduced in Symfony 8.1, to keep the transport's retry limit.

Example · php
// src/MessageHandler/SyncPricesHandler.php
declare(strict_types=1);

namespace App\MessageHandler;

use App\Message\SyncPrices;
use App\Pricing\PriceFeedInterface;
use App\Pricing\RateLimitedException;
use Symfony\Component\Messenger\Attribute\AsMessageHandler;
use Symfony\Component\Messenger\Exception\RecoverableMessageHandlingException;

#[AsMessageHandler]
final class SyncPricesHandler
{
    public function __construct(private readonly PriceFeedInterface $feed)
    {
    }

    public function __invoke(SyncPrices $message): void
    {
        try {
            $this->feed->sync($message->supplierId);
        } catch (RateLimitedException $e) {
            // Retried like any other failure, but after the delay we name
            // here (milliseconds) instead of the strategy's computed one.
            throw new RecoverableMessageHandlingException(
                'Supplier rate limit hit.',
                retryDelay: 60_000,
                previous: $e,
            );
        }
    }
}

Related

symfony messenger introsymfony messenger transportssymfony messenger patternssymfony transactionssymfony log context

Discussion

  • Be the first to comment on this lesson.