Operating Queues
Keeping workers alive with Supervisor or Horizon, the metrics worth watching, and what a growing queue depth means.
php artisan horizon # and php artisan horizon:terminate at the end of every deployA worker is an ordinary process. It exits on an error, on queue:restart, when it hits --max-time, and when the server reboots. Nothing in Laravel starts it again, so operating a queue means running a process monitor beside it. On Linux that is usually Supervisor, configured to run php artisan queue:work with autorestart=true and numprocs set to the number of workers you want.
Horizon
Horizon is a first-party package for Redis queues only. composer require laravel/horizon then php artisan horizon:install. One php artisan horizon process reads config/horizon.php and starts the workers described there, so worker counts, queues, tries and timeouts live in version control rather than in a Supervisor file. The keys map to the flags you already know: tries is --tries, memory is --memory, maxJobs is --max-jobs. You still need Supervisor, but only to keep the single horizon process alive. On deploy, run php artisan horizon:terminate after the new code is in place.
What to watch
The dashboard at /horizon shows throughput, runtime and wait time per queue; schedule horizon:snapshot every five minutes to populate it. Wait time is the number that matters to users. Queue depth, which Queue::size('default') also returns, is a derivative: a flat depth means jobs arrive as fast as they are finished, and a steadily rising depth means arrivals exceed throughput. Only two things fix that, more worker processes or a faster job, and a depth that rises while workers are idle means they are not watching that queue name.
Example
<?php
// config/horizon.php (excerpt)
return [
// Seconds of wait time that count as a long wait for notifications.
'waits' => [
'redis:default' => 60,
'redis:critical' => 30,
],
'environments' => [
'production' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['default', 'notifications'],
'balance' => 'auto',
'autoScalingStrategy' => 'time',
'minProcesses' => 1,
'maxProcesses' => 10,
'tries' => 3,
'timeout' => 60,
'memory' => 128,
'maxTime' => 3600,
'maxJobs' => 0,
],
// Heavy work, capped so it cannot take the whole box.
'supervisor-images' => [
'connection' => 'redis',
'queue' => ['images'],
'balance' => 'simple',
'processes' => 2,
'tries' => 3,
'timeout' => 300,
],
],
'local' => [
'supervisor-1' => [
'maxProcesses' => 3,
],
],
],
];When to use it
- Keep eight workers running across reboots without logging in to restart them.
- Move worker counts and timeouts out of a server config file and into the repository.
- Find out whether last night's slowdown was a flood of jobs or a slow job.
- Get an email when the critical queue's wait time goes over thirty seconds.
More examples
Supervisor, with and without Horizon
Use one file or the other, never both for the same queues. stopwaitsecs must be longer than your slowest job: it is how long Supervisor waits after asking the process to stop before killing it, and a kill mid-job is exactly the situation that makes a job run twice. numprocs is the only place worker count lives in the plain setup, which is the argument for Horizon: in the Horizon setup that number is in config/horizon.php and reviewed like any other change.
; /etc/supervisor/conf.d/laravel-worker.conf
; Plain workers: Supervisor runs eight copies of queue:work.
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/app/artisan queue:work redis --tries=3 --max-time=3600
autostart=true
autorestart=true
user=www-data
numprocs=8
redirect_stderr=true
stdout_logfile=/var/www/app/storage/logs/queue.log
stopwaitsecs=3600
; /etc/supervisor/conf.d/horizon.conf
; With Horizon: one process, and it starts the workers itself.
[program:horizon]
process_name=%(program_name)s
command=php /var/www/app/artisan horizon
autostart=true
autorestart=true
user=www-data
redirect_stderr=true
stdout_logfile=/var/www/app/horizon.log
stopwaitsecs=3600
; sudo supervisorctl reread && sudo supervisorctl update
; sudo supervisorctl start horizonDashboard access and metrics
Two things surprise people here. First, /horizon is reachable without authentication in the local environment and governed by the viewHorizon gate everywhere else, so an unedited gate means nobody can see it in production. Second, the throughput and wait-time graphs are built from snapshots, so they stay blank until horizon:snapshot is scheduled.
<?php
// app/Providers/HorizonServiceProvider.php
declare(strict_types=1);
namespace App\Providers;
use App\Models\User;
use Illuminate\Support\Facades\Gate;
use Laravel\Horizon\Horizon;
use Laravel\Horizon\HorizonApplicationServiceProvider;
class HorizonServiceProvider extends HorizonApplicationServiceProvider
{
public function boot(): void
{
parent::boot();
Horizon::routeMailNotificationsTo('[email protected]');
}
/**
* This gate decides who may open /horizon outside the local
* environment. Locally the dashboard is open.
*/
protected function gate(): void
{
Gate::define('viewHorizon', fn (User $user) => $user->is_admin);
}
}
// routes/console.php
// Without this, the metrics graphs stay empty.
// use Illuminate\Support\Facades\Schedule;
// Schedule::command('horizon:snapshot')->everyFiveMinutes();Reading queue depth
A rising depth on its own does not say what to change; the difference between arrival rate and throughput does. Here the workers are 145 jobs a minute short, so roughly half again as much worker capacity will hold the line, and the backlog already queued needs more than that to clear. The same arithmetic tells you when the answer is a faster job instead: if throughput per process is tiny because each job takes four seconds, adding processes is the expensive fix.
<?php
// Depth sampled once a minute.
$depth = collect([120, 260, 410, 555, 700]);
$growth = $depth->sliding(2)
->map(fn ($pair) => $pair->last() - $pair->first());
echo $growth->implode(', ');
// Output:
// 140, 150, 145, 145
// Jobs finished per minute, measured from the dashboard.
$throughput = 300;
$arrivals = $throughput + (int) $growth->average();
echo "arriving {$arrivals}/min, finishing {$throughput}/min";
// Output:
// arriving 445/min, finishing 300/min
Discussion