Can anyone help with the correct configuration within horizon.php to get a single supervisor to run multiple queues? I have tried:
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['default', 'queue2'],
'balance' => 'simple',
'processes' => 10,
'tries' => 3,
],
as well as:
'supervisor-1' => [
'connection' => 'redis',
'queue' => 'default, queue2',
'balance' => 'simple',
'processes' => 10,
'tries' => 3,
],
The second queue shows up correctly in horizon and I can send jobs to them but they just do not get processed.
I am provisioned on forge and have my queues setup using redis with the following queue.php config:
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => 'default', // Default Queue
'retry_after' => 90,
'block_for' => null,
],
in config/horizon.php
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['default', 'queue2'],
'balance' => 'simple',
'processes' => 10,
'tries' => 3,
],
in supervisor : --queue=default,queue2
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/App/artisan queue:listen redis --queue=default,queue2 --sleep=3 --tries=3
autostart=true
autorestart=true
user=root
numprocs=8
redirect_stderr=true
stdout_logfile= /var/www/html/App/storage/logs/worker.log
Related
I'm using Laravel forge and have setup Horizon for processing jobs.
The jobs are being processed I can see that in the forge logs. However, the jobs are not visible in my horizon dashboard. I can see that horizon is active, everything looks fine but they just don't show up.
My horizon config file is pretty basic:
'environments' => [
'prod' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['default'],
'balance' => 'simple',
'processes' => 20,
'tries' => 1,
],
],
'local' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['default'],
'balance' => 'simple',
'processes' => 3,
'tries' => 1,
],
],
],
The jobs are being dispatched on the default queue.
When I look into my failed_jobs table I see records showing up, but not in my horizon dashboard!
What could be wrong here? I already have restarted server/horizon/nginx. But nothing seems to work.
we're using Laravel 8.x and we've just switched our queue driver from database to redis/horizon. Now we're getting some strange behavior with some jobs.
The Job itself does converting stuff over some time between a few minutes up to 90 minutes. With the database-driver the job is executed properly. With horizon the job is executed for some time and then seems to get stucked. We've tested the job many times with the same payload and it get stucked at different runtimes and get marked as failed after the timeout (with the same timestamp as start).
The logfile of supervisor contains the following:
[2022-01-07 18:08:33][b69e5cf9-4f88-4005-a5c4-4722e46fe7d0] Processing: App\Jobs\ConvertJob
[2022-01-07 18:08:33][b69e5cf9-4f88-4005-a5c4-4722e46fe7d0] Failed: App\Jobs\ConvertJob
The (other) logfiles doesn't contain any (other) errors :-/
Our config for horizon looks like this (just the relevant parts):
'memory_limit' => 64,
'defaults' => [
'converter' => [
'connection' => 'redis',
'queue' => ['convert'],
'balance' => 'auto',
'maxProcesses' => 1,
'memory' => 128,
'tries' => 1,
'nice' => 0,
],
'worker' => [
'connection' => 'redis',
'queue' => ['default'],
'balance' => 'auto',
'maxProcesses' => 1,
'memory' => 128,
'tries' => 1,
'nice' => 0,
],
],
'environments' => [
'local' => [
'converter' => [
'minProcesses' => 4,
'maxProcesses' => 9,
'balanceMaxShift' => 3,
'balanceCooldown' => 2,
],
'worker' => [
'maxProcesses' => 6,
'balanceMaxShift' => 2,
'balanceCooldown' => 3,
]
],
],
Supervisor config:
[program:horizon]
process_name=%(program_name)s
command=php /app/artisan horizon
autostart=true
autorestart=true
user=%(ENV_SUPERVISOR_USER)s
redirect_stderr=true
stdout_logfile=/app/logs/sv-horizon.log
stopwaitsecs=5400
Queue config:
'redis' => [
'driver' => 'redis',
'connection' => 'queue',
'queue' => env('REDIS_QUEUE', 'default'),
'retry_after' => 5430,
'block_for' => null,
'after_commit' => false,
],
The Job itself sets tries to 1, timeout to 5400 seconds and failOnTimeout true.
Maybe anyone of you can give me a hint?
This is my config queue file. the queue is not working when I change
my connection to database. it only works when I use sync but it
doesn't run in the background. How do I make this work in the background? or how do I fix this problem?
<?php
return [
'connections' => [
'sync' => [
'driver' => 'sync',
],
'database' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'default',
'retry_after' => 90,
'after_commit' => false,
],
'beanstalkd' => [
'driver' => 'beanstalkd',
'host' => 'localhost',
'queue' => 'default',
'retry_after' => 90,
'block_for' => 0,
'after_commit' => false,
],
'sqs' => [
'driver' => 'sqs',
'key' => env('AWS_ACCESS_KEY_ID'),
'secret' => env('AWS_SECRET_ACCESS_KEY'),
'prefix' => env('SQS_PREFIX', 'https://sqs.us-east-1.amazonaws.com/your-account-id'),
'queue' => env('SQS_QUEUE', 'default'),
'suffix' => env('SQS_SUFFIX'),
'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
'after_commit' => false,
],
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => env('REDIS_QUEUE', 'default'),
'retry_after' => 90,
'block_for' => null,
'after_commit' => false,
],
],
'failed' => [
'driver' => env('QUEUE_FAILED_DRIVER', 'database-uuids'),
'database' => env('DB_CONNECTION', 'mysql'),
'table' => 'failed_jobs',
],
];
To execute queue run following command:
php artisan queue:work
Please check the setting worker listen to your process.
ps aux | grep php
Set worker listen:
php artisan queue:listen --queue=queu_name --tries=1 --memory=128 --timeout=300
I have a system where I spin up, every hour, tons of jobs and I would like them to be handled on a different redis connection, separate from the main one jobs are popped on to so they do not interfere with other jobs (ie, delaying them) that get pushed on to the queue.
So I did the following, I created a new redis connection:
'kingdom_jobs' => [
'url' => env('REDIS_URL'),
'host' => env('REDIS_HOST', '127.0.0.1'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => env('REDIS_CACHE_DB', 2),
],
Then created a new Queue Connection:
'kingdom_jobs' => [
'driver' => 'redis',
'connection' => 'kingdom_jobs', // Uses the new connection
'queue' => env('REDIS_QUEUE', 'default'),
'retry_after' => 90,
'block_for' => null,
],
Then in the loop that spins up all these jobs:
Kingdom::chunkById(250, function($kingdoms) use ($service) {
foreach ($kingdoms as $kingdom) {
UpdateKingdomJob::dispatch($kingdom)->onConnection('kingdom_jobs');
}
});
The issue?
Ya they get pushed to this new connection, I see them in horizon - over 500 of them - but they don't do anything. They don't get processed. Is there a way to tell horizon to process this additional connection?
I currently use php artisan horizon in development to process jobs - is there another step?
I assume from the docs, there is some step I am missing, I assume it has to do with php artisan queue:work --tries = 3 from the docs? Trying that did nothing. Do I need another instance of horizon for this new connection or do I configure horizon to know about this connection?
Help?
It is likely that you don't have a Horizon supervisor setup to watch your new queue.
Looking at the Horizon config:
'environments' => [
'production' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['default'],
'balance' => 'auto',
'minProcesses' => 1,
'maxProcesses' => 10,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
'tries' => 3,
],
],
],
It looks like we can just add another supervisor to the production environment (and others if you need it):
'environments' => [
'production' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['default'],
'balance' => 'auto',
'minProcesses' => 1,
'maxProcesses' => 10,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
'tries' => 3,
],
'supervisor-2' => [
'connection' => 'kingdom_jobs',
'queue' => ['default'],
'balance' => 'auto',
'minProcesses' => 1,
'maxProcesses' => 10,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
'tries' => 3,
],
],
],
I'm a bit confuse about how to run the job only once, because when I set the parameter "tries" to 1 and the job fails, it execute one more time. If I set the tries parameter to 3, the job runs 4 times. And finally if I set to 0, the jobs run indefinitely. Below my settings in config/horizon.php:
'production' =
'default' => [
'connection' => 'redis',
'queue' => [
'default',
'notifications',
'dom'
],
'balance' => 'auto',
'maxProcesses' => env('MAX_PROCESSES', 45),
'timeout' => 60,
'tries' => 1,
],
],
And below my settings in config/queue.php
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => 'default',
'retry_after' => 90,
],
And other question, what setting dispatch the "has been attempted to many times or run too along"?
Just set an attribute $tries = 1 to the Job, and on the catch of possible errors, call $this->fail();