we're using Laravel 8.x and we've just switched our queue driver from database to redis/horizon. Now we're getting some strange behavior with some jobs.
The Job itself does converting stuff over some time between a few minutes up to 90 minutes. With the database-driver the job is executed properly. With horizon the job is executed for some time and then seems to get stucked. We've tested the job many times with the same payload and it get stucked at different runtimes and get marked as failed after the timeout (with the same timestamp as start).
The logfile of supervisor contains the following:
[2022-01-07 18:08:33][b69e5cf9-4f88-4005-a5c4-4722e46fe7d0] Processing: App\Jobs\ConvertJob
[2022-01-07 18:08:33][b69e5cf9-4f88-4005-a5c4-4722e46fe7d0] Failed: App\Jobs\ConvertJob
The (other) logfiles doesn't contain any (other) errors :-/
Our config for horizon looks like this (just the relevant parts):
'memory_limit' => 64,
'defaults' => [
'converter' => [
'connection' => 'redis',
'queue' => ['convert'],
'balance' => 'auto',
'maxProcesses' => 1,
'memory' => 128,
'tries' => 1,
'nice' => 0,
],
'worker' => [
'connection' => 'redis',
'queue' => ['default'],
'balance' => 'auto',
'maxProcesses' => 1,
'memory' => 128,
'tries' => 1,
'nice' => 0,
],
],
'environments' => [
'local' => [
'converter' => [
'minProcesses' => 4,
'maxProcesses' => 9,
'balanceMaxShift' => 3,
'balanceCooldown' => 2,
],
'worker' => [
'maxProcesses' => 6,
'balanceMaxShift' => 2,
'balanceCooldown' => 3,
]
],
],
Supervisor config:
[program:horizon]
process_name=%(program_name)s
command=php /app/artisan horizon
autostart=true
autorestart=true
user=%(ENV_SUPERVISOR_USER)s
redirect_stderr=true
stdout_logfile=/app/logs/sv-horizon.log
stopwaitsecs=5400
Queue config:
'redis' => [
'driver' => 'redis',
'connection' => 'queue',
'queue' => env('REDIS_QUEUE', 'default'),
'retry_after' => 5430,
'block_for' => null,
'after_commit' => false,
],
The Job itself sets tries to 1, timeout to 5400 seconds and failOnTimeout true.
Maybe anyone of you can give me a hint?
Related
I'm using Laravel forge and have setup Horizon for processing jobs.
The jobs are being processed I can see that in the forge logs. However, the jobs are not visible in my horizon dashboard. I can see that horizon is active, everything looks fine but they just don't show up.
My horizon config file is pretty basic:
'environments' => [
'prod' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['default'],
'balance' => 'simple',
'processes' => 20,
'tries' => 1,
],
],
'local' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['default'],
'balance' => 'simple',
'processes' => 3,
'tries' => 1,
],
],
],
The jobs are being dispatched on the default queue.
When I look into my failed_jobs table I see records showing up, but not in my horizon dashboard!
What could be wrong here? I already have restarted server/horizon/nginx. But nothing seems to work.
I have a system where I spin up, every hour, tons of jobs and I would like them to be handled on a different redis connection, separate from the main one jobs are popped on to so they do not interfere with other jobs (ie, delaying them) that get pushed on to the queue.
So I did the following, I created a new redis connection:
'kingdom_jobs' => [
'url' => env('REDIS_URL'),
'host' => env('REDIS_HOST', '127.0.0.1'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => env('REDIS_CACHE_DB', 2),
],
Then created a new Queue Connection:
'kingdom_jobs' => [
'driver' => 'redis',
'connection' => 'kingdom_jobs', // Uses the new connection
'queue' => env('REDIS_QUEUE', 'default'),
'retry_after' => 90,
'block_for' => null,
],
Then in the loop that spins up all these jobs:
Kingdom::chunkById(250, function($kingdoms) use ($service) {
foreach ($kingdoms as $kingdom) {
UpdateKingdomJob::dispatch($kingdom)->onConnection('kingdom_jobs');
}
});
The issue?
Ya they get pushed to this new connection, I see them in horizon - over 500 of them - but they don't do anything. They don't get processed. Is there a way to tell horizon to process this additional connection?
I currently use php artisan horizon in development to process jobs - is there another step?
I assume from the docs, there is some step I am missing, I assume it has to do with php artisan queue:work --tries = 3 from the docs? Trying that did nothing. Do I need another instance of horizon for this new connection or do I configure horizon to know about this connection?
Help?
It is likely that you don't have a Horizon supervisor setup to watch your new queue.
Looking at the Horizon config:
'environments' => [
'production' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['default'],
'balance' => 'auto',
'minProcesses' => 1,
'maxProcesses' => 10,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
'tries' => 3,
],
],
],
It looks like we can just add another supervisor to the production environment (and others if you need it):
'environments' => [
'production' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['default'],
'balance' => 'auto',
'minProcesses' => 1,
'maxProcesses' => 10,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
'tries' => 3,
],
'supervisor-2' => [
'connection' => 'kingdom_jobs',
'queue' => ['default'],
'balance' => 'auto',
'minProcesses' => 1,
'maxProcesses' => 10,
'balanceMaxShift' => 1,
'balanceCooldown' => 3,
'tries' => 3,
],
],
],
I using laravel horizon for creating products using api.
but I facing issue of 429 too many request.
below is my horizon setup
'environments' => [
'production' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['default'],
'balance' => 'auto',
'minProcesses' => 1,
'maxProcesses' => 4,
'tries' => 1,
],
],
'local' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['default'],
'balance' => 'auto',
'minProcesses' => 1,
'maxProcesses' => 4,
'tries' => 1,
],
],
],
I got a solution.
I made one cron who will track the total count in Shopify pending basket.
if cron found countless then 30 then it will more 100 requests.
every minute cron will check this.
When trying to use Laravel Horizon instead of Laravel Forge's queue setup, I have an issue with "translating" two workers into the horizon config.php file.
Let's say we have two queues: "queue1" and "queue2" but one queue shall have different processes and a different timeout, how can I achieve this with only one supervisor?
The current setup is:
'production' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['queue1'],
'balance' => 'simple',
'processes' => 20,
'tries' => 3,
'timeout' => 30,
],
'supervisor-2' => [
'connection' => 'redis',
'queue' => ['queue2'],
'balance' => 'simple',
'processes' => 5, // different
'tries' => 1, // different
'timeout' => 60, // different
],
],
But I do not want to have 2 supervisors but only 1. How can this be achieved whilst maintaining different tries, processes and timeouts?
Can anyone help with the correct configuration within horizon.php to get a single supervisor to run multiple queues? I have tried:
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['default', 'queue2'],
'balance' => 'simple',
'processes' => 10,
'tries' => 3,
],
as well as:
'supervisor-1' => [
'connection' => 'redis',
'queue' => 'default, queue2',
'balance' => 'simple',
'processes' => 10,
'tries' => 3,
],
The second queue shows up correctly in horizon and I can send jobs to them but they just do not get processed.
I am provisioned on forge and have my queues setup using redis with the following queue.php config:
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => 'default', // Default Queue
'retry_after' => 90,
'block_for' => null,
],
in config/horizon.php
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['default', 'queue2'],
'balance' => 'simple',
'processes' => 10,
'tries' => 3,
],
in supervisor : --queue=default,queue2
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/App/artisan queue:listen redis --queue=default,queue2 --sleep=3 --tries=3
autostart=true
autorestart=true
user=root
numprocs=8
redirect_stderr=true
stdout_logfile= /var/www/html/App/storage/logs/worker.log