Laravel Custom Queue Not Respecting retry_after - php

Question

I am having some trouble with a custom laravel queue connection/queue. This particular connection/queue is being used for jobs which may be anywhere from 5 minutes to 10 hours (large data aggregations and data rebuilds)
I have a supervisor conf defined as
[program:laravel-worker-extended]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/artisan queue:work --queue=refreshQueue,rebuildQueue --sleep=3 --timeout=86400 --tries=2 --delay=360
autostart=true
autorestart=true
user=root
numprocs=4
redirect_stderr=true
stdout_logfile=/var/www/storage/logs/queue-worker.log
I have a queue connection defined as:
'refreshQueue' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'refreshQueue',
'retry_after' => 420, // Retry after 7 minutes
],
I’m adding a job to the queue with a Command via:
AggregateData::dispatch()->onConnection('refreshQueue')->onQueue('refreshQueue');
When DatabaseQueue is constructed, retryAfter is 420 as defined. however here are my job logs:
[2020-01-22 18:25:37] local.INFO: BEGINNING AGGREGATION
[2020-01-22 18:25:37] local.INFO: Aggregating data
[2020-01-22 18:27:08] local.INFO: BEGINNING AGGREGATION
[2020-01-22 18:27:08] local.ALERT: AGGREGATION FAILED: Aggregation in progress
Why does it continue to retry after 90 seconds when I explicitly tell it to retry after 420?
I’ve rebuilt my container, restarted the queue, and done about everything else I can to debug... and then waiting a while, I get this final log output:
[2020-01-22 18:25:37] local.INFO: BEGINNING AGGREGATION
[2020-01-22 18:25:37] local.INFO: Aggregating data
[2020-01-22 18:27:08] local.INFO: BEGINNING AGGREGATION
[2020-01-22 18:27:08] local.ALERT: AGGREGATION FAILED: Aggregation in progress
[2020-01-22 18:33:04] local.INFO: [COMPLETE] Aggregating data
[2020-01-22 18:33:04] local.INFO: Queue job finishedIlluminate\Queue\CallQueuedHandler#call
I can't quite grasp why the queue continues to retry the job after 90 seconds. Am I doing something wrong here?
Editing for some additional context here:
This method sets an in_progress flag when it begins, so that it cannot be run twice at the same exact time. The logs can be interpreted as:
BEGINNING AGGREGATION: First line in the handle() method of the job
AGGREGATION FAILED: Aggregation in progress: The failed() method of the job handles failures via exception. This line shows that it has both attempted the job again, and encountered the flag being set to 1 already meaning another job is processing currently. This flag gets reset to 0 when the job is complete or a different exception (not 'in-progress') is encountered.
Queue job finishedIlluminate\Queue\CallQueuedHandler#call Is further debugging I added in the service provider to listen for queue complete events.

Answer 1

This might have to do something with time timeout you're using. From the docs:
The --timeout value should always be at least several seconds shorter than your retry_after configuration value. This will ensure that a worker processing a given job is always killed before the job is retried. If your --timeout option is longer than your retry_after configuration value, your jobs may be processed twice.

Answer 2

I've figured out the issue here. In queue.php I was defining a connection refreshQueue. However, in my supervisor conf I was using:
command=php /var/www/artisan queue:work --queue=refreshQueue,rebuildQueue --sleep=3 --timeout=86400 --tries=2 --delay=360
as the command (--queue), where the command should have been:
command=php /var/www/artisan queue:work refreshQueue --sleep=3 --timeout=86400 --tries=2 --delay=360
Note the lack of --queue. The connection has the retry_after defined, not the queue itself.
This is a valuable lesson in the difference of connections vs queues.