Dispatching to queue always chooses the default connection - php

Question

I have an application running many different kinds of jobs that run for several seconds to an hour. This requires me to have two seperate queue connection due to retry_after being bound to a connection instead of a queue (even though they both use Redis, which is annoying but not the issue right now). One connection has a retry_after of 600 seconds, the other 3600 seconds (or one hour). My jobs implement ShouldQueue and use traits Queueable, SerializesModels, Dispatchable, InteractsWithQueue.
Now for the problem. I created a TestJob that sleeps for 900 seconds to make sure we pass the retry_after limit. When I try to dispatch a job on a specific connection using:
dispatch(new Somejob)->onConnection('redis-long-run')
or as I used to do it previously in the constructor of the job (which always used to work):
public function __construct() {
$this->onConnection('redis-long-run');
}
The job gets picked up by the queue worker, it runs for 600 seconds after which the worker restarts job, notices it has already run once and fails it. 300 seconds later, the job processes successfully. If my worker allow for more than one try, the duplicate jobs will run in parallel for 300 seconds.
In my test job I'm also printing out $this->connection which does show the correct connection being used so my guess is the broadcaster is just ignoring it completely.
I'm using Laravel 5.8.35 and PHP 7.3 in a Docker environment. Supervisor handles my workers.
Edit: I've confirmed the behavior persists after upgrading to Laravel v6.5.1
Steps To Reproduce:
Set your queue driver to Redis.
Create two different Redis connections in queue.php, one named redis having a retry_after of 600, the other named redis-long-run with a retry_after of 3600. In my case they also have different queue's, though I'm not sure this is required for this test.
'connections' => [
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => env('REDIS_QUEUE', 'default'),
'retry_after' => 600,
'block_for' => null,
],
'redis-long-run' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => 'long_run',
'retry_after' => 3600,
'block_for' => null,
]
]
Create a little command to dispatch our test job three times
<?php
namespace App\Console\Commands;
use App\Jobs\TestFifteen;
use Illuminate\Console\Command;
class TestCommand extends Command
{
/**
* The name and signature of the console command.
*
* #var string
*/
protected $signature = 'test:fifteen';
/**
* Create a new command instance.
*/
public function __construct()
{
parent::__construct();
}
/**
* Execute the console command.
*
* #return void
*/
public function handle()
{
for ($i = 1; $i <= 3; $i++) {
dispatch(new TestFifteen($i))->onConnection('redis-long-run');
}
}
}
Create the test job
<?php
namespace App\Jobs;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Events\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use function sleep;
use function var_dump;
class TestFifteen implements ShouldQueue
{
use Queueable, SerializesModels, Dispatchable, InteractsWithQueue;
private $testNumber;
public function __construct($testNumber)
{
$this->onConnection('redis-long-run');
$this->testNumber = $testNumber;
}
public function handle()
{
var_dump("Started test job {$this->testNumber} on connection {$this->connection}");
sleep(900);
var_dump("Finished test job {$this->testNumber} on connection {$this->connection}");
}
}
Run your queue workers. I use supervisor with the follow config for these workers.
[program:laravel-queue-default]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/artisan queue:work redis --queue=default --tries=3 --timeout=600
numprocs=8
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
[program:laravel-queue-long-run]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/artisan queue:work redis --queue=long_run --tries=1 --timeout=3540
numprocs=8
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0
Execute the artisan command php artisan test:fifteen
So am I doing something wrong or is the applied connection really not respected?
Also, what's the design philosophy behind not being able to just decide on a per job or queue basis what the retry_after should be and thus being able to use REDIS as my actual queue driver? Why can't I pick Redis as my queue handler and decide that queue-1 retries after 60 seconds and queue-2 retries after 120 seconds? I feels so unnatural having to set up two connections for this, when they use exactly the same Redis instance and everything.
Anyway, here's hoping some can shine some light on this issue. Thank you in advance.

Answer 1

From my understanding your connection is always redis, so instead you should specify the queue instead, when dispatching:
dispatch(new TestFifteen($i))->onQueue('long_run');
The connection is a different driver, like redis, sync, AWS, etc. and your queue is different configuration or multiple stacks of jobs to that connection.