My stack set-up consists of the following
Machine1 - Main Server (Running laravel)
Machine2 - MySql Server for the laravel codebase
Machine3 - Beanstalkd worker
I have setup Supervisord on Machine1 and added the following queue listener
[program:queue1]
command=php artisan queue:listen --queue=queue1 --tries=2
...
My laravel queue config file(app/config/queue.php) reads the following
'beanstalkd' => array(
'driver' => 'beanstalkd',
'host' => '--- Machine3 IP ---',
'queue' => 'queue1',
'ttr' => 60,
),
And I have installed beanstalkd on Machine3 along with Beanstalk console and can see my tasks being pushed to the queue and executing successfully. However I am not sure if Machine3 is actually executing them, and the reason for my suspicion is the High CPU usage on the main server as compared to no spikes in CPU usage on Machine3
I completely shutdown my beanstalkd Server to check if the queue still processes and the outcome was an error reported by laravel indicating it could not connect to the beanstalkd server.
I read somewhere that you need to have your laravel codebase on the beanstalkd server(Machine3) too, was that really the way to go?
Whichever machine you run queue:listen on is the machine that does the actual processing of the queue.
At the moment all you are doing is storing the queues on machine3, but processing them on machine1.
So you need to have machine3 run the queue:listen command if you want it to process the queue.
Related
I am using Laravel with Phpredis and I've created a webhook that adds a job to the queue. I've followed the docs for the interrogation but my jobs are not being queued.
.env
QUEUE_CONNECTION=redis
config/database.php
'client' => env('REDIS_CLIENT', 'phpredis'),
config/queue.php
...
'connections' => [
...
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => env('REDIS_QUEUE', 'default'),
'retry_after' => 90,
'block_for' => null,
],
...
],
...
I am using Windows with Xampp and redis-server.exe is running. This is what I am getting when the job is being added to the queue:
[9672] 03 Nov 21:44:03 - Accepted 127.0.0.1:52945
[9672] 03 Nov 21:44:03 - Client closed connection
This is how I'm adding the jobs to queue:
ProcessPhotos::dispatch($settings, $data, $id);
And this is how I'm trying to run the queued jobs:
php artisan queue:work
or
php artisan queue:listen
I am running one of the previous commands and nothing is happening and I'm also not receiving any errors. It's like the queue is empty (I've also checked the queue length using this code and I've got 000).
I've also tried to set a key into redis and that seemed to work. Does somebody know what's happening? I'm thinking to move to database if i can't get this solved ...
I've fixed the issue!
It turned out that it was something wrong with the server. (I've reinstalled again the Redis extension and it still wasn't working, then I changed the server version and it was working)
I reinstalled the Redis extension from here and switched to this server version. The rest of the settings were the same as in my previous post.
I am having some trouble with a custom laravel queue connection/queue. This particular connection/queue is being used for jobs which may be anywhere from 5 minutes to 10 hours (large data aggregations and data rebuilds)
I have a supervisor conf defined as
[program:laravel-worker-extended]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/artisan queue:work --queue=refreshQueue,rebuildQueue --sleep=3 --timeout=86400 --tries=2 --delay=360
autostart=true
autorestart=true
user=root
numprocs=4
redirect_stderr=true
stdout_logfile=/var/www/storage/logs/queue-worker.log
I have a queue connection defined as:
'refreshQueue' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'refreshQueue',
'retry_after' => 420, // Retry after 7 minutes
],
I’m adding a job to the queue with a Command via:
AggregateData::dispatch()->onConnection('refreshQueue')->onQueue('refreshQueue');
When DatabaseQueue is constructed, retryAfter is 420 as defined. however here are my job logs:
[2020-01-22 18:25:37] local.INFO: BEGINNING AGGREGATION
[2020-01-22 18:25:37] local.INFO: Aggregating data
[2020-01-22 18:27:08] local.INFO: BEGINNING AGGREGATION
[2020-01-22 18:27:08] local.ALERT: AGGREGATION FAILED: Aggregation in progress
Why does it continue to retry after 90 seconds when I explicitly tell it to retry after 420?
I’ve rebuilt my container, restarted the queue, and done about everything else I can to debug... and then waiting a while, I get this final log output:
[2020-01-22 18:25:37] local.INFO: BEGINNING AGGREGATION
[2020-01-22 18:25:37] local.INFO: Aggregating data
[2020-01-22 18:27:08] local.INFO: BEGINNING AGGREGATION
[2020-01-22 18:27:08] local.ALERT: AGGREGATION FAILED: Aggregation in progress
[2020-01-22 18:33:04] local.INFO: [COMPLETE] Aggregating data
[2020-01-22 18:33:04] local.INFO: Queue job finishedIlluminate\Queue\CallQueuedHandler#call
I can't quite grasp why the queue continues to retry the job after 90 seconds. Am I doing something wrong here?
Editing for some additional context here:
This method sets an in_progress flag when it begins, so that it cannot be run twice at the same exact time. The logs can be interpreted as:
BEGINNING AGGREGATION: First line in the handle() method of the job
AGGREGATION FAILED: Aggregation in progress: The failed() method of the job handles failures via exception. This line shows that it has both attempted the job again, and encountered the flag being set to 1 already meaning another job is processing currently. This flag gets reset to 0 when the job is complete or a different exception (not 'in-progress') is encountered.
Queue job finishedIlluminate\Queue\CallQueuedHandler#call Is further debugging I added in the service provider to listen for queue complete events.
This might have to do something with time timeout you're using. From the docs:
The --timeout value should always be at least several seconds shorter than your retry_after configuration value. This will ensure that a worker processing a given job is always killed before the job is retried. If your --timeout option is longer than your retry_after configuration value, your jobs may be processed twice.
I've figured out the issue here. In queue.php I was defining a connection refreshQueue. However, in my supervisor conf I was using:
command=php /var/www/artisan queue:work --queue=refreshQueue,rebuildQueue --sleep=3 --timeout=86400 --tries=2 --delay=360
as the command (--queue), where the command should have been:
command=php /var/www/artisan queue:work refreshQueue --sleep=3 --timeout=86400 --tries=2 --delay=360
Note the lack of --queue. The connection has the retry_after defined, not the queue itself.
This is a valuable lesson in the difference of connections vs queues.
Hi below is the config file of supervisor of one project
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /livesites/siteA.example.com/artisan queue:work database --sleep=3 --tries=3
autostart=true
autorestart=true
user=www-data
numprocs=8
redirect_stderr=true
stdout_logfile=/livesites/siteA.example.com/storage/logs/worker.log
Its running fine . I have another project (siteB.example.com) with the redis as QUEUE_CONNECTION in the .env. What should the the config file for that. Will there be any issue running two projects queues on same server ?
First If two projects are on different connections (Redis and Database) shouldn't be any problem.
But if connections are the same (both on Database or Redis), One solution might be using a different queue for each project.
for example in the siteA project push your jobs on siteA queue and in the siteB project push your jobs on siteB queue. Then create two separate supervisor config files and in each of them put --queue=siteA or --queue=siteB in the artisan command argument.
siteA.conf:
command=php /livesites/siteA.example.com/artisan queue:work database --queue=siteA --sleep=3 --tries=3
siteB.conf:
command=php /livesites/siteB.example.com/artisan queue:work database --queue=siteB --sleep=3 --tries=3
and finally, in your Laravel code dispatch each job to appropriate queue:
dispatch((new Job)->onQueue('siteA'));
in siteB project
dispatch((new Job)->onQueue('siteB'));
or you can globally change the default queue for each project in config/queue.php as below:
'database' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'siteA' // or siteB,
'retry_after' => 90,
]
I have a strange behavior when a Job runs:
On dev server (win 7 php 7.2.10) everything work fine,
on the production server Linux centOS php 7.0.10 it throws an Exception:
Illuminate\Queue\MaxAttemptsExceededException: A queued job has been attempted too many times. The job may have previously timed out.
config/queue.php
'database' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'default',
'retry_after' => 90,
],
this happens after a job is queued ... when it starts working ... after about 30 seconds (Failed)
the exception is in the failed_jobs table
I though it could be dependent by the php max_execution_time directive but when i do
php -r "echo ini_get('max_execution_time') . PHP_EOL;"
it shows me zero (no timeout ... which is correct)
the Job is queued in this way:
dispatch((new Syncronize($file))->onQueue('sync'));
The Sincronize Job has no timeout (has 1 try) and simply calls two artisan commands which work perfecly both on prod and on dev server if called from the shell.
https://pastebin.com/mnaHWq71
to start jobs on the dev server I use
php artisan queue:work --queue=sync,newsletter,default
on prod server I use this
https://pastebin.com/h7uv5gca
any idea of what can be the cause ?
Found the problem ...
was in my service /etc/init.d/myservice
cd /var/www/html/
case "$1" in
start)
php artisan queue:work --queue=sync,newsletter,default &
echo $!>/var/run/myservice.pid
echo "server daemon started"
;;
I didn't check if the process was already running so I launch it twice.
I saw 2 processes in ps axu and seems that this was the cause
This check solved
if [ -e /var/run/myservice.pid ]; then
echo "Service is running. Call stop first".
exit 1
fi
We're currently running two laravel applications on the same dedicated server. Each application utilizes laravel's queueing system for background jobs and notifications. Each uses redis for the driver. Neither define any specific queues, they are both using the default. Our supervisor .conf is as follows:
[program:site-one-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/siteone.com/artisan queue:work --sleep=5 --tries=1
autostart=true
autorestart=true
user=www-data
numprocs=4
redirect_stderr=true
stdout_logfile=/var/www/siteone.com/storage/logs/worker.log
[program:site-two-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/sitetwo.com/artisan queue:work --sleep=5 --tries=1
autostart=true
autorestart=true
user=www-data
numprocs=4
redirect_stderr=true
stdout_logfile=/var/www/sitetwo.com/storage/logs/worker.log
Everything worked as expected before adding the configuration for the second site. After adding it, we were testing and noticed that when invoking an event on sitetwo.com that triggered a notification to be sent to the queue, that the email addresses which should have received the notifications did not, and instead they were sent to two email addresses that only exist within the database for siteone.com!
Everything seems to function as expected as long as only one of the above supervisor jobs is running.
Is there somehow a conflict between the two different applications using the same queue name for processing? Did I botch the supervisor config? Is there something else that I'm missing here?
The name of the class is all Laravel cares about when reading the queue. So if you have 2 sites dispatching the job App\Jobs\CoolEmailSender, then whichever application picks it up first is going to process it, regardless of which invoked it.
I can think of 2 things here:
Multiple redis-instances
or
unique queue names passed to --queue
I just changed APP_ENV and APP_NAME into .env file and it worked for me.
For Example:
First .env: APP_ENV=local APP_NAME = localapp
Second .env: APP_ENV=staging APP_NAME=stagingapp
Maybe late but did you try to modify the queue config at config/queue.php
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => 'project1', // 'project2'...
'retry_after' => 90,
'block_for' => null,
],
run queue with --queue=project1
Note: This answer is for those who are having multiple domains, multiple apps, and one database.
You can dispatch & listen to your job from multiple servers by specifying the queue name.
App 1
dispatch((new Job($payload))->onQueue('app1'));
php artisan queue:listen --queue=app1
App 2
dispatch((new Job($payload))->onQueue('app2'));
php artisan queue:listen --queue=app2