I'm looking information about run X next jobs on queue of Laravel.
On my app, only can run 10 on some queue (name_of_any_queue) per minute.
On doc only see this:
php artisan queue:work --queue=name_of_any_queue --once Only process the next job on the queue
But this only run next job.
I've a task for running every minute command above.
It's possible? How can I do?
Two problems:
You cannot pass an argument to the queue to dictate how many items to process, other than one.
You cannot set a cron task to run less every minute.
Solution:
Use a loop and process --once 8 times within the loop.
Related
First of all, I know about queues and now have good experience with queues. The problem with the queue is, it is a queue. I want to execute multiple functions or commands together in the background. Queues will keep second command or function in a queue and will execute once the first one is done executing!
For example, I have a table with ~3,000,000 records and I want to process them faster. What I can do is divide them into 5 equal chunks and execute 5 commands altogether so that I can utilize my CPU as well as process data 5 times faster.
So, How can I do this with Laravel? Queues are not going to work because they execute stuff one after another. If your idea is to create multiple 5 multiple queues and supervisors to accomplish, that's not a standard way to do it I think.
Any Idea on what can be done in this case?
Finally, I found a solution. It is very very easy. So, Here is how it works.
First of all, I divide records into the number of chunks (For example, 5 chunks. It will divide 3 million items into 5 chunks having 600k items each)
Then, I can push a command to queues that I create instantaneously for the chunks and execute queue worker for that queue only once using --once option. To make it simple to understand, Here is the code that I am using.
$chunk_id = 0;
foreach($chunks as $chunk){
// Adding chunk to queue
Artisan::queue('process:items',[
'items' => $chunk,
])->onQueue('processChunk'.$chunk_id);
// Executing queue worker only once
exec('php artisan queue:work --queue=processChunk'.$chunk_id.' --once > storage/logs/process.log &');
$chunk_id++;
}
With exec command, we are executing queue worker for the specific queue created for the specific chunk. Also, we've added & at the end of the command which forces the command to execute in the background at OS level.
This is how it can be done. I tested it and it is working smoothly! Anything else to improve or are there any drawbacks of using this method?
Just to add something from my personal experience.
First install and configure supervisor for your OS accordingly, following are the confs for linux basaed OS e.g. Ubuntu
Supervisor confs: (/etc/supervisor/conf.d)
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/app/artisan queue:work database --sleep=3 --tries=3
autostart=true
autorestart=true
user=username
numprocs=25
redirect_stderr=true
stderr_events_enabled=true
stderr_logfile=/var/www/app/storage/logs/worker.error.log
stdout_logfile=/var/www/app/storage/logs/worker.log
Then create jobs according to your needs and dispatch.
Supervisor will process jobs simultaneously, in this particular case 25 jobs will be processing at a time.
I have an app built on laravel 5.1 which is using RabbitMQ for job processing. The worker command is as follows:
php artisan queue:work --queue=[queue-name] --daemon --tries=5
The issue which I am experiencing is that multiple jobs are getting processed at the same time even though I have only one worker for the queue. How and why is this happening? I have a table which tracks the status of the jobs and as per the table, the number of parallel processing jobs sometimes increase to 8 or 9. With one worker, this count is strange.
Note:
My application has 3500-4000 jobs for processing daily
The jobs are time consuming so dont know if this has something to do with the process timeout
In laravel you can start a queue listener with:
php artisan queue:listen
But how many workers (threads, processes) will be used to process the queue?
Is there any way to define the number of workers?
https://laravel.com/docs/queues#supervisor-configuration
You generate a config file where you define the number of workers.
numprocs=10
By running php artisan queue:listen only one process will be run and fetches the jobs from the queue. So the jobs will be fetched and processed one by one.
If you want to have more than one thread to process the queue jobs you need to run the listener many times in different consoles. But instead of running them manually you can use Supervisor to manage your threads then you will be able to configure the number of thread by setting numprocs parameter in Supervisor configuration setting
I'm having a bit of trouble grasping the concept of Queues in laravel 5.0. From what I understand, queues store a list of Commands to be executed by either by the php artisan queue:listen command or the php artisan queue:work --daemon command.
Correct me if I'm wrong, but the php artisan queue:listen just waits until there is a command in queue and then executes it, right? Then what does the php artisan queue:work --daemon command do in comparison? Does this command only work one at a time?
Anyways, a task I wish to complete is this... I want to periodically check if there are commands in the queue and if there are I wish to execute them. Since this is a periodic problem I assume a chron would be used, but how do i check if there are outstanding commands to be? Would I run a query on my queue table? If I do, how would I dispatch the command? Or should I just chron the php artisan queue:listen command
The main difference between queue:listen and queue:work is that the second one only sees if there's a job waiting, takes it and handles it. That's it.
The listener though runs as a background process checking for jobs available all the time. If there's a new job it is then handled (the process is slightly more difficult, but that's the main idea).
So basically if you need to handle your commands (jobs) as soon as they appear you might wanna use the queues.
And if you need to do something periodically (for example every 2 hours or every Monday at 9 AM etc.) you should go with cron + Schedule.
I wouldn't recommend combining them as you described. Also keep in mind that jobs can be delayed if you don't want then do be handled ASAP.
If I am running Beanstalk with Supervisor on a server with a Laravel 4 application, and I want it to process all queues asynchronously -- as many as it can at the same time -- can I have multiple listeners running at the same time? Will they be smart enough to not "take" the same to-do item from the queue, or will they all reach for the same one at the same time, and thus not work in the way I'm wanting? In short, I want to use Queues to process multiple tasks at a time -- can this be done?
php artisan queue:listen && php artisan queue:listen && php artisan queue:listen
In short, I want to use Queues to process multiple tasks at a time -- can this be done?
In short - yes it can be done. Every job taken by a worker is locked until it's release. It means that other workers will get different jobs to process.
IMO it's better to configure Supervisor to run multiple queue:work command. It will take only one job, process it and stop execution.
It's not encouraged to run PHP scripts in a infinite loop (as queue:listen does), because after some time they can have memory issues (leaks etc).
You can configure Supervisor to re-run finished workers.