First of all, I know about queues and now have good experience with queues. The problem with the queue is, it is a queue. I want to execute multiple functions or commands together in the background. Queues will keep second command or function in a queue and will execute once the first one is done executing!
For example, I have a table with ~3,000,000 records and I want to process them faster. What I can do is divide them into 5 equal chunks and execute 5 commands altogether so that I can utilize my CPU as well as process data 5 times faster.
So, How can I do this with Laravel? Queues are not going to work because they execute stuff one after another. If your idea is to create multiple 5 multiple queues and supervisors to accomplish, that's not a standard way to do it I think.
Any Idea on what can be done in this case?
Finally, I found a solution. It is very very easy. So, Here is how it works.
First of all, I divide records into the number of chunks (For example, 5 chunks. It will divide 3 million items into 5 chunks having 600k items each)
Then, I can push a command to queues that I create instantaneously for the chunks and execute queue worker for that queue only once using --once option. To make it simple to understand, Here is the code that I am using.
$chunk_id = 0;
foreach($chunks as $chunk){
// Adding chunk to queue
Artisan::queue('process:items',[
'items' => $chunk,
])->onQueue('processChunk'.$chunk_id);
// Executing queue worker only once
exec('php artisan queue:work --queue=processChunk'.$chunk_id.' --once > storage/logs/process.log &');
$chunk_id++;
}
With exec command, we are executing queue worker for the specific queue created for the specific chunk. Also, we've added & at the end of the command which forces the command to execute in the background at OS level.
This is how it can be done. I tested it and it is working smoothly! Anything else to improve or are there any drawbacks of using this method?
Just to add something from my personal experience.
First install and configure supervisor for your OS accordingly, following are the confs for linux basaed OS e.g. Ubuntu
Supervisor confs: (/etc/supervisor/conf.d)
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/app/artisan queue:work database --sleep=3 --tries=3
autostart=true
autorestart=true
user=username
numprocs=25
redirect_stderr=true
stderr_events_enabled=true
stderr_logfile=/var/www/app/storage/logs/worker.error.log
stdout_logfile=/var/www/app/storage/logs/worker.log
Then create jobs according to your needs and dispatch.
Supervisor will process jobs simultaneously, in this particular case 25 jobs will be processing at a time.
Related
I have a lot of tasks, which should be processed in proper order. I would like to divide them by "type" to different queues. Queues should be created dynamically. Then I would like to start X workers, which will process the tasks from the queues. But there is one important rule – from each queue, only one task should be processed at once. The reason is – each task of the specified type can change the state of the application, so I can't start processing a new task of the same type if the last one hasn't finished.
I would like to use Laravel queue system with Redis driver, but I'm not sure it's able to do that. How to prevent the queue system from taking more than one job from each queue at once? Any ideas?
Thank you in advance for your help.
If you are using supervisor then this is what you can do in your .conf file.
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/****/artisan queue:work
autostart=true
autorestart=true
user=root
numprocs=1 <----- this is what you are looking for
redirect_stderr=true
stdout_logfile=/var/www/html/****/storage/logs/supervisord.log
numprocs directive will instruct Supervisor to run 1 queue:work processes and monitor it, automatically restarting it if it fails. (Laravel Queue Supervisor Doc)
Job chaining allows you to specify a list of queued jobs that should be run in sequence after the primary job has executed successfully. If one job in the sequence fails, the rest of the jobs will not be run. To execute a queued job chain, you may use the withChain method on any of your dispatchable jobs.
If you would like to specify the default connection and queue that should be used for the chained jobs, you may use the allOnConnection and allOnQueue methods.
ProcessPodcast::withChain([
new OptimizePodcast,
new ReleasePodcast
])->dispatch()->allOnConnection('redis')->allOnQueue('podcasts');
See Laravel docs for more info.
In laravel you can start a queue listener with:
php artisan queue:listen
But how many workers (threads, processes) will be used to process the queue?
Is there any way to define the number of workers?
https://laravel.com/docs/queues#supervisor-configuration
You generate a config file where you define the number of workers.
numprocs=10
By running php artisan queue:listen only one process will be run and fetches the jobs from the queue. So the jobs will be fetched and processed one by one.
If you want to have more than one thread to process the queue jobs you need to run the listener many times in different consoles. But instead of running them manually you can use Supervisor to manage your threads then you will be able to configure the number of thread by setting numprocs parameter in Supervisor configuration setting
I'm having a bit of trouble grasping the concept of Queues in laravel 5.0. From what I understand, queues store a list of Commands to be executed by either by the php artisan queue:listen command or the php artisan queue:work --daemon command.
Correct me if I'm wrong, but the php artisan queue:listen just waits until there is a command in queue and then executes it, right? Then what does the php artisan queue:work --daemon command do in comparison? Does this command only work one at a time?
Anyways, a task I wish to complete is this... I want to periodically check if there are commands in the queue and if there are I wish to execute them. Since this is a periodic problem I assume a chron would be used, but how do i check if there are outstanding commands to be? Would I run a query on my queue table? If I do, how would I dispatch the command? Or should I just chron the php artisan queue:listen command
The main difference between queue:listen and queue:work is that the second one only sees if there's a job waiting, takes it and handles it. That's it.
The listener though runs as a background process checking for jobs available all the time. If there's a new job it is then handled (the process is slightly more difficult, but that's the main idea).
So basically if you need to handle your commands (jobs) as soon as they appear you might wanna use the queues.
And if you need to do something periodically (for example every 2 hours or every Monday at 9 AM etc.) you should go with cron + Schedule.
I wouldn't recommend combining them as you described. Also keep in mind that jobs can be delayed if you don't want then do be handled ASAP.
I've successfully configured and used gearman and its pecl php extension. I'm using it to execute a long process concerning long sql queries in a background. I'm using Yii btw, if that detail helps.
Here's how I use it :
public function actionProcessWithGearman(){
$output = shell_exec('gnome-terminal -e "php workers/worker.php" > /dev/null 2>/dev/null &');
$client = new GearmanClient();
$client->addServer();
$result = $client->doBackground('executeJob',//parameters);
}
Some details:
If you notice I run a gnome-terminal first, so that I can see the process rather than going directly with the php command, I also added /dev/null so that it will no longer wait for a response. And then the worker is woken up and runs the job.
Problem:
My problem arises when this action is executed several times or executed by several users in different clients, and as a result, multiple terminals running worker.php are being instantiated.
How do I have only one worker? and even if I can have several workers for several users in the different clients, how do I close a worker everytime the task is finished?
You can try adding returnCode() and jobStatus().
A sample here - http://php.net/manual/en/gearmanclient.dobackground.php
If you have a queue system in place the workers should always be running in the background and be ready to catch new jobs.
I think there is a misunderstanding on the queue system logic. You should just add a new job that will be processed by the worker, and not start the worker if you want a job to be done.
Gearman usually goes well with supervisor. Supervisor will control the worker and make sure it will be always available. If you need more than one job to be running at the same time then you can always add a new worker.
Just look for gearman + supervisor + php there are a lot of articles explaining how to set that up.
Letting supervisord taking care of the php processes launch and monitoring is the way to go.
This approach have a few pluses: for instance you can let supervisord start workers automatically, even multiple instances of it. And you can even control supervisord processes through xmlrpc, if you prefer to manage your workers from your own web interface (http://supervisord.org/api.html)
Sample supervisord conf
[program:worker_development]
process_name=worker_%(process_num)s
command=php worker.php
directory=/var/ww/myproject
autorestart=true
user=ubuntu
redirect_stderr=false
numprocs=2
startretries=5
If I am running Beanstalk with Supervisor on a server with a Laravel 4 application, and I want it to process all queues asynchronously -- as many as it can at the same time -- can I have multiple listeners running at the same time? Will they be smart enough to not "take" the same to-do item from the queue, or will they all reach for the same one at the same time, and thus not work in the way I'm wanting? In short, I want to use Queues to process multiple tasks at a time -- can this be done?
php artisan queue:listen && php artisan queue:listen && php artisan queue:listen
In short, I want to use Queues to process multiple tasks at a time -- can this be done?
In short - yes it can be done. Every job taken by a worker is locked until it's release. It means that other workers will get different jobs to process.
IMO it's better to configure Supervisor to run multiple queue:work command. It will take only one job, process it and stop execution.
It's not encouraged to run PHP scripts in a infinite loop (as queue:listen does), because after some time they can have memory issues (leaks etc).
You can configure Supervisor to re-run finished workers.