I'm using Laravel 5.5 and I'm trying to setup some fast queue processing. I've been running into one roadblock after another.
This site is an employer/employee matching service. So when an employer posts a job position, it needs to then run through all the employees in our system and calculate a number of variables to determine how well they match to the job. We have this all figured out, but it takes a long time to process one at a time when you have thousands of employees in the system. So, I set up to write a couple of tables. The first is a simple table that defines the position ID and the status. The second is a table listing all the employee IDs, the position ID, and the status of that employee being processed. This takes only a few seconds to write and then allows the user to move on in the application.
Then I have another server setup to run a cron every minute that checks for new entries in the first table. When found, it marks it out as started and then grabs all the employees and runs through each employee and starts a queued job in Laravel. The job I have defined does properly submit to the queue and running queue:work does in fact process the job properly. This is all tested.
However, the problem I'm running into is that I've tried database (MySQL), Redis and SQS for the queue and they are all very slow. I was using this same server to try to operate the queue:work (using Supervisor and attempting to run up to 300 processes) but then created 3 clones that don't run the cron but only run Supervisor (100 processes per clone) and killed Supervisor on the first server. With database it would process ok, though to run through 10k queued jobs would take hours, but with SQS and Redis I'm getting a ton of failures. The scripts are taking too long or something. I checked the CPUs on the clones running the workers and they are barely hitting 40% so I'm not over-taxing the servers.
I was just reading about Horizon and I'm not sure if it would help the situation. I keep trying to find information about how to properly setup a queue processing system with Laravel and just keep running into more questions than answers.
Is anyone familiar with this stuff and have any advice on how to set this up correctly so that it's very fast and failure free (assuming my code has no bugs)?
UPDATE: Following some other post advice, I figured I'd share a few more details:
I'm using Forge as the setup tool with AWS EC2 servers with 2G of RAM.
Each of the three clones has the following worker configuration:
command=php /home/forge/default/artisan queue:work sqs --sleep=10 --daemon --quiet --timeout=30 --tries=3
process_name=%(program_name)s_%(process_num)02d
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=forge
numprocs=100
stdout_logfile=/home/forge/.forge/worker-149257.log
The database is on Amazon RDS.
I'm curious if the Laravel cache will work with the queue system. There's elements of the queued script that are common to every run so perhaps if I queued that data up from the beginning it may save some time. But I'm not convinced it will be a huge improvement.
If we ignore the actual logic processed by each job, and consider the overhead of running jobs alone, Laravel's queueing system can easily handle 10,000 jobs per hour, if not several times that, in the environment described in the question—especially with a Redis backend.
For a typical queue setup, 100 queue worker processes per box seems extremely high. Unless these jobs spend a significant amount of time in a waiting state—such as jobs that make requests to web services across a network and use only a few milliseconds processing the response—the large number of processes running concurrently will actually diminish performance. We won't gain much by running more than one worker per processor core. Additional workers create overhead because the operating system must divide and schedule compute time between all the competing processes.
I checked the CPUs on the clones running the workers and they are barely hitting 40% so I'm not over-taxing the servers.
Without knowing the project, I can suggest that it's possible that these jobs do spend some of their time waiting for something. You may need to tune the number of workers to find the sweet spot between idle time and overcrowding.
With database it would process ok, though to run through 10k queued jobs would take hours, but with sqs and redis I'm getting a ton of failures.
I'll try to update this answer if you add the error messages and any other related information to the question.
I'm curious if the Laravel cache will work with the queue system. There's elements of the queued script that are common to every run so perhaps if I queued that data up from the beginning it may save some time.
We can certainly use the cache API when executing jobs in the queue. Any performance improvement we see depends on the cost of reproducing the data for each job that we could store in the cache. I can't say for sure how much time caching would save because I'm not familiar with the project, but you could profile sections of the code in the job to find expensive operations.
Alternatively, we could cache reusable data in memory. When we initialize a queue worker using artisan queue:work, Laravel starts a PHP process and boots the application once for all of the jobs that the worker executes. This is different from the application lifecycle for a typical PHP web app wherein the application reboots for every request and disposes state at the end of each request. Because every job executes in the same process, we can create an object that caches shared job data in the process memory, perhaps by binding a singleton into the IoC container, which the jobs can read much faster than even a Redis cache store because we avoid the overhead needed to fetch the data from the cache backend.
Of course, this also means that we need to make sure that our jobs don't leak memory, even if we don't cache data as described above.
I was just reading about Horizon and I'm not sure if it would help the situation.
Horizon provides a monitoring service that may help to track down problems with this setup. It may also improve efficiency a bit if the application uses other queues that Horizon can distribute work between when idle, but the question doesn't seem to indicate that this is the case.
Each of the three clones has the following worker configuration:
command=php /home/forge/default/artisan queue:work sqs --sleep=10 --daemon --quiet --timeout=30 --tries=3
(Sidenote: for Laravel 5.3 and later, the --daemon option is deprecated, and the queue:work command runs in daemon mode by default.)
Related
I am making an api that requires the job to be dispatched multiple times, however, each job takes 10 seconds, and it takes forever to process one by one. Is their anyway to run multiple job once?
GetCaptcha::dispatch($task_id)->afterCommit()->onQueue('default');
You can achieve that by running multiple workers at the same time.
From the Laravel docs:
To assign multiple workers to a queue and process jobs concurrently,
you should simply start multiple queue:work processes. This can either
be done locally via multiple tabs in your terminal or in production
using your process manager's configuration settings. When using
Supervisor, you may use the numprocs configuration value.
Read more here:
https://laravel.com/docs/9.x/queues#running-multiple-queue-workers
https://laravel.com/docs/9.x/queues#supervisor-configuration
For example I have two queues "high" and "low".
I have 1000 jobs in "high" queue and 0 in "low" queue.
Now I want to move for example 500 jobs from "high" queue to "low" queue.
For start it would be fine to find out how to move all jobs not only half of them.
I can get all jobs with this command :
\Redis::lrange('queues:' . $name, 0, -1);
But how to move them, any idea ?
To sum up the dicussion in the comments, here some recommendations and additional information.
Manually adjusting the Redis queue
It is not recommended to intervene with the redis queue manually. Do not alter the queue by hand. Instead, let the queue workers handle the queue that has an unexpected (high) load of work. You can also spawn additional queue workers temporarily to get the work done faster.
Maybe take the unbalanced queue loads into account when working on future features though.
Fixing the queue work load
To fix the queue work load, there are a number of solutions. What they have in common is that we share resources between the individual queues. The only difference is the way how this is achieved.
For the following options, I'll use a very basic example. Imagine a simple cloud application where users can buy some computation power (for whatever). To make things more interesting, the users of the application can also buy a priority queue ticket, which guarantees them priority processing. In other words, their requests should (but don't have to) be processed with priority.
1. Rescaling the queue worker processes
One way to share resources is to up- and down-scale queue worker processes based on the workload. This means we reduce the queue worker processes for one queue so that we have the resources for additional queue workers for the other queue.
In our example, we would probably expect our users to use 9 out of 10 times the non-priority processing because they do not want to pay the extra for faster processing. This means we would normally have 9 work items in the low priority queue for 1 work item in the high priority queue. So that the priority processing makes sense, we would now need something like 3 queue worker processes per queue. We would start the processes like this:
3x php artisan queue:work --queue=high
3x php artisan queue:work --queue=low
If now the high priority queue has suddenly a lot more work items (e.g. caused by a sale of the priority queue ticket), we would need to rescale our queue workers accordingly. To do so, we would have to manually kill some of the --queue=low worker processes and start more of the --queue=high workers.
Because this is quite cumbersome to do by hand (and we devs also need to sleep once in a while), there is a solution to this which is called Laravel Horizon. When configured properly balance mode set to auto, Horizon will make sure that queues with higher work loads do get more attention than queues with less work load. In other words, Horizon will try to achieve equal waiting times across queues.
2. Let queue workers work on multiple queues
A less complex approach is to let queue workers listen and work on multiple queues. This can be done by passing multiple, comma-separated queues to the --queue parameter: --queue=high,low
When doing so, we instruct the queue worker to handle work items on the high queue with priority over the work items on the low queue. That means the worker will always clear the high queue entirely before taking work from the low queue. If, after processing a job on the low queue, the worker finds an item on the high queue, he will jump back to this queue. So each time the worker looks for a new work item, he will first look on the high queue and if there are no work items, he will go to the low queue.
For our example above, we could for example deploy the following 6 queue workers:
4x php artisan queue:work --queue=high,low
2x php artisan queue:work --queue=low
In this case we would have (proportionally) a lot more worker resources for the high queue than for the low queue. But the resources would help out the other works of the low queue when they do not have any priority work to do.
We could of course also add the high queue as fallback to our low queue workers:
4x php artisan queue:work --queue=high,low
2x php artisan queue:work --queue=low,high
This way they would be able to help our high priority queue workers in case of work overloads like in the sale situation described above.
Conclusion
Simple solutions are often better. When your work loads do not explode all of a sudden and you can predict them quite good, then go for the second approach (but make use of supervisor so that you don't have to manually start the workers all the time and also to make sure they restart in case of a crash).
If, on the other hand, your work loads vary a lot and you cannot really tell in advance how many queue workers you need, Horizon may be worth a look. Personally, I also use Horizon for rather simple projects because it is easy to set up and it takes away work of me where I have to think a lot about loads and stuff.
I am working on a data importing functionality for a website. I am using Laravel 5.1 on homestead, and have given the machine 4GB of RAM and two virtual cpus. For importing the data I used the Laravel Excel package, with its chuncking mechanism that basically breaks the data set in chunks and adds the processing to the queue. The workflow for this is as follows:
the user uploads the excel file;
a job is dispatched to handle the excel file;
this job then chunks the file and dispatches multiple other jobs to process the chunked data;
each of these chunks dispatches another job to do some background logic on the inserted data when they are done.
I have setup a queue named 'import' in supervisor for these processes. Along with this queue, I also have a default one (for sending mails and other, less intensive, stuff) and a low priority queue for dispatching jobs from other jobs. Here is my supervisor configuration:
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=/usr/bin/php /home/vagrant/Code/iris/artisan queue:work --tries=1 --daemon --queue=default,import,background1
autostart=true
autorestart=true
user=vagrant
numprocs=8
redirect_stderr=true
stdout_logfile=/home/vagrant/Code/iris/storage/logs/worker.log
With smaller files this works well, but when I attempt to import a ~30k row document, the php processes spawned by supervisor run out of memory toward the end and in the laravel log I start seeing InvalidArgumentException: No handler registered for command [__PHP_Incomplete_Class] in Illuminate\Bus\Dispatcher (this happens especially when I run two imports in parallel or I attempt to download something through websockets) and I am really confused as to why this is happening. No one process exceeds the 512 MB limit as far as I can tell from using memory_get_usage(). Is this a garbage collector issue? Should I summon it manually?
And since I mentioned the websockets, I was also trying to create a separate queue (with higher priority) for handling web socket requests. I tried multiple supervisor configurations (created dedicated worker in supervisor configuration files, added the queue to the --queue= option in the config, etc.) but to no avail. I implemented report downloading through web sockets, and it works fine by itself, but when there are other things in the queue, the socket request gets handled after several items in the lower priority 'import' queue finish, leaving me to believe that I do not understand queue priorities very well. Is there a way to have a separate queue for socket requests that responds immediately to these requests?
UPDATE
I narrowed down the memory leak to the job that we were dispatching at the end of the import. However, I am not able to dispatch this job when attempting to import a very large file (30k+ rows). It errors out InvalidArgumentException: No handler registered for command [__PHP_Incomplete_Class] in Illuminate\Bus\Dispatcher (I think it's because it is running out of memory) and the php processes that were started by supervisor remain active and seem to consume the same amount of resources that they did at their peak (about 450 MB each). This keeps on going until I manually run php artisan queue:restart. For smaller files, the flow works as intendended: it runs the excel batches and at the end dispatches a job to the queue, which is also processed with success.
How can I make laravel queue:work to process jobs as many as possible? (With either redis or beanstalkd)
By default it is processing 1 job at the same time, but I need it to run multiple jobs at the same time, until CPU has free space.
Any help is appreciated.
It depends on how you are running the worker(s), and how you can increase the number of workers that are being started to get items from the queue and run them.
If you are using supervisord to run the workers, it could be as easy as increasing the numprocs in the configuration.
You would not usually set it to a very large number, as trying to run too many processes at once is likely to end up as a major problem when things run out of memory or CPU.
Have a look at this listener. The best it is that it runs workers depending of how many load you have in your queue, and it is very easy to configure. So, when you require much more workers, since it is spawning workers automatically, the new ones will handle the task... If there's no need of more workers, since the previously spawned are killed, there will be a minimum consuming only a few resources from your PC. You can tune depending of your needs and server capabilities.
https://github.com/smaugho/TunedQueue
I have a system running several kinds of jobs (Laravel 5.1 queues). I use database queue driver.
Supervisor makes sure 3 'instances' of queue:work are running as a daemon all the time.
ps aux | grep queue confirms this - three processes are waiting for jobs.
Sometimes I can see several job (without delay) records in the database table, but always only one of them has the reserved flag set to 1.
What am I doing wrong?
Why 3 daemon workers are not taking care of other jobs in the queue?
How can I make sure more than one job can be done at the same time?
UPDATE:
I wrote a job that sleeps for 15 seconds and then dispatches three new jobs (same class).
I ran this on my dev server and it worked. Three jobs were reserved at the same time. Once one of them is dealt with - another is taken from the queue and reserved. The behaviour one would expect.
Finally I ran the same situation on my production server and it did not work. One reserved job at once even though the environment is similar, three queue:work processes live etc.
I then asked supervisord to restart the workers and it started working as well.
So: it does work this way. The problem is I don't know what might have been causing the issue I had and when does it happen? How do I avoid this and how do I know if it's fine now?