I use Laravel 5.7 and 3 queues jobs, the time between jobs is too long/slow.
I foreach items of RSS feeds in the first job, and I dispatch this item in second job, etc... I don't enter in details but there are some ridiculous little calculations that must not take time.
The problem is that every dispatch to a job takes a lot of time. Horizon and Telescope do not allow me to debug.
The machine I use has 32 GB of RAM, and there are several processes (15 each) that turn the tails.
[program:mywebsite_feeder]
command=/RunCloud/Packages/php72rc/bin/php artisan queue:work redis --queue=feeder --tries=3 --sleep=0
directory=/home/runcloud/webapps/mywebsite
redirect_stderr=true
autostart=true
autorestart=true
user=runcloud
numprocs=15
process_name=%(program_name)s_%(process_num)s
I have this error in laravel.log:
production.ERROR: App\Jobs\FeederJob has been attempted too many times
or run too long. The job may have previously timed out.
by default laravel queues sleep 3 seconds when there are no jobs available.
u should use --sleep=0 option
I had the same issue and did a lot of searches but nothing help, even there is some issues about this bug in horizon Github but without a useful solution. the problem is about horizon and Redis bug for heavy tasks.
finally, I switch from Redis and horizon to SQL database (whatever you use in your project for me mssql) as queue connection and it fixed the problem
notice: use --timeout=0 in your artisan command
Related
I'm using Laravel 5.5 and I'm trying to setup some fast queue processing. I've been running into one roadblock after another.
This site is an employer/employee matching service. So when an employer posts a job position, it needs to then run through all the employees in our system and calculate a number of variables to determine how well they match to the job. We have this all figured out, but it takes a long time to process one at a time when you have thousands of employees in the system. So, I set up to write a couple of tables. The first is a simple table that defines the position ID and the status. The second is a table listing all the employee IDs, the position ID, and the status of that employee being processed. This takes only a few seconds to write and then allows the user to move on in the application.
Then I have another server setup to run a cron every minute that checks for new entries in the first table. When found, it marks it out as started and then grabs all the employees and runs through each employee and starts a queued job in Laravel. The job I have defined does properly submit to the queue and running queue:work does in fact process the job properly. This is all tested.
However, the problem I'm running into is that I've tried database (MySQL), Redis and SQS for the queue and they are all very slow. I was using this same server to try to operate the queue:work (using Supervisor and attempting to run up to 300 processes) but then created 3 clones that don't run the cron but only run Supervisor (100 processes per clone) and killed Supervisor on the first server. With database it would process ok, though to run through 10k queued jobs would take hours, but with SQS and Redis I'm getting a ton of failures. The scripts are taking too long or something. I checked the CPUs on the clones running the workers and they are barely hitting 40% so I'm not over-taxing the servers.
I was just reading about Horizon and I'm not sure if it would help the situation. I keep trying to find information about how to properly setup a queue processing system with Laravel and just keep running into more questions than answers.
Is anyone familiar with this stuff and have any advice on how to set this up correctly so that it's very fast and failure free (assuming my code has no bugs)?
UPDATE: Following some other post advice, I figured I'd share a few more details:
I'm using Forge as the setup tool with AWS EC2 servers with 2G of RAM.
Each of the three clones has the following worker configuration:
command=php /home/forge/default/artisan queue:work sqs --sleep=10 --daemon --quiet --timeout=30 --tries=3
process_name=%(program_name)s_%(process_num)02d
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
user=forge
numprocs=100
stdout_logfile=/home/forge/.forge/worker-149257.log
The database is on Amazon RDS.
I'm curious if the Laravel cache will work with the queue system. There's elements of the queued script that are common to every run so perhaps if I queued that data up from the beginning it may save some time. But I'm not convinced it will be a huge improvement.
If we ignore the actual logic processed by each job, and consider the overhead of running jobs alone, Laravel's queueing system can easily handle 10,000 jobs per hour, if not several times that, in the environment described in the question—especially with a Redis backend.
For a typical queue setup, 100 queue worker processes per box seems extremely high. Unless these jobs spend a significant amount of time in a waiting state—such as jobs that make requests to web services across a network and use only a few milliseconds processing the response—the large number of processes running concurrently will actually diminish performance. We won't gain much by running more than one worker per processor core. Additional workers create overhead because the operating system must divide and schedule compute time between all the competing processes.
I checked the CPUs on the clones running the workers and they are barely hitting 40% so I'm not over-taxing the servers.
Without knowing the project, I can suggest that it's possible that these jobs do spend some of their time waiting for something. You may need to tune the number of workers to find the sweet spot between idle time and overcrowding.
With database it would process ok, though to run through 10k queued jobs would take hours, but with sqs and redis I'm getting a ton of failures.
I'll try to update this answer if you add the error messages and any other related information to the question.
I'm curious if the Laravel cache will work with the queue system. There's elements of the queued script that are common to every run so perhaps if I queued that data up from the beginning it may save some time.
We can certainly use the cache API when executing jobs in the queue. Any performance improvement we see depends on the cost of reproducing the data for each job that we could store in the cache. I can't say for sure how much time caching would save because I'm not familiar with the project, but you could profile sections of the code in the job to find expensive operations.
Alternatively, we could cache reusable data in memory. When we initialize a queue worker using artisan queue:work, Laravel starts a PHP process and boots the application once for all of the jobs that the worker executes. This is different from the application lifecycle for a typical PHP web app wherein the application reboots for every request and disposes state at the end of each request. Because every job executes in the same process, we can create an object that caches shared job data in the process memory, perhaps by binding a singleton into the IoC container, which the jobs can read much faster than even a Redis cache store because we avoid the overhead needed to fetch the data from the cache backend.
Of course, this also means that we need to make sure that our jobs don't leak memory, even if we don't cache data as described above.
I was just reading about Horizon and I'm not sure if it would help the situation.
Horizon provides a monitoring service that may help to track down problems with this setup. It may also improve efficiency a bit if the application uses other queues that Horizon can distribute work between when idle, but the question doesn't seem to indicate that this is the case.
Each of the three clones has the following worker configuration:
command=php /home/forge/default/artisan queue:work sqs --sleep=10 --daemon --quiet --timeout=30 --tries=3
(Sidenote: for Laravel 5.3 and later, the --daemon option is deprecated, and the queue:work command runs in daemon mode by default.)
I have multiple queue workers on my Laravel 5.2 project. I am running on AWS. I am using ECS. I am using Redis for my queue driver.
I would like to know..
If I have 2 servers working the same queue
php /var/www/laravel/artisan queue:listen --env=production --timeout=30 --tries=1 --queue=mail
Will they both process the job, thus it gets processed twice? Or will it only get processed once and this will help with the load/redundancy?
Many Thanks in advance!
A job only exists on a queue once, as soon as a worker grabs the job, it is removed from the queue.
So as long as the different workers are accessing the same instance of the same queue, the jobs will only be executed once.
I have a beanstalkd queue implementation. I have created a benchmark tool to test the time of my queue processing. When I open my PHP worker in a terminal tab like this:
php artisan queue:work --daemon --sleep=0
It takes around 8 seconds to finish all my jobs. If I open total 4 terminal tabs and run the above line then test again with same load, it takes around 2 seconds. Which is great.
However when I run my workers using supervisord it takes around 7-8 seconds to process the same load, with 4 workers. Even if I increase my workers to 20, I don't see any impact. Sometimes I see increasing worker has a negative impact (11 seconds). I am sure all 20 workers are running.
So what's the issue here? How can I speed up my processing? In a real server I can't keep workers running in terminal tabs.
I will highly appreciate your help :)
Here's my supervisor config:
[program:app-worker]
process_name=%(program_name)s_%(process_num)02d
command=php artisan queue:work --daemon --sleep=0
directory=/home/vagrant/Code/app
autostart=true
autorestart=true
user=vagrant
redirect_stderr=true
numprocs=4
stdout_logfile=/home/vagrant/Code/app/storage/logs/worker.log
Even though this ticket is 2 years old, I think it's worth an answer.
My guess is this would have been caused by the explicit log file you're setting and the redirection of stderr, in addition, based on your config, looks like you're running this locally, and in a non-isolated environment, meaning your host OS operations and apps would be interfering with your benchmarks.
Either way, I'm sure you've solved this long ago :D
I'm having a bit of trouble grasping the concept of Queues in laravel 5.0. From what I understand, queues store a list of Commands to be executed by either by the php artisan queue:listen command or the php artisan queue:work --daemon command.
Correct me if I'm wrong, but the php artisan queue:listen just waits until there is a command in queue and then executes it, right? Then what does the php artisan queue:work --daemon command do in comparison? Does this command only work one at a time?
Anyways, a task I wish to complete is this... I want to periodically check if there are commands in the queue and if there are I wish to execute them. Since this is a periodic problem I assume a chron would be used, but how do i check if there are outstanding commands to be? Would I run a query on my queue table? If I do, how would I dispatch the command? Or should I just chron the php artisan queue:listen command
The main difference between queue:listen and queue:work is that the second one only sees if there's a job waiting, takes it and handles it. That's it.
The listener though runs as a background process checking for jobs available all the time. If there's a new job it is then handled (the process is slightly more difficult, but that's the main idea).
So basically if you need to handle your commands (jobs) as soon as they appear you might wanna use the queues.
And if you need to do something periodically (for example every 2 hours or every Monday at 9 AM etc.) you should go with cron + Schedule.
I wouldn't recommend combining them as you described. Also keep in mind that jobs can be delayed if you don't want then do be handled ASAP.
If I am running Beanstalk with Supervisor on a server with a Laravel 4 application, and I want it to process all queues asynchronously -- as many as it can at the same time -- can I have multiple listeners running at the same time? Will they be smart enough to not "take" the same to-do item from the queue, or will they all reach for the same one at the same time, and thus not work in the way I'm wanting? In short, I want to use Queues to process multiple tasks at a time -- can this be done?
php artisan queue:listen && php artisan queue:listen && php artisan queue:listen
In short, I want to use Queues to process multiple tasks at a time -- can this be done?
In short - yes it can be done. Every job taken by a worker is locked until it's release. It means that other workers will get different jobs to process.
IMO it's better to configure Supervisor to run multiple queue:work command. It will take only one job, process it and stop execution.
It's not encouraged to run PHP scripts in a infinite loop (as queue:listen does), because after some time they can have memory issues (leaks etc).
You can configure Supervisor to re-run finished workers.