I have a Laravel queued job which extracts links from a webpage. The timeout for the Queue listener configured through Laravel Forge is 240 seconds (4 minutes). However, jobs are taking up to 45 minutes to run.
My queue settings are:
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => 'default',
'retry_after' => 350,
],
And there are multiple job processes running - up to 35 processes. As you can imagine, this is eating up a lot of server memory. The processes just seem to hanging around. The command for these processes as shown in top is:
php7.1 artisan queue:work redis --once --queue=linkqueue --delay=0 --memory=128 --sleep=10 --tries=1 --env=local
How can a job run for 45 minutes if the timeout is 240 seconds? Why are there so many processes - shouldn't there just be one?
Also, any ideas why a script for extracting links should take 45 minutes to run?!
The script does work, that is, in most cases it runs as expected - it just takes ages. There are no errors reported/logged as far as I can see.
Code in the job is:
$dom = new DOMDocument;
$dom->loadHTML($html);
$links = $dom->getElementsByTagName('a');
foreach ($links as $a) {
$link = $a->getAttribute('href');
$newurl = new URL;
$newurl->url = $link;
$newurl->save();
}
Update: Another simple job runs just fine, in under a second. It is specifically just the link job above that is taking 10s of minutes. Could it be a RAM issue or something? Is there anything else I can do to diagnose the problem? When run as part of a console job, the extract links function itself runs in 1 or 2 seconds. It is only on the queue that it freaks out.
How can a job run for 45 minutes if the timeout is 240 seconds?
Because you have 'retry_after' => 350, on your queue connection. This means if Laravel does not hear from the job after 350 seconds - it assumes the job has failed and retries again. This is resulting in multiple processes of the one job in your situation.
If you are happy to allow your jobs to run for up to 45mins - then you should set retry_after to be a larger number. Say 3600 which is 1 hour.
That way a job will only start if it takes longer than 1 hour to run.
You can also do the following to make timeouts unlimited.
php artisan queue:listen --timeout=0
Related
I have a job scheduled to run every hour. 'withoutOverlapping' is set for this job and mutex expiry time is 24 hour.
But when I checked cron.log. I found that it is not running every hour. But it runs once in a day and at the end of job, there is a redis write error. I think this error is getting when clearing mutex (lock) of this job from redis after completion of job execution. Then after 24 hour, mutex expires automatically, then this job runs next hour.
To check mutex clears from redis or not, I need to see mutex of this job exists. So I tried below cod in tinker but it is showing zero events. Actually there are 7 jobs scheduked
>>> $s = new Illuminate\Console\Scheduling\Schedule();
=> Illuminate\Console\Scheduling\Schedule {#3543}
>>> count($s->events())
=> 0
>>>
So using tinker how can I achieve this ?
According to the Laravel docs I should be able to specify a job specific timeout:
If the timeout is specified on the job, it will take precedence over any timeout specified on the command line [...]
So, when I run artisan queue:listen without the --timeout option and I define the timeout inside the job (like Laravel tells me to):
public $timeout = 600;
I expect the timeout of that specific job to be 600 seconds. Unfortunately, I still get a ProcessTimedOutException. A custom timeout only works when I run the queue with --timeout=600.
I'm using Laravel 6 with PHP 7.4. As recommended by Laravel I've also enabled the pcntl PHP extension. For the queue I use the database driver with the following config:
'database' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'default',
'retry_after' => 90,
]
I opened a bug report because I couldn't get this to work. However, it seems like the timeout specified inside the job class only takes precedence over the timeout specified in the command line when running the queue with queue:work.
I've tested this and can confirm that it works with queue:work. According to a commenter on my bug report it doesn't work with queue:listen because:
queue:listen runs several processes while queue:work is a single process. queue:listen sets a timeout for the process it runs so we don't leave ghost processes running on the machine in case the master process was killed for some reason.
I'm using https://cron-job.org/ for a cron job but I have a problem. When running the script manually (for example, a script that gets data from a csv file), it works but if I run the script through that cron job, it fails because of that 30 seconds max_execution_time I guess.
But the problem is that in my script I'm already using:
ini_set('max_execution_time', 300);
It should be 5 minutes instead of 30 seconds before the cron job fails. What am I doing wrong?
Here is an image with that cron job history:
cron-job.org has some limits:
How and how long does cron-job.org visit my URLs?
cron-job.org visits your URLs at the configured dates/intervals and waits for the URL/script to finish execution. If your URL/script does not finish after 30 seconds, it will timeout and we will close the connection to prevent delays in the execution of the jobs of other users. Our system reads up to 1024 bytes of the output of your URLs/scripts. In case your script sends more data, job execution will be aborted. (Please also see the next question.)
You can read more here: FAQ.
This question may seem repetitive, as there are many threads around with the same subject, but thing is that most solutions seems to be linked with terminal coding, which i'm not comfortable with. The problem is simple i have a php script that needs to be executed very 10 seconds. Cron job in cpanel allows only upto 1 minute. What's the workaround to let cron work every 10 seconds ?
Let the cron job run after every minute and in your php script the following code example might help you out. I have used counter limit to 6 because this script will run after every ten seconds and six times in one minute.
<?php
for($i=0;$i<6;$i++){
sleep(10);
task();
}
function task(){
}
You can use simple bash-script like
#!/bin/sh
while [ true ]
do
php script.php
sleep 10
done
CRON jobs are the standard way to run some tasks periodically. Setting cron jobs require access to the terminal. However, some shared hosting providers don't provide this and you need to set it up through their interface.
If you hosting don't provide this you can third-party services that will call you url every 'X' seconds.
Here are few of them:
https://azure.microsoft.com/en-us/services/scheduler/
https://atrigger.com/
You can Google for more
Note: You can have publically expose the PHP file as an url
You can use the GUI Cpanel, select "once per minute" an try with something like the next command:
/path/to/bin/php /path/to/script.php; sleep 10; /path/to/bin/php /path/to/script.php; sleep 10; /path/to/bin/php /path/to/script.php; sleep 10; /path/to/bin/php /path/to/script.php; sleep 10; /path/to/bin/php /path/to/script.php; sleep 10; /path/to/bin/php /path/to/script.php
I'm running a PHP code via CLI by setting a cron-job. The script reads about 10000 records from database and runs 10000 new scripts (by exec command) without waiting for previous script to be done. I use this because I want all those tasks run fast. (each one takes about 10 seconds).
When number of tasks that are running gets large, CPU usage become 100% and can't work with server (CentOS). How can I handle this?
You need to limit the number of scripts running in parallel at any given time because running 10,000 concurrent scripts is clearly saturating your system. Instead, you should queue up each task and process 25 or 50 (whatever causes a reasonable amount of load) tasks at the same time.
Without much knowledge of how these scripts actually work, I can't really give you much advice code-wise, but you definitely need to have a queue in place to limit the number of concurrent instances of your script running at the same time.
Also check out semaphores, they might be useful for this producer/consumer model
I recently wrote a script that handle parallel execution of commands. It basically allows the operator to tune the number of concurrent processes at runtime.
If you feel that CPU usage is too high, just decrease the value in /etc/maxprunning
CMD="path-of-your-php-script"
function getMaxPRunning {
cat /etc/maxprunning
}
function limitProcs {
PRUNNING=`ps auxw | grep -v grep | grep "$CMD" | wc -l`
#echo "Now running $PRUNNING processes, MAX:$MAXPRUNNING"
[ $PRUNNING -ge $MAXPRUNNING ] && {
sleep 1
return
}
#echo "Launching new process"
sleep 0.2
$CMD &
}
MAXPRUNNING=`getMaxPRunning`
C=1
while [ 1 ];do
MAXPRUNNING=`getMaxPRunning`
limitProcs
done
If you want your php scripts to ignore an accidental parent's death, put this line at the top of the php script
pcntl_signal(SIGINT, SIG_IGN);