I wrote some tvguide scrapers in php. I run them from a script that is executed by a cronjob. This script is run every minute, and checks if a scraper needs to start. This way i can alter and manage these scraping jobs without having to modify the cron itself.
These scraping scripts vary in runtime, some take no more than 1 minute, and others can take up to 4 hours. When i run them one after another there is no problem. But when i try to run two script simultaniously - one or both scripts hang. Resulting in a email from the cron:
sh: line 1: 700865 Hangup /usr/local/bin/php /home/id789533/domains/erdesigns.eu/public_html/tvg_schedules/scraper.php --country=dk --provider=1 --scraper=tv2 2>&1
Where the /usr/local/..... is the command for the script, and which is called from the scheduler script.
I just cant find anything related to this message, and i have no idea how to fix it. I can send the script itself if needed.
All advise and help would be apreciated.
[Edit] I also took a look at the resource usage, and the load never gets higher than 150mb and 15% load. I have a limit of 400% and 1GB.
I execute the scripts from the php script like so:
shell_exec(sprintf("/usr/local/bin/php %s 2>&1", $scraper));
where $scraper is the filename. It executes the script like it should, but after a while i get the message sh: line 1: 000000 Hangup
I know for sure that it is not allocating to much memory, someone who can direct me to the right way? I dont know where to look right now.
PHP is a language intended for the web with features like a cap for maximum execution time to make sure scripts do not run indefinitely and thereby blocking resources. Therefore, PHP is not the best choice for this task.
If it is only a short script I would advice you to convert it into a BASH or Python script. However, if you want to stick to PHP, check your php.ini file for settings restricting execution time.
Related
I faced very strange problem on my hosting.
I have script and it can be triggered using URL like this
https://mywebsite.com/script.php
I need this script to be executed one time in two days.
So I created Cron job just like my hosting provider advised me.
wget -O /dev/null -q 'https://mywebsite.com/script.php'
It uses wget because script requires some extra scripts - so my hosting provider said that I need to create task like this.
It worked fine for about a month but for few weeks I have a problem.
For some reason that I and my hosting provider can't understand when I run script by opening URI in browser - script executed fine (I know it because of emails that are sended in 4 different steps of execution). But when Cron execute this scripts it executes infinitely - so I continue to receive emails for numerous times until I rename script or delete it.
Script execution time is about 2-3 minutes. So when I run it from URL and wait till it finishes - I get error on the screen that time of request (60 sec) is over. But I know that scripts executes fine till the last step.
What is the problem?
Wget
I had the same problem at some point with a php based cronjob. The Problem was, that wget itself can have a timeout. If this timeout is reached, wget will try again and again.
Try to use some wget options to make sure it runs as you want it to run.
Example:
wget -O /dev/null --tries=1 --timeout=600 'https://mywebsite.com/script.php'
--tries tells how many times it will try to execute if a timeout appears.
--timeout specifies the max exec. time in seconds.
Those options can be specified at cronjob level as well.
PHP Cronjobs
If possible it will may be a betther choice to let PHP run your cronjob directly. If you know the servers php directory you could create a cronjob like
/usr/bin/php /srv/www/yousite/html/script.php
In this case you have no third party programm like wget to rely on. If this helps depends on how the cronjob is built. If your cronjobs uses $_SERVER variables for example, this would not work.
There are some settings you want to check, before you use any PHP file as cronjob.
Keep in mind that the php configuration set within the php.ini could also have an impact on unwanted errors on PHP Cronjobs in general. In the php.ini there is a value called "max_execution_time" where the max seconds to process a php request is defined.
An other setting you might want to get your eye on is the "memory_limit" wich is also defined within the php.ini configuration. This configuration defines the max. memory a php request can use. As your cronjob seems to run for 2-3 minutes, that could mean that maybe a lot of data is stored in memory while you use it.
Be aware, that any request uses those limits. If you set them to high it may will cause problems with CPU load on your server, or with too many spawned php processes.
If you have a shared hosting service or something similar, you may not be able to change any of those settings.
I have developed a php script, that I want to run continuously.
For example, the script run.php executes a list of tasks, but I don't know how long it will take: it can take 30 sec, 1 min, 2 min, or more.
The problem is this script can't be executed simultaneous.
So I can't use the cron job because if I setup a cron job every minute, but the script is running during more than 1 minutes, I'll have bugs).
In fact, I want that this cron job to be Perpetual executed : everytime run.php is ended, the script run.php is reloaded, again and again...
I don't know how to resolve this problem.
Any help please ?
Thank you.
You've [at least] two options:
Under run.php code, determine if it is already running (by setting some external control system that is checked at the beginning of the script) and setup the crontab job as you described. [I recommend this approach].
Use something like (while :; do php run.php; done) & run it once, and put it inside the /etc/rc.local to make it run at every system start. I don't really like this approach, but it's a possible solution.
I have some limitations with my host and my scripts can't run longer than 2 or 3 seconds. But the time it will take to finish will certainly increase as the database gets larger.
So I thought about making the script stop what it is doing and call itself after 2 seconds, for example.
Firstly I tried using cURL and then I made some attempts with wget. But there is always a problem with waiting for the response and timeouts (with cURL, for example, I just need to ping the script, not wait for a response) or permissions with the server (functions that we use to run wget such as exec seems to be disabled on my server, or something like that).
What do you think is the best idea to make a PHP script ping/call itself?
On Unix/LInux systems I would personally recommend to schedule CRON JOBS to keep running the scripts at certain intervals
May be this SO Link will help you
Php scripts generally don't call other php scripts. It is possible to spawn a background process as illustrated here, but I don't think that's what you're after. If, so you'd be better off using cron as was discussed above.
Calling a function every X amount of seconds with the same script is certainly possible, but this does the opposite of what you want since it would only extend the run time of the script in question.
What you seem to be asking is, contrary to your comment, somewhat paradoxical. A process that calls method() every so often is still a long running process and is subject to the same restrictions as any other process on the server, regardless of the fact that it may be sitting idle for short intervals.
As far as I can see your options are:
Extend the php max_execution_time directive, or have your sysadmin do so if they are willing
Revise your script so that it completes within the time limit
Move to a new server
I'm creating a webservice for an Android app in PHP with MySQL. I want to continuously check whether any data is available. I haven't got any idea how to get data as a background process. How can I execute a query without any request or without calling file?
I searched and got some code like
$command = "php -d max_execution_time=50 -f myfile.php '".$param."' >/dev/null &";
exec($command);
But where should I put this code so this query will run continuously?
Yes, the ampersand trick will work. You can use something like supervisord to restart it every few hours, so that any memory leaks are dealt with. This also makes it less fragile if it were to crash or hang.
Also, you can use something like cron to run a task for 10 minutes, and then die off and wait for cron to start it again - bear in mind that with most background tasks, it doesn't matter if there's a short period the task is not running, since it will catch up. It's worth checking in each run whether the previous one is still running, and exit early if it is: that way you don't have two background tasks causing race-conditions when retrieving work from your database.
Finally you can use a job server, such as Gearman. This will allow you to send tasks to it in an asynchronous fashion, and they will be run by worker tasks (in either time or priority order). This is probably the most reliable approach, but it takes a bit more work to set up. There's a PHP module for this, but in my experience it's more of a hassle to use than Net_Gearman, which is available in PEAR.
I was wondering how to make a php daemon script that runs one time at the day?
Do you know any good frameworks with benefits?
or is it just small code?
Thanks
I was wondering how to make a php deamon script that runs one time at
the day?
In order to do this, get familiar with cron jobs. A cron job is a function that gets executed by the server on a time interval. Usually you'd edit your "crontab" by executing crontab -e
Then, once inside, you'd write the interval you want, followed by the command.
Typically it looks like:
30 18 * * * rm /home/someuser/tmp/* > /home/someuser/cronlogs/clean_tmp_dir.log
Since its PHP, you can either a) run your php command as a php cli command, OR b) you can make the command get executed when a particular page is run... and just execute that in cron via a curl -X GET 'http://url/' (etc.)
Also, note that you can write all of your stuff in a shell script file and actually run that file as your cron command... that reduces line-item complexity
cron
Sorry I haven't closed this one.
I actually discovered that my host didn't allowed cron jobs running. So I found a relevant homepage that offer a free service to make a request for me when I needed. In my case, I have specified a url link that should be requested to my RESTful API each day.
The link is here and works like a charm :)