I have a PHP script that pulls down a bunch of RSS feeds. To prevent overloading the publishers' servers, I use the PHP Sleep function to slow things down.
The entire script could last for a couple of hours.
If I run this from a Cron job on GoDaddy, it will happily work for 5 - 10 minutes and then return a server error. I checked and the PHP maximum execution time is 30 seconds, so I'm not sure if this is the cause of the problem.
If I run the job on my Mac, my local PHP also has a default maximum execution time of 30 seconds, but this script does work if I run it from the terminal, but I don't understand why.
How do I loop a script that will exceed 30 seconds without running into unreliability problems?
Help appreciated.
Short answer is use set_time_limit(0) to allow for a long-running script. Your terminal (CLI) PHP probably has it set to 0. You could also be running out of memory, especially on PHP 5.2 or older. Log all errors to a file, and inspect it.
You could rewrite your program to be able to work on a subset of the data during one run. The benefit of that approach is you could use it to run 24/7 or to run every five minutes, depending on what the PHP environment supports. You could also run multiple instances at a time, each working on their own data.
Related
To give a bit of background to my project first, I have a Wordpress website of which I have turned off the cron by tweaking the wp-config to:
define('DISABLE_WP_CRON', true);
I then set up a cron to the wp-cron file on the server to call the file every minute and this has been working as expected and as I need it to.
I use a plugin called 'Wp-Crontrol' to set up my own cron calls on various functions within my functions file.
Memory Limit: 2048M
PHP max execution time: 2700
The Issue
I have one function in particular that takes a report from a remote source and loops through each line entering each line into the database using the $wpdb class. If i place the function on a php page and go to it the function works perfectly as expected and enters all 6900 rows into the database after some time.
The way I would like it to work is to run a cron on that function (like I do with so many other things with no problems) but the issue is that when I set it up via cron it only seems to insert around 3000 rows before it just stops with no errors logged?
I am struggling to work out why running it manually would work perfectly but a scheduled cron of the same function during the night would only do half the job and not finish, stopping half way through?
I have turned on all error logging I can think of but nothing shows?
WordPress "cron" jobs are not the same thing as a system cron job. For instance, unless you have a high traffic site, it's pretty much impossible to set a gaurenteed 60 second cron with WordPress. WordPress crons are activated when someone puts an HTTP request into the system. A system cron runs off a daemon that runs in system memory. They are just different concepts and work differently.
Issue was processing time but for some reason it would not flag at all in any error messages. After I thinned out long processes it started behaving as expected.
I have a cron script which calls a PHP file every minute. That file checks to see if any actions need to be taken and, if so, takes those actions. When the execution time is long, however, this script doesn't always finish. I'm not seeing any error codes, and my built-in logging simply stops, since it only runs with the file being executed.
The thing is: the amount of time varies over a dozen seconds in range, so I'm not sure if it's 300 seconds to my timeout when sometimes the script seems to quit after just 280.
This is being run on an AWS ubuntu server.
Is there a list of all the places I might check for errors or timeouts? The two php.ini files are both set to a half-hour, and my script is not running for nearly that long.
I have a script that updates my database with listings from eBay. The amount of sellers it grabs items from is always different and there are some sellers who have over 30,000 listings. I need to be able to grab all of these listings in one go.
I already have all the data pulling/storing working since I've created the client side app for this. Now I need an automated way to go through each seller in the DB and pull their listings.
My idea was to use CRON to execute the PHP script which will then populate the database.
I keep getting Internal Server Error pages when I'm trying to execute a script that takes a very long time to execute.
I've already set
ini_set('memory_limit', '2G');
set_time_limit(0);
error_reporting(E_ALL);
ini_set('display_errors', true);
in the script but it still keeps failing at about the 45 second mark. I've checked ini_get_all() and the settings are sticking.
Are there any other settings I need to adjust so that the script can run for as long as it needs to?
Note the warnings from the set_time_limit function:
This function has no effect when PHP is running in safe mode. There is no workaround other than turning off safe mode or changing the time limit in the php.ini.
Are you running in safe mode? Try turning it off.
This is the bigger one:
The set_time_limit() function and the configuration directive max_execution_time only affect the execution time of the script itself. Any time spent on activity that happens outside the execution of the script such as system calls using system(), stream operations, database queries, etc. is not included when determining the maximum time that the script has been running. This is not true on Windows where the measured time is real.
Are you using external system calls to make the requests to eBay? or long calls to the database?
Look for particularly long operations by profiling your php script, and looking for long operations (> 45 seconds). Try to break those operations into smaller chunks.
Well, as it turns out, I overlooked the fact that I was testing the script through the browser. Which means Apache was handling the PHP process, which was executed with mod_fcgid, which had a timeout of exactly 45 seconds.
Executing the script directly from shell and CRON works just fine.
I have to run a pretty heavy task on PHP once a week (script that curls to various locations (websites, API's), gathers, sorts data and inserts it into a db). The whole script takes about 10 to 15 mintues to run on my mac (localhost) - guessing it'll run a bit faster on a server. Nevertheless - I'm currently looping through with AJAX, so when each task is finished, next one is launched. Now I need to run it weekly, automatically. So I think I can't do it with AJAX Anymore.
Do I have to just set the php.ini to let a script run for 30 mintues or there is a better way to do it ?
The maximum execution time of the PHP script is determined by the amount of time in which no output has been generated. So writing data into STDOUT (e.g. to a logfile) will keep the script running.
However, if you're running the script from command line, the max-execution-time will be defaulted to zero anyway and as already suggested, I'd start the script with a cronjob instead of an AJAX-Request or similar methods. I actually do that for most of my php-scripts performing administrative tasks like synchronizing data across several databases or similar purposes.
php.ini has nothing to do with scheduling jobs. It's simply definining PHP's startup settings. What you want is a cron job, as your title says.
For OSX cron setup, see http://hintsforums.macworld.com/showthread.php?s=&threadid=39005
As a possible alternative to using cron jobs, I found the sleep function. I have never used this before.
If I tell my script to run inside a kind of loop, and inside that loop I have an instruction like this
# sleeps for 86400 seconds or one day
sleep(86400);
will my script be launched again after 1 day? even if don't access it on my web browser again within that period?
I think is not possible, but I'm here to ask an expert about it.
The main problem with using PHP this way is, in my experience, not web server timeouts (there are ways to handle that with difficulty varying on the server and the platform) but memory leaks.
Straightforward PHP code tends to leak a lot of memory; most of the scripts I wrote were able to do hundreds of times as many work after I did some analysis and placed some unsets. And I was never able to prevent all the leaks this way. I'm also told there are memory leaks in the standard library, which, if true, makes it impossible to write daemons that would run for a long time in loops.
The script will timeout. You need to set it so that it won't timeout using set_time_limit.
I wouldn't do this I would either use a cron (that is a link) job if it is a regular task or an at (that is a link) job if the job is added at the run time of your script.
cron allows you to run a recurring job every day at 1pm for example whereas at allows you to schedule a job to run once for now +1day for example.
I have written a PHP 5.3 wrapper for the at queue if you choose to go down that route. It is available on GitHub https://github.com/treffynnon/PHP-at-Job-Queue-Wrapper
There is also time_sleep_until(). Maybe more useful to wake up on a specific time...
If you access the script through a web browser, it will be terminated after 30 seconds.
If you start the PHP script on the command line, this could work.
It would work, but your "startup time" will be subject to drift. Let's say your job takes 10 seconds to run, then sleeps 86400, runs another 10, sleeps 86400, etc.. You start it exactly at midnight on day 1. On Day 2 it'll run at 12:00:10am, on day 3 it's 12:00:20am, etc...
You can do some fancy math internally to figure out how long the run took, and subtract that from the next sleep call, but at the point, why not use cron? With cron the script will exit after each run, cleaning up memory and resources used. With your sleep method, you'll have to be VERY careful that you're not leaking resources somewhere, or things will eventually grind to a halt.
I had a similar problem before and found a php cron parsing class that will allow you to execute php similar to running crons. You can tie it to a commonly accessed script on your site if you don't have access to run crons directly.
I actually use this script as part of a larger cron job script:
a cron job runs every hour
an xml file for each sub-cron with a cron-like time component(i.e.- * */2 * * * php /home..)
the sub-cron script that will run if the current time meets the criteria of the sub-cron time component
a user interface is setup so that I don't have to manually add/remove sub-crons from the main cron
The cronParser class is here.
Many correct answers, but: Using sleep() means your script keeps running, and keeps using memory. Raising the default timeout of 30s will work, but again, this is bad idea. I suggest you use crontasks.
This is why legitimate cron jobs were invented. Just use crontab. Using a PHP script to do it will be EXTRAORDINARILY unreliable, buggy, and poorly timed.
Hope this is insightful.