I have a script that updates my database with listings from eBay. The amount of sellers it grabs items from is always different and there are some sellers who have over 30,000 listings. I need to be able to grab all of these listings in one go.
I already have all the data pulling/storing working since I've created the client side app for this. Now I need an automated way to go through each seller in the DB and pull their listings.
My idea was to use CRON to execute the PHP script which will then populate the database.
I keep getting Internal Server Error pages when I'm trying to execute a script that takes a very long time to execute.
I've already set
ini_set('memory_limit', '2G');
set_time_limit(0);
error_reporting(E_ALL);
ini_set('display_errors', true);
in the script but it still keeps failing at about the 45 second mark. I've checked ini_get_all() and the settings are sticking.
Are there any other settings I need to adjust so that the script can run for as long as it needs to?
Note the warnings from the set_time_limit function:
This function has no effect when PHP is running in safe mode. There is no workaround other than turning off safe mode or changing the time limit in the php.ini.
Are you running in safe mode? Try turning it off.
This is the bigger one:
The set_time_limit() function and the configuration directive max_execution_time only affect the execution time of the script itself. Any time spent on activity that happens outside the execution of the script such as system calls using system(), stream operations, database queries, etc. is not included when determining the maximum time that the script has been running. This is not true on Windows where the measured time is real.
Are you using external system calls to make the requests to eBay? or long calls to the database?
Look for particularly long operations by profiling your php script, and looking for long operations (> 45 seconds). Try to break those operations into smaller chunks.
Well, as it turns out, I overlooked the fact that I was testing the script through the browser. Which means Apache was handling the PHP process, which was executed with mod_fcgid, which had a timeout of exactly 45 seconds.
Executing the script directly from shell and CRON works just fine.
Related
I've got a sever which has an action triggered by a frequent cron job.
This is a php application build on Silverstripe (4.0).
The issue I'm facing is the php processes stay alive and also keep database connections open. This means after a few days the site stops working entirely once SQL stops accepting new connections.
The system has two tasks on cron jobs;
One takes a massive CSV file and spits it into smaller sub files which are then imported into the database. This one uses a lock file to prevent it running into a previously running instance. I'm not too sure if this is working.
The second task processes all the records which have been updated in large batches.
Either of these could be the source of the overloading but I'm not sure how to narrow it down.
What's the best way to diagnose the source of the issue?
In terms of debugging, this would be like any other task profiling the application with something like xdebug and kcachegrind. To ensure that processes do not run for too long you can limit the max_execution_time for the PHP.ini for the CLI.
To then let the CLI process run for a long time, but only just enough time add something to set the max execution time on a per row basis:
$allowed_seconds_per_row = 3;
foreach($rows_to_process as $row){
set_time_limit($allowed_seconds_per_row);
$this->process($row);
}
You can also register a shutdown function to record the state as the script ends.
It is likely that the memory is a key cause for failure and debugging focused on the memory usage and that can be controlled by unsetting variable data as needed.
To give a bit of background to my project first, I have a Wordpress website of which I have turned off the cron by tweaking the wp-config to:
define('DISABLE_WP_CRON', true);
I then set up a cron to the wp-cron file on the server to call the file every minute and this has been working as expected and as I need it to.
I use a plugin called 'Wp-Crontrol' to set up my own cron calls on various functions within my functions file.
Memory Limit: 2048M
PHP max execution time: 2700
The Issue
I have one function in particular that takes a report from a remote source and loops through each line entering each line into the database using the $wpdb class. If i place the function on a php page and go to it the function works perfectly as expected and enters all 6900 rows into the database after some time.
The way I would like it to work is to run a cron on that function (like I do with so many other things with no problems) but the issue is that when I set it up via cron it only seems to insert around 3000 rows before it just stops with no errors logged?
I am struggling to work out why running it manually would work perfectly but a scheduled cron of the same function during the night would only do half the job and not finish, stopping half way through?
I have turned on all error logging I can think of but nothing shows?
WordPress "cron" jobs are not the same thing as a system cron job. For instance, unless you have a high traffic site, it's pretty much impossible to set a gaurenteed 60 second cron with WordPress. WordPress crons are activated when someone puts an HTTP request into the system. A system cron runs off a daemon that runs in system memory. They are just different concepts and work differently.
Issue was processing time but for some reason it would not flag at all in any error messages. After I thinned out long processes it started behaving as expected.
I need to write a script that reads a big .csv and generates a file for each row.
Since on this server I can't modify timeout with set_time_limit, in my script that is manually executed in browser via http I get 10 row, execute my code for each, reach the end and auto reload for next 10 row, so on till I reach the end.
Now I need to convert this script into a web service, but I don't know how to avoid that the scripts timeout.
set_time_limit(900) // does not work
From reading here it seems it has a restriction.
This function has no effect when PHP is running in safe mode. There is no workaround other than turning off safe mode or changing the time limit in the php.ini.
Do you have safe mode enabled? If so, it seems you may have to disable it. You can also see other values that do the same/similar actions here, though they also require safe mode to be disabled.
I have to run a pretty heavy task on PHP once a week (script that curls to various locations (websites, API's), gathers, sorts data and inserts it into a db). The whole script takes about 10 to 15 mintues to run on my mac (localhost) - guessing it'll run a bit faster on a server. Nevertheless - I'm currently looping through with AJAX, so when each task is finished, next one is launched. Now I need to run it weekly, automatically. So I think I can't do it with AJAX Anymore.
Do I have to just set the php.ini to let a script run for 30 mintues or there is a better way to do it ?
The maximum execution time of the PHP script is determined by the amount of time in which no output has been generated. So writing data into STDOUT (e.g. to a logfile) will keep the script running.
However, if you're running the script from command line, the max-execution-time will be defaulted to zero anyway and as already suggested, I'd start the script with a cronjob instead of an AJAX-Request or similar methods. I actually do that for most of my php-scripts performing administrative tasks like synchronizing data across several databases or similar purposes.
php.ini has nothing to do with scheduling jobs. It's simply definining PHP's startup settings. What you want is a cron job, as your title says.
For OSX cron setup, see http://hintsforums.macworld.com/showthread.php?s=&threadid=39005
I have a massive amount of data that needs to be read from mysql, analyzed and based on the results split up and stored in new records.
five record takes about 20 seconds, but the records vary in length so I can't really estimate how long the program will take, however have calculated that the process should not take longer much longer than 5 hours, so I'd like to run it over night and feel quite sure that when I come back to the office the next morning the program is done.
Assuming the code is fail safe (I know right ;) how should set up Apache / PHP /Mysql settings so that when I execute the script so that I can be sure that the program will not time out and/or not run out of ram?
(it is basically running in a loop fetching sets of 100 rows until it can't anymore a loop so, I am hoping the fact that the variables are being reset at the beginning of each iteration will keep the memory usage constant.)
The actual size of the database when dumped is 14mb, so the volume of the data is not so high
(on a side note, it might also be that I haven't assigned the maximum resources to the server settings, so maybe that's why it takes 20 seconds to run 5 records)
Make sure you have removed any max_execution_time limits by setting this to 0 (unlimited) in your PHP.ini or by calling set_time_limit(0). This will ensure that PHP doesn't stop the script mid-execution.
If it all possible, you should run the script from the CLI so that you don't have to worry about Apache timing your request out (it shouldn't, but it might).
Since you are working with only 15 MB of data I wouldn't worry about memory usage (128 MB is the default in PHP). If you are really worried you can remove memory limits in PHP by modifying the memory_limit to be either a higher number of -1 (infinite memory).
Keep in mind modifying the PHP.ini will affect all scripts that are interpreted by that installation. I prefer to use the appropriate ini setting functions at the top of my scripts to prevent dangerous global changes.
On a side note: This doesn't really sound like a job for PHP. I'm not trying to discourage your use of PHP here, but there are other languages that are better suited for command line usage.
Better make your script exit the execution, and then restart it. Store the point
where it left last time. This will ensure you do not have memory leaks and script does not run out of memory due to some error in garbage collection,and that
the execution continues if there is unexpected failure.
A simple shell command would be :
while [ 1 ]; do php myPhpScript.php a; done
you can make other checks to ensure proper running.
I'd like to point out, by default scripts run via a CLI in PHP, default to having no time limit, unlike scripts run through CGI, mod_php etc.
And as stated avoid running this via Apache.
However if you MUST do this, consider breaking it down. You can make a page that could process 5-10 results, appends the dump file, then prints out either a meta refresh, or some JavaScript to reload the page with with a parameter telling it where it's up too, and to continue until done.
Not recommended though.
adding to some of the other good options here you might want to look at http://www.electrictoolbox.com/article/php/process-forking/ and also sending some requests to dev/null if you dont need them to give back feedback.
Don't do this using a web interface. Run it from the command line; but look to see if your code can be optimised, or set break points and do it in "chunks"
First of all http://php.net/manual/en/function.set-time-limit.php put set_time_limit(0); at the beginning of the script.
As for the memory you should take care of that by unsetting any variables, array, pointers that you do not need on each iteration.
Better run the script from the shell (CLI) or as cronjob.
As far as I know MySQL connections do not time out, so you should be safe by setting:
php_value max_execution_time X
in a .htaccess file or placing set_time_limit(X) at the beginning of your script where X is a comfortable value in seconds.