I have a script that is very long to execute, so when i run it it hit the max execution time on my webserver and end up timing out.
To illustrate that imagine i have a for loop that make some pretty intensive manipulation one million time. How could i spread this loop execution in several parts so that i don t hit the max execution time of my Webserver?
Many thanks,
If you have an application that is going to loop a known number of times (i.e. you are sure that it's going to finish some time) you can increase time limit inside the loop:
foreach ($data as $row) {
set_time_limit(10);
// do your stuff here
}
This solution will protect you from having one run-away iteration, but will let your whole script run undisturbed as long as you need.
Best solution is to use http://php.net/manual/en/function.set-time-limit.php to change the timeout. Otherwise, you can use 301 redirects to send to an updated URL on a timeout.
$threshold = 10000;
$t = microtime();
$i = isset( $_GET['i'] ) ? $_GET['i'] : 0;
for( $i; $i < 10000000; $i++ )
{
if( microtime - $t > $threshold )
{
header('Location: http://www.example.com/?i='.$i);
exit;
}
// Your code
}
The browser will only respect a few redirects before it stops, you're better to use javascript to force a page reload.
I someday used a technique where I splitted the work from one file into three parts. It was just an array of 120.000 elements with intensive operation. I created a splitter script which stored the arrays in a database of the size of 40.000 each one. Then I created an HTML file with a redirect to the first PHP file to compute the first 40.000 elements. After computing the first 40.000 elments I had again a HTML forward to the next PHP file and so on.
Not very elegant, but it worked :-)
If you have the right permissions on your hosting server, you could use the php interpreter to execute a php script and have it run in the background.
See Asynchronous shell exec in PHP.
if you are running a script that needs to execute for unknown time, you can use:
set_time_limit(0);
If possible you can make the script so that it handles a portion of the wanted operations. Once it completes say 10%, you via AJAX call the script again to execute the next 10%. But there are circumstances where this is not an ideal solution, it really depends on what you are doing.
I used this method to create a web-based crawler which only ran on my computer for instance. If it had to do the operations at once it would time out as well. So it was split into 200 "tasks", each called via Ajax once the previous completes. Works perfectly, and it's been over a year since it started running (crawling?)
Related
I have a foreach in cakephp that processes products from a distributor, but the thing is the lists have up to 200products each product can have 3 big pictures with 2 resizes.
So i have in total 1200 big actions to much for one request.
I breaked the foreach at each 10 products, removing them from the array and redirected to the same page. But after a while I get a redirect loop.
Any ideeas on how to avoid this?
If I add another page in this redirect freenzy will it work?
The redirect loop appears only when redirecting in the same page?
The thing is the loop will end, but the browser doesn't know that.
$this->data = $this->Session->read('Parser.data');
$limit = 0;
foreach ($this->data as $key => $data):
$limit++;
if ($limit == 4)
$this->redirect($this->here);
...
$this->Session->delete('Parser.data.' . $key);
endforeach;
$this->redirect(array('controller' => 'parser', 'action' => 'index')); //if $this->data is empty it redirects to upload page
The server work with any number of records from what I have tested, but I have this action along the lines:
$this->getImage(WWW_ROOT . $folder . DS, $new_path, $image['path']);
which looks like this:
protected function getImage($folder = null, $path = null, $from = null) {
if (isset($from) && !empty($from))
file_put_contents($folder . $path, file_get_contents($from));
}
this loads up the server's memory and crashes.
This is why I have to break the foreach a couple of times.
I also tried other functions to get the images as cUrl, but with same results!
Let me copy my answer from another very similar question:
Never use URLs to do these kind of tasks, it is simply plain wrong, insecure and can cause your script to die or the server to become not responding any more.
Lets say you have 10000 users and a script runtime of 30 sec, it is very likely that the script times out before it finished and you end up with just a part of your users being processed at this time. The other scenario with a high or infinite amount of script runtime can lock your server. Depending on the script or DB actions it might cause the server to have a high load and users who use the site while the script is running will encounter a horrible slow to non responding site.
Also you can't really run a loop on a single URL, well you could redirect from one to another that does the limit and offset thing to simulate a loop over the 100000 users. If you don't loop over the records but fetch all 100000 at the same time it's likely your script dies because of running out of memory.
You should create a shell that processes the users in a loop and always just processes batches of for example 10, 50 or 100 users.
When executing your shell I recommend to use it with the "nice" command together to limit the amount of CPU time the shell is allowed to use to prevent the shell from taking 100% CPU usage to keep your site responding.
Look at creating a shell
and setting up a cron in cake.
Context :
I'm making a PHP websocket server (here) running as a DAEMON in which there is obviously a main loop listening for sockets connections and incoming data so i can't just create an other loop with a sleep(x_number_of_seconds); in it because it'll freeze my whole server.
I can't execute an external script with a CRON job or fork a new process too (i guess) because I have to be in the scope of my server class to send data to connected client sockets.
Does anyone knows a magic trick to achieve this in PHP ? :/
Some crazy ideas :
Keeping track of the last loop execution time with microtime(true), and compare it with the current time on each loop, if it's about my desired X seconds interval, execute the method... which would result in a very drunk and inconsistent interval loop.
Run a JavaScript setInterval() in a browser that will communicate with my server trough a websocket and tell it to execute my method... i said they where crazy ideas !
Additional infos about what i'm trying to achieve :
I'm making a little online game (RPG like) in which I would like to add some NPCs that updates their behaviours every X seconds.
Is there an other ways of achieving this ? Am I missing something ? Should I rewrite my server in Node.js ??
Thanks a lot for the help !
A perfect alternative doesn't seams to exists so I'll use my crazy solution #1 :
$this->last_tick_time = microtime(true);
$this->tick_interval = 1;
$this->tick_counter = 0;
while(true)
{
//loop code here...
$t= microtime(true) - $this->last_tick_time;
if($t>= $this->tick_interval)
{
$this->on_server_tick(++$this->tick_counter);
$this->last_tick_time = microtime(true) - ($t- $this->tick_interval);
}
}
Basically, if the time elapsed since the last server tick is greater or equal to my desired tick interval, execute on_server_tick() method. And most importantly : we subtract the time overflow to make the next tick happen faster if this one happened too late. This way we fill the gaps and at the end, if the socket_select timeout is set to 1 second, we will never have a gap greater than 1.99999999+ second.
I also keep track of the tick counter, this way I can use modulo (%) to execute code on multiple intervals like this :
protected function on_server_tick($counter)
{
if($counter%5 == 0)
{
// 5 seconds interval
}
if($counter%10 == 0)
{
// 10 seconds interval
}
}
which covers all my needs ! :D
Don't worry PHP, I won't replace you with Node.js, you still my friend.
It looks to me like the websocket-framework you are using is too primitive to allow your server to do other useful things while waiting for connections from clients. The only call to PHP's socket_select() function is hard-coded to a one second timeout, and it does nothing when the time runs out. It really ought to allow a callback or an outside loop.
Look at the http://php.net/manual/en/function.socket-select.php manual page. The last parameter is a timeout time. socket_select() waits for incoming data on a socket or until the timeout time is up, which sounds like what you want to do, but the library has no provision for it. Then look at how the library uses it in core/classes/SocketServer.php.
I'm assuming you call run() and then it just never returns to your calling code until it gets a message on the socket, which prevents you from doing anything.
In PHP, I want to put a number of second delay on each iteration of the loop.
for ($i=0; $i <= 10; $i++) {
$file_exists=file_exists($location.$filename);
if($file_exists) {
break;
}
//sleep for 3 seconds
}
How can I do this?
Use PHP sleep() function. http://php.net/manual/en/function.sleep.php
This stops execution of next loop for the given number of seconds. So something like this
for ($i=0; $i <= 10; $i++) {
$file_exists=file_exists($location.$filename);
if($file_exists) {
break;
}
sleep(3); // this should halt for 3 seconds for every loop
}
I see what you are doing... your delaying a script to constantly check for a file on the filesystem (one that is being uploaded or being written by another script I assume). This is a BAD way to do it.
Your script will run slowly. Choking the server if several users are running that script.
Your server may timeout for some users.
HDD access is a costly resource.
There are better ways to do this.
You could use Ajax. And use a timeout to call your PHP script every few seconds. This will avoid the slow script loading. And also you can keep doing it constantly (the current for loop will only run for 33 seconds and then stop).
You can use a database. In some cases database access is faster than HDD access. Especially with views and caching. The script creating the file/uploading the file can set a flag in a table (i.e. file_exists) and then you can have a script that checks that field in your database.
You can use sleep(3) which sleeps the thread for 3 seconds.
Correction sleep method in php are in seconds.
Hare are two ways to sleep php script for some period of time. When you have your code and want to pause script working for some time use these functions.
In these examples the first part of code will be done on script run and the second part of code will be done but with time delay.
Using sleep() function you can define sleep time in seconds.
Example:
echo "Message 1";
// The first part of code.
$timeInSeconds = 3;
sleep($timeInSeconds);
// The second part of code.
echo "Message 2";
This way it is possible to sleep php script for 3 seconds. Using this function you can sleep script for whole number (integer) of seconds.
Using usleep() function you can define sleep time in microseconds. This sleep time is convenient for intervals that require more precise time than one second.
Example:
echo "Message 1";
// The first part of code.
$timeInMicroSeconds = 2487147;
usleep($timeInMicroSeconds);
// The second part of code.
echo "Message 2";
You can use this function if you want to sleep php for smaller time values than second (float). In this example I have put script to sleep for 2.487147 seconds.
Have you considered using a PHP Daemon script using supervisorD. I use it in multiple tasks that are required to be running all the time.
The catch is making sure that each time you are running your script you check for memory resources. If its too high, stop the process and then let it restart itself up again.
I have successfully used this process to be always checking database records for tasks to process.
It might be overkill but worth considering.
I wanted to execute a bunch of code for 5 seconds and if it has not finished executing within the specificed time frame I need to execute another piece of code..
Whether it's possible?
Ex..
There are two functions A and B
If A takes more than 30 seconds to execute the control should pass on to B
During function A you could periodically check how long the script has been executing, and if it goes over x seconds, run B:
function checkTime($start) {
$current = time();
$secondsToExecute = 5;
if (($start+$secondsToExecute) <= $current) {
func_b();
}
}
function func_a($start) {
// do some code
checkTime($start);
// do some code
checkTime($start);
// do some code
}
function func_b() {
// do something else
exit();
}
func_a(time());
http://php.net/manual/en/features.connection-handling.php
Set a time limit and a shutdown function, which checks if the status is 2 (timeout) and does your stuff if so.
One thing to note is that the time limit set this way only counts actual php processing time. Time spent with php waiting for another process or a database or http connection, etc, will not count and your time limit will not be considered reached.
If you need to count actual time that passed, even if it was not php processing time, you're going to have to go with the above suggested answer. Manually inserting that time check in places where it makes sense is the best, i.e. inside loops that you know may run too long, maybe even not on every iteration but on every N iterations, etc. Alternatively a more general approach is to use register_tick_function(), but that might lead to a noticeable performance hit with a low tick count, and you must take care to unregister it or use appropriate flags so you don't end up infinitely starting more and more calls to your timeout handling code once the timeout has happened.
Other approaches are also possible, you can register a handler for some signal using pcntl_signal() and have it sent to your process when the time limit is reached by an outside program ('man timeout' if you are on a linux box) or by a fork()-ed instance of your own php script, etc.
A site I am working with is starting to get a little sluggish, and I would like to refine it. I think the problem is with the PHP, but I can't be sure. How can I see how long functions are taking to perform?
If you want to test the execution time :
<?php
$startTime = microtime(true);
// Your content to test
$endTime = microtime(true);
$elapsed = $endTime - $startTime;
echo "Execution time : $elapsed seconds";
?>
Try the profiler feature in XDebug or Zend Debugger?
Two things you can do.
place Microtime calls everywhere although its not convenient if you want to test more than one function. So there is a simpler way to do it a better solution if you want to test many functions which i assume you would like to do.
just have a class (click on link to follow tutorial) where you can test how long all your functions take. Rather than place microtime everywhere. you just use this class. which is very convenient
http://codeaid.net/php/calculate-script-execution-time-%28php-class%29
the second thing you can do is to optimize your script is by taking a look at the memory usage.
By observing the memory usage of your scripts, you may be able optimize your code better.
PHP has a garbage collector and a pretty complex memory manager. The amount of memory being used by your script. can go up and down during the execution of a script. To get the current memory usage, we can use the memory_get_usage() function, and to get the highest amount of memory used at any point, we can use the memory_get_peak_usage() function.
view plaincopy to clipboardprint?
echo "Initial: ".memory_get_usage()." bytes \n";
/* prints
Initial: 361400 bytes
*/
// let's use up some memory
for ($i = 0; $i < 100000; $i++) {
$array []= md5($i);
}
// let's remove half of the array
for ($i = 0; $i < 100000; $i++) {
unset($array[$i]);
}
echo "Final: ".memory_get_usage()." bytes \n";
/* prints
Final: 885912 bytes
*/
echo "Peak: ".memory_get_peak_usage()." bytes \n";
/* prints
Peak: 13687072 bytes
*/
http://net.tutsplus.com/tutorials/php/9-useful-php-functions-and-features-you-need-to-know/
PK
You can also make it manually, by recording microtime() value in various places, like this:
<?
$TIMER['start']=microtime(TRUE);
// some code
$query="SELECT ...";
$TIMER['before q']=microtime(TRUE);
$res=mysql_query($query);
$TIMER['after q']=microtime(TRUE);
while ($row = mysql_fetch_array($res)) {
// some code
}
$TIMER['array filled']=microtime(TRUE);
// some code
$TIMER['pagination']=microtime(TRUE);
/and so on
?>
and then visualize it
<?
if ('127.0.0.1' === $_SERVER['REMOTE_ADDR']) {
echo "<table border=1><tr><td>name</td><td>so far</td><td>delta</td><td>per cent</td></tr>";
reset($TIMER);
$start=$prev=current($TIMER);
$total=end($TIMER)-$start;
foreach($TIMER as $name => $value) {
$sofar=round($value-$start,3);
$delta=round($value-$prev,3);
$percent=round($delta/$total*100);
echo "<tr><td>$name</td><td>$sofar</td><td>$delta</td><td>$percent</td></tr>";
$prev=$value;
}
echo "</table>";
}
?>
an IP address check implies that we are doing this profiling on the working site
Though I doubt it's PHP itself. Most likely it's database. So, pay most attention to query execution timing.
however, a "site" term is very broad. It includes also JS, CSS, images and stuff. So, I'd suggest to start form FirebFug's Net page to see what part of whole page takes more time.
Of course, refining can be done only after analysis of profiling results, and cannot be advised here without it.
Your best bet is Xdebug. Im happy as it comes bundled in my PHPed IDE. I can get profiler data at the click of a button.
So maybe you could consider that.
I had similar issues and so I created 2 new tables on the database and two new functions. One was audit_sql and the other was audit_code. Because I used an SQL abstraction class it was easy to time every single SQL call (I used php microtime as some others have suggested). So, I called microtime before and after the SQL call and stored the results on the database.
Similarly with pages. I called microtime at the start and end of each page and if necessary at the start and end of functons, divs - whatever I thought might be a culprit.
The general results were:
SQL calls to MySQL were almost instantaneous and were nto a problem at all. The only thing I would say is that even I was surprised at the number being executed! The site is generated from the database - even the menus, permissions etc. To produce the home page the SQL calls were measured in the 100s.
PHP was not the culprit. This was even more instantaneous that MySQL.
The culprit was.... (big build up!) calls to You Tube and Picassa and other sites like that. I host videos and photo albums on the site (well, I don't actually store them - they are stored on YT etc.) and on the home page are thumbnails that are extracted from You Tube and the like via the You Tube PHP API/Zend Framework. Because this is all http based to the other sites, each one was taking 1, 2 or 3 seconds. This was causing those divs containing these to take between 6 and 12 seconds and the home page up to 17 seconds.
The solution - store all thumbnails on my server. The first time one has to be served from the remote site (YT, Picassa etc.) so do that and then store it on your own site. Future times, you check if you have it and if so serve it always from your server. Cuts the page load time down to 2-3 seconds tops. Granted the first person to view the first home page load after someone has loaded more videos/images will take some time, but not thereafter. People will put a long one-off page load time down to their connection/the internet in general. Too many slow loads of your site and they will stop visiting!
I hope that helps somewhat.