I'd like to limit a specific section of PHP to X seconds - if it takes longer, kill the currently executing code (just the section, not the entire script) and run an alternate code.
Pseudo code example (Example use case here is an unstable API which is sometimes fast and other times its a black hole):
$completed = 1;
$seconds = 60;
while ($completed != -1 && $completed < 5) {
limit ($seconds) {
$api = new SomeAPI('user','secret','key');
$data = $api->getStuff('user="bob"');
$completed = -1;
} catch () {
$completed++;
sleep(10);
}
}
if ($completed === 5) echo "Error: API black-hole'd 5 times.\n";
else {
//Notice: data processing is OUTSIDE of the time limit
foreach ($data as $row) {
echo $row['name'].': '.$row['message']."\n";
}
}
HOWEVER, this should work for anything. Not just API/HTTP requests. E.g. an intensive database procedure.
In case you're reading too fast: set_time_limit and max_execution_time are not the answer as they affect the time limit for the entire script rather than just a section (unless I'm wrong on how those work, of course).
In the case of an API call, I would suggest using cURL, for which you can set a specific timeout for the API call.
For generic use, you can look at forking processes, which would give you the ability to time each process and kill it if it exceeds the expected time.
Of course if the section of code might be subject to long execution times due to a highly repetitive loop structure, you can provide your own timers to break out of the loop after a specified time interval.
I might not have directly answered your question, but really the point I wanted to get to is that you might have to use a different approach depending on what the code block actually does.
Related
I have a PHP script that processes data downloaded from multiple REST APIs into a standardized format and builds an array or table of this data. The script currently executes everything synchronously and therefore takes too long.
I have been trying to learn how to execute the function that fetches and processes the data, simultaneously or asynchronously so that the total time is the time of the slowest call. From my research it appears that ReactPHP or Amp are the correct tools.
However, I have been unsuccessful in creating test code that actually executes correctly. A simple example is attached, with mysquare() representing my more complex function. Due to a lack of examples on the net of exactly what I'm trying to achieve I have been forced to use a brute force method with 3 examples listed in my code.
Q1: Am I using the right tool for the job?
Q2: Can you fix my example code to execute asynchronously?
NB: I am a real beginner, so the simplest possible code example with a minimum of high level programming lingo would be appreciated.
<?php
require_once("../vendor/autoload.php");
for ($i = 0; $i <= 4; $i++) {
// Experiment 1
$deferred[$i] = new React\Promise\Deferred(function () use ($i) {
echo $x."\n";
usleep(rand(0, 3000000)); // Simulates long network call
return array($x=> $x * $x);
});
// Experiment 2
$promise[$i]=$deferred[$i]->promise(function () use ($i) {
echo $x."\n";
usleep(rand(0, 3000000)); // Simulates long network call
return array($x=> $x * $x);
});
// Experiment 3
$functioncall[$i] = function () use ($i) {
echo $x."\n";
usleep(rand(0, 3000000)); // Simulates long network call
return array($x=> $x * $x);
};
}
$promises = React\Promise\all($deferred); // Doesn't work
$promises = React\Promise\all($promise); // Doesn't work
$promises = React\Promise\all($functioncall); // Doesn't work
// print_r($promises); // Doesn't return array of results but a complex object
// This is what I would like to execute simulatenously with a variety of inputs
function mysquare($x)
{
echo $x."\n";
usleep(rand(0, 3000000)); // Simulates long network call
return array($x=> $x * $x);
}
Asynchronous doesn't mean multiple threads execute in parallel. 2 functions can only really run at the 'same time', if they (for example) do IO such as a HTTP request.
usleep() blocks, so you gain nothing. Both ReactPHP and Amp will have some kind of 'sleep' function themselves that's built right into the event loop.
For the same reason you will not be able to just use curl, because it will also block out of the box. You need to use the HTTP libraries that React and Amp provide and/recommend.
Since your end-goal is just doing HTTP requests, you could also not use any of these frameworks and just use the curl_multi functions. They're a bit hard to use though.
I'm answering my own question in an attempt to help other users, however this solution was developed alone without the help of an experienced programmer and so I do not know if it is ultimately the best way to do this.
TL;DR
I switched from ReactPHP because I didn't understand it to using amphp/parallel-functions which offers a simplified end user interface... sample code using this interface attached.
<?php
require_once("../vendor/autoload.php");
use function Amp\ParallelFunctions\parallelMap;
use function Amp\Promise\wait;
$start = \microtime(true);
$mysquare = function ($x) {
sleep($x); // Simulates long network call
//echo $x."\n";
return $x * $x;
};
print_r(wait(parallelMap([5,4,3,2,1,6,7,8,9,10], $mysquare)));
print 'Took ' . (\microtime(true) - $start) . ' milliseconds.' . \PHP_EOL;
The example code executes in 10.2 seconds which is slightly longer than the longest running instance of $mysquare().
In my actual use case I was able to fetch data via HTTP from 90 separate sources in around 5 seconds.
Notes:
The amphp/parallel-functions library appears to be using threads under the hood. From my preliminary experience this appears to require a lot more memory than just a single threaded PHP script, but I haven't yet ascertained the full impact. This was highlighted when I was passing a large array to $mysquare via the "use ($myarray)" expression and array was 65Mb. This brought the code to a standstill and it increased execution time exponentially so much so that it took orders of magnitude longer than synchronous execution. Also the memory usage peaked at over 5G! at one point leading me to believe that amphp was duplicating $myarray for each instance. Reworking my code to avoid the "use ($myarray)" expression fixed that problem.
I am tyring to make a php function that updates every second using php itself no other languages, just pure PHP codes.
function exp(){
//do something
}
I want it to return a value each second. Like update every second.
For an application server (not a web server), best practice is to use an event loop pattern instead of sleep. This gives you the ability to run multiple timers should the need arise (sleep is blocking so nothing else can run in the mean time). Web servers on the other hand should not really be executing any long-running scripts.
Whilst other languages give you event loops out of the box (node / js for example, with setInterval), PHP does not, so you have to either use a well known library or make your own). React PHP is a widely used event loop for PHP.
Here is a quick-and-dirty "hello world" implementation of an event loop
define("INTERVAL", 5 ); // 5 seconds
function runIt() { // Your function to run every 5 seconds
echo "something\n";
}
function checkForStopFlag() { // completely optional
// Logic to check for a program-exit flag
// Could be via socket or file etc.
// Return TRUE to stop.
return false;
}
function start() {
$active = true;
$nextTime = microtime(true) + INTERVAL; // Set initial delay
while($active) {
usleep(1000); // optional, if you want to be considerate
if (microtime(true) >= $nextTime) {
runIt();
$nextTime = microtime(true) + INTERVAL;
}
// Do other stuff (you can have as many other timers as you want)
$active = !checkForStopFlag();
}
}
start();
In the real world you would encapsulate this nicely in class with all the whistles and bells.
Word about threading:
PHP is single threaded under the hood (any OS threading must be manually managed by the programmer which comes with a significant learning curve). So every task in your event loop will hold up the tasks that follow. Node on the other hand, for example manages OS threads under the hood, taking that "worry" away from the programmer (which is a topic of much debate). So when you call setInterval(), the engine will work its magic so that the rest of your javascript will run concurrently.
Quick final note:
It could be argued that this pattern is overkill if all you want to do is have a single function do something every 5 seconds. But in the case where you start needing concurrent timers, sleep() will not be the right tool for the job.
sleep() function is the function that you are looking for:
while (true) {
my_function(); // Call your function
sleep(5);
}
While loop with always true
Call your function inside while loop
Wait for 5 seconds(sleep)
Return the beginning of the loop
By the way it's not a logical use case of endless loops in PHP if you are executing the script through a web protocol(HTTP, HTTPS, etc.) because you will get a timeout. A rational use case could be a periodic database updater or a web crawler.
Such scripts can be executed through command line using php myscript.php or an alternative (but not recommended) way is using set_time_limit to extend the limit if you insist on using a web protocol to execute the script.
function exp(){
//do something
}
while(true){
exp();
sleep(5);
}
Use sleep function to make execution sleep for 5 seconds
it will be better if you use setInterval and use ajax to perform your action
$t0 = microtime(true);
$i = 0;
do{
$dt = round(microtime(true)-$t0);
if($dt!= $i){
$i = $dt;
if(($i % 5) == 0) //every 5 seconds
echo $i.PHP_EOL;
}
}while($dt<10); //max execution time
Suppose exp() is your function
function exp(){
//do something
}
Now we are starting a do-while loop
$status=TRUE;
do {
exp(); // Call your function
sleep(5); //wait for 5 sec for next function call
//you can set $status as FALSE if you want get out of this loop.
//if(somecondition){
// $status=FALSE:
//}
} while($status==TRUE); //loop will run infinite
I hope this one helps :)
It's not preferable to make this in PHP, try to make on client side by calculating difference between time you got from database and current time.
you can make this in JS like this:
setInterval(function(){
// method to be executed;
},5000); // run every 5 seconds
I have a web page that contains some JavaScript and performs some Ajax calls. When trying to test it using Selenium, I randomly get "PHPUnit_Extensions_Selenium2TestCase_WebDriverException: Element is no longer attached to the DOM" message, maybe once in 5 runs.
Now I'm aware of the race issue between Ajax call and test engine, and I have taken steps to protect from it, but I still have some problem. My scenario is this: I change value of the select element 1 which triggers Ajax call that removes all option sub-elements of the select element 2 and generates new option sub-elements based on the Ajax response. Testing code:
$this->select($this->byId('select1'))->selectOptionByValue('value1');
$this->myWaitForElementToAppear('#select2>option[value="value2"]');
$this->select($this->byId('select2'))->selectOptionByValue('value2');
last line triggers the error. Here is the myWaitForElementToAppear method:
public function myWaitForElementToAppear($selector, $limit = 5) {
$start = time();
while(true) {
if($start + $limit < time()) {
break;
}
try {
$this->byCssSelector($selector);
break;
} catch(PHPUnit_Extensions_Selenium2TestCase_WebDriverException $e) {}
}
}
If I'm not mistaken, myWaitForElementToAppear method should ensure that desired option has been added by jQuery before it exits and thus allow it to be used on the next line. I should add that I've made sure that time-out doesn't happen here (since my method allows for it to happen) and I'm positive that it's not the case
Edit: I should add that putting sleep(1) after myWaitForElementToAppear call solves the problem, but I don't understand why the additional second is needed. Shouldn't call to myWaitForElementToAppear be enough?
There are some explanations here:
Firstly, time() has a very low precision, only returning the number of
whole seconds that have passed, which makes the whole thing quite
vague. Secondly, PHP has to sit there looping thousands of times while
it waits, essentially doing nothing. A much better solution is to use
the one of the two script sleep functions, sleep() and usleep(), which
take the amount of time to pause execution as their only parameter.
From php.net:
The idea of sleep and usleep is that by letting the cpu run a few idle
cycles so the other programs can have some cycles run of their own.
what results in better response times and lower overall system-load.
so if you have to wait for something, go to sleep for a few seconds
instead of occupying the cpu while doing absolute nothing but
waitting.
And you can use waitUntil from PHPUnit:
/* waitElementToDisappear */
$this->waitUntil(function($testCase) {
try {
$input = $testCase->byCssSelector("#select2>option[value="value2"]");
} catch (PHPUnit_Extensions_Selenium2TestCase_WebDriverException $e) {
if (PHPUnit_Extensions_Selenium2TestCase_WebDriverException::NoSuchElement == $e->getCode()) {
return true;
}
}
}, 5000);
/* waitElementToAppear */
$this->waitUntil(function($testCase) {
try {
$input = $testCase->byCssSelector("#select2>option[value="value2"]");
return true;
} catch (PHPUnit_Extensions_Selenium2TestCase_WebDriverException $e) {}
}, 5000);
So I have this function for making non-blocking curl requests. It works fine on what I've tested so far (small amounts of requests). But I need this to scale up to thousands of requests (maybe max 10,000). My issue is that I don't want to run into issues with too many parallel requests running at once.
What would you suggest to rate-limit the requests? Usleep? Requests in batches? The function is below:
function poly_curl($requests){
$queue = curl_multi_init();
$curl_array = array();
$count = 0;
foreach($requests as $request)
{
$curl_array[$count] = curl_init($request);
curl_setopt($curl_array[$count], CURLOPT_RETURNTRANSFER, true);
curl_multi_add_handle($queue, $curl_array[$count]);
$count++;
}
$running = NULL;
do {
curl_multi_exec($queue,$running);
} while($running > 0);
$res = array();
$count = 0;
foreach($requests as $request)
{
$res[$count] = curl_multi_getcontent($curl_array[$count]);
$count++;
}
$count = 0;
foreach($requests as $request){
curl_multi_remove_handle($queue, $curl_array[$count]);
$count++;
}
curl_multi_close($queue);
return $res;
}
I think curl_multi_exec is bad for this purpose, because even if you use batches in groups of 100, 99 request could be finished and still will have to wait for the last request completion.
But you need 100 parallel requests and when one finishes, another is immediately started. So you cannot use curl_multi_exec at all.
I would use normal producer-consumer algorithm with multiple (constant number) consumers with every consumer processing only one url. For example php-resque and COUNT=100 php resque.php
You may want to implement something that is called Exponential Backoff (wikipedia).
Basically, it is an algorithm that allows you dynamically scale your processes depending on some feedback.
You define a rate in your application, and on the first time-out, error, or anything you decide, you decrease this rate until the request finishes.
You may implement it easily using the HTTP response code for example.
Last time i was doing something like this it was including downloading and "parsing" files. Was able to proceed only 4 subpages at a time limited by very weak hardware processor (2 cores with HT). What time i ended up with two queuest: 1 for waiting, 2 for in-process. Every time a task gone from 2nd queue, new was taken from 1st one.
May saund complicated, but ended in two loops inside eachother and simple count()'s
Btw, considering so hight rate i would think of using Node.js - for simplicity - or anything more nonblocking and more suitable for deamons than PHP.. As long as threads are PHP weakpoint, it just does not suit there.
PS: nice & useful bit of code, thanks.
We used to face the same problem with C++ connection pooling code. The approach is those days involved some serious analysis.
But, the essence was that, we created a pool and requests would get processed depending on number of available requests. What we also did was assign a maximum number of connection pools.[This was determined by testing].
What you really need is a method to determine how many requests are being processed and put a limit to it. In your case that is $count
Just compare $count to a maximum value[say, $max] to it and stop there. define the value depending on the system the program runs. $max could be hardcoded or dynamic.
Firstly I want to give you the basic idea of what I am trying to do:
I'm trying to make a free web hosting service do some work for me. I've created one php page and MySQL db. The basic idea behind my PHP page is I have a while loop with condition of $shutdown, and some counter inside while loop to track whether code is running or not
<?php
/*
Connect to database etc. etc
*/
$shutdown = false;
// Main loop
while (!$shutdown)
{
// Check for user shutdown request
$strq = "SELECT * FROM TB_Shutdown;";
$result = mysql_query($strq);
$row = mysql_fetch_array($result);
if ($row[0] == "true")
{
$shutdown = true; // I know this statement is useless but nevermind
break;
}
//Increase counter
$strq = "SELECT * FROM TB_Counter;";
$result = mysql_query($strq);
$row = mysql_fetch_array($result);
if (intval($row[0]) == 60)
{
// Reset counter
$strq = "UPDATE TB_Counter SET value = 0";
$result = mysql_query($strq);
/*
I have some code to do some works at here its not important just curl stuff
*/
else
{
// Increase counter
$strq = "UPDATE TB_Counter SET value = " . (intval($row[0]) + 1);
$result = mysql_query($strq);
}
/*
I have some code to do some works at here its not important just curl stuff
*/
// Sleep
sleep(1);
}
?>
And I have a check.php which returns me the value from TB_Counter.
The problem is: I'm tracking the TB_Counter table every second. It stops after a while. If I close my webbrowser (which I called my main while php loop page from) it stops after like 2 minutes. If not after 5-7 mins I get the error "connection has been reset" on browser and loop stops.
What should I do to make my loop lasts forever?
You need to allow PHP to execute completely. There is an option in the PHP.INI file which says:
max_execution_time = 30;
This sets the maximum time in seconds a script is allowed to run
before it is terminated by the parser. This helps prevent poorly
written scripts from tying up the server. The default setting is 30.
When running PHP from the command line the default setting is 0.
The function set_time_limit:
Set the number of seconds a script is allowed to run. If this is
reached, the script returns a fatal error. The default limit is 30
seconds or, if it exists, the max_execution_time value defined in the
php.ini.
To check if PHP is running in safe mode, you can use this:
echo $phpinfo['PHP Core']['safe_mode'][0]
If it is going to be a huge process, you can consider running on Cron as a CronJob. A small explanation on it:
Cron is very simply a Linux module that allows you to run commands at predetermined times or intervals. In Windows, it’s called Scheduled Tasks. The name Cron is in fact derived from the same word from which we get the word chronology, which means order of time.
Using Cron, a developer can automate such tasks as mailing ezines that might be better sent during an off-hour, automatically updating stats, or the regeneration of static pages from dynamic sources. Systems administrators and Web hosts might want to generate quota reports on their clients, complete automatic credit card billing, or similar tasks. Cron has something for everyone!
Read more about Cron
You could use php function set_time_limit().
You should not handle this from a browser. Run a cron every minute doing the checks you need would be a better solution.
Next to that why would you update every second? Just write down a timestamp so you know when a request was made?
Making something to run forever is not doable. More important is to secure that your business process keeps running. So maybe it would be wise to put your business case here, you seem to need to count seconds and do something within a minute but it's not totally clear. So what do you need to do?