do while loop causing 100% CPU usage with curl_multi_exec - php

Question

This is a loop used in our script with curl. It's causing CPU usage to shoot up to 100%. A friend said "Your computer is looping so fast here it doesn't have time to process the request since it is constantly checking for a finish."... So my question is how can this loop be re-written to slow down? Thanks
$running = null;
do {
curl_multi_exec($mh, $running);
} while($running > 0);

Answer 1

Try this: http://php.net/manual/function.curl-multi-select.php

Answer 2

Adding a call to http://php.net/sleep or http://php.net/usleep in every iteration should reduce the CPU usage by allowing other running processes to be scheduled by the operating system.

Answer 3

Unfortunately you didn't post whole code. I suppose you are doing something like
$mh = curl_multi_init();
for ($i = 0; $i < $desiredThreadsNumber; $i++) {
$ch = curl_init();
// set up $ch here
curl_multi_add_handle($mh, $ch);
}
You should understand that you haven't run threads yet here. curl_multi_exec() runs all threads. But it can't run all $desiredThreadsNumber threads simultaneously. If you look on example on curl_multi_exec() php.net page, you will see that you must wait while curl_multi_exec() run all threads. In other words, you need next nested loop here:
$running = null;
do {
do {
$mrc = curl_multi_exec($mh, $running);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
} while($running > 0);
At the end let me suggest you to read this article http://www.onlineaspect.com/2009/01/26/how-to-use-curl_multi-without-blocking/ and use code snippet from there, I used it in 2 or 3 projects.

Answer 4

curl_multi_select (http://php.net/manual/function.curl-multi-select.php) is indeed the way to go, but there are a couple caveats.
First, if curl_multi_exec returns CURLM_CALL_MULTI_PERFORM, it has more data to be processed immediately, so should be run again. Also, it is important to check that curl_multi_exec did not fail immediately; in that case curl_multi_select could block forever.
This should work:
do {
while (CURLM_CALL_MULTI_PERFORM === curl_multi_exec($mh, $running)) {};
if (!$running) break;
while (curl_multi_select($mh) === 0) {};
} while (true);
If anyone sees a good way to avoid the while(true) without duplicating code, please point it out.

Answer 5

Tried all the solutions provided above but this one worked for me in highly loaded system where on each second there are more than 1k Multi Curl requests are being made.
//Execute Handles
$running = null;
do {
$mrc = curl_multi_exec($mh, $running);
} while($mrc == CURLM_CALL_MULTI_PERFORM);
while ($running && $mrc == CURLM_OK) {
if (curl_multi_select($mh) == -1) {
usleep(1);
}
do {
$mrc = curl_multi_exec($mh, $running);
} while ($mrc == CURLM_CALL_MULTI_PERFORM);
}

Answer 6

Try:
$running = null;
do {
do {
$mrc = curl_multi_exec($mh, $running);
} while ($mrc == CURLM_CALL_MULTI_PERFORM && curl_multi_select($mh) === 0 );
} while($running > 0 && $mrc == CURLM_OK );

Answer 7

You could add sleep(1), which sleeps for one second, into the loop.