PHP Multiple cURL requests to REST API stalls - php

Question

Currently I have a system that sends multiple requests to a REST API. It is structured something like this:
foreach ($data as $d)
{
$ch = curl_init( $url );
curl_setopt( $ch, CURLOPT_POST, 1);
curl_setopt( $ch, CURLOPT_HTTPHEADER, (array of data here));
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, 1);
$response = curl_exec( $ch );
$retry = 0;
while((curl_errno($ch) == 7 || curl_errno($ch) == 52) && $retry < 3)
{
$response = curl_exec($ch);
$retry++;
}
curl_close($ch);
(decode XML Response and loop)
}
(I can't expose the whole code so I have filled in the operations that are happening in brackes)
However after a few hundred requests the FastCGI script stalls. The REST API will still respond during this period if I query it in another fashion, but this batch client will not send anymore requests. After a few minutes, it will start responding again. I'm not sure why this is stalling, I can see via htop that there is no CPU activity on the threads at either end whilst this is happening.
Is there any reason why the cURL/PHP script would stall here?

Answer 1

if you allowed to use external PHP libraries;
I'd like to suggest this method:
https://github.com/php-curl-class/php-curl-class
// Requests in parallel with callback functions.
$multi_curl = new MultiCurl();
$multi_curl->success(function($instance) {
echo 'call to "' . $instance->url . '" was successful.' . "\n";
echo 'response: ' . $instance->response . "\n";
});
$multi_curl->error(function($instance) {
echo 'call to "' . $instance->url . '" was unsuccessful.' . "\n";
echo 'error code: ' . $instance->error_code . "\n";
echo 'error message: ' . $instance->error_message . "\n";
});
$multi_curl->complete(function($instance) {
echo 'call completed' . "\n";
});
$multi_curl->addGet('https://www.google.com/search', array(
'q' => 'hello world',
));
$multi_curl->addGet('https://duckduckgo.com/', array(
'q' => 'hello world',
));
$multi_curl->addGet('https://www.bing.com/search', array(
'q' => 'hello world',
));
$multi_curl->start();