Perform multiple simultaneous POST calls to the same API endpoint - php

I am trying to perform multiple POST REST Call. The catch: doing multiple POST calls at the same time. I am fully aware and have worked with the library guzzle but I haven't figured away to do this properly. I can perform GET calls asynchronously but nothing at the same level for POST calls. Then I came across pthreads and I read through the documentation and was a bit confused on how to even start it off. I have compiled php with the pthreads extension.
Could someone advise how to perform multiple POST calls at the same time and be able to gather the responses for later manipulation?
The below is a basic implementation that loops and waits. Very slow overall.
$postDatas = [
['field' => 'test'],
['field' => 'test1'],
['field' => 'test2'],
];
foreach ($postDatas as $postData) {
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://www.apisite.com",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_POSTFIELDS => json_encode($postData),
CURLOPT_HTTPHEADER => [
"cache-control: no-cache",
"connection: keep-alive",
"content-type: application/json",
"host: some.apisite.com",
],
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
echo $response;
}
}

That if the task is reduced to working with the API then you probably need to use http://php.net/manual/ru/function.curl-multi-exec.php
public function getMultiUrl() {
//If the connections are very much split the queue into parts
$parts = array_chunk($this->urlStack, self::URL_ITERATION_SIZE , TRUE);
//base options
$options = [
CURLOPT_USERAGENT => 'MyAPP',
CURLOPT_HEADER => false,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_POST => true,
];
foreach ($parts as $urls) {
$mh = curl_multi_init();
$active = null;
$connects = [];
foreach ($urls as $i => $url) {
$options[CURLOPT_POSTFIELDS] = $url['postData'];
$connects[$i] = curl_init($url['queryUrl']);
curl_setopt_array($connects[$i], $options);
curl_multi_add_handle($mh, $connects[$i]);
}
do {
$status = curl_multi_exec($mh, $active);
$info = curl_multi_info_read($mh);
if (false !== $info) {
var_dump($info);
}
} while ($status === CURLM_CALL_MULTI_PERFORM || $active);
foreach ($connects as $i => $conn) {
$content = curl_multi_getcontent($conn);
file_put_contents($this->dir . $i, $content);
curl_close($conn);
}
}
}

Related

Run curl for 20k in a sequence and save data in excel: fails

I need to run around 20K curl request, fetch the data from the request and save it in excel using Spout. Important is that I need to be run in sequence and save data in sequence.
I have set the PHP timeout to 0, but it still fails. Shows an empty page with no warning and no success message and no product is saved in xls file. If I run it for 1000 products or so then it works fine.
How can I fix this?
After getting the XML by sending the curl I am doing a foreach to get the required data and
$writer->openToFile('products.xlsx');
foreach ($xml->Body->product as $products) {
$values = [ $products ];
$rowFromValues = WriterEntityFactory::createRowFromArray($values);
$writer->addRow($rowFromValues);
}
$writer->close();
function getCurl($url, $soapBody) {
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_POSTFIELDS => $soapBody,
CURLOPT_HTTPHEADER => array(
"Content-Type: text/xml",
"cache-control: no-cache"
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
curl_close($curl);
return $response;
}
}

How do I find out how to translate a Postman query in cURL via PHP

I use postman for the first time.
My goal is to download a JSON file.
I managed to look at the code for cURL in postman:
But if I test the code in the command line, I can not log in.
Actually, I want to call the JSON file via PHP. I have tested initial approaches. In addition to the problem that I can not authenticate myself here, I think that I need more values. For example POSTFIELDS. Is that correct and if so, how do I find what I have to enter in Postman? Here is my initial Code:
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://api.divessi.com/divessi/index.php",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_POSTFIELDS => "..",
CURLOPT_HTTPHEADER => array(
"Authorization: Basic BASE64",
"Cache-Control: no-cache",
"Postman-Token: TOKEN",
"content-type: ??"
),
));
$response = curl_exec($curl);
$obj = json_decode($response);
$events = array();
if ($obj)
{
$events = $obj[0]->DATA;
}
$err = curl_error($curl);
curl_close($curl);
if ($err)
{
echo "cURL Error #:" . $err;
} else {
foreach ($events as $event) {
...
}

foreach only running function (`sendThis`) for first item in array

I'm trying to send each item in an array to a function. The function is a curl request, but I don't think that matters.
$a=array('string1','string2');
foreach ($a as $value) {
sendThis($value);
}
That code runs the function just once for the first item in the array (string1). How do I keep this running for every item in the array?
sendThis function is similar to the following:
function sendThis($value){
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://api.xyz.com/v2/users/".$value."/",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "GET",
CURLOPT_POSTFIELDS => "",
CURLOPT_HTTPHEADER => array(
"Authorization: XYZ123",
"cache-control: no-cache"
),
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
echo $response;
}}

Throttling curl in PHP issue

I'm trying to throttle my curl requests to keep under the GET request limits set by the server's RESTful API. Essentially, I'm looping through an array of IDs and looking to extract a base_bid to an array. I'm trying to add a delay between each request and have tried using PHP's usleep() function to do so but this only seems to add a delay for the first iteration and not onwards.
I'm using the code below:
$test = array();
for ($i=0; $i < count($campaigns); $i++) {
$ch = curl_init('https://api.appnexus.com/campaign?id='.$campaigns[$i]['id'].'');
$options = array(
CURLOPT_CUSTOMREQUEST => 'GET',
CURLOPT_RETURNTRANSFER => true,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_HTTPHEADER => array(
'Content-Type: application/json',
'Authorization:'.$token[1].''
));
curl_setopt_array($ch, $options);
$base = curl_exec($ch);
$info = curl_getinfo($ch);
//Attempting to throttle curl here
usleep(5000);
curl_close($ch);
if ($base === false || $info['http_code'] != 200) {
$output = 'Status code: '.$info['httpcode'].'';
} else {
preg_match('/"base_bid":([0-9\.]+)/', $base ,$bid);
$test[] = array(
'id' => $campaigns[$i]['id'],
'base_bid' => $bid[1]
);
}
};
Is this how usleep() is meant to be implemented in a loop for curl requests? Any comments would be highly valued!
Thanks,
Sam

php curl multi init HTTP 401 Unauthorized

I have used a curl single init to issue an HTTP Get and all worked fine.
Now I tried to use a multi init (as I need to get multiple URLs) and I get a 401 message with "This request requires HTTP authentication" on the response to the Get.
Same Curl options where used on both cases.
Here is the code for th multi init and below it the single init function.
protected function _multiQueryRunkeeper($uri, $subscribersInfo,$acceptHeader) {
$curlOptions = array(
CURLOPT_URL => 'https://api.runkeeper.com' . $uri,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_TIMEOUT => 8,
CURLOPT_HTTPAUTH => CURLAUTH_ANY,
CURLOPT_HTTPGET => true
);
$curl_array = array();
$mh = curl_multi_init();
foreach ($subscribersInfo as $i => $subscriber) {
$curl_array[$i] = curl_init();
curl_setopt_array($curl_array[$i],$curlOptions);
curl_setopt($curl_array[$i], CURLOPT_HEADER,
array('Authorization: Bearer '.$subscriber['token'],
'Accept: application/vnd.com.runkeeper.' . $acceptHeader));
curl_multi_add_handle($mh,$curl_array[$i]);
}
$running = NULL;
do {
usleep(10000);
curl_multi_exec($mh,$running);
} while($running > 0);
$subscribersWorkoutFeed = array();
foreach($subscribersInfo as $i => $subscriber)
{
$subscribersWorkoutFeed[$i] = curl_multi_getcontent($curl_array[$i]);
curl_multi_remove_handle($mh, $curl_array[$i]);
}
curl_multi_close($mh);
return $subscribersWorkoutFeed;
}
protected function _singleQueryRunkeeper($uri, $subscriberToken,$acceptHeader) {
try{
// get fitness user's fitness activities from Runkeeper
$this->_curl = isset($this->_curl)? $this->_curl : curl_init();
$curlOptions = array(
CURLOPT_URL => 'https://api.runkeeper.com' . $uri,
CURLOPT_HTTPHEADER => array('Authorization: Bearer '.$subscriberToken,
'Accept: application/vnd.com.runkeeper.' . $acceptHeader),
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_TIMEOUT => 8,
CURLOPT_HTTPGET => true
);
curl_setopt_array($this->_curl,$curlOptions);
$response = curl_exec($this->_curl);
if($response == false) {
if (Zend_Registry::isRegistered('logger')) {
$logger = Zend_Registry::get('logger');
$logger->log('Curl error on _singleQueryRunkeeper: '
. curl_error($this->_curl), Zend_Log::INFO);
}
return null;
}
$data = Zend_Json::decode($response);
return($data);
} catch(Exception $e){
if (Zend_Registry::isRegistered('logger')) {
$logger = Zend_Registry::get('logger');
$logger->log('exception occured on getUsersLatestWorkoutsFromRK. Curl error'
. curl_error($this->_curl), Zend_Log::INFO);
}
}
}

Categories