I'm trying to throttle my curl requests to keep under the GET request limits set by the server's RESTful API. Essentially, I'm looping through an array of IDs and looking to extract a base_bid to an array. I'm trying to add a delay between each request and have tried using PHP's usleep() function to do so but this only seems to add a delay for the first iteration and not onwards.
I'm using the code below:
$test = array();
for ($i=0; $i < count($campaigns); $i++) {
$ch = curl_init('https://api.appnexus.com/campaign?id='.$campaigns[$i]['id'].'');
$options = array(
CURLOPT_CUSTOMREQUEST => 'GET',
CURLOPT_RETURNTRANSFER => true,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_HTTPHEADER => array(
'Content-Type: application/json',
'Authorization:'.$token[1].''
));
curl_setopt_array($ch, $options);
$base = curl_exec($ch);
$info = curl_getinfo($ch);
//Attempting to throttle curl here
usleep(5000);
curl_close($ch);
if ($base === false || $info['http_code'] != 200) {
$output = 'Status code: '.$info['httpcode'].'';
} else {
preg_match('/"base_bid":([0-9\.]+)/', $base ,$bid);
$test[] = array(
'id' => $campaigns[$i]['id'],
'base_bid' => $bid[1]
);
}
};
Is this how usleep() is meant to be implemented in a loop for curl requests? Any comments would be highly valued!
Thanks,
Sam
Related
What I'm trying to accomplish is taking a simple array ($territories)
$territories = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18]
And assign each value as a variable, so that it can be used inside another function, and needs to run for each value in the first array.
The function being called is getting its data from another api, so that variable is required to correctly call the function.
Most of what I've done so far is working, except it's only running the function once.
The function needs to run all 18 times, and return 18 separate arrays.
The complete end goal here is to check the user's current zipcode against each array, and then to perform a function if there's a match to each array.
Below is what I have so far. I've left out API credentials for confidentiality.
//get territories in Nitro API
function get_territories(){
$url = 'https://nitro.powerhrg.com/admin/api/v1/territories.json';
$token = '';
// Nitro API request via cURL
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => '',
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => 'GET',
CURLOPT_HTTPHEADER => array(
'Content-Type: application/json',
'Authorization: '. $token.'',
),
));
// store response as string and return it
$result = json_decode(curl_exec($curl), true);
curl_close($curl);
return $result;
}
//get zipcodes from territory ID in Nitro API
function get_zipcodes($territory_id){
$url = 'https://nitro.powerhrg.com/directory/api/v1/zip_codes.json?territory_id='.$territory_id;
$token = '';
$body = array(
'id' => $territory_id,
);
// Nitro API request via cURL
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => '',
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 0,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => 'GET',
CURLOPT_HTTPHEADER => array(
'Content-Type: application/json',
'Authorization: '. $token.'',
),
CURLOPT_POSTFIELDS => json_encode($body)
));
// store response as string and return it
$result = json_decode(curl_exec($curl), true);
curl_close($curl);
return $result;
}
// show JSON array of all territory data by id (testing)
function display_zipcodes_raw(){
// get list of territory IDs
$territories = get_territories();
$values = [];
foreach ($territories as &$territory) {
$values[] = $territory['id'];
$id = 0;
foreach ($values as $key=>$value) {
$id = $value;
}
$zipcodes = get_zipcodes($id);
$results = [];
foreach ($zipcodes as &$zip) {
$results[] = $zip['zip_code'];
}
return 'zipcode results: '.json_encode($results).'<br>';
}
}
add_shortcode( 'display_zipcodes_raw', 'display_zipcodes_raw' );
Try the below
function display_zipcodes_raw()
{
$zipcodes = [];
foreach (get_territories() as $territory) {
$zipcodes[$territory['id']] = array_column(get_zipcodes($territory['id']), 'zip_code');
}
return json_encode($zipcodes);
}
As a recommendation, global functions that reference other global functions that serve a single responsibility like you have here should probably be collected together and placed into a class, object-orientated programming is truly a beautiful thing. Good luck on your adventure!
You have l unnecessary variables and loops, it's much more simple:
// show JSON array of all territory data by id (testing)
function display_zipcodes_raw(){
// get list of territory IDs
$territories = get_territories();
$results = [];
foreach ($territories as $territory) {
$zipcodes = get_zipcodes($territory['id']);
foreach ($zipcodes as $zip) {
$results[] = $zip['zip_code'];
}
}
return 'zipcode results: '.json_encode($results).'<br>';
}
I am trying to perform multiple POST REST Call. The catch: doing multiple POST calls at the same time. I am fully aware and have worked with the library guzzle but I haven't figured away to do this properly. I can perform GET calls asynchronously but nothing at the same level for POST calls. Then I came across pthreads and I read through the documentation and was a bit confused on how to even start it off. I have compiled php with the pthreads extension.
Could someone advise how to perform multiple POST calls at the same time and be able to gather the responses for later manipulation?
The below is a basic implementation that loops and waits. Very slow overall.
$postDatas = [
['field' => 'test'],
['field' => 'test1'],
['field' => 'test2'],
];
foreach ($postDatas as $postData) {
$curl = curl_init();
curl_setopt_array($curl, array(
CURLOPT_URL => "https://www.apisite.com",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_ENCODING => "",
CURLOPT_MAXREDIRS => 10,
CURLOPT_TIMEOUT => 30,
CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,
CURLOPT_CUSTOMREQUEST => "POST",
CURLOPT_POSTFIELDS => json_encode($postData),
CURLOPT_HTTPHEADER => [
"cache-control: no-cache",
"connection: keep-alive",
"content-type: application/json",
"host: some.apisite.com",
],
));
$response = curl_exec($curl);
$err = curl_error($curl);
curl_close($curl);
if ($err) {
echo "cURL Error #:" . $err;
} else {
echo $response;
}
}
That if the task is reduced to working with the API then you probably need to use http://php.net/manual/ru/function.curl-multi-exec.php
public function getMultiUrl() {
//If the connections are very much split the queue into parts
$parts = array_chunk($this->urlStack, self::URL_ITERATION_SIZE , TRUE);
//base options
$options = [
CURLOPT_USERAGENT => 'MyAPP',
CURLOPT_HEADER => false,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_POST => true,
];
foreach ($parts as $urls) {
$mh = curl_multi_init();
$active = null;
$connects = [];
foreach ($urls as $i => $url) {
$options[CURLOPT_POSTFIELDS] = $url['postData'];
$connects[$i] = curl_init($url['queryUrl']);
curl_setopt_array($connects[$i], $options);
curl_multi_add_handle($mh, $connects[$i]);
}
do {
$status = curl_multi_exec($mh, $active);
$info = curl_multi_info_read($mh);
if (false !== $info) {
var_dump($info);
}
} while ($status === CURLM_CALL_MULTI_PERFORM || $active);
foreach ($connects as $i => $conn) {
$content = curl_multi_getcontent($conn);
file_put_contents($this->dir . $i, $content);
curl_close($conn);
}
}
}
I'm trying to send an image via POST using PHP's CURL methods.
I am sending the image to an API that expects the POST field 'photo_file' to be an image.
I set the 'photo_file' to # followed by the file name.
However, when I make the request the API receives the literal string '#filename' instead of the file contents.
Here is the relevant code:
Calling code:
$data = array(
'campaign_id' => $_POST['campaign_id'],
);
$img_source = realpath($img_source); // "/Users/andrew/Sites/roo/9699d27bb09fda3133701ca9af084e3d.jpg"
$response = $y->upload_photo($img_source, $data);
$y->upload_photo():
public function upload_photo($img_source, $data) {
$fields = array(
'campaign_id' => $data['campaign_id'],
'photo_file' => '#' . $img_source
);
return $this->request('photos', $fields, 'post');
}
$this->request() (relevant parts):
public function request($function, $fields = array(), $method = 'get') {
$ch = curl_init();
$url = $this->host . $function;
$curlConfig = array(
CURLOPT_URL => $url,
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_HTTPHEADER => array('Accept: application/json', 'Expect:'),
CURLOPT_VERBOSE => 1,
CURLOPT_HEADER => 1,
);
if ($method == 'post') {
$curlConfig[CURLOPT_POST] = 1;
$curlConfig[CURLOPT_POSTFIELDS] = true;
$curlConfig[CURLOPT_POSTFIELDS] = $fields;
}
curl_setopt_array($ch, $curlConfig);
$result = curl_exec($ch);
list($header_blob, $body) = explode("\r\n\r\n", $result, 3);
$this->http_status_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
return json_decode($body);
}
The API is returning an error, and when its dev looked at the logs he saw that the request on his end was:
'campaign_id' => '4'
'photo_file' => '#/Users/andrew/Sites/roo/9699d27bb09fda3133701ca9af084e3d.jpg'
so adding # to the beginning of the file didn't actually send the contents of the file, like I understand it's supposed to. Also, the API works fine with other platforms that use it, so the problem is definitely on my end.
I've been banging my head against the wall on this one. I feel like my problem is something really silly, but I just can't figure out what.
I have a proxy script, which is handling everything (POST data and uploads).
When I print_r($_FILES) on the back-end server, it's an empty array, and $_POST contains the file path.
$ch = curl_init();
$options = array(
CURLOPT_HEADER => 1,
CURLOPT_HTTPHEADER => array(
"Host: {$host}",
"Cache-Control: no-cache"
),
CURLOPT_RETURNTRANSFER => true,
CURLOPT_SSL_VERIFYHOST => false,
CURLOPT_SSL_VERIFYPEER => 0,
CURLOPT_URL => "http://{$this->config->ipAddress}{$url}",
CURLOPT_FOLLOWLOCATION => false,
CURLOPT_COOKIE => $cookie_string,
CURLOPT_USERAGENT => $_SERVER["HTTP_USER_AGENT"]
);
if (! empty($_POST)) {
if (! empty($_FILES)) {
//$options[CURLOPT_HTTPHEADER][] = "Content-type: multipart/form-data";
$files = "";
foreach ($_FILES as $fid => $file) {
$files .= "{$fid}=#" . $file["tmp_name"] . ";type={$file["type"]};name={$file["name"]}&";
}
}
$postString = (! empty($files) ? $files : '') . http_build_query($_POST);
$options[CURLOPT_POST] = 1;
$options[CURLOPT_POSTFIELDS] = $postString;
}
curl_setopt_array($ch, $options);
$output = curl_exec($ch);
$info = curl_getinfo($ch);
curl_close($ch);
The reason I'm using http_build_query instead of just feeding an array is because we will encounter a multidimensional array.
Thanks!
I'm willing to bet your form enctype isn't "multipart/form-data" b/c your not passing the post string as an array.
http://php.net/manual/en/function.curl-setopt.php search for "CURLOPT_POSTFIELDS"
below is an excerpt from that page.
"If value is an array, the Content-Type header will be set to multipart/form-data. As of PHP 5.2.0, value must be an array if files are passed to this option with the # prefix."
You can find out what is going on with the curl by setting the curl option "curl_setopt($ch, CURLOPT_VERBOSE, TRUE); " and then "print_r( curl_getinfo($ch) );" after the "curl_exec($ch);" and it will output all the information about the request and response.
I have this function here:
public static function call($action, array $args)
{
$post_args = array(
'action' => $action,
'args' => $args
);
$stream = json_encode($post_args);
$headers = array(
'Content-type: application/json',
'Accept: application/json',
'Expect:'
);
$userpwd = self::$_user.':'.sha1(sha1(self::$_pass));
$ch = curl_init();
$args = array(
CURLOPT_URL => self::$_url,
CURLOPT_FOLLOWLOCATION => TRUE,
CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_POST => TRUE,
CURLOPT_POSTFIELDS => $stream,
CURLOPT_HTTPHEADER => $headers,
CURLOPT_USERPWD => $userpwd
);
curl_setopt_array($ch, $args);
$res = curl_exec($ch);
$data = json_decode($res, true);
if (isset($data) === TRUE
&& empty($data) === FALSE
) {
$res = $data;
}
return $res;
}//end call()
and at the URL where I'm posting, I'm just doing:
echo file_get_contents('php://input');
but getting nothing, even though I do post data. What could be the problem? I'm at a dead end.
Also, why do I need CURLOPT_FOLLOWLOCATION set to TRUE when I'm just posting to a simple virtual host URL on my local machine, not doing any redirects.
EDIT:
tried redoing it with fopen like so:
public static function call($action, array $args)
{
$post_args = array(
'action' => $action,
'args' => $args
);
$stream = json_encode($post_args);
$userpwd = self::$_user.':'.sha1(sha1(self::$_pass));
$opts = array(
'http' => array(
'method' => 'POST',
'header' => array(
"Authorization: Basic ".base64_encode($userpwd),
"Content-type: application/json"
),
'content' => $stream
)
);
$context = stream_context_create($opts);
$res = '';
$fp = fopen(self::$_url, 'r', false, $context);
if($fp){
while (!feof($fp)){
$res .= fread($fp, 128);
}
}
return $res;
}//end call()
no success. The connection works with curl and with fopen, since I pass the status along with result (which is just the php://input stream). Any ideas?
Can you be sure about that curl_exec function ends successfully. Also, why don't you use fopen for this purpose. I have written a JSON RPC client and server. I'm sending requests with fopen, and it works perfect.
$httpRequestOptions =
array(
'http'=>array(
'method'=>'POST',
'header'=>'Content-type: application/json',
'content'=>$requestJSON
)
);
$context = stream_context_create($httpRequestOptions);
// send request
if($fileHandler = #fopen($serverURL, 'r', false, $context)){
I'm not writing the rest. You can use this code I have written.
Found out the problem.
I was calling http://localhost/api, since I thought that it would load the index.php automatically, and then I was going to change the default file name for the folder.
The problem was that I didn't add index.php at the end - I should've called http://localhost/api/index.php.
This way it worked with cURL and with fopen.
Any ideas how to call the API without revealing the filename?