I have a class with the function below. When I call that function directly it works, but if I call that function inside the loop I got an error: Cannot connect to host. Can somebody help me? :) Thanks in advance.
public function exportProfile($xml) {
$_myRequest = curl_init("https://www.example.com/postProfile");
curl_setopt($_myRequest, CURLOPT_POST, 1);
curl_setopt($_myRequest, CURLOPT_USERPWD, "testUser:testPassword");
curl_setopt($_myRequest, CURLOPT_HTTPHEADER, Array("Content-Type: application/xml"));
curl_setopt($_myRequest, CURLOPT_POSTFIELDS,$xml );
curl_setopt($_myRequest, CURLOPT_RETURNTRANSFER, 1);
// do request, the response text is available in $_response
$_response = curl_exec($_myRequest);
$err = curl_error($_myRequest) ;
echo $err;
$_statusCode = curl_getinfo($_myRequest, CURLINFO_HTTP_CODE);
// close cURL resource, and free up system resources
curl_close($_myRequest);
return simplexml_load_string($_response);
}
I can't reproduce that problem with a simple example. Your example code seems to have a syntax error in CURLOPT_USERPWD, but I assume that is not related.
It sounds quite likely that there could there be rate limiting on the site you are querying, i.e. when you hit them a few times in a row they lock you out.
Perhaps you could debug by trying to run the query multiple times outside of the loop, i.e. calling it 1/2/3 times manually.
Related
So currently I am working with a REST API provided by a third party tool that requires me to make thousands of requests per run of my script.
For the most part, everything works well the only issue is that it takes some time. Now that the logic is finished and complete, I am looking to improve the performance of the script by playing with the CURL requests.
Some notes:
Using a third party app (like POSTMAN) I get a faster response on
average per request ~ 600ms(POSTMAN) vs 1300ms(PHP Curl). I was kind of able to achieve this rate which means I think I have the best optimization I can get
I am currently using curl_multi in other parts of my script, but the
part I am currently targeting has one CURL request depend on the
return value of another.
These are all GET requests, I have POST,PUT,DELETE, and PATCH, but
those are used rather sparingly so the area I am targeting are the
linear GET requests.
I have done some research and for the most part everyone recommends using curl_mutli as the default, but I can't really do that as the requests are chained. I was looking at the PHP documentation and I thought about going past my basic GET request and adding some more options.
Original Code
So below is my first simplistic take, creates a new curl object, sets the request type, transfer, header, credentials, and SSL verification (Needed this to bypass the third part tool moving to a cloud instance). For the specific query I was testing this ran at about ~1460ms
(From test code below = 73sec/50runs = 1.46seconds)
function executeGET($getUrl)
{
/* Curl Options */
$ch = curl_init($this->URL . $getUrl);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));
curl_setopt($ch, CURLOPT_USERPWD, /*INSERT CREDENTIALS*/);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
/* Result handling and processing */
$result = curl_exec($ch);
return $result;
}
First Attempt at optimization
Next I started going through already posted stack overflow questions and the PHP documentation to look at different CURL options to see what would affect the request. The first thing I found was the IP resolve, I forced that to default to IPV4 and that sped it up by ~ 100ms. The big one however was the encoding which sped it up by about ~300ms.
(From test code below = 57sec/50runs = 1.14seconds)
function executeGET($getUrl)
{
/* Curl Options */
$ch = curl_init($this->URL . $getUrl);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));
curl_setopt($ch, CURLOPT_USERPWD, /*INSERT CREDENTIALS*/);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4 ); //NEW
curl_setopt($ch, CURLOPT_ENCODING, ''); //NEW
/* Result handling and processing */
$result = curl_exec($ch);
return $result;
}
Re-using same curl object
The last part I decided to try was using a single curl object but just changing the URL before each request. My logic was that all the options were the same I was just querying a different endpoint. So I merged the newly discovered options for IPRESOLVE and the ENCODING along with the re-use and this is what I got in its most striped down version.
(From test code below = 32sec/50runs = 0.64seconds)
private $ch = null;
function executeREUSEGET($getUrl)
{
/* Curl Options */
if ($this->ch == null) {
$this->ch = curl_init();
curl_setopt($this->ch, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($this->ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($this->ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));
curl_setopt($this->ch, CURLOPT_USERPWD, /*INSERT CREDENTIALS*/);
curl_setopt($this->ch, CURLOPT_SSL_VERIFYPEER, false); //Needed to bypass SSL for multi calls
curl_setopt($this->ch, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4 ); //NEW
curl_setopt($this->ch, CURLOPT_ENCODING, '');
}
curl_setopt($this->ch, CURLOPT_URL, $this->URL . $getUrl);
/* Result handling and processing */
$result = curl_exec($this->ch);
return $result;
}
Testing Code
Reuse GET taken in seconds is: 32
Normal GET(with extra options) taken in seconds is: 58
Normal GET taken in seconds is: 73
$common = new Common();
$startTime = time();
for ($x = 0; $x < 50; $x++){
$r = $common->executeREUSEGET('MY_COMPLEX_QUERY');
}
echo 'Time taken in seconds is: '.(time()-$startTime);
$startTime = time();
for ($x = 0; $x < 50; $x++){
$r = $common->executeGET('MY_COMPLEX_QUERY');
}
echo 'Reuse taken in seconds is: '.(time()-$startTime);
Conclusion
This was my thought process when going my code and trying to speed up my request to more closely match what I was receiving using POSTMAN. I was hoping to get some feedback on what I can improve on to speed up these specific GET requests or if there is even anything else I can do?
EDIT: I didn't really do much testing with the curl_multi in terms of optimization, but now that I have my default GET request down under a second, would it be better to convert those curl_multi into my executeReuseGet? The difference is when I use curl_multi I just need the data, none of it depends on previous input like my executeGets.
I made a mistake during PHP web-service development.
The problem is that web service re-call itself by curl without return anything.
My mystake example code exposed into server at the following endpoint:
/server/get-my-infinite-loop-error
public function myInfiniteLoopError(){
$url = '/server/get-my-infinite-loop-error' // my web-call mistake
$curl = curl_init($url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($curl);
// never code executed right?
curl_close($curl);
$data = json_decode($result);
// json response
echo $data;
}
Now i am still stuck since almost an hour.
Do you have any idea to face the problem?
Thanks a lot in advance.
I'm new to cURL & I'm trying to retrieve data from a dynamic URL. I've successfully retrieved data from 1 page only, but what I want is retrieving data from all pages. The problem here is that pages are variable; I don't know how many time the code should loop to loop over all pages. i.e. the number of pages varies from one case to another, and a good program is the one that works in as many cases as possible. Consequently, putting the links in an array and looping over them isn't the right solution for this.
Here's a quick explanation of the URL I'm trying to retrieve data from:
https://link-search.api.cj.com/v2/link-search?website-id=[Your-ID]&link-type=banner
&advertiser-ids=1513033&records-per-page=100&page-number=' . $num
Did you notice the last variable $num? That should represent the number of the page from where data will be retrieved from. In some cases, it could be only 1, and in some other cases, it could be 10, 12 or 15 (it varies) depending on the parameters I choose, whether I want to see everything or filter some information.
Now here is the problem. How to dynamically increment that number as long as the request returns data? and if not, cURL should stop running?
Here's the code:
<?php
$num = 1;
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, "XGET");
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($curl, CURLOPT_ENCODING, "UTF-8");
curl_setopt($curl, CURLOPT_URL, 'https://link-search.api.cj.com/v2/link-search?website-id=[Your-ID]&link-type=banner&advertiser-ids=1513033&records-per-page=100&page-number=' . $num);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_COOKIE,true);
$result = curl_exec($curl);
$xml = simplexml_load_string($result) or die("Error: Cannot create object");
if (curl_errno($curl)) {
echo 'Error:' . curl_error($curl);
}
curl_close($curl);
?>
You can define your curl request inside the function and call that function like this:
<?php
$GLOBALS['num'] = 1;
function curlRequest()
{
// HERE DEFINE YOUR CURL REQUEST
// https://yourUrl.com?$GLOBALS['num']
if (curl_errno($curl)) {
echo 'Error:' . curl_error($curl);
// Exit from function in case there is no output
return;
} else {
$GLOBALS['num']++;
// Call the function to fetch NEXT page
curlRequest();
}
}
// Call the function for first time
curlRequest();
Assuming that curl will fire error if page number not exist or you can implement if condition from the result you are getting. Hope you will get some idea from this.
The main part here is if & else. You should implement if condition on the output you are getting and if page not exist then output will be different so you can simply return from the function in this case.
Please excuse my terminology if I get anything wrong, I'm new to Google API
I'm trying to add events via the Places API, for a single venue (listed under the bar category). I've followed the instructions here:
https://developers.google.com/places/documentation/actions?utm_source=welovemapsdevelopers&utm_campaign=places-events-screencast#event_intro
and this is the URL I am posting to (via PHP)
https://maps.googleapis.com/maps/api/place/event/add/format?sensor=false&key=AIzaSyAFd-ivmfNRDanJ40pWsgP1 (key altered)
which returns a 404 error. If I have understood correctly, I have set the sensor to false as I am not mobile, and created an API key in Google apis, with the PLACES service turned on.
Have I missed a vital step here, or would a subsequent error in the POST submission cause a 404 error? I can paste my code in but I thought I'd start with the basics.
Many thanks for your help, advice and time. It's very much apprecciated.
I've added this line to my CurlCall function which I believe should specify a POST, and the result is still the same.
curl_setopt($ch, CURLOPT_POST, 1);
so the whole functions reads
function CurlCall($url,$topost)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_HTTPHEADERS, array('Content-Type: application/json'));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_VERBOSE, TRUE);
curl_setopt($ch, CURLOPT_POSTFIELDS, $topost);
$body = curl_exec($ch);
curl_close($ch);
echo $body;
echo '<br>';
echo 'URL ' . $url;
echo '<br>';
return $body;
}
Are you sure that you are using the POST method and not GET?
You need to specify the format of your request in the URL by changing the word format to either json or xml depending on how you are going to structure and POST your request.
XML:
https://maps.googleapis.com/maps/api/place/event/add/xml?sensor=false&key=your_api_key
JSON:
https://maps.googleapis.com/maps/api/place/event/add/json?sensor=false&key=your_api_key
When I execute the following code it takes between 10-12 seconds to respond.
Is the problem with Twitter or with our server?
I really need to know as this is part of the code to display tweets on our website and a 12 second load time is just not acceptable!
function get_latest_tweets($username)
{
print "<font color=red>**". time()."**</font><br>";
$path = 'http://api.twitter.com/1/statuses/user_timeline/' . $username.'.json?include_rts=true&count=2';
$jason = file_get_contents($path);
print "<font color=red>**". time()."**</font><br>";
}
Thanks
When you put the URL into your browser (http://api.twitter.com/1/statuses/user_timeline/username.json?include_rts=true&count=2) how long does it take for the page to appear? If it's quick then you need to start the search at your server.
use curl instead of file_get_contents() to request, so that response will be compressed. Here is the curl function which iam using.
function curl_file_get_contents($url)
{
$curl = curl_init();
curl_setopt($curl,CURLOPT_URL,$url); //The URL to fetch. This can also be set when initializing a session with curl_init().
curl_setopt($curl,CURLOPT_RETURNTRANSFER,TRUE); //TRUE to return the transfer as a string of the return value of curl_exec() instead of outputting it out directly.
curl_setopt($curl,CURLOPT_ENCODING , "gzip");
curl_setopt($curl, CURLOPT_FAILONERROR, TRUE); //To fail silently if the HTTP code returned is greater than or equal to 400.
curl_setopt($curl,CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, TRUE);
$contents = curl_exec($curl);
curl_close($curl);
return $contents;
}