So currently I am working with a REST API provided by a third party tool that requires me to make thousands of requests per run of my script.
For the most part, everything works well the only issue is that it takes some time. Now that the logic is finished and complete, I am looking to improve the performance of the script by playing with the CURL requests.
Some notes:
Using a third party app (like POSTMAN) I get a faster response on
average per request ~ 600ms(POSTMAN) vs 1300ms(PHP Curl). I was kind of able to achieve this rate which means I think I have the best optimization I can get
I am currently using curl_multi in other parts of my script, but the
part I am currently targeting has one CURL request depend on the
return value of another.
These are all GET requests, I have POST,PUT,DELETE, and PATCH, but
those are used rather sparingly so the area I am targeting are the
linear GET requests.
I have done some research and for the most part everyone recommends using curl_mutli as the default, but I can't really do that as the requests are chained. I was looking at the PHP documentation and I thought about going past my basic GET request and adding some more options.
Original Code
So below is my first simplistic take, creates a new curl object, sets the request type, transfer, header, credentials, and SSL verification (Needed this to bypass the third part tool moving to a cloud instance). For the specific query I was testing this ran at about ~1460ms
(From test code below = 73sec/50runs = 1.46seconds)
function executeGET($getUrl)
{
/* Curl Options */
$ch = curl_init($this->URL . $getUrl);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));
curl_setopt($ch, CURLOPT_USERPWD, /*INSERT CREDENTIALS*/);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
/* Result handling and processing */
$result = curl_exec($ch);
return $result;
}
First Attempt at optimization
Next I started going through already posted stack overflow questions and the PHP documentation to look at different CURL options to see what would affect the request. The first thing I found was the IP resolve, I forced that to default to IPV4 and that sped it up by ~ 100ms. The big one however was the encoding which sped it up by about ~300ms.
(From test code below = 57sec/50runs = 1.14seconds)
function executeGET($getUrl)
{
/* Curl Options */
$ch = curl_init($this->URL . $getUrl);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));
curl_setopt($ch, CURLOPT_USERPWD, /*INSERT CREDENTIALS*/);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4 ); //NEW
curl_setopt($ch, CURLOPT_ENCODING, ''); //NEW
/* Result handling and processing */
$result = curl_exec($ch);
return $result;
}
Re-using same curl object
The last part I decided to try was using a single curl object but just changing the URL before each request. My logic was that all the options were the same I was just querying a different endpoint. So I merged the newly discovered options for IPRESOLVE and the ENCODING along with the re-use and this is what I got in its most striped down version.
(From test code below = 32sec/50runs = 0.64seconds)
private $ch = null;
function executeREUSEGET($getUrl)
{
/* Curl Options */
if ($this->ch == null) {
$this->ch = curl_init();
curl_setopt($this->ch, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($this->ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($this->ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));
curl_setopt($this->ch, CURLOPT_USERPWD, /*INSERT CREDENTIALS*/);
curl_setopt($this->ch, CURLOPT_SSL_VERIFYPEER, false); //Needed to bypass SSL for multi calls
curl_setopt($this->ch, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4 ); //NEW
curl_setopt($this->ch, CURLOPT_ENCODING, '');
}
curl_setopt($this->ch, CURLOPT_URL, $this->URL . $getUrl);
/* Result handling and processing */
$result = curl_exec($this->ch);
return $result;
}
Testing Code
Reuse GET taken in seconds is: 32
Normal GET(with extra options) taken in seconds is: 58
Normal GET taken in seconds is: 73
$common = new Common();
$startTime = time();
for ($x = 0; $x < 50; $x++){
$r = $common->executeREUSEGET('MY_COMPLEX_QUERY');
}
echo 'Time taken in seconds is: '.(time()-$startTime);
$startTime = time();
for ($x = 0; $x < 50; $x++){
$r = $common->executeGET('MY_COMPLEX_QUERY');
}
echo 'Reuse taken in seconds is: '.(time()-$startTime);
Conclusion
This was my thought process when going my code and trying to speed up my request to more closely match what I was receiving using POSTMAN. I was hoping to get some feedback on what I can improve on to speed up these specific GET requests or if there is even anything else I can do?
EDIT: I didn't really do much testing with the curl_multi in terms of optimization, but now that I have my default GET request down under a second, would it be better to convert those curl_multi into my executeReuseGet? The difference is when I use curl_multi I just need the data, none of it depends on previous input like my executeGets.
Related
I'm trying to get all links at concrete web-page. I need only 'a' tags with concrete parameter. But firstly, as far as I know, I have to download the whole page.
I use this (mostly not mine) code:
<?php
function file_get_contents_curl($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); //Устанавливаем параметр, чтобы curl возвращал данные, вместо того, чтобы выводить их в браузер.
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FORBID_REUSE, true);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$startUrl = 'address';
$data = file_get_contents_curl($startUrl);
echo($data);
?>
By this I'm getting error "Too many requests". The question is: can I change the amount of requests for finging elements of links array?
I think about curl_multi, but as far as I understand, it assumes that I already have the array and only need to make multiple threads.
Help, please.
I create one small API what collect contact form informations from one website and store in database on another website in managment application what I also build.
On website where is contact form is this code:
// collect all fields
$save = array(
'oAuth'=>'{KEY}',
'secret'=>'{SECRET-KEY}',
'category'=>'Category',
'name'=>'Jon Doe',
'email'=>'jon#doe.com',
'phone'=>'123 456 7890',
// ... etc. other fields
);
// made GET request
$fields=array();
foreach($save as $key=>$val){
$fields[]=$key."=".rawurlencode($val);
}
// Set cross domain URL
$url='http://api.mydomain.com/';
// Send informations in database
$cURL = curl_init();
curl_setopt($cURL,CURLOPT_URL, $url.'?'.join("&",$fields));
curl_setopt($cURL, CURLOPT_RETURNTRANSFER, true);
curl_setopt($cURL, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($cURL, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
curl_setopt($cURL, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($cURL, CURLOPT_TIMEOUT, 2);
curl_setopt($cURL, CURLOPT_HTTPHEADER, array('Accept: application/json'));
$output=curl_exec($cURL);
curl_close($cURL);
usleep(1000);
// redirect
header("Location: http://somemysite.com/thank-you-page");
session_destroy();
exit;
My question is do to use cURL like now for this or to use header() function to send GET?
I ask because I not use output here and sometimes redirection start before cURL finish request.
What is faster way to send GET info and not lost data?
The redirection will not start before cURL finishes. What may happen is that cURL fails to load you remote page, and you don't handle that error. You should add something like this:
if($output === false || curl_getinfo($cURL, CURLINFO_HTTP_CODE) != 200) {
// Here you handle your error
}
If you are looking for an alternative to load remote webpages, you can set allow_url_fopen to true in your php.ini and then use this simple function instead of cURL:
$output = file_get_contents($url.'?'.join("&",$fields));
I am making a command line application. I need to send out multiple POST requests via cURL simultaneously after I have performed log in procedures - meaning outgoing requests must send session id etc.
The chain of events is as follows:
I open cURL connection with curl_init
I log in to remote site sending POST request with curl_exec and get returned HTML code as response
I send multiple POST requests to same site simultaneously.
I was thinking of using something like that:
// Init connection
$ch = curl_init();
// Set curl options
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookies.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookies.txt');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_POST, 1);
// Perform login
curl_setopt($ch, CURLOPT_URL, "http://www.mysite/login.php");
$post = array('username' => 'username' , 'password' => 'password');
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($post));
$result = curl_exec($ch);
// Send multiple requests after being logged on
curl_setopt($ch, CURLOPT_TIMEOUT_MS, 1);
for($i = 0 ; $i < 10 ; $i++){
$post = array('myvar' => 'changing_value');
curl_setopt($ch, CURLOPT_URL, 'www.myweb.ee/changing_url');
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($post));
curl_exec($ch);
}
But this doesn't seem to work as only the first request in loop seems to be sent.
Using curl_multi_init would probably one solution but I don't know if i can pass it the same cURL handle multiple times with changed options for each.
I don't need any response from server for those simultaneous requests but it would be awesome if it also can be done somehow.
It would be perfect if someone could push me in the right direction how to do it.
You'll need to create a new curl handle for every request, and then register it with http://www.php.net/manual/en/function.curl-multi-add-handle.php
here is some code i ripped out and adapted from my code base, have in mind that you should add error checking in there.
function CreateHandle($url , $data) {
$curlHandle = curl_init($url);
$defaultOptions = array (
CURLOPT_COOKIEJAR => 'cookies.txt' ,
CURLOPT_COOKIEFILE => 'cookies.txt' ,
CURLOPT_ENCODING => "gzip" ,
CURLOPT_FOLLOWLOCATION => true ,
CURLOPT_RETURNTRANSFER => true ,
CURLOPT_POST => 1,
CURLOPT_POSTFIELDS => $data
);
curl_setopt_array($curlHandle , $defaultOptions);
return $curlHandle;
}
function MultiRequests($urls , $data) {
$curlMultiHandle = curl_multi_init();
$curlHandles = array();
$responses = array();
foreach($urls as $id => $url) {
$curlHandles[$id] = CreateHandle($url , $data[$id]);
curl_multi_add_handle($curlMultiHandle, $curlHandles[$id]);
}
$running = null;
do {
curl_multi_exec($curlMultiHandle, $running);
} while($running > 0);
foreach($curlHandles as $id => $handle) {
$responses[$id] = curl_multi_getcontent($handle);
curl_multi_remove_handle($curlMultiHandle, $handle);
}
curl_multi_close($curlMultiHandle);
return $responses;
}
There's a faster, more efficient option ... that doesn't require that you use any curl at all ...
http://uk3.php.net/manual/en/book.pthreads.php
http://pthreads.org
See github for latest source, releases on pecl ....
I will say this, file_get_contents may seem appealing, but PHP was never designed to run threaded in this manner, it's socket layers and the like give no thought to consumption you might find that it's better to fopen and sleep inbetween little reads to conserve CPU usage ... however you do it it will be much better ... and how you do it depends on what kind of resources you want to dedicate the task ...
Please excuse my terminology if I get anything wrong, I'm new to Google API
I'm trying to add events via the Places API, for a single venue (listed under the bar category). I've followed the instructions here:
https://developers.google.com/places/documentation/actions?utm_source=welovemapsdevelopers&utm_campaign=places-events-screencast#event_intro
and this is the URL I am posting to (via PHP)
https://maps.googleapis.com/maps/api/place/event/add/format?sensor=false&key=AIzaSyAFd-ivmfNRDanJ40pWsgP1 (key altered)
which returns a 404 error. If I have understood correctly, I have set the sensor to false as I am not mobile, and created an API key in Google apis, with the PLACES service turned on.
Have I missed a vital step here, or would a subsequent error in the POST submission cause a 404 error? I can paste my code in but I thought I'd start with the basics.
Many thanks for your help, advice and time. It's very much apprecciated.
I've added this line to my CurlCall function which I believe should specify a POST, and the result is still the same.
curl_setopt($ch, CURLOPT_POST, 1);
so the whole functions reads
function CurlCall($url,$topost)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_HTTPHEADERS, array('Content-Type: application/json'));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_VERBOSE, TRUE);
curl_setopt($ch, CURLOPT_POSTFIELDS, $topost);
$body = curl_exec($ch);
curl_close($ch);
echo $body;
echo '<br>';
echo 'URL ' . $url;
echo '<br>';
return $body;
}
Are you sure that you are using the POST method and not GET?
You need to specify the format of your request in the URL by changing the word format to either json or xml depending on how you are going to structure and POST your request.
XML:
https://maps.googleapis.com/maps/api/place/event/add/xml?sensor=false&key=your_api_key
JSON:
https://maps.googleapis.com/maps/api/place/event/add/json?sensor=false&key=your_api_key
I need to put a string of data like so: '< client>...<\client>' onto an XMl server (example url:'http://example.appspot.com/examples') using PHP.
(Context: Adding a new client's details to the server).
I have tried using CURLOPT_PUT, with a file and with just a string (since it requires CURLOPT_INFILESIZE and CURLOPT_INFILE) but it does not work!
Are there any other PHP functions that could be used to do such a thing? I have been looking around but PUT requests information is sparse.
Thanks.
// Start curl
$ch = curl_init();
// URL for curl
$url = "http://example.appspot.com/examples";
// Put string into a temporary file
$putString = '<client>the RAW data string I want to send</client>';
/** use a max of 256KB of RAM before going to disk */
$putData = fopen('php://temp/maxmemory:256000', 'w');
if (!$putData) {
die('could not open temp memory data');
}
fwrite($putData, $putString);
fseek($putData, 0);
// Headers
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
// Binary transfer i.e. --data-BINARY
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_URL, $url);
// Using a PUT method i.e. -XPUT
curl_setopt($ch, CURLOPT_PUT, true);
// Instead of POST fields use these settings
curl_setopt($ch, CURLOPT_INFILE, $putData);
curl_setopt($ch, CURLOPT_INFILESIZE, strlen($putString));
$output = curl_exec($ch);
echo $output;
// Close the file
fclose($putData);
// Stop curl
curl_close($ch);
since I haven't worked with cURL so far I can't really answer to that topic. If you'd like to use cURL I'd suggest looking at the server log and see what actually didn't work (so: Was the output of the request really what it's supposed to be?)
If you don't mind switching over to another technology/library I'd suggest you to use the Zend HTTP Client which is really straight forward to use, simple to include and should satisfy all your needs. Especially as performing a PUT Request is as simple as that:
<?php
// of course, perform require('Zend/...') and
// $client = new Zend_HTTP_Client() stuff before
// ...
[...]
$xml = '<yourxmlstuffhere>.....</...>';
$client->setRawData($xml)->setEncType('text/xml')->request('PUT');
?>
Code sample is from: Zend Framework Docs # RAW-Data Requests
Another way to add string body to the PUT request with CURL in PHP is:
<?php
$data = 'My string';
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'PUT'); // Define method type
curl_setopt($ch, CURLOPT_POSTFIELDS, $data); // Set data to the body request
?>
I hope this helps!