I am making a command line application. I need to send out multiple POST requests via cURL simultaneously after I have performed log in procedures - meaning outgoing requests must send session id etc.
The chain of events is as follows:
I open cURL connection with curl_init
I log in to remote site sending POST request with curl_exec and get returned HTML code as response
I send multiple POST requests to same site simultaneously.
I was thinking of using something like that:
// Init connection
$ch = curl_init();
// Set curl options
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookies.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookies.txt');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_POST, 1);
// Perform login
curl_setopt($ch, CURLOPT_URL, "http://www.mysite/login.php");
$post = array('username' => 'username' , 'password' => 'password');
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($post));
$result = curl_exec($ch);
// Send multiple requests after being logged on
curl_setopt($ch, CURLOPT_TIMEOUT_MS, 1);
for($i = 0 ; $i < 10 ; $i++){
$post = array('myvar' => 'changing_value');
curl_setopt($ch, CURLOPT_URL, 'www.myweb.ee/changing_url');
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($post));
curl_exec($ch);
}
But this doesn't seem to work as only the first request in loop seems to be sent.
Using curl_multi_init would probably one solution but I don't know if i can pass it the same cURL handle multiple times with changed options for each.
I don't need any response from server for those simultaneous requests but it would be awesome if it also can be done somehow.
It would be perfect if someone could push me in the right direction how to do it.
You'll need to create a new curl handle for every request, and then register it with http://www.php.net/manual/en/function.curl-multi-add-handle.php
here is some code i ripped out and adapted from my code base, have in mind that you should add error checking in there.
function CreateHandle($url , $data) {
$curlHandle = curl_init($url);
$defaultOptions = array (
CURLOPT_COOKIEJAR => 'cookies.txt' ,
CURLOPT_COOKIEFILE => 'cookies.txt' ,
CURLOPT_ENCODING => "gzip" ,
CURLOPT_FOLLOWLOCATION => true ,
CURLOPT_RETURNTRANSFER => true ,
CURLOPT_POST => 1,
CURLOPT_POSTFIELDS => $data
);
curl_setopt_array($curlHandle , $defaultOptions);
return $curlHandle;
}
function MultiRequests($urls , $data) {
$curlMultiHandle = curl_multi_init();
$curlHandles = array();
$responses = array();
foreach($urls as $id => $url) {
$curlHandles[$id] = CreateHandle($url , $data[$id]);
curl_multi_add_handle($curlMultiHandle, $curlHandles[$id]);
}
$running = null;
do {
curl_multi_exec($curlMultiHandle, $running);
} while($running > 0);
foreach($curlHandles as $id => $handle) {
$responses[$id] = curl_multi_getcontent($handle);
curl_multi_remove_handle($curlMultiHandle, $handle);
}
curl_multi_close($curlMultiHandle);
return $responses;
}
There's a faster, more efficient option ... that doesn't require that you use any curl at all ...
http://uk3.php.net/manual/en/book.pthreads.php
http://pthreads.org
See github for latest source, releases on pecl ....
I will say this, file_get_contents may seem appealing, but PHP was never designed to run threaded in this manner, it's socket layers and the like give no thought to consumption you might find that it's better to fopen and sleep inbetween little reads to conserve CPU usage ... however you do it it will be much better ... and how you do it depends on what kind of resources you want to dedicate the task ...
Related
So currently I am working with a REST API provided by a third party tool that requires me to make thousands of requests per run of my script.
For the most part, everything works well the only issue is that it takes some time. Now that the logic is finished and complete, I am looking to improve the performance of the script by playing with the CURL requests.
Some notes:
Using a third party app (like POSTMAN) I get a faster response on
average per request ~ 600ms(POSTMAN) vs 1300ms(PHP Curl). I was kind of able to achieve this rate which means I think I have the best optimization I can get
I am currently using curl_multi in other parts of my script, but the
part I am currently targeting has one CURL request depend on the
return value of another.
These are all GET requests, I have POST,PUT,DELETE, and PATCH, but
those are used rather sparingly so the area I am targeting are the
linear GET requests.
I have done some research and for the most part everyone recommends using curl_mutli as the default, but I can't really do that as the requests are chained. I was looking at the PHP documentation and I thought about going past my basic GET request and adding some more options.
Original Code
So below is my first simplistic take, creates a new curl object, sets the request type, transfer, header, credentials, and SSL verification (Needed this to bypass the third part tool moving to a cloud instance). For the specific query I was testing this ran at about ~1460ms
(From test code below = 73sec/50runs = 1.46seconds)
function executeGET($getUrl)
{
/* Curl Options */
$ch = curl_init($this->URL . $getUrl);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));
curl_setopt($ch, CURLOPT_USERPWD, /*INSERT CREDENTIALS*/);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
/* Result handling and processing */
$result = curl_exec($ch);
return $result;
}
First Attempt at optimization
Next I started going through already posted stack overflow questions and the PHP documentation to look at different CURL options to see what would affect the request. The first thing I found was the IP resolve, I forced that to default to IPV4 and that sped it up by ~ 100ms. The big one however was the encoding which sped it up by about ~300ms.
(From test code below = 57sec/50runs = 1.14seconds)
function executeGET($getUrl)
{
/* Curl Options */
$ch = curl_init($this->URL . $getUrl);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));
curl_setopt($ch, CURLOPT_USERPWD, /*INSERT CREDENTIALS*/);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4 ); //NEW
curl_setopt($ch, CURLOPT_ENCODING, ''); //NEW
/* Result handling and processing */
$result = curl_exec($ch);
return $result;
}
Re-using same curl object
The last part I decided to try was using a single curl object but just changing the URL before each request. My logic was that all the options were the same I was just querying a different endpoint. So I merged the newly discovered options for IPRESOLVE and the ENCODING along with the re-use and this is what I got in its most striped down version.
(From test code below = 32sec/50runs = 0.64seconds)
private $ch = null;
function executeREUSEGET($getUrl)
{
/* Curl Options */
if ($this->ch == null) {
$this->ch = curl_init();
curl_setopt($this->ch, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($this->ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($this->ch, CURLOPT_HTTPHEADER, array('Content-Type: application/json'));
curl_setopt($this->ch, CURLOPT_USERPWD, /*INSERT CREDENTIALS*/);
curl_setopt($this->ch, CURLOPT_SSL_VERIFYPEER, false); //Needed to bypass SSL for multi calls
curl_setopt($this->ch, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4 ); //NEW
curl_setopt($this->ch, CURLOPT_ENCODING, '');
}
curl_setopt($this->ch, CURLOPT_URL, $this->URL . $getUrl);
/* Result handling and processing */
$result = curl_exec($this->ch);
return $result;
}
Testing Code
Reuse GET taken in seconds is: 32
Normal GET(with extra options) taken in seconds is: 58
Normal GET taken in seconds is: 73
$common = new Common();
$startTime = time();
for ($x = 0; $x < 50; $x++){
$r = $common->executeREUSEGET('MY_COMPLEX_QUERY');
}
echo 'Time taken in seconds is: '.(time()-$startTime);
$startTime = time();
for ($x = 0; $x < 50; $x++){
$r = $common->executeGET('MY_COMPLEX_QUERY');
}
echo 'Reuse taken in seconds is: '.(time()-$startTime);
Conclusion
This was my thought process when going my code and trying to speed up my request to more closely match what I was receiving using POSTMAN. I was hoping to get some feedback on what I can improve on to speed up these specific GET requests or if there is even anything else I can do?
EDIT: I didn't really do much testing with the curl_multi in terms of optimization, but now that I have my default GET request down under a second, would it be better to convert those curl_multi into my executeReuseGet? The difference is when I use curl_multi I just need the data, none of it depends on previous input like my executeGets.
I create one small API what collect contact form informations from one website and store in database on another website in managment application what I also build.
On website where is contact form is this code:
// collect all fields
$save = array(
'oAuth'=>'{KEY}',
'secret'=>'{SECRET-KEY}',
'category'=>'Category',
'name'=>'Jon Doe',
'email'=>'jon#doe.com',
'phone'=>'123 456 7890',
// ... etc. other fields
);
// made GET request
$fields=array();
foreach($save as $key=>$val){
$fields[]=$key."=".rawurlencode($val);
}
// Set cross domain URL
$url='http://api.mydomain.com/';
// Send informations in database
$cURL = curl_init();
curl_setopt($cURL,CURLOPT_URL, $url.'?'.join("&",$fields));
curl_setopt($cURL, CURLOPT_RETURNTRANSFER, true);
curl_setopt($cURL, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($cURL, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
curl_setopt($cURL, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($cURL, CURLOPT_TIMEOUT, 2);
curl_setopt($cURL, CURLOPT_HTTPHEADER, array('Accept: application/json'));
$output=curl_exec($cURL);
curl_close($cURL);
usleep(1000);
// redirect
header("Location: http://somemysite.com/thank-you-page");
session_destroy();
exit;
My question is do to use cURL like now for this or to use header() function to send GET?
I ask because I not use output here and sometimes redirection start before cURL finish request.
What is faster way to send GET info and not lost data?
The redirection will not start before cURL finishes. What may happen is that cURL fails to load you remote page, and you don't handle that error. You should add something like this:
if($output === false || curl_getinfo($cURL, CURLINFO_HTTP_CODE) != 200) {
// Here you handle your error
}
If you are looking for an alternative to load remote webpages, you can set allow_url_fopen to true in your php.ini and then use this simple function instead of cURL:
$output = file_get_contents($url.'?'.join("&",$fields));
I'm trying to make a curl request to my laravel server, in that request I have to check whether the user of my laravel application is logged in or not. I use this code:
$transferAmount = 200;
//set POST variables
$url = URL::route('post-spend-partner');
$fields = array(
'transferAmount' => urlencode($transferAmount),
'cancelUrl' => urlencode(URL::route('get-return-page-example')),
'returnUrl' => urlencode(URL::route('get-return-page-example')),
);
// New Connection
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $fields);
curl_setopt($ch, CURLOPT_URL, $url);
curl_exec($ch);
curl_close($ch);
In the requested url I'm just checking if I'm logged in or not, but it always returns false:
public function postSpendPartner() {
echo "Authenticated? " . (Auth::check() ? 'Yes' : 'No');
}
I know for sure that I'm logged in, if I try the exact same thing with Ajax it completely works!
Does anyone know what I could try, to solve this problem?
Best regards!
Fabrice
Some facts: HTTP is stateless. Session IDs need to be passed to the server in order to continue the session. Session IDs are (most of the time) stored in cookies. Cookies are included in the request.
Using a cookiejar could indeed be one possible solution. The fact that it works using Ajax, and not by re-submitting the request from your server might be because of the session-verification mechanism on the server: Some session implementations lock session IDs to the initial IP address. If the contents of your cookiejar file check out, that might be the culprit.
That aside: re-submitting the request via Curl from your server is a severe codesmell to me. A proper solution would to implement something such as OAuth.
Try sending your cookies as a header with your curl request.
// ...
$cookie_header = "Cookie:";
$headers = [];
foreach($_COOKIE as $key => $val) {
// Do sanitize cookie values
$cookie_header .= " ".$key."=".$value.";";
}
$headers[] = $cookie_header;
curl_setopt($curl, CURLOPT_HTTPHEADER, $headers);
// ...
You could filter out unnecessary cookie values from $cookie_header.
I tried to do a flickr search using cURL and Flickr API. When trying to print the response, it prints "1". What is wrong with my code?
$params = array(
'api_key' => 'b838e46f6e8eada6a62fac7e2b25ffcc',
'method' => 'flickr.photos.search',
'format' => 'php_serial',
'text' =>'cars'
);
$encoded_params = array();
foreach($params as $k => $v){
$encoded_params[] = urlencode($k).'='.urlencode($v);
}
$ch = curl_init();
$timeout = 0; // set to zero for no timeout
curl_setopt ($ch, CURLOPT_URL, 'https://api.flickr.com/services/rest/?'.implode('&', $encoded_params));
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$file_contents = curl_exec($ch);
print_r($file_contents);
curl_close($ch);
$rsp_obj = unserialize($file_contents);
//echo 'https://api.flickr.com/services/rest/?'.implode('&', $encoded_params);
echo print_r($rsp_obj);
If cURL can not verify the certificate of the remote site, then you can set the option CURLOPT_SSL_VERIFYPEER to false, so that it will not try to do that.
This has of course some security implications – the remote site might not be who they pretend they are, but if you’re only doing a search that’s a rather minor concern, especially when you’re only testing locally for now. For a production app on a server you should maybe look into getting that fixed though, especially when you’ll be doing something that isn’t a pure search later on.
Please take a look at this sample code:
function http_response($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, TRUE);
curl_setopt($ch, CURLOPT_NOBODY, TRUE); // remove body
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$head = curl_exec($ch);
$httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
echo $httpCode ;
}
this code will print the httpCode of the given url. I have couple of questions:
Can I get rid of some setopt() lines here and still getting httpCode?
What about if I want to check multiple urls at the same time? Can I modify the code to do that?
Can I do the same functionality in a simpler way using libraries different than cURL?
Thanks :)
You should be able to remove CURLOPT_HEADER and CURLOPT_NOBODY and still get the same result.
You could do that like this:
$urls = array(
'http://google.com',
'http://facebook.com'
);
$status = array();
foreach($urls as $url){
$status[$url] = http_response($url);
}
Try print_r($status); after this and you'll see the result.
You could do this with file_get_contents and $http_response_header, to learn more: http://www.php.net/manual/en/reserved.variables.httpresponseheader.php I would however recommend using cURL anyway.
*2. to check multiple urls you have to use this function in a loop, in any programming language 1 response from a server = 1 connection to that server. If you want to use 1 function to get responses from multiple servers you can always pass an array to the function and do the loop inside the function
*3. you can try this way:
function get_contents() {
file_get_contents("http://example.com");
var_dump($http_response_header);
}
get_contents();