All,
I have to request a URL which returns a JSON request. I am using PHP and CURL to do this. Currently it takes around 3-4 seconds for the request and response.
Following is the curl code
$ch = curl_init();
$devnull = fopen('/tmp/curlcookie.txt', 'w');
curl_setopt($ch, CURLOPT_STDERR, $devnull);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_URL, $desturl);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 0);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT,1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postdata);
$ret = curl_exec($ch);
curl_close($ch);
if ($devnull)
{
fclose($devnull);
}
Following is the CURL_GETINFO array
Array
(
[url] => https://xx.xx.xxx.xx/portalsite/tester
[content_type] => application/json
[http_code] => 200
[header_size] => 198
[request_size] => 835
[filetime] => -1
[ssl_verify_result] => 20
[redirect_count] => 0
[total_time] => 2.054561
[namelookup_time] => 6.5E-5
[connect_time] => 0.016048
[pretransfer_time] => 0.123947
[size_upload] => 699
[size_download] => 46735
[speed_download] => 22746
[speed_upload] => 340
[download_content_length] => 0
[upload_content_length] => 0
[starttransfer_time] => 1.743973
[redirect_time] => 0
)
How can I speed up the CURL processing time?
Thanks
According to this answer (similar problem) cURL can be slow if you are on Mac OS X and you access to your project with xxxx.local (with 127.0.0.1 myproject.localin your /etc/hosts/
As #lepix said:
It is because the .local tld is reserved for Bonjour service, and this
since Mac OS X Lion (10.7).
Hope it will help, thanks to lepix.
it looks like most of the time is waiting for the server to respond (starttransfer_time - pretransfer_time = 1.620026)... maybe the server is doing some database or other operation that takes time?
I had some problem like that, using wget it was fast (1 second max), using cURL it took about 5 seconds to get the page, when tcpdump-ing I have found that cURL try to do a revers DNS lookup, and if the server doesn't have the revers DNS registred it will slow you down, I set up an reverse DNS on my local DNS server so every request to that site using cURL now goes very fast. I didn't find a way to disable revers DNS lookup from cURL settings.
My sugestion is to scan your trafic to see where it is waiting so long.
Related
I am calling a REST endpoint in PHP using cURL to fetch some JSON data:
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($ch);
echo $result;
curl_close($ch);
It takes 2.5 seconds to fetch the data using the above code on my localhost. The same code takes around 7.5 seconds when run on the live server. When the URL is opened directly on a browser it takes only 1.5 seconds.
My question is: Why does it take so long for cURL to fetch data on the live server and how can I solve this problem?
Below is the output of curl_getinfo($ch) on the server:
Array
(
[content_type] => application/json
[http_code] => 200
[header_size] => 420
[request_size] => 113
[filetime] => -1
[ssl_verify_result] => 0
[redirect_count] => 0
[total_time] => 7.305496
[namelookup_time] => 0.150378
[connect_time] => 0.473187
[pretransfer_time] => 0.473237
[size_upload] => 0
[size_download] => 1291504
[speed_download] => 176785
[speed_upload] => 0
[download_content_length] => -1
[upload_content_length] => 0
[starttransfer_time] => 1.787901
[redirect_time] => 0
[redirect_url] =>
[certinfo] => Array
(
)
[primary_port] => 80
[local_port] => 53962
)
I found the solution to my problem. As I had mentioned in the question, the service was loading the fastest in browsers. So, I checked the 'Request Headers' of the request in the 'Network' tab of Google Chrome Inspector. I copied those headers and used them in my cURL request in PHP. After scraping those headers I found that all I needed to do was to add an Accept-Encoding header. I passed a value of gzip like so:
curl_setopt($ch, CURLOPT_ENCODING, 'gzip');
but setting it to an empty string also works.
curl_setopt($ch, CURLOPT_ENCODING, '');
According to the php.net manual for CURLOPT_ENCODING:
The contents of the "Accept-Encoding: " header. This enables decoding
of the response. Supported encodings are "identity", "deflate", and
"gzip". If an empty string, "", is set, a header containing all
supported encoding types is sent.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "set ur url");
curl_setopt($ch, CURLOPT_ENCODING , "gzip");
curl_setopt($ch, CURLOPT_ENCODING, '');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
curl_close($ch);
Please check this example
I have a php page that does an order lookup. In the order summary page, I do a while loop to bring up all of the orders. Normally doing this is just fine and isn't an issue. Recently I decided to add the ability to see an order "status" based on its JIRA issue/ticket status. So for instance, if the JIRA ticket state is open, then the order status is "open" as well. If it the JIRA ticket is closed, the order status is "closed" as well.
The way I am doing JIRA ticket lookups is via a CURL GET request. Here is an example of the code:
// CHECK JIRA TICKET
$username = 'xxxx';
$password = 'xxxx';
$proxy = 'http://xxxx.xxxx.xxxx:8080/';
$url = "http://xxxx.xxxx.xxxx/rest/api/2/issue/".$row['jira_ticket']."?fields=status";
$ch = curl_init();
$headers = array(
'Accept: application/json',
'Content-Type: application/json'
);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "GET");
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_USERPWD, "$username:$password");
curl_setopt($ch, CURLOPT_POST, false);
curl_setopt($ch, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4);
$ticket_result = curl_exec($ch);
curl_close($ch);
In order for me to get a JIRA ticket status for each order, I am running this CURL GET request within a while loop. Example of code:
$results = mysqli query...;
while ($row = mysqli_fetch_array($results))
{
....define some variables
....the CURL code
....etc
}
The issue I am having is that the more orders I have, the longer it takes to load the page. I know for certain this is due to the CURL GET request within the while loop, because if I remove it, then the page loads quickly. When I debug the CURL request, the longest time I am seeing is starttransfer_time. Example:
[http_code] => 405
[header_size] => 639
[request_size] => 239
[filetime] => -1
[ssl_verify_result] => 0
[redirect_count] => 0
[total_time] => 0.294348
[namelookup_time] => 0.004216
[connect_time] => 0.004885
[pretransfer_time] => 0.004928
[size_upload] => 0
[size_download] => 1013
[speed_download] => 3441
[speed_upload] => 0
[download_content_length] => 1013
[upload_content_length] => -1
[starttransfer_time] => 0.294315
[redirect_time] => 0
Sometimes it can go up to .5 seconds. So my question...is there any way I can speed up the CURL request? If not, is there a better way to approach what I'm attempting to do?
EDIT:
I have a workaround, though not really an answer. For the time being I will just store the "order status" as a column in MySQL, and have the actual JIRA CURL request in a different file run as a background cron job that runs every so often. It's not perfect, but at least this allows for the pages to come up much quicker.
I'm working on a script to upload files to http://imagerelay.com. This is my first time using a REST API and cURL with PHP. There are two phases to the upload. First you submit metadata to the ImageRelay which includes the filesize for your local file. This returns an ID number which you use to build the URL for uploading the chunks (using CLI 'split' to generate). However, when I upload a file using cURL there are an extra 219 bytes added to each file. I suspect this is related to how cURL works, but I'm not sure how to correct the overage without manually increasing the filesize in phase 1 by 219bytes * x number of chunks.
Here is the response array received from the cURL request. The actual file chunk size is 1M (1,048,576 bytes), whereas the response shows 219 bytes more at 1048795
[http_code] => 201
[header_size] => 583
[request_size] => 346
[filetime] => -1
[ssl_verify_result] => 0
[redirect_count] => 0
[total_time] => 2.039208
[namelookup_time] => 1.2E-5
[connect_time] => 0.040014
[pretransfer_time] => 0.13825
[size_upload] => 1048795
[size_download] => 1
[speed_download] => 0
[speed_upload] => 514314
[download_content_length] => 1
[upload_content_length] => 1048795
[starttransfer_time] => 0.182222
Here's the relevant section of the code. $file represents the chunk, and $resource is pre-determined as it increments each chunk.
$ch = curl_init( $resource );
curl_setopt($ch, CURLOPT_POST, true );
$pdfData = array( 'file' => new CurlFile( "file-chunks/{$file}", 'application/octet-stream') );
curl_setopt($ch, CURLOPT_POSTFIELDS, $params );
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
$response = curl_exec($ch);
I appreciate any help that can be provided. I've been working on this for a week and scouring the internet but I can't figure out why the filesize of the chunk doesn't match the upload_content_length or size_upload. I do not have the same issue when doing this over the command line with curl (and not PHP over CLI). Thank you!
[edit]Removed typo in first sentence (accidental paste!)[/edit]
[edit2]Added bounty[/edit]
Try to send file like this
$headers = array(
'Content-Type: multipart/form-data'
);
$file = "/full/path/to/file.pdf"; // Full path to file
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
$result = curl_exec($ch);
curl_close($ch);
Try to use the option CURLOPT_VERBOSE and check STDERR to see the complete payload.
You can also try to use WireShark to check what goes on the wire.
(It could be scary at first but fairly easy to use).
I've looked at the prior posts about cURL and HTTP code 0, but they aren't helping.
I can cURL into www.bambooping.com with script below from localhost - ie, test_curl.php on localhost calls test_curl2.php on bambooping.com. However, if I run it on bambooping.com, I get HTTP code 0. (I know calling this on same host is dumb - it's just to isolate problem.)
On bambooping.com safe_mode is not set, and curl is compiled in (ie, should be since I can cURL in). This is very strange - the calling host is preventing the cURL. Why would calling out with cURL fail like this, yet calling into that same host with cURL be ok?
test_curl.php:
<?php
error_reporting(E_ALL); ini_set("display_errors", 1);
function curl_download($Url){
// is cURL installed yet?
if (!function_exists('curl_init')){
die('Sorry cURL is not installed!');
}
// OK cool - then let's create a new cURL resource handle
$ch = curl_init();
// Now set some options (most are optional)
// Set URL to download
curl_setopt($ch, CURLOPT_URL, $Url);
// Set a referer
// curl_setopt($ch, CURLOPT_REFERER, $_SERVER['REQUEST_URI']);
// make it blank - then it is ignored - otherwise, checked and error returned!
curl_setopt($ch, CURLOPT_REFERER, '');
// User agent
curl_setopt($ch, CURLOPT_USERAGENT, "MozillaXYZ/1.0");
// Include header in result? (0 = yes, 1 = no)
curl_setopt($ch, CURLOPT_HEADER, 0);
// Should cURL return or print out the data? (true = return, false = print)
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
// Timeout in seconds
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
// curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
// Download the given URL, and return output
$output = curl_exec($ch);
print_r(curl_getinfo($ch));
// Close the cURL resource, and free system resources
curl_close($ch);
return $output;
}
$str = curl_download("http://www.bambooping.com/test_curl2.php");
echo $str;
?>
test_curl2.php
<?php
echo "I am here";
?>
The curl_getinfo is:
Array
(
[url] => http://www.bambooping.com/test_curl2.php
[content_type] =>
[http_code] => 0
[header_size] => 0
[request_size] => 0
[filetime] => -1
[ssl_verify_result] => 0
[redirect_count] => 0
[total_time] => 0
[namelookup_time] => 4.3E-5
[connect_time] => 0
[pretransfer_time] => 0
[size_upload] => 0
[size_download] => 0
[speed_download] => 0
[speed_upload] => 0
[download_content_length] => 0
[upload_content_length] => 0
[starttransfer_time] => 0
[redirect_time] => 0
)
Ideas? I'm fresh out...
Thanks -
Please check there is a curl error
it's say the problem to you
<?php
if(curl_errno($ch)) echo 'Curl error: ' . curl_error($ch);
?>
The server for www.bambooping.com is probably sitting behind a firewall that prevents outgoing HTTP requests. Even though it's the same server, the request still needs to go out into the wild to resolve the DNS.
You could either edit the hosts file on your server to include 127.0.0.1 www.bampooing.com. Or you could change the URL to http://127.0.0.1/test_curl2.php, as this localhost domain is probably not blocked by the firewall.
I am unable to make a php curl request with port number , without port number i am getting response properly.
Through terminal when i do curl http://www.sample.com:8088 i am getting response back properly. But not through php curl on doing curl_inf()
i am getting
Array ( [url] => http://www.sample.com:8088/ [content_type] => [http_code] => 0 [header_size] => 0 [request_size] => 0 [filetime] => 0 [ssl_verify_result] => 0 [redirect_count] => 0 [total_time] => 0 [namelookup_time] => 0 [connect_time] => 0 [pretransfer_time] => 0 [size_upload] => 0 [size_download] => 0 [speed_download] => 0 [speed_upload] => 0 [download_content_length] => -1 [upload_content_length] => -1 [starttransfer_time] => 0 [redirect_time] => 0 [certinfo] => Array ( ) )
My code
$url = "http://www.sample.com:8088/";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_TIMEOUT, 400);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, True);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, True);
$report=curl_getinfo($ch);
print_r($report);
$result = curl_exec($ch);
curl_close($ch);
echo $result;
this code give me response from ubuntu , but not giving response from cent os if port number is specified.
Please let me know the how to fix it.
THanks in advance.
Try to set the port like this:
curl_setopt($ch, CURLOPT_PORT, $_SERVER['SERVER_PORT']);
or:
curl_setopt($ch, CURLOPT_PORT, 8088);
Try to set
curl_setopt($ch,CURLOPT_PORT, 8088);
See more info http://php.net/manual/en/function.curl-setopt.php
Add
curl_setopt($ch, CURLOPT_PORT, 8088);
Curl_setopt reference
"setsebool httpd_can_network_connect on" resolves the problem for me
Make sure that the port 8080 is enabled in Cent OS
and try this curl_setopt($ch, CURLOPT_PORT, 8088);
I went to server admin with this problem and he changed configuration in centos
in folder /etc/sysconfig/
he edited file selinux and change SELINUX=disabled and restated it.
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
and my problem got solved.