I seem to be facing a weird problem. I have a PHP script in laptop (running under wamp) which uses 'curl' and works perfectly fine at work. But when at home and connected to my ISP, the same
script fails to return any data. Even a sample code like below fails:
$ch = curl_init("http://www.yahoo.com/");
curl_setopt($ch, CURLOPT_URL, "http://www.example.com/");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_exec($ch);
var_dump(curl_getinfo($ch));
curl_getinfo dows not provide any clues to further debug. The HTTP CODE is zero. Do I need to add any headers>
I am able to access example.com through the browser, so no network issues I guess. curl_err and curl_errno is null.
<pre>
array (size=22)
'url' => string 'http://www.example.com/' (length=23)
'content_type' => null
'http_code' => int 0
'header_size' => int 0
'request_size' => int 0
'filetime' => int -1
'ssl_verify_result' => int 0
'redirect_count' => int 0
'total_time' => float 0
'namelookup_time' => float 0
'connect_time' => float 0
'pretransfer_time' => float 0
'size_upload' => float 0
'size_download' => float 0
'speed_download' => float 0
'speed_upload' => float 0
'download_content_length' => float -1
'upload_content_length' => float -1
'starttransfer_time' => float 0
'redirect_time' => float 0
'certinfo' =>
array (size=0)
empty
'redirect_url' => string '' (length=0)
</pre>
null
null
After thinking this through I think this is what you should do:
Ensure you can connect to the site from PHP:
<?php
$urlNoHttp = 'www.example.com'; // try 'example.com', too
$hostIP = gethostbyname($urlNoHttp);
var_dump($hostIP);
If it does not get the IP you have connection issues, so proceed to point 2.
Otherwise - it's not the network issues, it's curl setting itself - try point 3 and following.
Try pinging/wget'ting your destination from command line (ping www.example.com). If you get responses - your host is reachable for your OS and not the PHP, so
If you got the IP - then try reading about curl_setopt() int PHP docs. I'd start with setting proper HTTP headers (like Content-type, User-Agent, non-empty body to the request etc.). Also, try adding curl_setopt($c, CURLOPT_FRESH_CONNECT, 1); which forces not using a cached version of the url.
If it's still not working - try mixing in some proxy options, like curl_setopt($c, CURLOPT_HTTPPROXYTUNNEL, TRUE); to force "transparcence" of proxies, if there are any.
Generally you have to figure out what changes when you switch network environments. You have to know if there are any proxies involved, some strange NAT systems or maybe your router's firewall has some strange settings, that assume this kind of request as insecure and block it.
Hope this helps.
Related
This is the code I am using:
curl -k https://www.ashleydirect.com/graphics/ad_images/T908-6.jpg
This works fine (the "-k" flag is necessary for it to work or it times out)
I then use this code in PHP:
$ch = curl_init("https://www.ashleydirect.com/graphics/ad_images/T908-6.jpg");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($ch);
curl_close($ch);
And it times out -- I've tried a ton of variations, but $result is always false.
This is the PHP cURL information when I do phpinfo():
cURL support enabled
cURL Information 7.38.0
Age 3
Features
AsynchDNS No
Debug No
GSS-Negotiate No
IDN Yes
IPv6 Yes
Largefile Yes
NTLM Yes
SPNEGO No
SSL Yes
SSPI No
krb4 No
libz Yes
CharConv No
Protocols dict, file, ftp, ftps, gopher, http, https, imap, imaps, pop3, pop3s, rtsp, smtp, smtps, telnet, tftp
Host x86_64-unknown-linux-gnu
SSL Version OpenSSL/1.0.1e
ZLib Version 1.2.3
Any ideas would be greatly appreciated.
UPDATE
Here is the information from curl_getinfo($ch):
array (
'url' => 'https://www.ashleydirect.com/graphics/ad_images/T908-6.jpg',
'content_type' => NULL,
'http_code' => 0,
'header_size' => 0,
'request_size' => 0,
'filetime' => -1,
'ssl_verify_result' => 1,
'redirect_count' => 0,
'total_time' => 59.27538100000000298450686386786401271820068359375,
'namelookup_time' => 0.00975999999999999957867036215475309290923178195953369140625,
'connect_time' => 0.05170500000000000095923269327613525092601776123046875,
'pretransfer_time' => 0,
'size_upload' => 0,
'size_download' => 0,
'speed_download' => 0,
'speed_upload' => 0,
'download_content_length' => -1,
'upload_content_length' => -1,
'starttransfer_time' => 0,
'redirect_time' => 0,
'certinfo' =>
array (
),
'primary_ip' => '65.207.240.29',
'primary_port' => 443,
'local_ip' => '172.24.32.132',
'local_port' => 54461,
'redirect_url' => '',
)
UPDATE 2
Response from curl_error:
Unknown SSL protocol error in connection to www.ashleydirect.com:443
UPDATE 3 - Solution
I wanted to clearly put the solution I came up with, thanks to #Valery Viktorovsky who pointed out they only accept TLS 1.0.
The solution, then was to add this:
// Set to TLS 1.0 (CURL_SSLVERSION_TLSv1_0)
curl_setopt($ch, CURLOPT_SSLVERSION, 4);
More information here
Your php code is fine. Do a curl_getinfo($ch) and curl_error($ch) aftercurl_exec to see the response code returned by the server.
Update: I tested ashleydirect.com SSL certificate and it supports only TLS 1.0. So make sure your php version supports TLSv1.0.
It works fine for me, both the command line without -k and the PHP code.
If for some reason it times out you can set a bigger timeout:
// if it times out on establishing the connection
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10); // seconds
// if it times out while waiting for the response
curl_setopt($ch, CURLOPT_TIMEOUT, 60); // seconds
Also, the call curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false) is usually accompanied by:
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
to get the complete effect.
I'm trying to get the data returned from this link with CURL. However, I'm getting false as the output.
Here's my code and output:
Code:
$url = "https://airport.api.aero/airport/distance/LOS/ABV?user_key=MY_USER_KEY_GOES_HERE&units=mile";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
$data = curl_exec($ch);
var_dump( curl_getinfo($ch) );
curl_close($ch);
var_dump($data);
Dump of curl_getinfo call
array (size=26)
'url' => string 'https://airport.api.aero/airport/distance/LOS/ABV?user_key=MY_USER_KEY&units=mile' (length=102)
'content_type' => null
'http_code' => int 0
'header_size' => int 0
'request_size' => int 0
'filetime' => int -1
'ssl_verify_result' => int 0
'redirect_count' => int 0
'total_time' => float 0.702
'namelookup_time' => float 0.249
'connect_time' => float 0.468
'pretransfer_time' => float 0
'size_upload' => float 0
'size_download' => float 0
'speed_download' => float 0
'speed_upload' => float 0
'download_content_length' => float -1
'upload_content_length' => float -1
'starttransfer_time' => float 0
'redirect_time' => float 0
'redirect_url' => string '' (length=0)
'primary_ip' => string '107.22.197.175' (length=14)
'certinfo' =>
array (size=0)
empty
'primary_port' => int 443
'local_ip' => string '192.168.1.17' (length=12)
'local_port' => int 53767
Dump of $data
boolean false
Is there something I'm doing wrong?
I was able to successfully send a get request (you should hide your key from your question lol)
Try this:
$url = "https://airport.api.aero/airport/distance/LOS/ABV";
$qry_str = "?units=mile&user_key=MY_USER_KEY_GOES_HERE";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url . $qry_str);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_TIMEOUT, '3');
curl_setopt($ch, CURLOPT_HEADER, 0);
$content = trim(curl_exec($ch));
curl_close($ch);
print $content;
Found the solution here. Since the url was using https, I needed to include this option curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);. That did it.
Are you able to make a curl request via command line?
i.e curl 'https://airport.api.aero/airport/distance/LOS/ABV?user_key=MY_USER_KEY_GOES_HERE&units=mile'
I'm experiencing weird behaviour of a cURL request that I do in my PHP code.
I'm running the code locally on a standard WAMP Apache server.
Here's the code:
$auth_info = "...";
$some_url = "...";
$channel = curl_init();
curl_setopt($channel, CURLOPT_URL, $some_url);
curl_setopt($channel, CURLOPT_HTTPHEADER,
array("Authorization: BASIC " . base64_encode($auth_info))
);
curl_setopt($channel, CURLOPT_RETURNTRANSFER, true);
curl_setopt($channel, CURLOPT_HEADER, true);
$response = curl_exec($channel);
var_dump(curl_getinfo($channel));
$header_size = curl_getinfo($channel, CURLINFO_HEADER_SIZE);
$header = substr($response, 0, $header_size);
$body = substr($response, $header_size);
var_dump($header);
var_dump($body);
curl_close($channel);
If I'm executing this little PHP code snippet via my CLI (Powershell, since I'm running on Windows), everything is fine, all var_dumps work and I can see the $header and $body and everything and the data I expect is actually present.
Now for the weird behaviour. If I'm opening the script file with the above snippet in any browser, it just gives me:
array (size=26)
'url' => string 'the_url_here' (length=39)
'content_type' => null
'http_code' => int 0
'header_size' => int 0
'request_size' => int 0
'filetime' => int -1
'ssl_verify_result' => int 0
'redirect_count' => int 0
'total_time' => float 1.234
'namelookup_time' => float 0
'connect_time' => float 0.109
'pretransfer_time' => float 0
'size_upload' => float 0
'size_download' => float 0
'speed_download' => float 0
'speed_upload' => float 0
'download_content_length' => float -1
'upload_content_length' => float -1
'starttransfer_time' => float 0
'redirect_time' => float 0
'redirect_url' => string '' (length=0)
'primary_ip' => string 'here_is_the_ip' (length=12)
'certinfo' =>
array (size=0)
empty
'primary_port' => int 443
'local_ip' => string 'here_is_my_ip' (length=13)
'local_port' => int -> my_local_port_here
boolean false
boolean false
I'm completely puzzled since I cannot see a difference between the script beeing started by the CLI and beeing started by the browser. Has anyone got an idea on this?
EDIT Note: If I'm using Guzzle for the request, everything works fine, both in CLI and browser. I'm still interested in an answer why cURL is causing problems here.
Have you tried logging verbose output for the Curl request?
Normally I find this the best way to figure out what's going on under the hood...https://stackoverflow.com/a/14436877/682754
Also not as popular but this approach looks simple to implement and is a lot cleaner...https://stackoverflow.com/a/26611776/682754
I am trying to use PHP CURL method to try to automate a login into an application.
The url of the application is:
https://www.test.accounts.xerox.com/auth/login.jsf?Xcntry=USA&Xlang=en_US&next_page=https://uscbopoc.external.xerox.com:4430/eservice_enu/
My PHP code to invoke the CURL request is:
$url = 'https://www.test.accounts.xerox.com/auth/login.jsf?Xcntry=USA&Xlang=en_US';
$cookie = "cookie.txt";
$postData = array(
'_id171:username' => 'XXXX',
'_id171:password' => 'XXXX',
'_id171:submitLogin' => '_id171_SUBMIT',
'nextpage' => 'next_page=https://uscbopoc.external.xerox.com:4430/eservice_enu/'
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
//curl_setopt($ch, CURLOPT_HTTPHEADER
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.4 (KHTML, like Gecko) Chrome/22.0.1229.94 Safari/537.4");
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch,CURLOPT_HTTPHEADER,array(
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3',
'Accept-Encoding: gzip,deflate,sdch',
'Accept-Language: en-US,en;q=0.8',
'Cache-Control: max-age=0',
'Connection: keep-alive',
'Content-Type:application/x-www-form-urlencoded',
'Host:www.test.accounts.xerox.com',
'Origin:https://www.test.accounts.xerox.com'
));
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch,CURLOPT_REFERER,'https://www.test.accounts.xerox.com/auth/login.jsf?Xcntry=USA&Xlang=en_US&next_page=https://uscbopoc.external.xerox.com:4430/eservice_enu/');
curl_setopt($ch, CURLOPT_POSTFIELDS, $postData);
curl_setopt ($ch, CURLOPT_COOKIEJAR,"c:\\wamp\www\\cookie.txt");
curl_setopt ($ch, CURLOPT_COOKIEFILE,"c:\\wamp\www\\cookie.txt");
curl_close($ch);
return $output;
Below is my curl_getinfo function output
array
'url' => string 'https://www.test.accounts.xerox.com/auth/login.jsf?Xcntry=USA&Xlang=en_US' (length=73)
'content_type' => string 'text/html;charset=UTF-8' (length=23)
'http_code' => int 200
'header_size' => int 648
'request_size' => int 778
'filetime' => int -1
'ssl_verify_result' => int 20
'redirect_count' => int 0
'total_time' => float 0.969
'namelookup_time' => float 0
'connect_time' => float 0.078
'pretransfer_time' => float 0.407
'size_upload' => float 568
'size_download' => float 96789
'speed_download' => float 99885
'speed_upload' => float 586
'download_content_length' => float -1
'upload_content_length' => float 568
'starttransfer_time' => float 0.469
'redirect_time' => float 0
'certinfo' =>
array
empty
'redirect_url' => string '' (length=0)
But I always get a login page back as response. I think the main problem might be with the cookie but I am new to PHP and to CURL so I am not sure of that.
I also had a few general questions about CURL if anybody could point me to the right direction that will be great:
What is the best way to determine the parameters (Headers, POST Fields) required for a post request? (I know one is Firebug) but is there a tool available?
Ans: I used HTTPFox (a firefox plugin) and found it perfect to
determine the post request parameters and also to capture and view
cookies information
What is the best way to debug the CURL post requests?
Should we be giving Login Page as URL or url of the action attribute provided in form?
Ans: So far for the post requests using the value of Action parameter
has worked for me.
EDIT --------
The code started working after I provided the absolute path for parameters
CURLOPT_COOKIEJAR
CURLOPT_COOKIEFILE
I have updated the code to reflect the same and I also had to perform a simple get first to get the initial set of cookies that were needed for post to work.
The GET request was with same curl parameters.
What is the best way to determine the parameters (Headers, POST Fields) required for a post request? (I know one is Firebug) but is there a tool available?
Yes, Firebug is a tool and available. Point in case here is that you need the inspection there where the request works, so often a tool inside the browser like Firebug or Chromium developer tools.
What is the best way to debug the CURL post requests?
Curl verbose option.
Is there any tutorials available online to explain advanced CURL features with examples?
Sure, just search stackoverflow. Just one example here for the verbose:
Getting content body from http post using php CURL
Should we be giving Login Page as URL or url of the action attribute provided in form?
That depends on what you need to do, it can not be that generally answered.
I'm testing this code snippet on a Windows 7 machine running EasyPHP and PHP5, when I run the code it still returns code 200, I'm using a non-existent url so it shouldn't, should it? Is this because a redirection takes place for example to http://www.123-reg.co.uk? Thats what I suspect but I am struggling to find how I get around that.
$url = "http://www.bnkbkvbirbcx.co.uk";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, TRUE);
curl_setopt($ch, CURLOPT_NOBODY, TRUE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, FALSE);
$hr = curl_exec($ch);
$httpreturncode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
curl_close($ch);
echo $httpreturncode;
inserting var_dump($hr, curl_getinfo($ch)) produces this output;
string 'HTTP/1.1 200 OK
Date: Fri, 28 Sep 2012 20:18:16 GMT
Server: Apache
Cache-control: no-cache, no-store
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Pragma: no-cache
Connection: close
Content-Type: text/html; charset=UTF-8
' (length=224)
array
'url' => string 'http://www.bnkbkvbirbcx.co.uk' (length=29)
'content_type' => string 'text/html; charset=UTF-8' (length=24)
'http_code' => int 200
'header_size' => int 224
'request_size' => int 62
'filetime' => int -1
'ssl_verify_result' => int 0
'redirect_count' => int 0
'total_time' => float 0.063
'namelookup_time' => float 0.016
'connect_time' => float 0.031
'pretransfer_time' => float 0.031
'size_upload' => float 0
'size_download' => float 0
'speed_download' => float 0
'speed_upload' => float 0
'download_content_length' => float -1
'upload_content_length' => float 0
'starttransfer_time' => float 0.063
'redirect_time' => float 0
'certinfo' =>
array
empty
'redirect_url' => string '' (length=0)
I would like to suggest to set another DNS server for your computer/server. It seems that your provider redirects non-existing domains to their homepage.
I don't know your provider virginmedia but with the german telekom there is an option to deactivate this option on their homepage.
An option would be to use an open/free DNS server like them from Google which are aviable via 8.8.8.8 and 8.8.4.4
I hope that helps!
Well I finally got to the bottom of this, I feel stupid it took me so long now! My isp DOES intervene with a "handy feature" as follows: "If an address you enter doesn't locate a site, this handy feature will convert the incorrect address into a web search, so instead of an error message you will get a list of our closest matches". I also discovered I can choose to "switch off" this handy feature which I did and now I get the results I'd expect. Thanks for all your input, as you can tell from some of my shakey posts I'm new to stackoverflow but have found it good to share my problem.
The script below might be helpful to whomever ends their search here.
The fakePhrase is used to detect ISP "Search Assist" adware HTTP responses.
#!/bin/bash
fakePhrase="verizon"
siteList=(
'http://google.com'
'https://google.com'
'http://wikipedia.org'
'https://wikipedia.org'
'http://cantgettherefromhere'
'http://searchassist.verizon.com'
)
exitStatus=0
function isUp {
http=`curl -sL -w "%{http_code}" "$1" -o temp_isUp`
fakeResponse=`cat temp_isUp | grep $fakePhrase`
if [ -n "$fakeResponse" ]; then
http=$fakePhrase
fi
case $http in
[2]*)
;;
[3]*)
echo 'Redirect'
;;
[4]*)
exitStatus=4
echo "$1 is DENIED with ${http}"
;;
[5]*)
exitStatus=5
echo "$1 is ERROR with ${http}"
;;
*)
exitStatus=6
echo "$1 is NO RESPONSE with ${http}"
;;
esac
}
for var in "${siteList[#]}"
do
isUp $var
done
if [ "$exitStatus" -eq "0" ]; then
echo 'All up'
fi
rm temp_isUp
exit $exitStatus