Unable to make external HTTP Requests using PHP cURL behind Proxy - php

I'm running a local LAMP web server inside a virtualbox VM configured through vagrant which is sitting behind a corporate proxy.
From the web server I'm trying to make external HTTP requests using php-curl but they're simply timing out; However requesting local addresses via php-curl works fine.
Snippet:
$ch = curl_init();
$fp = fopen('/tmp/curl-debug.txt', 'w');
$options = array(
CURLOPT_URL => 'http://stackoverflow.com',
CURLOPT_RETURNTRANSFER => true,
CURLOPT_HEADER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_AUTOREFERER => true,
CURLOPT_CONNECTTIMEOUT => 10,
CURLOPT_TIMEOUT => 15,
CURLOPT_MAXREDIRS => 10,
CURLOPT_VERBOSE => true,
CURLOPT_STDERR => $fp
);
curl_setopt_array( $ch, $options );
$response = curl_exec($ch);
curl_close($ch);
Debug log:
* Hostname was NOT found in DNS cache
* Trying 23.201.173.230...
* Connection timed out after 10001 milliseconds
* Closing connection 0
Helpful info:
My OS: Windows 7
VM OS: Ubuntu 14.04.4
PHP v5.5.9-1
Apache 2.4.7
VM has a private_network ip of 192.168.33.10
It's worth mentioning I'm behind a corporate proxy which the vm is configured to play nicely with AFAIK.
If I try running curl from the command line of the virtual machine, everything works as expected
E.g: curl http://stackoverflow.com/ returns html.
nslookup from inside VM:
Server: 10.0.2.3
Address: 10.0.2.3#53
Non-authoritative answer:
Name: stackoverflow.com
Address: 104.16.35.249
Name: stackoverflow.com
Address: 104.16.34.249
Name: stackoverflow.com
Address: 104.16.37.249
Name: stackoverflow.com
Address: 104.16.33.249
Name: stackoverflow.com
Address: 104.16.36.249
I've read around that /etc/resolv.conf could play a role in this and have made sure that the user group www-data has read access to it. My resolv.conf:
nameserver 10.0.2.3
search proxy.internal
I've tried changing the nameserver to 8.8.8.8 no luck and tried changing search to the ip that proxy.internal resolves to.
I've also tried reaching the IP which the domains resolve to via php-curl to bypass the dns but it still timesout.
I'm out of ideas, and google results!

My first thought is to make sure the php-curl extension is installed properly and working, by using grep on your apache php.ini. I realize your question states you have php-curl working on local addresses, but no further details were provided so I would be inclined to double check.
My next thought is that the corporate proxy is interfering - try booting the box on a different internet connection (your house, tethered to your phone, whatever) and see if that solves your problem. If it doesn't at least you can rule out the proxy as being the immediate issue.

The problem was the proxy server.
I falsely assumed that because my virtual machine was configured to play nicely with the proxy, all Apache requests would go via the VM.
Solution - tell php-curl about the proxy:
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, true);
curl_setopt($ch, CURLOPT_PROXYPORT, $PORT);
curl_setopt($ch, CURLOPT_PROXY, $PROXY_ADDR);
curl_setopt($ch, CURLOPT_PROXYUSERPWD, 'username_here:password_here');

Related

CentOS PHP curl unable to negotiate an acceptable set of security parameters

On an Ubuntu 14.04.3 this code works fine:
$url_login = "https://test.example.com/login.do";
$cert_file = '/var/www/html/test/cert.pem';
$ssl_key = '/var/www/html/test/cert_private.pem';
$post_fields = 'userAction=1&cancelReason=&cancelType=&account=&memoType=&userText=&userid=99999999&password=xxxxxxxxxxxxxxxx';
$ch = curl_init();
$options = array(
CURLOPT_RETURNTRANSFER => 1,
CURLOPT_HEADER => 1,
CURLOPT_FOLLOWLOCATION => 1,
CURLOPT_SSL_VERIFYHOST => 0,
CURLOPT_SSL_VERIFYPEER => 0,
CURLOPT_USERAGENT => 'Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)',
CURLOPT_VERBOSE => 0,
CURLOPT_URL => $url_login ,
CURLOPT_SSLCERT => $cert_file ,
CURLOPT_SSLCERTTYPE, 'PEM',
CURLOPT_SSLKEY => $ssl_key,
CURLOPT_COOKIESESSION => 1,
CURLOPT_POST => true,
CURLOPT_POSTFIELDS => $post_fields
);
curl_setopt_array($ch , $options);
$output = curl_exec($ch);
The php on Ubuntu is using curl with openssl.
On a Centos 7 if fails with:
Curl Error : SSL peer was unable to negotiate an acceptable set of security parameters.
curl is here with nss.
The "cert.pem" contains only the client certificate with the cert-chain, and the "cert_private.pem" contains the private key not password protected. (-----BEGIN RSA PRIVATE KEY-----).
How can i get the above PHP code work with both? openssl and nss implementations of curl?
How about correcting:
CURLOPT_SSLCERTTYPE, 'PEM',
to
CURLOPT_SSLCERTTYPE => 'PEM',
?
I've also come across this problem using client certificate authentication with nss, while openssl works fine.
After much testing, this is what I've established with the server we're trying to contact:
curl using TLS v1.2 (default in some cases) with client certificate fails
curl using TLS v1.2 with client cert required by server, but not used by client, connects successfully. However client is not authenticated.
curl using TLS v1.0 with client certificate is successful
The above happens regardless of cipher suite, generally we're using rsa_aes_256_cbc_sha_256.
The quick workaround is to force TLS v1.0:
CURLOPT_SSLVERSION => 4,
Clearly this isn't ideal, and your server may not support it.
Another option is to compile curl with openssl or even GnuTLS (although I haven't tested the latter) instead of nss. Again, this may not be an option.
So far this points to a problem with NSS. I'll update this answer if further debugging generates any useful information.
Just for reference, this is the full error message using curl on the command line:
* NSS error -12227 (SSL_ERROR_HANDSHAKE_FAILURE_ALERT)
* SSL peer was unable to negotiate an acceptable set of security parameters.
* Closing connection 0
curl: (35) SSL peer was unable to negotiate an acceptable set of security parameters.
Update 2015-11-24: Further testing with Wireshark and ssltap shows the initial handshake is succeeding and the connection gets as far as the client sending ChangeCipherSpec, followed by its encrypted "Finished" message.
The server should then decrypt the client's "Finished" message, verify the hash and MAC and respond with its own encrypted "Finished" message. Instead, the server is responding with "handshake_failure" at this point.
This should provide a clue as to where NSS is failing.
Chrome, Openssl and Charles Proxy can all authenticate using the client certificate. Firefox (using NSS) and curl (with NSS) both fail at this point.
Update 2015-11-27: Additional information provided by the server's operations team suggests this may be an issue with a non-compliant server. The problem only arises when using TLS 1.2 under certain circumstances. This would explain why some SSL libraries, such as OpenSSL, are flexible enough to work around it.
NSS may be more strict in its compliance with RFCs. I'll update the answer if/when we hear more from the operations team managing the server.
Update 2017-01-25: The webserver software and load balancers are custom built for a specific bank's payment gateway. We've recently tried again with a new client and the server now appears to work with both Curl built with either NSS or OpenSSL and are no longer seeing the error. In summary: the workaround was to use a different SSL library and wait for the developers to fix the server software.

cURL call with a port in the url/location not working - couldn't connect to host

I am trying to make a cURL call to a url that looks like this:
https://example.com:9000/test
When I execute the following code, I get curl error 7 couldn't connect to host.
$headers = array(
CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_CONNECTTIMEOUT => 5,
CURLOPT_TIMEOUT => 10,
CURLOPT_URL => 'https://example.com:9000/test',
);
$headers[CURLOPT_SSL_VERIFYPEER] = FALSE;
$headers[CURLOPT_SSL_VERIFYHOST] = 2;
$ch = curl_init();
curl_setopt_array($ch, $headers);
$response = curl_exec($ch);
If I set the url to https://example.com/test, I am able to connect to the host, just not to what I need to get.
I have also tried setting <code>CURLOPT_PORT => 9000</code> with the same result (error 7).
One other note, I am able to use cURL with the url on some machines but not others. My Windows machine works fine, but the linux server I'm on is the one having issues. Another linux server seems to work fine as well.
EDIT:
Server is shared hosting on hostgator.com.
If anyone else has this problem, contact your server host. I contacted hostgator.com and they responded with the following:
I have opened the outbound port 9000
for you.
Everything works now.
check the url in IE browser. if it works fine then remove the timeout params and try.

PHP locally hosted, connecting to external resources through a proxy server

I'm trying to do some locally hosted facebook development but I'm on a university network, and therefore all outgoing connections from my computer need to pass through our proxy server. The main problem I'm having is that I can't seem to find any documentation for setting up apache to USE a proxy server, rather than to ACT as a proxy server.
Upon thinking about this however, perhaps when I do a "cURL" request or an fopen, that apache does not perform the retrieving of data, and it is instead the PHP drivers that do this. Older versions allowed you to setup a global proxy in the PHP.ini file, but not in PHP 5.
I have to use code to actually physically set the defaults and cannot find any config files where I can set them permanently. For example, this sets up streams so fopen can function:
$r_default_context = stream_context_get_default
(
array
(
'http' => array
( // All HTTP requests are passed through the local NTLM proxy server on port 8080.
'proxy' => 'tcp://proxy.munged.edu:8080',
'request_fulluri' => True,
),
)
);
but this will not set everything which is required as to use cURL, I have to do this:
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_PROXY, "http://proxy.munged.edu:8080");
curl_setopt($ch, CURLOPT_PROXYPORT, 8080);
Is there anyone who knows how to set all things that require outgoing connections to use this proxy as I don't won't code that's specific to this computer (because my plan was to work on my code locally and then upload it to some webspace when it's done: the change/upload/refresh cycle is ALOT more time consuming than just that change/refresh cycle)
edit:
just to clarify, i have been including all this in a file called "proxyconfig.php" then checking for it's existance, and include()-ing it at the top. if there's no way to set up the defaults in config files, having the methods to set up all the things that the facebook.php page used for their API requires would be awesom.
Your method is correct, assuming that the application is in iframe mode (FBML applications require Facebook being able to callback to your server).
If the issue is wanting to be able to develop locally and deploy to a remote site with minimal modification to your files, I'd recommend extending BaseFacebook as a new class called LocalBaseFacebook and changing CURL_OPTS to:
public static $CURL_OPTS = array(
CURLOPT_CONNECTTIMEOUT => 10,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_TIMEOUT => 10,
CURLOPT_USERAGENT => 'facebook-php-3.0',
CURLOPT_PROXY => 'http://proxy.munged.edu:8080',
CURLOPT_PROXYPORT => 8080
);
When deploying out, make a switch when instantiating the Facebook class based on hostname or some uniquely identifying property / configuration (you could even use a $_GET variable such as ?is_local=1) and attach that to the end of your Canvas URL.

cURL - Operation not permitted error

I'm trying to use cURL with PHP and its giving me this error:
"Failed to connect to 208.77.188.166: Operation not permitted"
I'm pretty sure its a server issue - but just in case, here is my code:
<?php
$ch = curl_init();
$url ="http://www.example.com";
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec($ch) or die(curl_error($ch));
echo $output;
?>
cURL is enabled on Apache, I've tried changing permissions of the file to 777.
Any ideas?
It's possible that you need to enable allow_url_fopen (reference) -- you can do this in an .htaccess file if it's on apache.
You can enable this by putting this in an .htaccess file:
php allow_url_fopen on
Make sure you set all required CURL options:
CURLOPT_RETURNTRANSFER => true,
CURLOPT_HEADER => false,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_ENCODING => "",
CURLOPT_USERAGENT => "spider",
CURLOPT_AUTOREFERER => true,
CURLOPT_CONNECTTIMEOUT => 120,
CURLOPT_TIMEOUT => 120,
CURLOPT_MAXREDIRS => 10,
I will also suggest you echo out errors also using something like this:
$hostconnect = curl_init($url);
$errmsgcurl = curl_error($hostconnect);
echo $errmsgcurl;
The above code is not tested and it just serves as an example.
Also would suggest trying out your code on a local apache server this way you can tell where the problem sits easily.
Many shared hosting providers prohibit outbound connections. Bluehost, for example, requires that you purchase a static IP before allowing outbound connections. Then you need to make sure CURL knows what outbound interface to use.
The error you are receiving is most likely do to a firewall blocking all outbound connections. Many shared hosting providers are blocking outgoing port 80 connections to try to stop rampant errors in PHP scripts that allow remote includes to then be used as an attack vector against the server.
Please contact your host, and if this is the case you will need to find an alternate way to connect to the remote host, or move hosting companies.
You should try using a version of curl installed on the server or on your workstation (command line version) and try to replicate the error, you may need to set a referrer header in the curl request, but that all depends on the server you are trying to contact.
Could be a proxy issue or some kind of authentication problem on the server - can you access this URL using a regular web browser ?

reading SSL page with CURL (php) [duplicate]

This question already has answers here:
PHP - SSL certificate error: unable to get local issuer certificate
(19 answers)
Closed 1 year ago.
I am trying to download the content of a secure (uses https) webpage using php and curl libraries.
However, reading failed and I get error 60: "SSL certificate problem, verify that the CA cert is OK."
also "Details: SSL3_GET_SERVER_CERTIFICATE:certificate verify failed"
So...pretty self explanatory error msg's.
My question is: How do I send an SSL certificate (the right one?) and get this page to verify it and let me in?
Also, here is my options array in case you are wondering:
$options = array(
CURLOPT_RETURNTRANSFER => true, // return web page
CURLOPT_HEADER => false, // don't return headers
CURLOPT_FOLLOWLOCATION => true, // follow redirects
CURLOPT_ENCODING => "", // handle all encodings
CURLOPT_USERAGENT => "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:x.x.x) Gecko/20041107 Firefox/x.x", // who am i
CURLOPT_AUTOREFERER => true, // set referer on redirect
CURLOPT_CONNECTTIMEOUT => 120, // timeout on connect
CURLOPT_TIMEOUT => 120, // timeout on response
CURLOPT_MAXREDIRS => 10, // stop after 10 redirects
CURLOPT_SSL_VERIFYHOST => 1,
);
Any suggestions would be great,
Andrew
It sounds like you might be misinterpreting the error. It looks to me like the site you're connecting to is self-signed or some other common problem. Just like the usual browser warning, you're easiest work around is to disable the checks.
You'll need to set CURLOPT_SSL_VERIFYPEER and CURLOPT_SSL_VERIFYHOST to FALSE. This should disable the two main checks. They may not both be required, but this should at least get you going.
To be clear, this disables a feature designed to protect you. Only do this if you have verified the certificate and server by some other means.
More info on the PHP site: curl_setopt()
If you want to use SSL peer verification (turning it off is not always good idea) you may use next solution on Windows globally for all applications:
Download file with root certificates from here:
http://curl.haxx.se/docs/caextract.html
Add to php.ini:
curl.cainfo=C:/path/to/cacert.pem
that's all magic, CURL can now verify certificates.
(as I know there is no such problem on Linux, at least on Ubuntu)
Even after following advice on SO.. You may still have problems with an error like:
error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error
the problem is with the SSL version. Use the following for version 3
curl_setopt($ch, CURLOPT_SSLVERSION,3)
I am assuming that u have enabled verification of peer and host as well and are pointing to an actual certificate file. Eg.
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($ch, CURLOPT_CAINFO, getcwd() . "/cacert.pem");
This is a "problem" with openssl and VeriSign.
I had a similar problem and my openssl was missing the intermediate ssl certificate used by VeriSign to sign the server certificate.
https://knowledge.verisign.com/support/ssl-certificates-support/index?page=content&id=AR657
I had to import these intermediate certificates from the VeriSign Homepage or Firefox cert-database-export into my local ca-certificates list and after this step I was able to use wget/curl to use the protected connection without any errors.
If it's a developer machine - you can also add this certificate in you system.
Something like this - https://www.globalsign.com/support/intermediate/intermediate_windows.php
It's for WinXP, but it works also on other versions of windows.
You're not SENDing the SSL cert. It appears there's a problem with the SSL cert as it is installed on the host you are contacting. Use option -k or --insecure, to get past the complaint.
Ah. See Ryan Graham's answer
This is apparently on openssl bug. Tomcat can be configured to work around this in /etc/tomcat7/server.xml by restricting the available cipher list:
<Connector protocol="HTTP/1.1" SSLEnabled="true" ... ciphers="SSL_RSA_WITH_RC4_128_SHA"/>

Categories