How to manually resolve hosts in HTTP(S) connections in PHP - php

Curl has a feature for manually specifying which IP to resolve a host to. For example:
curl https://www.google.com --resolve "www.google.com:443:173.194.72.112"
This is especially useful when using HTTPS. If it was just a HTTP request, I could have achieved the same by specifying the IP address directly, and adding a host header. But in HTTPS that would break the connection since the SSL certificate host would be compared to the IP address and not the host header.
My question is, how can I do the same thing in PHP?

Although #deceze's answer is correct, a live example might be useful. I needed CURLOPT_RESOLVE because I was trying to connect directly to the IP address with an additional Host: www.example.com header, but since the server was using SNI, this didn't work.
I used CURLOPT_RESOLVE to solve my problem. This code allows me to connect to the SNI server on a IP address of my choosing:
$resolve = array(sprintf(
"%s:%d:%s",
$hostname,
$port,
$host_ip
));
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RESOLVE, $resolve);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($ch);
curl_close($ch);

According to the changelog, support for CURLOPT_RESOLVE was added in 5.5.0.
Note that at the time of writing it's not even documented yet, but according to this bug report it takes an array as argument.

Even though it's an old question is worth noticing that when using CURL to multiple servers using the CURLOPT_RESOLVE option, there is a DNS cache which needs to be cleared before using CURL again, otherwise the CURL will point to the first server regardless of the curl_setopt($ch, CURLOPT_RESOLVE, $resolve); setting.
The only way to make this work is to add to the $resolve array a resolve string with the last sever used prefixed with a '-':
$servers = [ '192.0.2.1', '192.0.2.2', '192.0.2.3' ];
foreach ($servers as $idx => $server) {
$resolve = [];
// Remove the last server used from the DNS cache
if($idx){
$last_server = $server[$idx-1];
$resolve[] = "-example.com:443:{$last_server}";
}
// resolve the new server
$resolve[] = "example.com:443:{$server}";
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_DNS_CACHE_TIMEOUT, 0);
curl_setopt($ch, CURLOPT_RESOLVE, $resolve);
curl_setopt($ch, CURLOPT_URL, "https://example.com/some/path");
curl_setopt($ch, CURLOPT_VERBOSE, 1);
$result = curl_exec($ch);
$info = curl_getinfo($ch);
echo $info['primary_ip']."\n";
curl_close($ch);
}
As pointed here: https://bugs.php.net/bug.php?id=74135

Related

Whats the correct way use PHP cURL for a GET request with query parameters and alternate port number? (for SOLR)

I'm trying to use PHP cURL to send a GET request to Apache Solr to receive search results, and I'm running into some trouble. This is more to do with PHP cURL I think than Solr...but i digress...here's what I have so far.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://example.com/solr/example/select?".$this->query);
curl_setopt($ch, CURLOPT_PORT, 8983);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
curl_setopt($ch, CURLOPT_TIMEOUT, '4');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$data = json_decode(curl_exec($ch), TRUE);
curl_close($ch);
Currently, the request times out at 4 seconds...so I'm wondering if maybe the port# isn't being set properly...I also tried to include it in the URL itself with no luck. The weird part is that I can echo the constructed URL, add the port# manually, copy/paste into the browser and it works! But, for some reason it doesn't with the code above.
Any help would be greatly appreciated! Thanks in advance!
The above code actually works as intended...my issue was local firewall rules blocking remote connection to certain port #'s (as opposed to remote server firewall rules as skrilled suspected)

Curl function is not working?

I am trying to fetch a data from a site using curl method. Before i can able to get data by using curl.Recently client taken ssl certificate for that site from then onwards i can't able to get the data using curl. But when i try the URL in browser i can able to see data. Can any one tell me why it is not working.did i need to try in some another way.
Below is my code
// create curl resource
$ch = curl_init();
// set url
curl_setopt($ch, CURLOPT_URL, "https://qa.myhealth.today/myhealth-portal/nirvahak/public/validateSessionttt?username=pavithra#gmail.com");
//return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// $output contains the output string
$output = curl_exec($ch);
// close curl resource to free up system resources
curl_close($ch);
$output=json_decode($output);
print_r($output);
There are 2 fixes:
1) SSL certified websites do not trust any non-SSL requests. So, we need to win trust of SSL enabled client.
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
2) Provide him the security certificate.
Go the the respective site. Save the SSL certificate and pass the certificate path to the CURL request.
So, the final code should be:
// create curl resource
$ch = curl_init();
$url = "https://qa.myhealth.today/myhealth-portal/nirvahak/public/validateSessionttt?username=pavithra#gmail.com";
// set url
curl_setopt($ch, CURLOPT_URL, $url);
//return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
curl_setopt($ch, CURLOPT_CAINFO, getcwd() . "/CAcerts/BuiltinObjectToken-EquifaxSecureCA.crt");
// $output contains the output string
$output = curl_exec($ch);
// close curl resource to free up system resources
curl_close($ch);
$output=json_decode($output);
print_r($output);
Detailed instructions on how to do it are place here:
Set CURLOPT_SSL_VERIFYHOST and CURLOPT_SSL_VERIFYPEER to false.
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
CURLOPT_SSL_VERIFYHOST
1 to check the existence of a common name in the SSL peer certificate. 2 to check the existence of a common name and also verify that it matches the hostname provided. In production environments the value of this option should be kept at 2 (default value).
CURLOPT_SSL_VERIFYPEER
FALSE to stop cURL from verifying the peer's certificate. Alternate certificates to verify against can be specified with the CURLOPT_CAINFO option or a certificate directory can be specified with the CURLOPT_CAPATH option.
Source: http://php.net/manual/en/function.curl-setopt.php

Accessing .onion sites without TOR [duplicate]

Locked. Comments on this question have been disabled, but it is still accepting new answers and other interactions. Learn more.
I'm trying to connect to a Tor hidden service using the following PHP code:
$url = 'http://jhiwjjlqpyawmpjx.onion/'
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_PROXY, "http://127.0.0.1:9050/");
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5);
$output = curl_exec($ch);
$curl_error = curl_error($ch);
curl_close($ch);
print_r($output);
print_r($curl_error);
When I run it, I get the following error:
Couldn't resolve host name
However, when I run the following command from my command line in Ubuntu:
curl -v --socks5-hostname localhost:9050 http://jhiwjjlqpyawmpjx.onion
I get a response as expected.
The PHP cURL documentation says this:
--socks5-hostname
Use the specified SOCKS5 proxy (and let the proxy resolve the host name).
I believe the reason it works from the command line is because Tor (the proxy) is resolving the .onion hostname, which it recognizes. When running the PHP code above, my guess is that cURL or PHP is trying to resolve the .onion hostname and doesn't recognize it. I've searched for a way to tell cURL/PHP to let the proxy resolve the hostname, but I can't find a way.
There is a very similar Stack Overflow question, cURL request using socks5 proxy fails when using PHP, but it works through the command line.
You need to set option CURLOPT_PROXYTYPE to CURLPROXY_SOCKS5_HOSTNAME, which sadly wasn't defined in old PHP versions, circa pre-5.6; if you have earlier in but you can explicitly use its value, which is equal to 7:
curl_setopt($ch, CURLOPT_PROXYTYPE, 7);
I use Privoxy and cURL to scrape Tor pages:
<?php
$ch = curl_init('http://jhiwjjlqpyawmpjx.onion'); // Tormail URL
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 1);
curl_setopt($ch, CURLOPT_PROXY, "localhost:8118"); // Default privoxy port
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_HTTP);
curl_exec($ch);
curl_close($ch);
?>
After installing Privoxy you need to add this line to the configuration file (/etc/privoxy/config). Note the space and '.' a the end of line.
forward-socks4a / localhost:9050 .
Then restart Privoxy.
/etc/init.d/privoxy restart
Try to add this:
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 1);
TL;DR: Set CURLOPT_PROXYTYPE to use CURLPROXY_SOCKS5_HOSTNAME if you have a modern PHP, the value 7 otherwise, and/or correct the CURLOPT_PROXY value.
As you correctly deduced, you cannot resolve .onion domains via the normal DNS system, because this is a reserved top-level domain specifically for use by Tor and such domains by design have no IP addresses to map to.
Using CURLPROXY_SOCKS5 will direct the cURL command to send its traffic to the proxy, but will not do the same for domain name resolution. The DNS requests, which are emitted before cURL attempts to establish the actual connection with the Onion site, will still be sent to the system's normal DNS resolver. These DNS requests will surely fail, because the system's normal DNS resolver will not know what to do with a .onion address unless it, too, is specifically forwarding such queries to Tor.
Instead of CURLPROXY_SOCKS5, you must use CURLPROXY_SOCKS5_HOSTNAME. Alternatively, you can also use CURLPROXY_SOCKS4A, but SOCKS5 is much preferred. Either of these proxy types informs cURL to perform both its DNS lookups and its actual data transfer via the proxy. This is required to successfully resolve any .onion domain.
There are also two additional errors in the code in the original question that have yet to be corrected by previous commenters. These are:
Missing semicolon at end of line 1.
The proxy address value is set to an HTTP URL, but its type is SOCKS; these are incompatible. For SOCKS proxies, the value must be an IP or domain name and port number combination without a scheme/protocol/prefix.
Here is the correct code in full, with comments to indicate the changes.
<?php
$url = 'http://jhiwjjlqpyawmpjx.onion/'; // Note the addition of a semicolon.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_PROXY, "127.0.0.1:9050"); // Note the address here is just `IP:port`, not an HTTP URL.
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5_HOSTNAME); // Note use of `CURLPROXY_SOCKS5_HOSTNAME`.
$output = curl_exec($ch);
$curl_error = curl_error($ch);
curl_close($ch);
print_r($output);
print_r($curl_error);
You can also omit setting CURLOPT_PROXYTYPE entirely by changing the CURLOPT_PROXY value to include the socks5h:// prefix:
// Note no trailing slash, as this is a SOCKS address, not an HTTP URL.
curl_setopt(CURLOPT_PROXY, 'socks5h://127.0.0.1:9050');
Here is a simple function to help you.
Tho to save time first of all you need to make sure that you check if the proxy is valid to not with simple check with fsocketopen()
try {
$fp = fsockopen($ip, $port, $errno, $errstr, 10);
fclose($fp);
return true;
} catch (\Throwable $th) {
return false;
}
If socket returns true then go for requestUrl function
private function requestUrl($url, $proxy)
{
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_PROXY, $proxy);
curl_setopt($curl, CURLOPT_HEADER, 1);
curl_setopt($curl, CURLOPT_HTTPPROXYTUNNEL, 1);
curl_setopt($curl, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5);
curl_setopt($curl, CURLOPT_TIMEOUT, 10);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, FALSE);
$contents = curl_exec($curl);
//Check for errors.
// if (curl_errno($curl)) {
// return new \Exception(curl_error($curl));
// }
curl_close($curl);
return $contents;
}

500 Server error file_get_contents

I have two project on same server. I want some data form on my website so I am using file_get_contents; most of the time I get the 500 internal error
I checked that my url fopen is on using phpinfo().
With default settings, file_get_content() doesn't work behind a proxy or it cannot handle timeouts. It's normally recommended to read local files.
Therefore use cURL instead.
Below function could be used for the job:
function http_request($uri, $time_out = 10, $headers = 0)
{
// Initializing
$ch = curl_init();
// Set URI
curl_setopt($ch, CURLOPT_URL, trim($uri));
curl_setopt($ch, CURLOPT_HEADER, $headers);
// 1 - if output is not needed on the browser
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// Time-out in seconds
curl_setopt($ch, CURLOPT_TIMEOUT, $time_out);
// Executing
$result = curl_exec($ch);
// Closing the channel
curl_close($ch);
return $result;
}
Let me know whether your're using Linux or Windows to give you cURL installation tips

i have a curl that works in localhost but not in live website. i cannot figure out why

i have this link
http://www.bata.com.sg, this website actually exists
that works in my curl code that checks if the page exists.
it works in my localhost code, but it keeps failing in my live website.
I have tested using other domains like http://www.yahoo.com.sg, it works all the time on my localhost AND my live website.
i copied this code http://w-shadow.com/blog/2007/08/02/how-to-check-if-page-exists-with-curl/ word for word.
i dont understand why it fails with this particular url.
my website is hosted with site5.
i noticed that i keep getting false(boolean) for this line
curl_exec($ch);
I get this for curl_error Couldn't resolve host 'www.bata.com.sg'
please advise.
You need to talk to customer support of site5 to figure out why their server can not resolve www.bata.com.sg
Until you get an answer from them, try the following code.
Key points
It connect to the IP address www.bata.com.sg resolves to - 194.228.50.32
Then sends Host: www.bata.com.sg header
In essence, it works the same way as Curl would if it could resolve the address.
<?php
// this is the IP address that www.bata.com.sg resolves to
$server = '194.228.50.32';
$host = 'www.bata.com.sg';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $server);
/* set the user agent - might help, doesn't hurt */
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)');
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
/* try to follow redirects */
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
/* timeout after the specified number of seconds. assuming that this script runs
on a server, 20 seconds should be plenty of time to verify a valid URL. */
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 15);
curl_setopt($ch, CURLOPT_TIMEOUT, 20);
$headers = array();
$headers[] = "Host: $host";
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
curl_setopt($ch, CURLOPT_VERBOSE, true);
/* don't download the page, just the header (much faster in this case) */
curl_setopt($ch, CURLOPT_NOBODY, true);
curl_setopt($ch, CURLOPT_HEADER, true);
$response = curl_exec($ch);
curl_close($ch);
var_dump($response);
I have figured out the reason.
Using this website i was told that there was an issue with the nameservers of bata.com.sg
http://www.intodns.com/bata.com.sg
anyway the answers above have been useful as well. i have learned something from them.
This might be a firewall issue. sometimes the hosting company might restrict what the webserver can consult.
also can you make sure curl is present in the php_info(). I think you didn't mentioned any error so.
also you get give a try to
file_get_contents('http://www.yahoo.com.sg');

Categories