I am using php and google's new CSE in my website . The URL for request is like
https://www.googleapis.com/customsearch/v1?key=INSERT-YOUR-KEY&cx=017576662512468239146:omuauf_lfve&q=lectures
The connection is https and file_get_content() is getting failed. How can i get it working?
Since i need to host the site on some external web-hosting servers, I am in need of solution which don't alter php configuration file or work with default options found on most of web-hosting sites.
You can use curl:
## HTTPS url that you are targeting.
$url = "https://www.googleapis.com/customsearch/v1?key=INSERT-YOUR-KEY&cx=017576662512468239146:omuauf_lfve&q=lectures";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_USERAGENT, 'Opera/9.23 (Windows NT 5.1; U; en)');
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
## Below two option will enable the HTTPS option.
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
$result = curl_exec($ch);
echo $result;
If you can't modify the config yourself, talk to the hosting provider. It most probably has to do with the lacking of openssl in the php core or startup.
If you are doing this for someone else and you have no control or contact with the host (the "some external host"), explain the situation and help them find a suitable host with the appropriate features enabled.
Related
I am using PHP Simple HTML DOM Parser, here you can check more about it: http://simplehtmldom.sourceforge.net/
And also i am using a CURL because this web adress http://www.sportsdirect.com is not loading on the normal examples from the SimpleHTMLDom.
So here is the code i use:
<?php
include_once('../simple_html_dom.php');
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, 'http://www.sportsdirect.com/');
curl_setopt($curl, CURLOPT_HEADER, 0);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 10);
$str = curl_exec($curl);
curl_close($curl);
$html= str_get_html($str);
echo $html->plaintext;
?>
When i try to load the script it gives me: 500 Internal Server Error
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator, webmaster#superweb.bg and inform them of the time the error occurred, and anything you might have done that may have caused the error.
More information about this error may be available in the server error log.
Additionally, a 404 Not Found error was encountered while trying to use an ErrorDocument to handle the request.
This script is just not working for this web adress, because when i try to load other website like mandmdirectDOTcom it is woking OKEY!
Where is my mistake and how i can make this thing works?
Try this for the curl fetch. It works for me in this case. This is a standard set of curl options & settings I use that work well:
include_once('simple_html_dom.php');
$url = "http://www.sportsdirect.com";
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($curl, CURLOPT_SSLVERSION, 3);
curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($curl, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
$str = curl_exec($curl);
curl_close($curl);
$html = str_get_html($str);
echo $html->plaintext;
I believe the issue with your original curl settings was the missing user agent. Try the same script with the CURLOPT_USERAGENT line commented out to see what I mean.
Many servers have firewall settings that disallow curl requests from users making requests without a proper user agent setting. The user agent I have set here is a fairly generic Firefox user agent, so feel free to experiment with that to use something else.
Try setting a Host header in the request. It's possible that the target domain is on a shared server, and without a Host header, the server doesn't know what to do.
curl_setopt($curl, CURLOPT_HTTPHEADER, array('Host: www.sportsdirect.com'));
Hi I have a server which has several virtual hosts set up on it.
I wanted to make a curl request to this server's ip using php.
Also I wanted to make this request to a specific hostname on the server's ip.
Is there a way to do it?
A bit more elaboration :
I want to make a curl requests between my servers using internal LAN, using their internal IP. The issue is that I have several sites hosted on this server. So when i make a curl request to the internal IP of the server.. something like (curl_init(xxx.xxx.xxx.xxx)), I want to be able to be tell apache to go to a particular folder pointed to by a virtual host. I hope that made the question a bit more clear.. – Vishesh Joshi 3 mins ago edit
You can set the host header in the curl request:
<?php
$ch = curl_init('XXX.XXX.XXX.XXX');
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: subdomain.hostname.com'));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
echo curl_exec($ch);
For HTTPS sites use CURLOPT_RESOLVE which exists in every PHP version since PHP 5.5.
<?php
$ch = curl_init('https://www.example.com/');
// note: array used here
curl_setopt($ch, CURLOPT_RESOLVE, array(
"www.example.com:443:172.16.1.1",
));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_VERBOSE, true);
$result = curl_exec($ch);
Sample output:
* Added www.example.com:443:172.16.1.1 to DNS cache
* Hostname www.example.com was found in DNS cache
* Trying 172.16.1.1...
Base on Leigh Simpson,
It works, but I need query string attach with it.
That's what I work around:
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://xxx.xxx.xxx.xxx/index.php?page=api&action=getdifficulty");
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: subdomain.hostname.com'));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
echo curl_exec($ch);
?>
I've noticed a little problem with CURL in PHP. Whenever I request a https:// connection it returns "false", and every website that I try to reach while I have my PHP page open reports to have an Untrusted certificate.
This is my request method:
private function request($url, $params, $method = "GET") {
if ($method == "GET")
$url = $this->structGET($url, $params);
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
if (isset($_SERVER['HTTP_USER_AGENT'])) {
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
} else {
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.X.Y.Z Safari/525.13.');
}
curl_setopt($ch, CURLOPT_TIMEOUT, 30);
$header[] = 'Accept-Language: EN';
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
if ($method == "POST") {
curl_setopt($ch, CURLOPT_POST, true);
if ($params)
curl_setopt($ch, CURLOPT_POSTFIELDS, $params);
}
$result = curl_exec($ch);
curl_close($ch);
return $result;
}
And this is what Chrome returns when I try visiting Facebook.
The site's security certificate is not trusted!
You attempted to reach
www.facebook.com, but the server presented a certificate issued by an
entity that is not trusted by your computer's operating system. This
may mean that the server has generated its own security credentials,
which Google Chrome cannot rely on for identity information, or an
attacker may be trying to intercept your communications. You cannot
proceed because the website operator has requested heightened security
for this domain.
Don't use techniques that disable certificate verification. While they may "solve" your problem on the surface, they only ignore the problem, rather than fixing it. Never do this in production code.
The most likely cause is that you're on a network where there is a MITM corporate proxy. However legitimate these devices may be, they are effectively MITM devices.
What they do is that they will replace the original certificate with a certificate issued using their own internal CA, so as to be able to monitor the traffic.
If this device was legitimately set up by your network administrator, you should be able to get its CA certificate (in those circumstances, the CA certificate would typically be installed on all end-user machines centrally administered).
It's quite likely that, as a developer, you may have installed your own machine, and might not have the CA certificate installed. Ask your network administrator for that CA certificate, and install it with the certificates used by your browser and by curl within PHP (two different locations). Where the curl default location is may depend on the system you're using, but you can also configure it via CURLOPT_CAINFO.
Presumably, you're developing within a local network, but may possibly deploy that service on a different network when it's done. Make sure that this is configurable.
Yeah that happens when cURL tries to see if the SSL is verified. Facebook usually has a verified signature but may be because of network, it is returning invalid (happens in my case: using fortiguard proxy, facebook blocked!)
So, what you can do is, you can choose to ignore that error totally.
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
That should fix it. But, if you want to fix it properly, then you should probably use a proxy or something or get a certificate for the server(if it is yours).
You need to add these two lines for SSL
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
Hi I have a server which has several virtual hosts set up on it.
I wanted to make a curl request to this server's ip using php.
Also I wanted to make this request to a specific hostname on the server's ip.
Is there a way to do it?
A bit more elaboration :
I want to make a curl requests between my servers using internal LAN, using their internal IP. The issue is that I have several sites hosted on this server. So when i make a curl request to the internal IP of the server.. something like (curl_init(xxx.xxx.xxx.xxx)), I want to be able to be tell apache to go to a particular folder pointed to by a virtual host. I hope that made the question a bit more clear.. – Vishesh Joshi 3 mins ago edit
You can set the host header in the curl request:
<?php
$ch = curl_init('XXX.XXX.XXX.XXX');
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: subdomain.hostname.com'));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
echo curl_exec($ch);
For HTTPS sites use CURLOPT_RESOLVE which exists in every PHP version since PHP 5.5.
<?php
$ch = curl_init('https://www.example.com/');
// note: array used here
curl_setopt($ch, CURLOPT_RESOLVE, array(
"www.example.com:443:172.16.1.1",
));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_VERBOSE, true);
$result = curl_exec($ch);
Sample output:
* Added www.example.com:443:172.16.1.1 to DNS cache
* Hostname www.example.com was found in DNS cache
* Trying 172.16.1.1...
Base on Leigh Simpson,
It works, but I need query string attach with it.
That's what I work around:
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://xxx.xxx.xxx.xxx/index.php?page=api&action=getdifficulty");
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Host: subdomain.hostname.com'));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
echo curl_exec($ch);
?>
Setup:
1. Joomla 1.5 website on a LAMP server (CentOS 5.2/Apache 2.2/PHP 5.2.9/mysql 5)
2. Joomla module for currency conversion added. Module uses google finance to convert currency
3. LAMP stack resides in the intranet behind a proxy. The server environment variables for http_proxy, yum.conf proxy have been setup, and kernel successfully updated.
4. phpinfo() clearly shows curl is installed
5. module mentioned in '2.' allows 3 methods to connect to google finance, fread(), file_get_contents() and using the cURL libraries. As the box is behind a proxy, only the cURL libraries method should work.
Problem:
on a WAMP stack, the curl library method works fine. On the lamp stack, however, the module is unable to communicate with google finance, and throws an error mentioning connect timed out. Here's some code to make it clearer.
if (isset($_GET['process'])) {
$url = "http://finance.google.com/finance/converter?a={
$_GET['a']}&from={$_GET['from']}&to={$_GET['to']}";
$app->get_page($url);
$data = $app->process();
}
function get_page($url) {
if ($url!='') {
echo $url;
$ch = curl_init ();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, $this->binary);
$this->html = curl_exec($ch);
curl_close($ch);
}
}
I even tried adding a curl_setopt($ch, CURLOPT_PROXY,'10.x.xx.xx:8080'); after curl_init(), to no avail. I've compiled apache with libcurl and php enabled, and I need to know the following:
1. How to instruct php to route outgoing requests(streams) through the proxy?
2. Do I need to configure cURL (libcurl) with the proxyname and port?
3. I've switched iptables off, so the linux firewall is not in the picture anymore, is there anything else I need to do to allow outgoing requests?
4. I've setup the proxy so that my LAMP stack is unblocked for all content, cURL works off the command line, but not from php/apache. What am I missing? Any environment variables? Any switches?
Thanks in advance for your time.
Shrinivas
Here's an example using a local SOCKS5 proxy on port 1090:
<?php
$url = 'www.whatismyip.com/automation/<your unique whatismyip hash>';
function get_page($url, $proxy=true) {
if ($url!='') {
$ch = curl_init ();
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
if ($proxy) {
curl_setopt($ch, CURLOPT_PROXY, 'localhost');
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5);
curl_setopt($ch, CURLOPT_PROXYPORT, 1090);
}
$html = curl_exec($ch);
curl_close($ch);
return $html;
}
}
var_dump(get_page($url));
var_dump(get_page($url, false));
You'd probably want to use curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_HTTP); and curl_setopt($ch, CURLOPT_PROXYPORT, 8080); instead.