I have two project on same server. I want some data form on my website so I am using file_get_contents; most of the time I get the 500 internal error
I checked that my url fopen is on using phpinfo().
With default settings, file_get_content() doesn't work behind a proxy or it cannot handle timeouts. It's normally recommended to read local files.
Therefore use cURL instead.
Below function could be used for the job:
function http_request($uri, $time_out = 10, $headers = 0)
{
// Initializing
$ch = curl_init();
// Set URI
curl_setopt($ch, CURLOPT_URL, trim($uri));
curl_setopt($ch, CURLOPT_HEADER, $headers);
// 1 - if output is not needed on the browser
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// Time-out in seconds
curl_setopt($ch, CURLOPT_TIMEOUT, $time_out);
// Executing
$result = curl_exec($ch);
// Closing the channel
curl_close($ch);
return $result;
}
Let me know whether your're using Linux or Windows to give you cURL installation tips
Related
Locked. Comments on this question have been disabled, but it is still accepting new answers and other interactions. Learn more.
I'm trying to connect to a Tor hidden service using the following PHP code:
$url = 'http://jhiwjjlqpyawmpjx.onion/'
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_PROXY, "http://127.0.0.1:9050/");
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5);
$output = curl_exec($ch);
$curl_error = curl_error($ch);
curl_close($ch);
print_r($output);
print_r($curl_error);
When I run it, I get the following error:
Couldn't resolve host name
However, when I run the following command from my command line in Ubuntu:
curl -v --socks5-hostname localhost:9050 http://jhiwjjlqpyawmpjx.onion
I get a response as expected.
The PHP cURL documentation says this:
--socks5-hostname
Use the specified SOCKS5 proxy (and let the proxy resolve the host name).
I believe the reason it works from the command line is because Tor (the proxy) is resolving the .onion hostname, which it recognizes. When running the PHP code above, my guess is that cURL or PHP is trying to resolve the .onion hostname and doesn't recognize it. I've searched for a way to tell cURL/PHP to let the proxy resolve the hostname, but I can't find a way.
There is a very similar Stack Overflow question, cURL request using socks5 proxy fails when using PHP, but it works through the command line.
You need to set option CURLOPT_PROXYTYPE to CURLPROXY_SOCKS5_HOSTNAME, which sadly wasn't defined in old PHP versions, circa pre-5.6; if you have earlier in but you can explicitly use its value, which is equal to 7:
curl_setopt($ch, CURLOPT_PROXYTYPE, 7);
I use Privoxy and cURL to scrape Tor pages:
<?php
$ch = curl_init('http://jhiwjjlqpyawmpjx.onion'); // Tormail URL
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 1);
curl_setopt($ch, CURLOPT_PROXY, "localhost:8118"); // Default privoxy port
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_HTTP);
curl_exec($ch);
curl_close($ch);
?>
After installing Privoxy you need to add this line to the configuration file (/etc/privoxy/config). Note the space and '.' a the end of line.
forward-socks4a / localhost:9050 .
Then restart Privoxy.
/etc/init.d/privoxy restart
Try to add this:
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 1);
TL;DR: Set CURLOPT_PROXYTYPE to use CURLPROXY_SOCKS5_HOSTNAME if you have a modern PHP, the value 7 otherwise, and/or correct the CURLOPT_PROXY value.
As you correctly deduced, you cannot resolve .onion domains via the normal DNS system, because this is a reserved top-level domain specifically for use by Tor and such domains by design have no IP addresses to map to.
Using CURLPROXY_SOCKS5 will direct the cURL command to send its traffic to the proxy, but will not do the same for domain name resolution. The DNS requests, which are emitted before cURL attempts to establish the actual connection with the Onion site, will still be sent to the system's normal DNS resolver. These DNS requests will surely fail, because the system's normal DNS resolver will not know what to do with a .onion address unless it, too, is specifically forwarding such queries to Tor.
Instead of CURLPROXY_SOCKS5, you must use CURLPROXY_SOCKS5_HOSTNAME. Alternatively, you can also use CURLPROXY_SOCKS4A, but SOCKS5 is much preferred. Either of these proxy types informs cURL to perform both its DNS lookups and its actual data transfer via the proxy. This is required to successfully resolve any .onion domain.
There are also two additional errors in the code in the original question that have yet to be corrected by previous commenters. These are:
Missing semicolon at end of line 1.
The proxy address value is set to an HTTP URL, but its type is SOCKS; these are incompatible. For SOCKS proxies, the value must be an IP or domain name and port number combination without a scheme/protocol/prefix.
Here is the correct code in full, with comments to indicate the changes.
<?php
$url = 'http://jhiwjjlqpyawmpjx.onion/'; // Note the addition of a semicolon.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_PROXY, "127.0.0.1:9050"); // Note the address here is just `IP:port`, not an HTTP URL.
curl_setopt($ch, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5_HOSTNAME); // Note use of `CURLPROXY_SOCKS5_HOSTNAME`.
$output = curl_exec($ch);
$curl_error = curl_error($ch);
curl_close($ch);
print_r($output);
print_r($curl_error);
You can also omit setting CURLOPT_PROXYTYPE entirely by changing the CURLOPT_PROXY value to include the socks5h:// prefix:
// Note no trailing slash, as this is a SOCKS address, not an HTTP URL.
curl_setopt(CURLOPT_PROXY, 'socks5h://127.0.0.1:9050');
Here is a simple function to help you.
Tho to save time first of all you need to make sure that you check if the proxy is valid to not with simple check with fsocketopen()
try {
$fp = fsockopen($ip, $port, $errno, $errstr, 10);
fclose($fp);
return true;
} catch (\Throwable $th) {
return false;
}
If socket returns true then go for requestUrl function
private function requestUrl($url, $proxy)
{
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_PROXY, $proxy);
curl_setopt($curl, CURLOPT_HEADER, 1);
curl_setopt($curl, CURLOPT_HTTPPROXYTUNNEL, 1);
curl_setopt($curl, CURLOPT_PROXYTYPE, CURLPROXY_SOCKS5);
curl_setopt($curl, CURLOPT_TIMEOUT, 10);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($curl, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($curl, CURLOPT_SSL_VERIFYHOST, FALSE);
$contents = curl_exec($curl);
//Check for errors.
// if (curl_errno($curl)) {
// return new \Exception(curl_error($curl));
// }
curl_close($curl);
return $contents;
}
I have a strange problem which may or may not be tied to Plesk. This PHP script intends to fetch a page on the same server when executed on the same domain e.g. http://quotationsbook.com/sometestpage.php
<?php
error_reporting('E_ALL');
ini_set('display_errors', 1);
function curlFileGetContents($urlreq) {
$ch = curl_init();
$timeout = 5;
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_MAXREDIRS, 50);
curl_setopt($ch, CURLOPT_URL, $urlreq);
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
$request_result = curl_exec($ch);
if (curl_errno($ch))
$response = 0;
else
$response = $request_result;
curl_close($ch);
return $response;
}
$url = 'http://quotationsbook.com';
$data = curlFileGetContents($url);
echo '<pre>';
print_r($data);
echo '</pre>';
?>
However, it does not fetch the page requested, it always fetches a Plesk error page.
In terms of the PHP var allow_url_fopen it is set to On.
The problem only occurs for the same domain where the code is hosted, not for other domains. i.e. it only occurs on http://quotationsbook.com where what I'm trying to fetch is under http://quotationsbook.com/*, it does not occur when I try to fetch say, http://google.com
The answer to your question is actually in your last paragraph.
The problem only occurs for the same domain where the code is hosted,
not for other domains. i.e. it only occurs on
http://quotationsbook.com where what I'm trying to fetch is under
http://quotationsbook.com/*, it does not occur when I try to fetch
say, http://google.com
That is your clue.
I must pose a question and that is why use CURL if you are on the same server? What are you trying to accomplish? There are probably better solutions out there than using CURL.
Assuming you need to use CURL, it's likely a firewall or other security issue. Make sure the port being accessed is open and not blocked.
See PHP Curl does not work on localhost?
I am trying to fetch file from the same server i am running my php script in which i am cUrl to fetch it.
It does not download file and get timeout.
I am able to get the file using same url from browser.
cUrl is able to get the file if the url is anything other than the same server.
Are their any settings i need to modify to support file download using cUrl on same server.
Appreciate your help here.
My code:
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_URL, $file_url);
$content = curl_exec($ch);
curl_close($ch);
try this with all parameters of curl
function curl_download($Url){
// is cURL installed yet?
if (!function_exists('curl_init')){
die('Sorry cURL is not installed!');
}
// create a new cURL resource handle
$ch = curl_init();
// Now set some options (most are optional)
// Set URL to download
curl_setopt($ch, CURLOPT_URL, $Url);
// Set a referer
//curl_setopt($ch, CURLOPT_REFERER, "http://www.example.org/yay.htm");
// User agent
curl_setopt($ch, CURLOPT_USERAGENT, "MozillaXYZ/1.0");
// Include header in result? (0 = yes, 1 = no)
curl_setopt($ch, CURLOPT_HEADER, 0);
// Should cURL return or print out the data? (true = return, false = print)
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
// Timeout in seconds
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
// Download the given URL, and return output
$output = curl_exec($ch);
// Close the cURL resource, and free system resources
curl_close($ch);
return $output;
}
Use Google Chrome' Copy as cURL (open "dev tools", "Network" tab, right-click on a request) on a successfully fetched file, paste to a terminal, then execute.
If everything is fine, and you are able to get a file via curl at terminal, then there is something with headers and/or request params. If not, it's more likely that something wrong with your network configuration.
Also, check web server logs, do you even get to a web server with your script?
Are you using 127.0.0.1 or localhost or full TLD? Is browser connecting to Internet via proxy?
The workaround I found is this:
Test if you are on your own server, then use a simple include for the same script:
if (($ser = servername()) != $floc) {
return(GetFileContent("$floc/$page.php"));
} else { include "$page.php"; }
I use php curl to get content from a php script in the same domain url. But I get curl_exec error. The curl error code is 28 or operation timed out. After days of debugging, I found that it works on non script page like htm, but not php, it also works if the url is a script on different domain. I have been debugging for days and found no solutions. Helps appreciated.
$url = 'http://...';
$agent = '';
$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 8);
curl_setopt($ch, CURLOPT_TIMEOUT, 8);
$result = curl_exec ($ch);
print "<pre>\n";
print_r(curl_getinfo($ch));
// get error info echo "\n\ncURL error number:" .curl_errno($ch);
// print error info echo "\n\ncURL error:" . curl_error($ch);
print "</pre>\n";
curl_close ($ch);
echo $result;
cURL error number:28 cURL error:
Operation timed out after 8000
milliseconds with 0 bytes received
Okay: $url = http://.../page.htm
Fail: $url = http://.../page.php
You're not setting the User Agent. Some servers actually do not even respond to such requests (thus the client doesn't know the connection dropped).
Add the following to your code:
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER['HTTP_USER_AGENT']);
Additionally, I use the following in my CURL functionality:
curl_setopt($ch,CURLOPT_FOLLOWLOCATION,true);
curl_setopt($ch,CURLOPT_MAXREDIRS,50);
if(substr($url,0,8)=='https://'){
// The following ensures SSL always works. A little detail:
// SSL does two things at once:
// 1. it encrypts communication
// 2. it ensures the target party is who it claims to be.
// In short, if the following code is allowed, CURL won't check if the
// certificate is known and valid, however, it still encrypts communication.
curl_setopt($ch,CURLOPT_HTTPAUTH,CURLAUTH_ANY);
curl_setopt($ch,CURLOPT_SSL_VERIFYPEER,false);
}
Maybe because of session locks
Try to drop session_start() from page which you try to get using cURL
Also see this session.auto-start
Set into CURL setting
curl_setopt($ch, CURLOPT_TIMEOUT, 300); // 300 seconds
If still have issue so run the script with following location disable.
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);
I use the following command in some old scripts:
curl -Lk "https:www.example.com/stuff/api.php?"
I then record the header into a variable and make comparisons and so forth. What I would really like to do is convert the process to PHP. I have enabled curl, openssl, and believe I have everything ready.
What I cannot seem to find is a handy translation to convert that command line syntax to the equivalent commands in PHP.
I suspect something in the order of :
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
// What goes here so that I just get the Location and nothing else?
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
// Get the response and close the channel.
$response = curl_exec($ch);
curl_close($ch);
The goal being $response = the data from the api “OK=1&ect”
Thank you
I'm a little confused by your comment:
// What goes here so that I just get the Location and nothing else?
Anyway, if you want to obtain the response body from the remote server, use:
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$response = curl_exec($ch);
If you want to get the headers in the response (i.e.: what your comment might be referring to):
curl_setopt($ch, CURLOPT_HEADER, 1);
If your problem is that there is a redirection between the initial call and the response, use:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);