file_get_contents not retrieving page contents - php

OK, before saying this is a duplicate just read a bit....
I have been trying to echo contents of URL that has allow_url_fopen disabled for HOURS now, I have tried every solution posted on stack overflow. EXAMPLE:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
$result = curl_exec($ch);
curl_close($ch);
Doesn't WORK
function curl_get_contents($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
Doesn't WORK
$url = "http://www.google.com";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$data = curl_exec($ch);
curl_close($ch);
echo $data;
Doesn't WORK
fopen("cookies.txt", "w");
$url="http://adfoc.us/1575051";
$ch = curl_init();
$header=array('GET /1575051 HTTP/1.1',
'Host: adfoc.us',
'Accept:text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language:en-US,en;q=0.8',
'Cache-Control:max-age=0',
'Connection:keep-alive',
'Host:adfoc.us',
'User-Agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.116 Safari/537.36',
);
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,0);
curl_setopt( $ch, CURLOPT_COOKIESESSION, true );
curl_setopt($ch,CURLOPT_COOKIEFILE,'cookies.txt');
curl_setopt($ch,CURLOPT_COOKIEJAR,'cookies.txt');
curl_setopt($ch,CURLOPT_HTTPHEADER,$header);
$result=curl_exec($ch);
curl_close($ch);
Doesn't WORK
// create the Gateway object
$gateway = new Gateway();
// set our url
$gateway->init($url);
// get the raw response, ignore errors
$response = $gateway->exec();
Doesn't WORK
$file = "http://www.example.com/my_page.php";
if (function_exists('curl_version'))
{
$curl = curl_init();
curl_setopt($curl, CURLOPT_URL, $file);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
$content = curl_exec($curl);
curl_close($curl);
}
else if (file_get_contents(__FILE__) && ini_get('allow_url_fopen'))
{
$content = file_get_contents($file);
}
else
{
echo 'You have neither cUrl installed nor allow_url_fopen activated. Please setup one of those!';
}
This doesn't work.
The page I am trying to use file_get_contents on is not on my website. I am trying to use file_get_contents so i can make a simple API for the site owner by reading a page and checking if a certain word is present on the page.
But yeah if anyone has any suggestions PLEASE post below :)

You can check first weather the site is available or not for example a sample code
Code taken from here:
<?php
$cURL = curl_init('http://www.technofusions.com/');
curl_setopt ( $cURL , CURLOPT_RETURNTRANSFER , true );
// Follow any kind of redirection that are in the URL
curl_setopt ( $cURL , CURLOPT_FOLLOWLOCATION , true );
$result = curl_exec ( $cURL );
// Getting HTTP response code
$answer = curl_getinfo ( $cURL , CURLINFO_HTTP_CODE );
curl_close ( $cURL );
if ( $answer == ' 404 ' ) {
echo ' The site not found (ERROR 404)! ' ;
} else {
echo ' It looks like everything is working fine ... ' ;
}
?>
For a full answer you can got to this tutorial Curl IN PHP

Related

why am i getting Forbidden when i try to use curl in php file

I m trying to integrate a payment method on a website, the first thing I did, I tried a curl code to test it using git console and it works just fine, then I tried to execute the curl command using PHP. I created a file then I used this code:
<?php
$endpoint_url = 'https://secure.payinspect.com';
$params = [
'action'=>'SALE',
'order_id'=>'ORDER12345',
'order_amount'=>'1.99',
'order_currency'=>'USD',
'order_description'=>'Product',
'card_number'=>'4111111111111111',
'card_exp_month'=>'05',
'card_exp_year'=>'2020',
'card_cvv2'=>'000',
'payer_first_name'=>'John',
'payer_last_name'=>'Doe',
'payer_address'=>'BigStreet',
'payer_country'=>'US',
'payer_state'=>'CA',
'payer_city'=>'City',
'payer_zip'=>'123456',
'payer_email'=>'doe#example',
'payer_phone'=>'199999999',
'payer_ip'=>'123.123.123.123',
'term_url_3ds'=>'http://client.site.com/return.php',
'recurring_init'=>'N',
'hash'=>'e3dd86f469f40a5cfedf96a82ff257af'
];
$buff = [];
foreach ($params as $k => $v) {
array_push($buff, "{$k}={$v}");
}
$url = $endpoint_url . implode('&', $buff);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_close($ch);
if ($result===false){ print curl_error($curl); }
$response = json_decode($result, true);
echo $result;
?>
but I got this error :
Forbidden
You don't have permission to access
I googled for this error and I tried to add this line
curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36');
$result = curl_exec($ch);
but i still got the same error . so what causes this problem and how cauld i fix it
Assuming everything else is correct - this might fix the problem.
<?php
$endpoint_url = 'https://secure.payinspect.com';
$params = [
'action'=>'SALE',
'order_id'=>'ORDER12345',
'order_amount'=>'1.99',
'order_currency'=>'USD',
'order_description'=>'Product',
'card_number'=>'4111111111111111',
'card_exp_month'=>'05',
'card_exp_year'=>'2020',
'card_cvv2'=>'000',
'payer_first_name'=>'John',
'payer_last_name'=>'Doe',
'payer_address'=>'BigStreet',
'payer_country'=>'US',
'payer_state'=>'CA',
'payer_city'=>'City',
'payer_zip'=>'123456',
'payer_email'=>'doe#example',
'payer_phone'=>'199999999',
'payer_ip'=>'123.123.123.123',
'term_url_3ds'=>'http://client.site.com/return.php',
'recurring_init'=>'N',
'hash'=>'e3dd86f469f40a5cfedf96a82ff257af'
];
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $endpoint_url);
// -- this sets the request method to POST ----
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($params));
// --- end ----
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_close($ch);
if ($result===false) { print curl_error($curl); }
$response = json_decode($result, true);
echo $result;
I can't say for certain (which means this isn't a great answer) but by suffixing the parameters to the URL, in the way you're doing currently, you're creating a GET request rather than a POST one.
It's quite likely that the receiving service is expecting to see your hash value (and everything else) in the POST data - and as it doesn't see it there, it rejects your request completely.

What on my PHP Curl call do I need to add to read a response's custom headers?

I tried a few several ways to read the responses custom header but have not been able to. I know the response I get is served by nginx and the custom header names start with X-......
$endpoint = 'url here';
$ch = curl_init( $endpoint );
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_HEADERFUNCTION, 'cbFunc');
$result = curl_exec($ch);
print_r( curl_getinfo($ch ) );
The PHP manual is an excellent reference guide an a good starting point when you run into problems like this.
CURLOPT_HEADERFUNCTION [Set value to] A callback accepting five parameters.
hence
log_headers('init');
$endpoint = 'url here';
$ch = curl_init( $endpoint );
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_HEADERFUNCTION, 'log_headers');
$result = curl_exec($ch);
$headers=log_headers();
print_r($headers);
function log_headers($ch=false, $headers=false)
{
static $hdrs;
if (is_array($hrs) && $ch===$headers===false) {
return $hdrs[];
} elseif ($ch==='init') {
$hdrs=array();
return 0;
}
$hdrs[]=$headers;
return strlen($headers);
}

Cannot extract the data from the website using PHP cURL

I am working on a project which needs to get the data from other webpage:
https://eth.ethfans.org/#/miner?0x2998850087633a4806191960c94ed535d97da598
I am trying to use the function cRUL:
<?php
$url = "https://eth.ethfans.org/#/miner?0x2998850087633a4806191960c94ed535d97da598";
$ch = curl_init();
$timeout = 5;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$contents = curl_exec($ch);
curl_close($ch);
echo $contents;
?>
However, I can only get the layout of the site, but I cannot get the data inside.
Can anyone help for this ?
Thanks in Advance.
Regards,
Alex
Use str_get_html to fetch the data from the layout:
$get_html = str_get_html($contents);
Example:
function check()
{
$url = "https://stackoverflow.com/questions/49248329/cannot-extract-the-data-from-the-website-using-php-curl";
$get_html = $this->get_curl($url);
#print_r($get_html); exit;
$get_html = str_get_html($get_html);
$fb = NULL;
foreach ($get_html->find('a') as $v) { // you can get what data from the layout
if(strpos($v->href, 'facebook'))
{
echo $fb = $v->href;
echo "\n";
break;
}
}
unset($get_html);
}
public function get_curl($url)
{
ob_start();
$ch = curl_init($url);
$headers = [
'Accept-Language: en-US,en;q=0.5',
'Cache-Control: no-cache',
'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:28.0) Gecko/20100101 Firefox/51.0',
];
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch,CURLOPT_URL, $url);
$response = curl_exec ($ch);
curl_close ($ch);
ob_end_flush();
return $response;
}
you're hitting the wrong url, the page you're hitting only contains the layout and the javascript required to fetch the actual data, then the javascript fetch the data from https://eth.ethfans.org/api/page/miner?value=2998850087633a4806191960c94ed535d97da598 , so, do as the javascript does, and fetch that url.

How to get japanese wiki link from my english wiki link?

Task: We have wikipedia English page and need to retrieve the same page address in Japanese.
suggested to parse http://en.wikipedia.org/wiki/Mini 4wd?action=raw results (there are other languages links in the bottom), but this way is too inefficient. Are there any other ways is the one real option?
We found some API in Wiki that seems fine for single word. but for two words like - Kamen rider, mini 4wd ... it doesn't work.
My code is not working
$url = 'https://en.wikipedia.org/w/api.php?action=query&prop=langlinks&format=json&lllimit=100&llprop=url&lllang=ja&titles=Kamen rider';
$url = rawurldecode(urlencode($url));
echo $url;
// outputs: https://en.wikipedia.org/w/api.php?action=query&prop=langlinks&format=json&lllimit=100&llprop=url&lllang=ru&titles=Mini+4wd
// and then the rest your logic whatever it is the rest
$header[] = "Accept: application/json";
$header[] = "Accept-Encoding: gzip";
$ch = curl_init();
curl_setopt( $ch, CURLOPT_HTTPHEADER, $header );
curl_setopt($ch,CURLOPT_ENCODING , "gzip");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
curl_setopt( $ch, CURLOPT_URL, $url );
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, true );
$response = json_decode(curl_exec($ch));
/* echo '<pre>';
print_r($response);
echo '</pre>'; */
exit;
Two words doesn't work because its not properly formatted. Kamen<space>rider and mini<space>4wd has spaces. You need it to be converted first. Consider this example:
$url = 'https://en.wikipedia.org/w/api.php?action=query&prop=langlinks&format=json&lllimit=100&llprop=url&lllang=ru&titles=Mini 4wd';
$url = rawurldecode(urlencode($url));
echo $url;
// outputs: https://en.wikipedia.org/w/api.php?action=query&prop=langlinks&format=json&lllimit=100&llprop=url&lllang=ru&titles=Mini+4wd
// and then the rest your logic whatever it is the rest
$contents = file_get_contents($url);
$contents = json_decode($contents, true);
// echo '<pre>';
// print_r($contents);
// echo '</pre>';
Sample Fiddle
Kindly try this code it works. Your $keywords = 'Mini 4wd';
$url = 'https://en.wikipedia.org/w/api.php?action=query&prop=langlinks&format=json&lllimit=100&llprop=url&lllang=ja&titles='.$keywords.'&redirects=';
$url1 = rawurldecode(urlencode($url));
$ch = curl_init();
// set url
curl_setopt($ch, CURLOPT_URL, $url1);
//return the transfer as a string
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
// $output contains the output string
$output = curl_exec($ch);
$info = curl_getinfo($ch);
$exec = explode('"url":"',$output);
$exe = explode('",',$exec[1]);
$URL = $exe[0];
Output :
<p>Wikipedia Help here as <?php echo $URL;?></p>

Retriving a rss feed with jfeed and curl?

I have been fighting with this for hours now I am trying to retrive a rss feed from maxhire:
rsslink, parse the content and display it using jfeed. now i am aware of the ajax not allowing for cross domain and i have been using the proxy.php that jfeed comes packaged with, but to no avail it just tells me there are to many redirects in the url so i have increased them like so:
<?php
header('Content-type: text/html');
$context = array(
'http'=>array('max_redirects' => 99)
);
$context = stream_context_create($context);
// hand over the context to fopen()
$handle = fopen($_REQUEST['url'], "r", false, $context);
if ($handle) {
while (!feof($handle)) {
$buffer = fgets($handle, 4096);
echo $buffer;
}
fclose($handle);
}
?>
but still no luck it just returns a message telling me that the object has been moved. So i have moved on to using curl like so:
$ch = curl_init('http://www.maxhire.net/cp/?EC5A6C361E43515B7A591C6539&L=EN');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, false);
$result = curl_exec($ch);
var_dump($result);
to retrive the xml page locally but it just returns the same error the object has moved:
<body>string(237) "<title>Object moved</title>
<h2>Object moved to here.</h2>
"
</body>
then redirects me to a url locally with : &AspxAutoDetectCookieSupport=1 added to the end.
Can someone please explain what i'm doing wrong?
Right I managed to get curl working by faking the useragent and the cookies and i am using a custom metafield in wordpress to assign the url like so:
<?php
$mykey_values = get_post_custom_values('maxhireurl');
foreach ( $mykey_values as $key => $value ) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $value);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.6 (KHTML, like Gecko) Chrome/16.0.897.0 Safari/535.6');
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_REFERER, "http://www.maxhire.net");
$html = curl_exec($ch);
curl_close($ch);
echo $html;
}
?>

Categories