I can't get the content of a specific URL - php

I can't get the content of this URL (I get a blank page): https://www.euronews.com/api/watchlive.json
I have always used this function and never had problems:
function file_get_contents_curl($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 3);
curl_setopt($ch, CURLOPT_TIMEOUT, 3);
curl_setopt($ch, CURLOPT_REFERER, $referer);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
curl_setopt($ch, CURLOPT_HEADER, 5);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$data = json_decode(file_get_contents_curl("https://www.euronews.com/api/watchlive.json"), true); $url = $data['url']; echo $url;
Any idea?
EDIT: Unknown problem caused (possibly) by cURL module in Raspbian (Raspberry)

I think you need to echo the function if you want to see the content. I tried this on my end and I got the content. althought I have some undefined variable error because of $referrer, $header, $useragent and $cookie. but it runs and get the result. here is how I call the function
echo file_get_contents_curl("https://www.euronews.com/api/watchlive.json");

You're getting more than you expect in your function response, so the json_decode is failing. You need to set CURLOPT_HEADER to false instead of 5.
curl_setopt($ch, CURLOPT_HEADER, FALSE);
You can verify what you're getting by doing
$response = file_get_contents_curl("https://www.euronews.com/api/watchlive.json");
var_dump($response);
$data = json_decode($response, true);
$url = $data['url'];
echo $url;

Related

How to get full html content of specific url?

I used several method to get html content of aptoide.com in php.
1) file_get_contents();
2) readfile();
3) curl as php function
function get_dataa($url) {
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (compatible; Konqueror/4.0; Microsoft Windows) KHTML/4.0.80 (like Gecko)");
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
4)PHP Simple HTML DOM Parser
include_once('simple_html_dom.php');
$url="http://aptoide.com";
$html = file_get_html($url);
But all of them give empty output for aptoide.com
Is there a way to get full html content of that url ?
echo file_get_contents('http://www.aptoide.com/'); works fine for me.
So it's possible that aptoide.com has been blocked you. If you want to change your IP (as you said in comment) you have to use this:
$url = 'http://aptoide.com.com/';
$proxy = '127.0.0.1:9095'; // Your proxy
// $proxyauth = 'user:password'; // Proxy authentication if required
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
//curl_setopt($ch, CURLOPT_PROXYUSERPWD, $proxyauth);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
$curl_scraped_page = curl_exec($ch);
curl_close($ch);
echo $curl_scraped_page;
use your curl get_dataa function with this line added:
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
because that page is redirecting to www.aptide.com
full function:
function get_dataa($url) {
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (compatible; Konqueror/4.0; Microsoft Windows) KHTML/4.0.80 (like Gecko)");
$data = curl_exec($ch);
curl_close($ch);
return $data;
}

PHP CURL - Session expired

I'm trying to make an voip call with PHP CURL and MEGAVOIP.
The problem is i can't manage the session to access the page protected by a password.
I looked which variables are posted to the login page to post it with Curl.
But my code doesn't work.
Following Colin Morelli and Waygood's advices, I just added those lines in both commands:
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookies_file);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookies_file);
but it's still the same:
Megavoip returns: SESSION EXPIRED
So here is my full code:
<?php
ini_set("display_errors", 1);
$username="***";
$password="***";
$url="https://www.megavoip.com/login";
$url2="https://www.megavoip.com/phone_to_phone/";
$timeout = 10;
$cookies_file = 'cookies.txt';
// HERE I GET THE TOKEN
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookies_file);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookies_file);
curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
$content = curl_exec($ch);
preg_match_all('/<input[^>]+>/i',$content, $result);
preg_match_all('/(id|value)=("[^"]*")/i',$result[0][5], $img);
$img1=str_replace('"', '', $img[0][0]);
$img2=str_replace('"', '', $img[0][1]);
$img1=substr($img1,3);
$img2=substr($img1,6);
$postdata = "login%5Busername%5D=".$username."&login%5Bpassword%5D=".$password."&page_referrer=login&".$img1."=".$img2;
// HERE I SEND THE VARIABLES
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookies_file);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookies_file);
curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postdata);
$content = curl_exec($ch);
// IF LOGGED HERE I'LL MAKE THE CALL
curl_close($ch);
echo $content;
exit;
?>
Any ideas to help me?
This is a test account so feel free to use my login and password if you want to have a look on this and help me!
Thank you a lot in advance.
Ok you need to forward the cookie/session after login,
you need to first extract the cookie from Header after login like following
// HERE I GET THE TOKEN
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
.... ....
.... ....
$content = curl_exec($ch);
preg_match('/^Set-Cookie:\s*([^;]*)/mi', $content, $m);
parse_str($m[1], $cookies);
$cookie = $cookies['NAMEOFCOOKIEUNEEDHERE'];
After that need to use $cookie variable in curl options like that
// HERE I SEND THE VARIABLES
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_COOKIE, 'NAMEOFCOOKIEUNEEDHERE='.$cookie);
I hope your problem will be resolved.
Thanks
Moin

How to solve the irrecognizable characters while curl this page

My code is here, the problem is different with my other question, here I meet the problem is after I successfully curl down the page, but all I see the characters are irrecognizable, eg. "��Ƶ�̳̣�2012�棩", how to make them appear normally?
$cookie_file = tempnam('./temp','cookie');
$login_url = 'http://bbs.php100.com/login.php';
$post_fields = 'cktime=3600&step=2&pwuser=username&pwpwd=password';
$ch = curl_init($login_url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_fields);
curl_exec($ch);
curl_close($ch);
$url = 'http://bbs.php100.com/index.php';//or specific page
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 0);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file);
$contents = curl_exec($ch);
//preg_match("",$contents,$arr);
//echo $arr[1];
curl_close($ch);
You need to save the contents of the page to variable and convert encoding.
$url = 'http://bbs.php100.com/index.php';//or specific page
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HEADER, 0);
// Note "1"! It is needed for curl_exec() to return contents of the page
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file);
$contents = curl_exec($ch);
$contents = iconv('gbk','utf8',$contents);
echo $contents;
If you are using not UTF-8 encoding, set second parameter of iconv according to your needs.

How to use CURL instead of file_get_contents?

I use file_get_contents function to get and show external links on my specific page.
In my local file everything is okay, but my server doesn't support the file_get_contents function, so I tried to use cURL with the below code:
function file_get_contents_curl($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
echo file_get_contents_curl('http://google.com');
But it returns a blank page. What is wrong?
try this:
function file_get_contents_curl($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
This should work
function curl_load($url){
curl_setopt($ch=curl_init(), CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$response = curl_exec($ch);
curl_close($ch);
return $response;
}
$url = "http://www.google.com";
echo curl_load($url);
I encountered such a problem accessing Google Drive content via the direct link.
After calling file_get_contents returned 302 Moved temporarily
//Any google url. This example is fake for Google Drive direct link.
$url = "https://drive.google.com/uc?id=0BxQKKJYjuNElbFBNUlBndmVHHAj";
$html = file_get_contents($url);
echo $html; //print none because error 302.
With the code below it worked again:
//Any google url. This example is fake for Google Drive direct link.
$url = "https://drive.google.com/uc?id=0BxQKKJYjuNElbFBNUlBndmVHHAj";
$ch = curl_init();
curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 3);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
$html = curl_exec($ch);
curl_close($ch);
echo $html;
I tested it today, 03/19/2018
//You can try this . It should work fine.
function curl_tt($url){
$ch = curl_init();
curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 3);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
echo curl_tt("https://google.com");

curl fetching content of redirected webpage

I try to login in a site which has a combined login.
The redirect to this site works fine and the redirect back as well ... but only if I stop the PHP programm with die(). Unfortunately it is not stored in $return, which I would need to proceed.
Any ideas, what I am doing wrong? Much appreciated!
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_POST, TRUE);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_UNRESTRICTED_AUTH, TRUE);
curl_setopt($ch, CURLOPT_POSTREDIR, TRUE);
curl_setopt($ch, CURLOPT_URL, URI_LOGIN);
$content = curl_exec($ch);
// PREPARING THE LOGIN
$referer = URI_LOGIN . '?ReturnUrl=' . urlencode('/returnUrl/');
curl_setopt($ch, CURLOPT_URL, $referer);
curl_setopt($ch, CURLOPT_REFERER, $referer);
curl_setopt($ch, CURLOPT_COOKIEJAR, COOKIE);
$post = ...;
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($post) );
curl_setopt($ch, CURLOPT_USERAGENT, '...');
curl_setopt($ch, CURLINFO_HEADER_OUT, TRUE);
$content = curl_exec($ch);
// LOGIN
curl_setopt($ch, CURLOPT_URL, COMBINED_LOGIN);
$post = ...;
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($post) );
$return = curl_exec($ch);
// ...
?>
There was a problem with javascript. It stucked on an 'in-between'-page and I had to make another curl request to get to the right page.
Now it is working fine!

Categories