My scraping code is not working , I used php cURL , But not returning page content . So please help me , how can I get all content .
function getSslPage($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_REFERER, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$result = curl_exec($ch);
curl_close($ch);
return $result;
}
$url = "https://www.woolworths.com.au/Shop/SearchProducts?search=Fresh%20Apple";
$ch = getSslPage($url);
echo $ch;
Add this right after your curl_exec() call and copy/paste here the output:
if (curl_errno($ch)) {
throw new Exception(curl_error($ch));
}
Related
I can't get the content of this URL (I get a blank page): https://www.euronews.com/api/watchlive.json
I have always used this function and never had problems:
function file_get_contents_curl($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 3);
curl_setopt($ch, CURLOPT_TIMEOUT, 3);
curl_setopt($ch, CURLOPT_REFERER, $referer);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
curl_setopt($ch, CURLOPT_HEADER, 5);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_USERAGENT, $useragent);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$data = json_decode(file_get_contents_curl("https://www.euronews.com/api/watchlive.json"), true); $url = $data['url']; echo $url;
Any idea?
EDIT: Unknown problem caused (possibly) by cURL module in Raspbian (Raspberry)
I think you need to echo the function if you want to see the content. I tried this on my end and I got the content. althought I have some undefined variable error because of $referrer, $header, $useragent and $cookie. but it runs and get the result. here is how I call the function
echo file_get_contents_curl("https://www.euronews.com/api/watchlive.json");
You're getting more than you expect in your function response, so the json_decode is failing. You need to set CURLOPT_HEADER to false instead of 5.
curl_setopt($ch, CURLOPT_HEADER, FALSE);
You can verify what you're getting by doing
$response = file_get_contents_curl("https://www.euronews.com/api/watchlive.json");
var_dump($response);
$data = json_decode($response, true);
$url = $data['url'];
echo $url;
not getting response using curl. I have put all solution but did not get response.
$ch = curl_init();
$header = array('api_key:xxxxxxxxxxxxxxxx','Content-Type: application/json');
curl_setopt($curl, CURLOPT_RETURNTRANSFER, 0);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, $method);
if($postdata!=""){
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postdata);
}
$response = curl_exec($ch);
curl_close($ch);
$result = json_decode($response,true);
print_r($result); // not display result
this example not displaying any result but it send to specific place.
Add this in your code and then check.
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
For your used-case you have to change your code.
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,"https://api.nhs.uk/organisations/FNM60");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$response = curl_exec($ch);
curl_close($ch);
$result = json_decode($response,true);
print_r($result);
here we are getting response without echo on the top of the page but we don't need to that type of out put so we have disabled using span tag.
echo "<span style='display:none;'>"; //to hide the curl response
$ch = curl_init();
$header = array('api_key:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx','Content-Type: application/json');
curl_setopt($curl, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, $method);
if($postdata!=""){
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postdata);
}
$response = curl_exec($ch);
$info = curl_getinfo($ch);
$responseBody = json_decode($response);
print_r($responseBody);
echo $response; // display 1(one)
curl_close($ch);
echo "</span>";
and result is : 1
check full code
I tried accessing a website using curl and it just output Object moved to here. I used curl_error but doesn't show any errors.
$url = "https:somewebsite.com";
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE); // Accepts all CAs
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
//curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec($ch);
if(curl_error($ch))
{
echo 'error:' . curl_error($ch);
}
curl_close($ch);
echo $output;
?>
I run below code and it worked, but I see blank screen when I try with this URL : https://www.reddit.com/r/funny/comments/3aguyp/i_told_my_boyfriend_the_new_puppy_wasnt_allowed/
<?php include('simple_html_dom.php');
$html = getSslPage('https://www.reddit.com/r/pics/comments/3a53qw/grandmother_mother_and_daughter/');
function getSslPage($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_REFERER, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$result = curl_exec($ch);
curl_close($ch);
return $result;
}
$html = str_get_html($html);
echo $html;
The 2 page have the same DOM structure.
I use file_get_contents function to get and show external links on my specific page.
In my local file everything is okay, but my server doesn't support the file_get_contents function, so I tried to use cURL with the below code:
function file_get_contents_curl($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
echo file_get_contents_curl('http://google.com');
But it returns a blank page. What is wrong?
try this:
function file_get_contents_curl($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
This should work
function curl_load($url){
curl_setopt($ch=curl_init(), CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$response = curl_exec($ch);
curl_close($ch);
return $response;
}
$url = "http://www.google.com";
echo curl_load($url);
I encountered such a problem accessing Google Drive content via the direct link.
After calling file_get_contents returned 302 Moved temporarily
//Any google url. This example is fake for Google Drive direct link.
$url = "https://drive.google.com/uc?id=0BxQKKJYjuNElbFBNUlBndmVHHAj";
$html = file_get_contents($url);
echo $html; //print none because error 302.
With the code below it worked again:
//Any google url. This example is fake for Google Drive direct link.
$url = "https://drive.google.com/uc?id=0BxQKKJYjuNElbFBNUlBndmVHHAj";
$ch = curl_init();
curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 3);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
$html = curl_exec($ch);
curl_close($ch);
echo $html;
I tested it today, 03/19/2018
//You can try this . It should work fine.
function curl_tt($url){
$ch = curl_init();
curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 3);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
echo curl_tt("https://google.com");