I have a problem with cURL. It takes over 40 seconds to fetch a web page.
The function is:
function get_page(){
$url = get_url();
$timeout = 1000;
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, $CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$return_data = curl_exec($ch);
print_r (curl_getinfo($ch));
curl_close($ch);
return $return_data;
}
Also, it seems that $return_data = curl_exec($ch) actually dumps the page.
I managed to solve this problem by changing the DNS to 8.8.8.8
Related
I have a curl set up like this:
function file_get_contents_curl($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_TIMEOUT, 400);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch,CURLOPT_BINARYTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
$data = curl_exec($ch);
return $data;
}
When I call it with the following:
$html = file_get_contents_curl("https://training.sap.com/service-id/module.php/core/as_login.php?AuthId=sf-sp&ReturnTo=%2Fsuccessfactors-community%2Fpermission-check");
You can see that link has an authentication, and I do not want to spend time on it. I just want that curl session to die after maybe 3 seconds or x seconds, or determine before hand if it is going to be a never ending loop on the curl for one reason such a SSL or password requirement etc.
My scraping code is not working , I used php cURL , But not returning page content . So please help me , how can I get all content .
function getSslPage($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_REFERER, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$result = curl_exec($ch);
curl_close($ch);
return $result;
}
$url = "https://www.woolworths.com.au/Shop/SearchProducts?search=Fresh%20Apple";
$ch = getSslPage($url);
echo $ch;
Add this right after your curl_exec() call and copy/paste here the output:
if (curl_errno($ch)) {
throw new Exception(curl_error($ch));
}
I am trying to use this PHP script to get the shortened link from bit.ly API.. It works fine but my question is there any way to make this script more efficient or take some unnecessary parts out of it. Also my main question is that when I use:
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
I have to use the trim function on $data but when I use:
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
I don't have to do that.. Why is it causing a line break after the link when I use 1 instead of true?
<?php
function get_bitly_short_url($url, $format = 'txt')
{
$connectURL = 'http://api.j.mp/v3/shorten?login=(MY USERNAME)&apiKey=(MY API)&uri=' . urlencode($url) . '&format=' . $format;
return curl_get_result($connectURL);
}
function curl_get_result($url)
{
$ch = curl_init();
$timeout = 5;
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$data = curl_exec($ch);
curl_close($ch);
return trim($data);
}
$short_url = get_bitly_short_url('http://google.com');
?>
1 !== true in common case, but in your case, should be no difference,
double check all other things...
I make simple test for you:
<?php
$url = 'http://ziptasticapi.com/92530';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
$data = curl_exec($ch);
curl_close($ch);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
$data1 = curl_exec($ch);
curl_close($ch);
var_dump($data);
echo "\n";
var_dump($data1);
Results:
string(52) "{"country":"US","state":"CA","city":"LAKE ELSINORE"}"
string(52) "{"country":"US","state":"CA","city":"LAKE ELSINORE"}"
So, bugs in php happens, but no bugs this time
No difference, check other parts of your system
Luck!
Hi I want to be able to do the following:
<?php
function get_data($url)
{
$ch = curl_init();
$timeout = 5;
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,$timeout);
curl_setopt($ch,CURLOPT_SSL_VERIFYPEER, false);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$data = get_data('https://torcache.net/torrent/7975CDEEDCEC6092729DAEAE302CB9BD7D633B0B.torrent');
?>
However it seems that torcache is returning a html page and then the torrent is seved a few seconds after, is there anyway for curl to get the actual torrent? At the minute $data just contains the html page torcache returns?
Attempted to set referer as:
curl_setopt($ch, CURLOPT_REFERER, 'https://torcache.net/torrent/7975CDEEDCEC6092729DAEAE302CB9BD7D633B0B.torrent');
But not working, I get this response:
<head><title>302 Found</title></head>
<body bgcolor="white">
<center><h1>302 Found</h1></center>
<hr><center>nginx/1.2.0</center>
</body>
</html>
Thanks
SOLVED:
function get_data($url)
{
$ch = curl_init();
$timeout = 5;
curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch,CURLOPT_CONNECTTIMEOUT,$timeout);
curl_setopt($ch,CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch,CURLOPT_REFERER, 'https://torcache.net/');
curl_setopt($ch,CURLOPT_ENCODING,"gzip");
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
Added "curl_setopt($ch,CURLOPT_ENCODING,"gzip");" this also as the data was gzipped!
Figure out how they check whether to server the HTML page or the torrent file. My guess is the HTTP_REFERER. Spoof it.
If $_GET['tor'] contains the info_hash from torrentz.eu site, this should do the trick.
$ch = curl_init('http://torcache.net/torrent/'.strtoupper($_GET['tor']).'.torrent');
curl_setopt($ch, CURLOPT_POSTFIELDS, null);
curl_setopt($ch, CURLOPT_POST, FALSE);
curl_setopt($ch, CURLOPT_HTTPGET, TRUE);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/x-bittorrent','Referer: http://torcache.net/torrent/'.strtoupper($_GET['tor']).'.torrent'));
curl_setopt($ch, CURLOPT_REFERER, 'http://torcache.net/torrent/'.strtoupper($_GET['tor']).'.torrent');
curl_setopt($ch, CURLOPT_ENCODING,"gzip");
$result = curl_exec($ch);
echo $result;
I use file_get_contents function to get and show external links on my specific page.
In my local file everything is okay, but my server doesn't support the file_get_contents function, so I tried to use cURL with the below code:
function file_get_contents_curl($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
echo file_get_contents_curl('http://google.com');
But it returns a blank page. What is wrong?
try this:
function file_get_contents_curl($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
This should work
function curl_load($url){
curl_setopt($ch=curl_init(), CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$response = curl_exec($ch);
curl_close($ch);
return $response;
}
$url = "http://www.google.com";
echo curl_load($url);
I encountered such a problem accessing Google Drive content via the direct link.
After calling file_get_contents returned 302 Moved temporarily
//Any google url. This example is fake for Google Drive direct link.
$url = "https://drive.google.com/uc?id=0BxQKKJYjuNElbFBNUlBndmVHHAj";
$html = file_get_contents($url);
echo $html; //print none because error 302.
With the code below it worked again:
//Any google url. This example is fake for Google Drive direct link.
$url = "https://drive.google.com/uc?id=0BxQKKJYjuNElbFBNUlBndmVHHAj";
$ch = curl_init();
curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 3);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
$html = curl_exec($ch);
curl_close($ch);
echo $html;
I tested it today, 03/19/2018
//You can try this . It should work fine.
function curl_tt($url){
$ch = curl_init();
curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 3);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
echo curl_tt("https://google.com");