I've been using PHP curl to get data I need from remote website. Here is the cURL function I used:
function get_content($adr)
{
$ch = curl_init();
curl_setopt ($ch, CURLOPT_URL, $adr);
curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_setopt ($ch, CURLOPT_HEADER, 0);
ob_start();
curl_exec ($ch);
curl_close ($ch);
$string = ob_get_contents();
ob_end_clean();
return $string;
}
$myrul = "http://remoteurl.com";
$result = get_content($myrul);
But how do I get the headers for the response?
If my comment above is correct, change:
curl_setopt($ch, CURLOPT_HEADER, 0);
to:
curl_setopt($ch, CURLOPT_HEADER, 1);
and parse the returned headers out however you see fit. Note that just changing the above in your function will return both headers and content, so if you want only headers returned:
function http_head_curl($url,$timeout=10)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_TIMEOUT, $timeout); // in seconds
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_NOBODY, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$res = curl_exec($ch);
if ($res === false) {
throw new RuntimeException("cURL exception: ".curl_errno($ch).": ".curl_error($ch));
}
return trim($res);
}
Related
I am trying to find where I'll be redirected at. So I tried to functions for this, but none of those are working properly.
the links is here. when you try to enter, you will be redirected:
https://lions-mansion.jp/MA141070/
so I tried use cURL,
function redirect1($url) {
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT ,0);
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
$data = curl_exec($ch);
$data = curl_getinfo($ch,CURLINFO_EFFECTIVE_URL );
curl_close($ch);
return $data;
}
and also this:
function redirect($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$result = curl_exec($ch);
if (preg_match('~Location: (.*)~i', $result, $match)) {
$location = trim($match[1]);
}
return $result;
}
But I couldn't find the redirected url.
this page does not use a redirect-scheme that libcurl understands (it uses a html <meta http-equiv="REFRESH"-redirect, unsupported by libcurl), so libcurl can neither tell you where it is being redirected, nor can libcurl auto-follow the redirect (because libcurl does not understand it)
you need to parse out the redirect url yourself from the HTML, eg
function redirect1($url) {
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_BINARYTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT ,0);
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
$data = curl_exec($ch);
$domd=#DOMDocument::loadHTML($data);
$xp=new DOMXPath($domd);
// <META http-equiv="REFRESH" content="0;URL=http://sumai.tokyu-land.co.jp/branz/roppongi4/?iad=daikyo" />
$location=$xp->query("//meta[#http-equiv='REFRESH']")->item(0)->getAttribute("content");
// 0;URL=http://sumai.tokyu-land.co.jp/branz/roppongi4/?iad=daikyo
$location=substr($location,stripos($location,'URL=')+4);
curl_close($ch);
return $location;
}
var_dump(redirect1('https://lions-mansion.jp/MA141070/'));
output:
C:\projects\misc>php re.php
string(57) "http://sumai.tokyu-land.co.jp/branz/roppongi4/?iad=daikyo"
If keep you CURLOPT_RETURNTRANSFER to true, after executing the CURL command you can use this function call to get the redirect of effective URL:
$finalUrl = curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
I am new to PHP and trying to use cURL function to fetch the view of a page, but I am unable to fetch for particularly www.zomato.com, while all other websites are working fine using this code.
<?php
set_time_limit(0);
function getSslPage($url) {
$ch = curl_init();
//curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, TRUE);
curl_setopt($ch, CURLOPT_CAINFO, $_SERVER['DOCUMENT_ROOT']."/mypro/cacert.pem");
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_REFERER, $url);
//curl_setopt($ch, CURLOPT_PORT, 44455);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12');
$result = curl_exec($ch);
echo curl_error($ch);
echo curl_errno($ch);
curl_close($ch);
return $result;
}
ECHO getcwd();
$a=getSslPage("https://www.zomato.com/ncr/open-house-connaught-place-new-delhi/");
echo ($a);
?>
I get error no.- 056 when trying to use this code.
I have to download large videos with curl. Cause the site where I download from put 403 blocks, And I can pass thru with only Curl. Thats my php curl function code. I need this with the download but it shouldnt eat up my memory..
function fakeip()
{
//return long2ip( mt_rand(0, 65537) * mt_rand(0, 65535) );
return "127.0.0.1";
}
function curlx($feed,$coo=null,$ref=null)
{
$ch = curl_init();
$timeout = 0;
curl_setopt ($ch, CURLOPT_URL, $feed);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, array("REMOTE_ADDR: ".fakeip(),"X-Client-IP: ".fakeip(),"Client-IP: ".fakeip(),"HTTP_X_FORWARDED_FOR: ".fakeip(),"X-Forwarded-For: ".fakeip()));
curl_setopt($ch,CURLOPT_USERAGENT,'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:40.0) Gecko/20100101 Firefox/40.0');
//curl_setopt($ch, CURLOPT_INTERFACE, "88.150.225.55");
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
if(!empty($coo))
{
curl_setopt($ch, CURLOPT_COOKIEFILE, $coo);
curl_setopt($ch, CURLOPT_COOKIEJAR, $coo);
}
if(empty($ref))
{
curl_setopt($ch, CURLOPT_REFERER,$feed);
}
else
{
curl_setopt($ch, CURLOPT_REFERER,$ref);
}
curl_setopt ($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$veri= curl_exec($ch);
curl_close($ch);
return str_replace(array("\n", "\t", "\r",), null, $veri);
}
Here is my cUrl code,
function curl_cookieset() {
$fp = fopen("cookie.txt", "w");
fclose($fp);
/*visit the homepage to set the cookie properly */
$ch = curl_init ("http://www.autoscout24.de/ListGN.aspx?vis=1&state=A&atype=C&cy=D&page=1&results=20&ustate=N,U&sort=price&rfde=True&custtype=P&zipc=D");
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.001 (windows; U; NT4.0; en-US; rv:1.0) Gecko/25250101");
curl_setopt($ch, CURLOPT_HEADER, 0);
$output = curl_exec ($ch);
}
function curl_download($Url ){
if (!function_exists('curl_init')){
die('Sorry cURL is not installed!');
}
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $Url);
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.001 (windows; U; NT4.0; en-US; rv:1.0) Gecko/25250101');
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$output = curl_exec($ch);
curl_close($ch);
return $output;
}
And this is how I use it,
curl_cookieset();
while(true) {
//some other process
$result = curl_download($url);
//some other process
}
This works expectedly on localhost, but when I try to run on an external server, It doesn't set cookie and I don't get the same result.
Note: It creates cookie.txt
cookie.txt chmod = 666
What should be the problem?
I am trying to use curl with a proxy. I have the following code:
function getPage($proxy, $url, $referer, $agent, $header, $timeout) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_HEADER, $header);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_PROXY, $proxy);
curl_setopt($ch, CURLOPT_HTTPPROXYTUNNEL, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_REFERER, $referer);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
$result['EXE'] = curl_exec($ch);
$result['INF'] = curl_getinfo($ch);
$result['ERR'] = curl_error($ch);
curl_close($ch);
return $result;
}
$result = getPage(
'75.125.147.82:3128', // use valid proxy
'http://www.google.com/',
'http://www.google.com/',
'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.8) Gecko/2009032609 Firefox/3.0.8',
1,
5);
if (empty($result['ERR'])) {
echo $results['EXE'];
} else {
echo $result['ERR'];
}
The output of this script is 'couldn't connect to host'
Anyone know whats wrong?
You might check what the status code is of where you're connecting to:
$status = curl_getinfo($http, CURLINFO_HTTP_CODE);
Might give you an idea of what is going on.
well
delete this code
curl_setopt($ch,CURLOPT_HTTPPROXYTUNNEL, 1);
it works for me,I'm testing this code as well
good luck