PHP curl extract download link - php

I'm trying to extract a download link from this website using cURL.
http://abelhas.pt/Asantino/TUTORIAIS/SONY+VEGAS/ZLICED+TRAILER/ZLICED+TRAILER/ZLICED+TRAILER+PRE-RENDERD+FX,36948354.mp4(video)
Opening the link gives u a download button, pressing it will then call a ajax script that returns some J SON data from following URL
http://abelhas.pt/action/License/Download
This URL expects 2 parameters ( fileId(int) & __RequestVerificationToken(str)) as POST parameters and this header needs to be send aswell (X-Requested-With: XMLHttpRequest)
This is the script I'm using to login & fetch the __RequestVerificationToken && the fileId from the Item page
$options = array(
'http'=>array(
'method'=>"GET",
'header'=>"Accept-language: en\r\n" .
"Cookie:".$cookie.";\r\n" . // check function.stream-context-create on php.net
"User-Agent: Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B334b Safari/531.21.102011-10-16 20:23:10\r\n"
)
);
$context = stream_context_create($options);
$html = file_get_html($url,false,$context);
$token = $html->find('input[name=__RequestVerificationToken]',0)->value;
$id = $html->find('a.fileCopyAction',0)->rel;
return array(
'id' => $id,
'token' => $token
);
Followed by this code that I use to extract the download link(this does not work)
$res = array();
$options = array(
CURLOPT_RETURNTRANSFER => true, // return web page
CURLOPT_HEADER => false, // do not return headers
CURLOPT_FOLLOWLOCATION => true, // follow redirects
CURLOPT_USERAGENT => "spider", // who am i
CURLOPT_AUTOREFERER => true, // set referer on redirect
CURLOPT_CONNECTTIMEOUT => 120, // timeout on connect
CURLOPT_TIMEOUT => 120, // timeout on response
CURLOPT_MAXREDIRS => 10, // stop after 10 redirects
CURLOPT_POSTFIELDS => 'fileId='.$file_id.'&__RequestVerificationToken='.$token,
CURLOPT_HTTPHEADER => array(
'Accept:*/*',
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Language: en-US,en;q=0.8',
'Cache-Control: max-age=0',
'Connection: keep-alive',
'Content-Type: application/x-www-form-urlencoded',
'Cookie: '.$cookie,
'Host: abelhas.pt',
'Origin: http://abelhas.pt',
'X-Requested-With: XMLHttpRequest',
),
);
$ch = curl_init("http://abelhas.pt/action/License/Download");
curl_setopt_array( $ch, $options );
$content = curl_exec( $ch );
$err = curl_errno( $ch );
$errmsg = curl_error( $ch );
$header = curl_getinfo( $ch );
curl_close( $ch );
$res['content'] = $content;
$res['url'] = $header['url'];
return $res;
This returns J SON data but not what the website returns if u inspect it with chrome/firefox.
So the question remains : How to do this to get the same results as if visiting it urself using a browser?
Thanks in advance.

Related

Php curl with proxy response 502

I am getting 502 as response from a specific site while scrapping, it works with other sites and without proxy works too, can anyone help ?
function doo ( $proxyip, $port, $auth, $url )
{
$options = array(
CURLOPT_RETURNTRANSFER => true,
CURLOPT_HEADER => false,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_ENCODING => "",
CURLOPT_USERAGENT => "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36", // tell yourself i am this
CURLOPT_AUTOREFERER => true,
CURLOPT_CONNECTTIMEOUT => 120,
CURLOPT_TIMEOUT => 120,
CURLOPT_MAXREDIRS => 10,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_PROXY => $proxyip,
CURLOPT_PROXYPORT => $port,
CURLOPT_PROXYUSERPWD => $auth,
CURLOPT_HTTPPROXYTUNNEL => 1,
CURLOPT_PROXYTYPE => 'HTTP'
);
$ch = curl_init( $url );
curl_setopt_array( $ch, $options );
$content = curl_exec( $ch );
$err = curl_errno( $ch );
$errmsg = curl_error( $ch );
$header = curl_getinfo( $ch );
curl_close( $ch );
$header['errno'] = $err;
$header['errmsg'] = $errmsg;
$header = $content;
return $header;
}
Exact Error:
Failure when receiving data from the peer: Received HTTP code 502 from proxy after CONNECT

php Curl pass http cookie

after a user logged into my website a session is started with name "xy-login".
I now want to use cURL to get the content of another webressource which accepts the same session (F.e. xylogin=lqb3a4fma4ggf4rr49ktfggel).
With wget I would do something like this:
$opts = array('http' => array('header'=> 'Cookie: ' . $_SERVER['HTTP_COOKIE']."\r\n"));
$contents = file_get_contents('http://localhst/xy/print', false, $context);
But how would I need to change this function to pass the session (as with wget)?
https://stackoverflow.com/a/14953910/1474777
function get_web_page( $url )
{
$user_agent='Mozilla/5.0 (Windows NT 6.1; rv:8.0) Gecko/20100101 Firefox/8.0';
$options = array(
CURLOPT_CUSTOMREQUEST =>"GET", //set request type post or get
CURLOPT_POST =>false, //set to GET
CURLOPT_USERAGENT => $user_agent, //set user agent
CURLOPT_COOKIEFILE =>"cookie.txt", //set cookie file
CURLOPT_COOKIEJAR =>"cookie.txt", //set cookie jar
CURLOPT_RETURNTRANSFER => true, // return web page
CURLOPT_HEADER => false, // don't return headers
CURLOPT_FOLLOWLOCATION => true, // follow redirects
CURLOPT_ENCODING => "", // handle all encodings
CURLOPT_AUTOREFERER => true, // set referer on redirect
CURLOPT_CONNECTTIMEOUT => 120, // timeout on connect
CURLOPT_TIMEOUT => 120, // timeout on response
CURLOPT_MAXREDIRS => 10, // stop after 10 redirects
);
$ch = curl_init( $url );
curl_setopt_array( $ch, $options );
$content = curl_exec( $ch );
$err = curl_errno( $ch );
$errmsg = curl_error( $ch );
$header = curl_getinfo( $ch );
curl_close( $ch );
$header['errno'] = $err;
$header['errmsg'] = $errmsg;
$header['content'] = $content;
return $header;
}

To develop custom curl functionality in drupal7

I am newbie to drupal and trying to develop a custom module for curl functionality of php which will get the content of page and will display the whole content
as it is.I tried but it is not displaying the content below is the code that I have tried.
function mycurl_menu() {
$items['mycurl'] = array(
'title' => 'My curl demo',
'page callback' => 'mycurl_curl',
'access callback' => TRUE,
'type' => MENU_CALLBACK,
);
return $items;
}
function mycurl_curl() {
$user_agent = "Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.65 Safari/537.36";
$postdata ="name=amit&pass=amit";
$options = array(
CURLOPT_CUSTOMREQUEST =>"GET", //set request type post or get
CURLOPT_POST =>false, //set to GET
CURLOPT_USERAGENT => $user_agent, //set user agent
CURLOPT_COOKIEFILE =>"cookie.txt", //set cookie file
CURLOPT_COOKIEJAR =>"cookie.txt", //set cookie jar
CURLOPT_RETURNTRANSFER => true, // return web page
CURLOPT_HEADER => false, // don't return headers
CURLOPT_FOLLOWLOCATION => true, // follow redirects
CURLOPT_ENCODING => "", // handle all encodings
CURLOPT_AUTOREFERER => true, // set referer on redirect
CURLOPT_CONNECTTIMEOUT => 120, // timeout on connect
CURLOPT_TIMEOUT => 120, // timeout on response
CURLOPT_MAXREDIRS => 10, // stop after 10 redirects
);
$url = "http://localhost/drupal_commerce/product";
$ch = curl_init( $url );
curl_setopt_array( $ch, $options );
$content = curl_exec( $ch );
$err = curl_errno( $ch );
$errmsg = curl_error( $ch );
$header = curl_getinfo( $ch );
curl_close( $ch );
$header['errno'] = $err;
$header['errmsg'] = $errmsg;
$header['content'] = $content;
return $header;
}
You should return $header['content'] since the menu callback is no supposed to return an array, and check the permissions of the page you are fetching with curl so that is it available.

PHP: Cannot retrieve correct ajax response using curl or file_get_contents

Start page
Frame page
End page
I can't figure out what I'm missing. I've tried mimicking the firefox request headers, but it doesn't work.
Also, the frame page reaches the end page using a javascript ajax request. It posts the data to $post_to_link (see code below) then navigates to the expected result (not my current result), where the megaupload link is located.
Expected output:
/membersonly/components/com_iceplayer/GMorBMlet.php?url=http%3A%2F%2Fwww.megaupload.com%2F%3Fd%3DVNICBFWL&
Current output:
file_get_contents outputs 3
curl outputs error 403 forbidden access
Here is my code:
// call it like so...
echo GetHosterLink( 1148, 252636, '', '37fn8Oklq', 15, -75 );
// $s is incremented every second you are 'visiting' the referer page
// $m decreases below zero when you move your mouse `down` on the start page
function GetHosterLink( $id, $link_id, $cap, $sec, $s, $m )
{
$link_page = str_replace( '[ID]', $id, 'http://www.icefilms.info/membersonly/components/com_iceplayer/video.php?vid=[ID]' );
$post = "id={$link_id}&s={$s}&iqs=&url=&m={$m}&cap=&sec={$sec}&t={$id}";
$header = implode( "\r\n", array(
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7",
"Accept-Encoding: gzip, deflate",
"Accept-Language: en-us,en;q=0.5",
"Cache-Control: no-cache",
"Connection: keep-alive",
"Content-Length: " . strlen( $post ),
"Content-Type: application/x-www-form-urlencoded; charset=UTF-8",
"Host: www.icefilms.info",
"Pragma: no-cache",
"Referer: http://www.icefilms.info/membersonly/components/com_iceplayer/video.php?h=374&w=631&vid={$id}&img=",
"User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0"
));
$opts = array('http' =>
array(
'method' => 'POST',
'header' => implode("\r\n",
array(
'Content-type: application/x-www-form-urlencoded',
'Content-length: ' . strlen( $post ),
'Referer: ' . $link_page . '&h=374&w=631',
'User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0',
'Host: www.icefilms.info'
)
),
'content' => http_build_query(
array(
'id' => $link_id,
's' => $s,
'sec' => $sec,
't' => $id,
'm' => $m,
'iqs' => '',
'url' => '',
'cap' => ''
)
)
)
);
$context = stream_context_create($opts);
$post_to_link = 'http://www.icefilms.info/membersonly/components/com_iceplayer/video.phpAjaxResp.php';
$get_result = file_get_contents( $post_to_link, false, $context );
$f_result = cURL::DoRequest( $post_to_link, $post, '',
array( array( CURLOPT_HTTPHEADER, $header ) ) );
$f_r = array(
'result' => $f_result,
'get_result' => $get_result,
'get_opts' => $opts,
'get_response' => $http_response_header,
'req_post' => $post,
'req_href' => $post_to_link,
'req_header' => $header
);
return ( $f_r );
}
Here is my curl.php file:
class cURL
{
public static function DoRequest( $url, $post = '',
$cookie_file = '', $variables = array() )
{
$curl = curl_init();
#session_start();
$cookie = ( 'PHPSESSID=' . session_id() . '; path=/' );
#session_write_close();
curl_setopt( $curl, CURLOPT_URL, $url );
curl_setopt( $curl, CURLOPT_RETURNTRANSFER, 1 );
curl_setopt( $curl, CURLOPT_COOKIE, $cookie );
if ( !empty( $cookie_file ) )
{
curl_setopt( $curl, CURLOPT_COOKIEFILE, $cookie_file );
curl_setopt( $curl, CURLOPT_COOKIEJAR, $cookie_file );
}
if ( !empty( $post ) )
{
//curl_setopt( $curl, CURLOPT_POST, 1 );
curl_setopt( $curl, CURLOPT_POSTFIELDS, $post );
}
foreach ( $variables as $var )
curl_setopt( $curl, $var[0], $var[1] );
$result = curl_exec( $curl );
curl_close( $curl );
return ( $result );
}
}
Try visiting initial page (the one's url you have in $link_page) with curl and make sure the file you point to in:
curl_setopt( $curl, CURLOPT_COOKIE, $cookie );
exists and is writeable.
Then request the $post_to_link url with same curl resource.
By visiting the initial page you are getting cookies and making sure there is a valid session for your next request. That also secures the referer you then provide in headers. There are many ways to figure "automatic" requests, and things like checking cookie and if you actually visited "referer"-ed link are quite common.

PHP Curl 302 authentication with cookies

I am trying to learn to use PHP curl and it seemed to go well until I have tried to authenticate to changeip.com. Here is the function I use to make a Curl call:
function request($ch, $url, $params = array())
{
$options = array
(
CURLOPT_URL => $url,
CURLOPT_USERAGENT => 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8',
//CURLOPT_COOKIESESSION => TRUE,
CURLOPT_FOLLOWLOCATION => TRUE,
CURLOPT_HEADER => TRUE,
CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_SSL_VERIFYPEER => FALSE,
CURLOPT_SSL_VERIFYPEER => FALSE,
CURLINFO_HEADER_OUT => TRUE,
CURLOPT_CONNECTTIMEOUT => 30,
CURLOPT_TIMEOUT => 30,
CURLOPT_MAXREDIRS => 30,
CURLOPT_VERBOSE => TRUE,
CURLOPT_COOKIEJAR => __DIR__ . DIRECTORY_SEPARATOR . 'cookies.txt',
CURLOPT_COOKIEFILE => __DIR__ . DIRECTORY_SEPARATOR . 'cookies.txt',
CURLOPT_HTTPHEADER => array
(
'Host: www.changeip.com',
'Pragma:',
'Expect:',
'Keep-alive: 115',
'Connection: keep-alive',
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language: en-us,en;q=0.5',
//'Accept-Encoding: gzip,deflate',
'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7',
'Content-Type: application/x-www-form-urlencoded',
),
);
if(!empty($params['referrer']))
{
$options[CURLOPT_REFERER] = $params['referrer'];
}
if(!empty($params['post']))
{
$options[CURLOPT_POST] = TRUE;
$options[CURLOPT_POSTFIELDS] = $params['post'];
}
curl_setopt_array($ch, $options);
$return = array();
$return['body'] = curl_exec($ch);
$info = curl_getinfo($ch);
//die(var_dump( curl_getinfo($ch, CURLINFO_HEADER_OUT) ));
$return['header'] = http_parse_headers(substr($return['body'], 0, $info['header_size']));
$return['body'] = substr($return['body'], $info['header_size']);
/*if(!empty($return['header']['Location']))
{
$params['referrer'] = $url;
return request($ch, substr($url, 0, strrpos($url, '/')+1) . $return['header']['Location'], $params);
}*/
return $return;
}
And here is the actual call:
// chaneip
$ch = curl_init();
// login
$params = array();
$params['post'] = array
(
'p' => 'aaaaaa2',
'u' => 'aaaaaa2',
);
$params['referrer'] = 'https://www.changeip.com/login.asp';
$return = request($ch, 'https://www.changeip.com/loginverify.asp?', $params);
However, this script does not retrieve valid cookies from changeip.com, i.e., does not authenticate. I have tried to compare Curl sent headers with HTTPLiveHeaders expecting to find any difference but in the end I didn't find anything. Can anyone advice me what is missing to make this work?
Commonly given question:
is cookie.txt 0777? Yes and the script does actually create some sort of cookie:
www.changeip.com FALSE / FALSE 0 ACloginAddrs 6
www.changeip.com FALSE / FALSE 0 ASPSESSIONIDCCSSCQRA DNHKGDICMKHFIJADMAPPMHHC
But it isn't a valid cookie.
$options[CURLOPT_POSTFIELDS] = http_build_query($params['post']);
Fixed the issue.

Categories