PHP Curl 302 authentication with cookies - php

I am trying to learn to use PHP curl and it seemed to go well until I have tried to authenticate to changeip.com. Here is the function I use to make a Curl call:
function request($ch, $url, $params = array())
{
$options = array
(
CURLOPT_URL => $url,
CURLOPT_USERAGENT => 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8',
//CURLOPT_COOKIESESSION => TRUE,
CURLOPT_FOLLOWLOCATION => TRUE,
CURLOPT_HEADER => TRUE,
CURLOPT_RETURNTRANSFER => TRUE,
CURLOPT_SSL_VERIFYPEER => FALSE,
CURLOPT_SSL_VERIFYPEER => FALSE,
CURLINFO_HEADER_OUT => TRUE,
CURLOPT_CONNECTTIMEOUT => 30,
CURLOPT_TIMEOUT => 30,
CURLOPT_MAXREDIRS => 30,
CURLOPT_VERBOSE => TRUE,
CURLOPT_COOKIEJAR => __DIR__ . DIRECTORY_SEPARATOR . 'cookies.txt',
CURLOPT_COOKIEFILE => __DIR__ . DIRECTORY_SEPARATOR . 'cookies.txt',
CURLOPT_HTTPHEADER => array
(
'Host: www.changeip.com',
'Pragma:',
'Expect:',
'Keep-alive: 115',
'Connection: keep-alive',
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Language: en-us,en;q=0.5',
//'Accept-Encoding: gzip,deflate',
'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7',
'Content-Type: application/x-www-form-urlencoded',
),
);
if(!empty($params['referrer']))
{
$options[CURLOPT_REFERER] = $params['referrer'];
}
if(!empty($params['post']))
{
$options[CURLOPT_POST] = TRUE;
$options[CURLOPT_POSTFIELDS] = $params['post'];
}
curl_setopt_array($ch, $options);
$return = array();
$return['body'] = curl_exec($ch);
$info = curl_getinfo($ch);
//die(var_dump( curl_getinfo($ch, CURLINFO_HEADER_OUT) ));
$return['header'] = http_parse_headers(substr($return['body'], 0, $info['header_size']));
$return['body'] = substr($return['body'], $info['header_size']);
/*if(!empty($return['header']['Location']))
{
$params['referrer'] = $url;
return request($ch, substr($url, 0, strrpos($url, '/')+1) . $return['header']['Location'], $params);
}*/
return $return;
}
And here is the actual call:
// chaneip
$ch = curl_init();
// login
$params = array();
$params['post'] = array
(
'p' => 'aaaaaa2',
'u' => 'aaaaaa2',
);
$params['referrer'] = 'https://www.changeip.com/login.asp';
$return = request($ch, 'https://www.changeip.com/loginverify.asp?', $params);
However, this script does not retrieve valid cookies from changeip.com, i.e., does not authenticate. I have tried to compare Curl sent headers with HTTPLiveHeaders expecting to find any difference but in the end I didn't find anything. Can anyone advice me what is missing to make this work?
Commonly given question:
is cookie.txt 0777? Yes and the script does actually create some sort of cookie:
www.changeip.com FALSE / FALSE 0 ACloginAddrs 6
www.changeip.com FALSE / FALSE 0 ASPSESSIONIDCCSSCQRA DNHKGDICMKHFIJADMAPPMHHC
But it isn't a valid cookie.

$options[CURLOPT_POSTFIELDS] = http_build_query($params['post']);
Fixed the issue.

Related

getting a bad request response when doing a curl call

I am receiving a bad request from the server when I make a curl request. Not sure where I went wrong, but it looks like that some of the CURLOPT options are incorrect. Do you have any idea what needs to be changed?
This is the code:
$url = "https://order.pizzahut.com/site/ajax/find_nearby";
$ch = curl_init($url);
$useragent = 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3'; // Setting useragent of a popular browser
$header = array("Accept: application/json, text/plain, */*");
$options = [
CURLOPT_USERAGENT => $useragent,
CURLOPT_COOKIESESSION => true,
CURLOPT_COOKIEJAR => "cookie.txt",
CURLOPT_REFERER =>"https://www.google.com",
CURLOPT_CUSTOMREQUEST=>"POST",
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTPHEADER =>$header
];
curl_setopt_array($ch, $options);
loadPage("AK", "ANCHORAGE",$ch);
function loadPage($abbreviation,$city,$ch){
global $conn;
$city = strtolower($city);
$near = "$city,"."$abbreviation";
echo $near;
$postData = [
'occasion' => "C",
'near' => "$near",
'zip' => "$near",
'address' => "",
'dine_in' => false,
'store_limit' =>"10",
'locator' =>true
];
$payload = json_encode($postData);
# Convert the post data array to URL encoded string
// $postDataStr = http_build_query($postData);
echo $payload;
$options[CURLOPT_POSTFIELDS] = $payload;
$html = curl_exec($ch);
echo $html;
}
yes you are right, its post request only issue with header it requires valid Cookie in header. I just copied Cookie from browser and paste in header and its works. So the solution is:
loadPage("AK", "ANCHORAGE");
function loadPage($abbreviation,$city){
$city = strtolower($city);
$near = "$city,"."$abbreviation";
$postData = [
'occasion' => "C",
'near' => "$near",
'zip' => "$near",
'address' => "",
'dine_in' => false,
'store_limit' =>"10",
'locator' =>true
];
$url = "https://order.pizzahut.com/site/ajax/find_nearby";
$ch = curl_init($url);
$useragent = 'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3'; // Setting useragent of a popular browser
$header = array(
"Accept: application/json, text/plain, */*",
"Cookie: failover_checked=1; PHPpoolSSL=!4maklee1+k5+tBLWWPR8rI/3M2NcIXVsgM+5XMIVRK/fJJleEnP4hyVoopcRquvCAs/7CbLq/Vfd0cc=; TS011d6839=01166f2bd0e693828ed3fb37eef086402c5626aca6908460b9722b154cb51f244b1dd882278db1bb1f9ac97a3f2d3617b12e7c7b5c; www-origin=yeti-HR-HARYANA-; www-origin-reset=1; exp_last_visit=1143192473; optimizelyEndUserId=oeu1458555784471r0.9519811202417545; _gat_UA-34361514-1=1; QOSESSID=6ohua5ugd55mjupf0p6k32uia6; _dc_gtm_UA-34361514-1=1; exp_last_activity=1458560242; exp_tracker=a%3A5%3A%7Bi%3A0%3Bs%3A12%3A%22site%2Flocator%22%3Bi%3A1%3Bs%3A4%3A%22home%22%3Bi%3A2%3Bs%3A5%3A%22index%22%3Bi%3A3%3Bs%3A23%3A%22site%2Fajax%2Fclear_session%22%3Bi%3A4%3Bs%3A38%3A%22site%2Fajax%2Fcoupons%2FremoveNationalCoupon%22%3B%7D; TS01bded03=01166f2bd034b629d99e25089af83586e4bf2347f7b1f70b91972e835cfd37e486120c510cfabe2bd9856390632a8276d8250ae05869b06cc1d6a7bc77ee826c8b208bc7361686c0b1cbd41a8530948d8fa923b0b3; TS0118ed69=01166f2bd0c9a8c11fd75a87aad59a90e93a811cbea110792544f73601f99f85ef0de952c0bb1c1706650ace601420df60560355a33fa2ecea9f8f89e1fdd807c0f5717ab8fe2b84c6305f1add271e312787310fa30c484c0174c988835bcea2ced9db5fcf; optimizelySegments=%7B%22209642986%22%3A%22referral%22%2C%22209669694%22%3A%22gc%22%2C%22209692476%22%3A%22false%22%2C%22209740042%22%3A%22none%22%2C%22307825635%22%3A%22true%22%7D; optimizelyBuckets=%7B%7D; _ga=GA1.2.700288496.1458555787; optimizelyPendingLogEvents=%5B%5D"
);
$options = [
CURLOPT_USERAGENT => $useragent,
CURLOPT_COOKIESESSION => true,
// CURLOPT_COOKIEJAR => "cookie.txt",
// CURLOPT_REFERER =>"https://www.google.com",
CURLOPT_POST=>true,
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HTTPHEADER =>$header
];
// $payload = json_encode($postData);
$options[CURLOPT_POSTFIELDS] = http_build_query($postData);
curl_setopt_array($ch, $options);
$html = curl_exec($ch);
echo $html;
}
Output
{"success":true,"response":[{"address":"118 E 6th Ave","city":"Anchorage","state":"AK","zip":"99501","country":"US","phone":"(907)337-2323","lat":61.216325,"long":-149.882477,"distance":0.81,"StoreNumber":"011168","OnlineOrdering":true,"WingStreet":true,"Buffet":false,"DineIn":false},{"address":"100 E Benson Blvd","city":"Anchorage","state":"AK","zip":"99503","country":"US","phone":"(907)337-2323","lat":61.193863,"long":-149.882482,"distance":1.82,"StoreNumber":"011169","OnlineOrdering":true,"WingStreet":true,"Buffet":true,"DineIn":true},{"address":"2205 E Tudor Rd #39","city":"Anchorage","state":"AK","zip":"99507","country":"US","phone":"(907)743-9828","lat":61.1811,"long":-149.840628,"distance":2.59,"StoreNumber":"014640","OnlineOrdering":true,"WingStreet":true,"Buffet":false,"DineIn":false},{"address":"341 Boniface Pkwy, Unit G","city":"Anchorage","state":"AK","zip":"99504","country":"US","phone":"(907)337-2323","lat":61.220935,"long":-149.777626,"distance":2.7,"StoreNumber":"008615","OnlineOrdering":true,"WingStreet":true,"Buffet":false,"DineIn":false},{"address":"3727 Spenard Highway","city":"Anchorage","state":"AK","zip":"99517","country":"US","phone":"(907)337-2323","lat":61.186142,"long":-149.914382,"distance":2.86,"StoreNumber":"010576","OnlineOrdering":true,"WingStreet":true,"Buffet":false,"DineIn":false},{"address":"1921 W Dimond Blvd Ste 112","city":"Anchorage","state":"AK","zip":"99515","country":"US","phone":"(907)337-2323","lat":61.139149,"long":-149.915007,"distance":5.74,"StoreNumber":"009150","OnlineOrdering":true,"WingStreet":true,"Buffet":false,"DineIn":false},{"address":"12870 Old Seward Hwy","city":"Anchorage","state":"AK","zip":"99515","country":"US","phone":"(907)337-2323","lat":61.103643,"long":-149.858735,"distance":7.87,"StoreNumber":"010684","OnlineOrdering":true,"WingStreet":true,"Buffet":false,"DineIn":false},{"address":"11740 Old Glenn Hwy","city":"Eagle River","state":"AK","zip":"99577","country":"US","phone":"(907)337-2323","lat":61.326675,"long":-149.571538,"distance":12.15,"StoreNumber":"008181","OnlineOrdering":true,"WingStreet":true,"Buffet":true,"DineIn":true},{"address":"851 E Parks Hwy","city":"Wasilla","state":"AK","zip":"99654","country":"US","phone":"(907)373-6600","lat":61.58137,"long":-149.429086,"distance":28.87,"StoreNumber":"010324","OnlineOrdering":true,"WingStreet":true,"Buffet":true,"DineIn":true}],"store_info":{"011168":{"address":"118 E 6th Ave","city":"Anchorage","state":"AK","state_name":"Alaska","zip":99501,"phone":"(907) 337-2323","status":"","carryout_open":false,"carryout_close":false,"carryout_future_order":true,"delivery_open":"10:00AM","delivery_close":"11:30PM","delivery_future_order":true,"closure_reason":"","onlineStatus":"online","firstTimeDiscount":false,"currentlyOpen":false,"deliveryAvailable":false,"carryoutAvailable":false,"acceptFutureOrders":true,"landmark":null,"futuredelivery":true,"futurecarryout":true,"pizzaEvoTestStore":true,"offlineStatusMsg":null,"showDeliveryTime":"Y","promiseTimeDel":35,"promiseTimeLastUpdateDel":{"date":"2016-03-21 01:47:00","timezone_type":3,"timezone":"America\/Chicago"},"promiseTimeDelStale":true,"showCarryoutTime":"Y","promiseTimeCo":15,"promiseTimeLastUpdateCo":{"date":"2016-03-20 22:49:00","timezone_type":3,"timezone":"America\/Chicago"},"promiseTimeCoStale":true,"promiseTime":35},"011169":{"address":"100 E Benson Blvd","city":"Anchorage","state":"AK","state_name":"Alaska","zip":99503,"phone":"(907) 337-2323","status":"","carryout_open":false,"carryout_close":false,"carryout_future_order":true,"delivery_open":"11:30AM","delivery_close":"11:00PM","delivery_future_order":true,"closure_reason":"","onlineStatus":"online","firstTimeDiscount":false,"currentlyOpen":false,"deliveryAvailable":false,"carryoutAvailable":false,"acceptFutureOrders":true,"landmark":null,"futuredelivery":true,"futurecarryout":true,"pizzaEvoTestStore":true,"offlineStatusMsg":null,"showDeliveryTime":"Y","promiseTimeDel":35,"promiseTimeLastUpdateDel":{"date":"2016-03-08 21:12:00","timezone_type":3,"timezone":"America\/Chicago"},"promiseTimeDelStale":true,"showCarryoutTime":"Y","promiseTimeCo":15,"promiseTimeLastUpdateCo":{"date":"2016-03-20 23:09:00","timezone_type":3,"timezone":"America\/Chicago"},"promiseTimeCoStale":true,"promiseTime":35},"014640":{"address":"2205 E Tudor Rd #39","city":"Anchorage","state":"AK","state_name":"Alaska","zip":99507,"phone":"(907) 743-9828","status":"","carryout_open":false,"carryout_close":false,"carryout_future_order":true,"delivery_open":"10:00AM","delivery_close":"12:00AM","delivery_future_order":true,"closure_reason":"","onlineStatus":"online","firstTimeDiscount":false,"currentlyOpen":false,"deliveryAvailable":false,"carryoutAvailable":false,"acceptFutureOrders":true,"landmark":null,"futuredelivery":true,"futurecarryout":true,"pizzaEvoTestStore":true,"offlineStatusMsg":null,"showDeliveryTime":"Y","promiseTimeDel":35,"promiseTimeLastUpdateDel":{"date":"2016-03-21 03:09:00","timezone_type":3,"timezone":"America\/Chicago"},"promiseTimeDelStale":true,"showCarryoutTime":"Y","promiseTimeCo":15,"promiseTimeLastUpdateCo":{"date":"2016-03-21 03:07:00","timezone_type":3,"timezone":"America\/Chicago"},"promiseTimeCoStale":true,"promiseTime":35},"008615":{"address":"341 Boniface Pkwy, Unit G","city":"Anchorage","state":"AK","state_name":"Alaska","zip":99504,"phone":"(907) 337-2323","status":"","carryout_open":false,"carryout_close":false,"carryout_future_order":true,"delivery_open":"10:00AM","delivery_close":"12:00AM","delivery_future_order":true,"closure_reason":"","onlineStatus":"online","firstTimeDiscount":false,"currentlyOpen":false,"deliveryAvailable":false,"carryoutAvailable":false,"acceptFutureOrders":true,"landmark":null,"futuredelivery":true,"futurecarryout":true,"pizzaEvoTestStore":true,"offlineStatusMsg":null,"showDeliveryTime":"Y","promiseTimeDel":35,"promiseTimeLastUpdateDel":{"date":"2016-03-21 02:40:00","timezone_type":3,"timezone":"America\/Chicago"},"promiseTimeDelStale":true,"showCarryoutTime":"Y","promiseTimeCo":15,"promiseTimeLastUpdateCo":{"date":"2016-03-20 23:40:00","timezone_type":3,"timezone":"America\/Chicago"},"promiseTimeCoStale":true,"promiseTime":35},"010576":{"address":"3727 Spenard Highway","city":"Anchorage","state":"AK","state_name":"Alaska","zip":99517,"phone":"(907) 743-9877","status":"","carryout_open":false,"carryout_close":false,"carryout_future_order":true,"delivery_open":"12:00PM","delivery_close":"12:00AM","delivery_future_order":true,"closure_reason":"","onlineStatus":"online","firstTimeDiscount":false,"currentlyOpen":false,"deliveryAvailable":false,"carryoutAvailable":false,"acceptFutureOrders":true,"landmark":null,"futuredelivery":true,"futurecarryout":true,"pizzaEvoTestStore":true,"offlineStatusMsg":null,"showDeliveryTime":"Y","promiseTimeDel":35,"promiseTimeLastUpdateDel":{"date":"2016-03-21 01:47:00","timezone_type":3,"timezone":"America\/Chicago"},"promiseTimeDelStale":true,"showCarryoutTime":"Y","promiseTimeCo":15,"promiseTimeLastUpdateCo":{"date":"2016-03-20 19:57:00","timezone_type":3,"timezone":"America\/Chicago"},"promiseTimeCoStale":true,"promiseTime":35},"009150":{"address":"1921 W Dimond Blvd Ste 112","city":"Anchorage","state":"AK","state_name":"Alaska","zip":99515,"phone":"(907) 337-2323","status":"","carryout_open":false,"carryout_close":false,"carryout_future_order":true,"delivery_open":"10:00AM","delivery_close":"12:00AM","delivery_future_order":true,"closure_reason":"","onlineStatus":"online","firstTimeDiscount":false,"currentlyOpen":false,"deliveryAvailable":false,"carryoutAvailable":false,"acceptFutureOrders":true,"landmark":null,"futuredelivery":true,"futurecarryout":true,"pizzaEvoTestStore":true,"offlineStatusMsg":null,"showDeliveryTime":"Y","promiseTimeDel":35,"promiseTimeLastUpdateDel":{"date":"2016-03-21 01:48:00","timezone_type":3,"timezone":"America\/Chicago"},"promiseTimeDelStale":true,"showCarryoutTime":"Y","promiseTimeCo":15,"promiseTimeLastUpdateCo":{"date":"2016-03-21 01:34:00","timezone_type":3,"timezone":"America\/Chicago"},"promiseTimeCoStale":true,"promiseTime":35},"010684":{"address":"12870 Old Seward Hwy","city":"Anchorage","state":"AK","state_name":"Alaska","zip":99515,"phone":"(907) 337-2323","status":"","carryout_open":false,"carryout_close":false,"carryout_future_order":true,"delivery_open":"12:00PM","delivery_close":"11:30PM","delivery_future_order":true,"closure_reason":"","onlineStatus":"online","firstTimeDiscount":false,"currentlyOpen":false,"deliveryAvailable":false,"carryoutAvailable":false,"acceptFutureOrders":true,"landmark":null,"futuredelivery":true,"futurecarryout":true,"pizzaEvoTestStore":true,"offlineStatusMsg":null,"showDeliveryTime":"Y","promiseTimeDel":35,"promiseTimeLastUpdateDel":{"date":"2016-03-21 01:53:00","timezone_type":3,"timezone":"America\/Chicago"},"promiseTimeDelStale":true,"showCarryoutTime":"Y","promiseTimeCo":15,"promiseTimeLastUpdateCo":{"date":"2016-03-21 01:53:00","timezone_type":3,"timezone":"America\/Chicago"},"promiseTimeCoStale":true,"promiseTime":35},"008181":{"address":"11740 Old Glenn Hwy","city":"Eagle River","state":"AK","state_name":"Alaska","zip":99577,"phone":"(907) 694-1086","status":"","carryout_open":false,"carryout_close":false,"carryout_future_order":true,"delivery_open":"10:00AM","delivery_close":"12:00AM","delivery_future_order":true,"closure_reason":"","onlineStatus":"online","firstTimeDiscount":false,"currentlyOpen":false,"deliveryAvailable":false,"carryoutAvailable":false,"acceptFutureOrders":true,"landmark":null,"futuredelivery":true,"futurecarryout":true,"pizzaEvoTestStore":true,"offlineStatusMsg":null,"showDeliveryTime":"Y","promiseTimeDel":35,"promiseTimeLastUpdateDel":{"date":"2016-03-21 01:31:00","timezone_type":3,"timezone":"America\/Chicago"},"promiseTimeDelStale":true,"showCarryoutTime":"Y","promiseTimeCo":15,"promiseTimeLastUpdateCo":{"date":"2016-03-20 22:31:00","timezone_type":3,"timezone":"America\/Chicago"},"promiseTimeCoStale":true,"promiseTime":35},"010324":{"address":"851 E Parks Hwy","city":"Wasilla","state":"AK","state_name":"Alaska","zip":99654,"phone":"(907) 373-6600","status":"","carryout_open":false,"carryout_close":false,"carryout_future_order":true,"delivery_open":"12:00PM","delivery_close":"11:00PM","delivery_future_order":true,"closure_reason":"","onlineStatus":"online","firstTimeDiscount":false,"currentlyOpen":false,"deliveryAvailable":false,"carryoutAvailable":false,"acceptFutureOrders":true,"landmark":null,"futuredelivery":true,"futurecarryout":true,"pizzaEvoTestStore":true,"offlineStatusMsg":null,"showDeliveryTime":"Y","promiseTimeDel":35,"promiseTimeLastUpdateDel":{"date":"2016-03-21 01:12:00","timezone_type":3,"timezone":"America\/Chicago"},"promiseTimeDelStale":true,"showCarryoutTime":"Y","promiseTimeCo":15,"promiseTimeLastUpdateCo":{"date":"2016-03-21 01:12:00","timezone_type":3,"timezone":"America\/Chicago"},"promiseTimeCoStale":true,"promiseTime":35}},"closed_event":"011168","dimensions":{"exp":"b327ba0a88f0d185c2648ce42a7543c6","usertype":"Guest","storeid":"national","requesttime":"06:39:25"},"ga_tracking":{"ga_label":"Bing Success","errors":null},"user_location":{"city":"Haryana","state":"HR","state_name":null}}
First you are setting curl options with $options without CURLOPT_POSTFILEDS and then you set $options CURLOPT_POSTFIELDS within loadPage function, but those new options are not passed to curl handler by curl_setopt_array();
use curl_setopt_array just after
$options[CURLOPT_POSTFIELDS] = $payload;

PHP curl extract download link

I'm trying to extract a download link from this website using cURL.
http://abelhas.pt/Asantino/TUTORIAIS/SONY+VEGAS/ZLICED+TRAILER/ZLICED+TRAILER/ZLICED+TRAILER+PRE-RENDERD+FX,36948354.mp4(video)
Opening the link gives u a download button, pressing it will then call a ajax script that returns some J SON data from following URL
http://abelhas.pt/action/License/Download
This URL expects 2 parameters ( fileId(int) & __RequestVerificationToken(str)) as POST parameters and this header needs to be send aswell (X-Requested-With: XMLHttpRequest)
This is the script I'm using to login & fetch the __RequestVerificationToken && the fileId from the Item page
$options = array(
'http'=>array(
'method'=>"GET",
'header'=>"Accept-language: en\r\n" .
"Cookie:".$cookie.";\r\n" . // check function.stream-context-create on php.net
"User-Agent: Mozilla/5.0 (iPad; U; CPU OS 3_2 like Mac OS X; en-us) AppleWebKit/531.21.10 (KHTML, like Gecko) Version/4.0.4 Mobile/7B334b Safari/531.21.102011-10-16 20:23:10\r\n"
)
);
$context = stream_context_create($options);
$html = file_get_html($url,false,$context);
$token = $html->find('input[name=__RequestVerificationToken]',0)->value;
$id = $html->find('a.fileCopyAction',0)->rel;
return array(
'id' => $id,
'token' => $token
);
Followed by this code that I use to extract the download link(this does not work)
$res = array();
$options = array(
CURLOPT_RETURNTRANSFER => true, // return web page
CURLOPT_HEADER => false, // do not return headers
CURLOPT_FOLLOWLOCATION => true, // follow redirects
CURLOPT_USERAGENT => "spider", // who am i
CURLOPT_AUTOREFERER => true, // set referer on redirect
CURLOPT_CONNECTTIMEOUT => 120, // timeout on connect
CURLOPT_TIMEOUT => 120, // timeout on response
CURLOPT_MAXREDIRS => 10, // stop after 10 redirects
CURLOPT_POSTFIELDS => 'fileId='.$file_id.'&__RequestVerificationToken='.$token,
CURLOPT_HTTPHEADER => array(
'Accept:*/*',
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
'Accept-Language: en-US,en;q=0.8',
'Cache-Control: max-age=0',
'Connection: keep-alive',
'Content-Type: application/x-www-form-urlencoded',
'Cookie: '.$cookie,
'Host: abelhas.pt',
'Origin: http://abelhas.pt',
'X-Requested-With: XMLHttpRequest',
),
);
$ch = curl_init("http://abelhas.pt/action/License/Download");
curl_setopt_array( $ch, $options );
$content = curl_exec( $ch );
$err = curl_errno( $ch );
$errmsg = curl_error( $ch );
$header = curl_getinfo( $ch );
curl_close( $ch );
$res['content'] = $content;
$res['url'] = $header['url'];
return $res;
This returns J SON data but not what the website returns if u inspect it with chrome/firefox.
So the question remains : How to do this to get the same results as if visiting it urself using a browser?
Thanks in advance.

Not receiving transfer on different host

I'm trying to fetch html from a page with curl and php. I have the following code, which works perfectly on my host:
<?php
$curl = curl_init();
$URLs[] = "http://www.stackoverflow.com";
echo request( $curl, $URLs[0] );
function request($curl, $url, $post = null)
{
echo (isset($post) ? 'POST ' . $url : 'GET ' . $url) . PHP_EOL;
try
{
curl_setopt_array($curl, array
(
CURLOPT_RETURNTRANSFER => true,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_HEADER => true,
CURLOPT_AUTOREFERER => true,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_URL => $url,
CURLOPT_USERAGENT => 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0',
CURLOPT_HTTPHEADER => array
(
'Accept-Language: en-US;q=0.6,en;q=0.4',
'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
)
));
return curl_exec($curl);
}
catch(Exception $exception)
{
echo $exception;
}
}
curl_close($curl);
?>
When I put it on my webhost, I get no response. I checked for curl error and it says 'No Url set!'. Any hints please? Thanks!
try this other format of curl handling, initializing host in curl_init function:
$url = ....
$arrayConfig = ....
$curl = curl_init($url);
curl_setopt_array($curl,$arrayConfig);
$content = curl_exec($curl);
$err = curl_errno($curl);
$errmsg = curl_error($curl) ;
$header = curl_getinfo($curl);
curl_close($curl);

How to use PHP curl to fetch www.yahoo.com page?

Try to write a simple crawler method. When I use PHP curl to get the www.yahoo.com page, I fetch nothing. How can I fetch the page?
My code is in the following.
public function getWebPage($url, $timeout = 120) {
$options = array(
CURLOPT_RETURNTRANSFER => true,
CURLOPT_HEADER => false,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_ENCODING => "",
CURLOPT_USERAGENT => "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.19) Gecko/20081216 Ubuntu/8.04 (hardy) Firefox/2.0.0.19",
CURLOPT_AUTOREFERER => true,
CURLOPT_CONNECTTIMEOUT => $timeout,
CURLOPT_TIMEOUT => $timeout,
CURLOPT_MAXREDIRS => 10,
);
$ch = curl_init($url);
curl_setopt_array($ch, $options);
$content = curl_exec($ch);
$err = curl_errno($ch);
$errmsg = curl_error($ch);
$header = curl_getinfo($ch);
curl_close($ch);
return $content;
}
The yahoo.com runs on secure socket layer. So add this cURL param to your existing set.
CURLOPT_SSL_VERIFYPEER => false,
and also disable the USERAGENT..
The working code.. (tested)
<?php
class A
{
public function getWebPage($url, $timeout = 120) {
$options = array(
CURLOPT_RETURNTRANSFER => true,
CURLOPT_HEADER => false,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_ENCODING => "",
//CURLOPT_USERAGENT => "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.19) Gecko/20081216 Ubuntu/8.04 (hardy) Firefox/2.0.0.19",
CURLOPT_AUTOREFERER => true,
CURLOPT_CONNECTTIMEOUT => $timeout,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_TIMEOUT => $timeout,
CURLOPT_MAXREDIRS => 10,
);
$ch = curl_init($url);
curl_setopt_array($ch, $options);
$content = curl_exec($ch);
$err = curl_errno($ch);
$errmsg = curl_error($ch);
$header = curl_getinfo($ch);
curl_close($ch);
return $content;
}
}
$a = new A;
echo $a->getWebPage('www.yahoo.com');

PHP curl not getting the intended page content, Firefox does. Possible causes?

When I use the curl to fetch a page on an ecommerce site, it always gives me the same front page (ignoring the starting item parameter); whereas when I go to the url in a browser it works as usual.
Simplified code:
// s is the starting item count, no idea what yp4p_page is for exactly yet.
$url = 'http://list.taobao.com/market/baobao.htm?cat=40&yp4p_page=4&s=176';
$ch = curl_init($url);
$header[0] = 'Accept: text/xml,application/xml,application/xhtml+xml,'
. 'text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5';
$header[] = 'Cache-Control: max-age=0';
$header[] = 'Connection: keep-alive';
$header[] = 'Keep-Alive: 300';
$header[] = 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7';
$header[] = 'Accept-Language: en-us,en;q=0.5';
//$cookieFile = tempnam('/tmp', 'curlcookie');
$cookieFile = dirname(__FILE__) . DIRECTORY_SEPARATOR . 'curlcookies.txt';
$options = array(
CURLOPT_RETURNTRANSFER => true,
CURLOPT_HEADER => false,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_ENCODING => 'gzip,deflate',
CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0) Gecko/20100101 Firefox/6.0 FirePHP/0.6',
CURLOPT_AUTOREFERER => true,
CURLOPT_CONNECTTIMEOUT => 120,
CURLOPT_TIMEOUT => 120,
CURLOPT_MAXREDIRS => 10,
CURLOPT_SSL_VERIFYHOST => 0,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_VERBOSE => 1,
CURLOPT_HTTPHEADER => $header,
CURLOPT_COOKIEFILE => $cookieFile,
CURLOPT_COOKIEJAR => $cookieFile,
);
curl_setopt_array($ch, $options);
$strPageHTML = curl_exec($ch);
curl_close($ch);
I'm sorry for the Chinese site, but if you look at the items listed and their url as returned by curl, their id's are always the same as the ones on the front page (where s = 0) when they should be different.
What am I doing wrong?
Edit 1: added cookie to code, still doesn't work.
Edit 2: edited the cookie line to clear any confusion. Also the contents of the cookies are as follows:
# Netscape HTTP Cookie File
# http://curl.haxx.se/rfc/cookie_spec.html
# This file was generated by libcurl! Edit at your own risk.
#HttpOnly_.taobao.com TRUE / FALSE 0 cookie2 d686d4be95b4b56b61292118b43e1333
#HttpOnly_.taobao.com TRUE / FALSE 1316321978 _tb_token_ eeab7e3e5ea9e
.taobao.com TRUE / FALSE 1321505978 t 3c473872e51e93b0cf172375b31f503a
.taobao.com TRUE / FALSE 0 uc1 cookie14=UoLdHCGrCsSKAg%3D%3D
.taobao.com TRUE / FALSE 0 v 0
.taobao.com TRUE / FALSE 0 _lang zh_CN:GBK
You should take a look at cookies generated by the website, or even some CSRF tokens that would be inserted to keep you away from doing some parsing job.
When I inspect the webpage at first load, I can find this:
Set-Cookie:cookie2=b1d92ddac8aa82350a6ff5e892a8637d;Domain=.taobao.com;Path=/;HttpOnly
_tb_token_=fde3979ee6b13;Domain=.taobao.com;Path=/;Expires=Sat, 17-Sep-2011 07:09:40 GMT;HttpOnly
t=91f29eb410a21a04bf36025823c4b2ad; Domain=.taobao.com; Expires=Wed, 16-Nov-2011 07:09:40 GMT; Path=/
uc1=cookie14=UoLdHCDBHbn1eg%3D%3D; Domain=.taobao.com; Path=/
Maybe these cookies are used to identify you while navigating through categories.
Searching for "token" in the DOM made some results too.
Instead of accessing the page by pretending to be a user, is it possible to access the information you require via their api (http://open.taobao.com/)?
This page uses a lot of cookies, I would not be surprised a session cookie is required to load the page. See what happens when enabling that
curl_setopt($DATA_POST, CURLOPT_COOKIEFILE, 'cookiefile.txt');
curl_setopt($DATA_POST, CURLOPT_COOKIEJAR, 'cookiefile.txt');
// s is the starting item count, no idea what yp4p_page is for exactly yet.
$url = 'http://list.taobao.com/market/baobao.htm?cat=40&yp4p_page=4&s=176';
$ch = curl_init($url);
$header[0] = 'Accept: text/xml,application/xml,application/xhtml+xml,'
. 'text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5';
$header[] = 'Cache-Control: max-age=0';
$header[] = 'Connection: keep-alive';
$header[] = 'Keep-Alive: 300';
$header[] = 'Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7';
$header[] = 'Accept-Language: en-us,en;q=0.5';
$cookieFile = "cookie_china"; // I've changed this value and it seems to be working fine, I get the same results
$options = array(
CURLOPT_RETURNTRANSFER => true,
CURLOPT_HEADER => false,
CURLOPT_FOLLOWLOCATION => true,
CURLOPT_ENCODING => 'gzip,deflate',
CURLOPT_USERAGENT => 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:6.0) Gecko/20100101 Firefox/6.0 FirePHP/0.6',
CURLOPT_AUTOREFERER => true,
CURLOPT_CONNECTTIMEOUT => 120,
CURLOPT_TIMEOUT => 120,
CURLOPT_MAXREDIRS => 10,
CURLOPT_SSL_VERIFYHOST => 0,
CURLOPT_SSL_VERIFYPEER => false,
CURLOPT_VERBOSE => 1,
CURLOPT_HTTPHEADER => $header,
CURLOPT_COOKIEFILE => $cookieFile,
CURLOPT_COOKIEJAR => $cookieFile,
);
curl_setopt_array($ch, $options);
$strPageHTML = curl_exec($ch);
curl_close($ch);

Categories