I'm trying to download a URL : http://es.extpdf.com/nagore-pdf.html using the following code. But I'm getting statuscode as 0 in return. But when accessing it from : http://web-sniffer.net/ it shows 301 redirected. My code seems to be working fine for 301 redirected URLs too.
What could be the problem?
<?php
print disavow_download_url("http://es.extpdf.com/nagore-pdf.html");
function disavow_download_url($url) {
$custom_headers = array();
$custom_headers[] = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
$custom_headers[] = "Pragma: no-cache";
$custom_headers[] = "Cache-Control: no-cache";
$custom_headers[] = "Accept-Language: en-us;q=0.7,en;q=0.3";
$custom_headers[] = "Accept-Charset: utf-8,windows-1251;q=0.7,*;q=0.7";
$ch = curl_init();
$useragent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:9.0.1) Gecko/20100101 Firefox/9.0.1";
curl_setopt($ch, CURLOPT_USERAGENT, $useragent); // set user agent
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
//curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_HTTPHEADER, $custom_headers);
//these two from https
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 15);
curl_setopt($ch, CURLOPT_TIMEOUT, 10); //timeout in seconds
$txResult = curl_exec($ch);
$statuscode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
print "statuscode=$statuscode\n";
print "result=$txResult\n";
}
The url is accessible from USA, not from your region. It worked for the web-sniffer because their server is hosted at USA(or somewhere which region is allowed by the extpdf).
I have used an USA proxy with the curl and it returned me data.
curl_setopt($ch, CURLOPT_PROXY, "100.9.90.1:3128"); // change IP, Port
Related
I need to get the source code of a page that is only accessible when I am logged in. However, when I use the following code, it doesn't recognize that I'm logged in, and asks me to re-login.
$opts = array('http' => array('header'=> 'Cookie: ' . $COOKIEFILE."\r\n"));
$context = stream_context_create($opts);
$file = file_get_contents('http://www.example.com/', false, $context);
echo $file;
I have researched and found examples similar to mine, but they don't help me. When I use the above code it doesn't recognize my cookies.
Full code:
$USERNAME = 'myEmail';
$PASSWORD = 'myPassword';
$COOKIEFILE = tempnam ("/tmp", "CURLCOOKIE");
$ch = curl_init();
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 30);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, $COOKIEFILE);
curl_setopt($ch, CURLOPT_COOKIEFILE, $COOKIEFILE);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 120);
curl_setopt($ch, CURLOPT_TIMEOUT, 120);
curl_setopt($ch, CURLOPT_URL,
'https://accounts.google.com/ServiceLogin?hl=en&service=alerts&continue=http://www.google.com/alerts/manage');
$data = curl_exec($ch);
$formFields = getFormFields($data);
$formFields['Email'] = $USERNAME;
$formFields['Passwd'] = $PASSWORD;
unset($formFields['PersistentCookie']);
$post_string = '';
foreach($formFields as $key => $value) {
$post_string .= $key . '=' . urlencode($value) . '&';
}
$post_string = substr($post_string, 0, -1);
curl_setopt($ch, CURLOPT_URL, 'https://accounts.google.com/ServiceLoginAuth');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_string);
$result = curl_exec($ch);
$ch = curl_init ("http:/example.com");
curl_setopt ($ch, CURLOPT_COOKIEFILE, $COOKIEFILE);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
$output = curl_exec ($ch);
var_dump($output);
Allen, in situations like yours, when I'm having trouble login with curl, what I normally do is use Live Http Headers for firefox to log my requests and then use the request headers with curl CURLOPT_HTTPHEADER, the cookies will be included in the headers.
Flow:
Log in to the website using your browser, once logged, log any request with Live Http Headers for firefox, copy and paste the request to an array, something like :
$myHeaders = array(
"Host: stackoverflow.com",
"User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:41.0) Gecko/20100101 Firefox/41.0",
"Accept: application/json, text/javascript, */*; q=0.01",
"Accept-Language: en-US,en;q=0.5",
"Accept-Encoding: gzip, deflate",
"DNT: 1",
"Content-Type: application/x-www-form-urlencoded; charset=UTF-8",
"X-Requested-With: XMLHttpRequest",
"Referer: http://stackoverflow.com/questions/32933636/use-cookie-with-curl-php",
"Content-Length: 482",
"Cookie: mycookie :)",
"Connection: keep-alive",
"Pragma: no-cache"
);
Then, use the headers with curl:
curl_setopt($ch, CURLOPT_HTTPHEADER, $myHeaders );
Notes:
1- No need for CURLOPT_COOKIEJAR or CURLOPT_COOKIEFILE
2- This works with almost every site I've tried.
3- Some cookies may expire after a while, others, like google, never expire.
Onsite booking process now i am using rest api calling to get the data about booking process.But now the problem is that when I set the form url is :-
$url = 'https://book.api.ean.com/ean-services/rs/hotel/v3/res?
minorRev=99
&cid=55505
&sig=1893d9f7e3e9fbd3f8a36f43cd61287d
&apiKey=1bn8n4or4tjajq23fe4l6m18lp
&customerUserAgent=Mozilla/5.0 (Windows NT 6.1; WOW64; rv:38.0) Gecko/20100101 Firefox/38.0
&customerIpAddress=223.30.152.118
&customerSessionId=e80df6de9008af772cfb48a389465415
&locale=en_US
¤cyCode=USD
&hotelId=106347
&arrivalDate=10/30/2015
&departureDate=11/01/2015
&supplierType=E
&rateKey=469e1aff-49de-4944-a64d-25d96ccde3aa
&roomTypeCode=200127420
&rateCode=200706716
&chargeableRate=257.20
&room1=2,5,7
&room1FirstName=test
&room1LastName=testers
&room1BedTypeId=23
&room1SmokingPreference=NS
&email=test#yourSite.com
&firstName=tester
&lastName=testing
&homePhone=2145370159
&workPhone=2145370159
&creditCardType=CA
&creditCardNumber=5401999999999999
&creditCardIdentifier=123
&creditCardExpirationMonth=11
&creditCardExpirationYear=2015
&address1=travelnow
&city=Bellevue
&stateProvinceCode=WA
&countryCode=US
&postalCode=98004';
and when i manually posted the data it will get the response But when I am using curl to post the url which i have posted previous it will face the error.
My curl code is :-
$header[] = "Accept: application/json";
$header[] = "Accept-Encoding: gzip";
$header[] = "Content-length: 0";
$ch = curl_init();
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'POST');
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
curl_setopt($ch, CURLOPT_ENCODING, "gzip");
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_VERBOSE, true);
$verbose = fopen('php://temp', 'rw+');
curl_setopt($ch, CURLOPT_STDERR, $verbose);
$result = curl_exec($ch);
After posting data i will get the response
{"HotelRoomReservationResponse":{"EanWsError":{"itineraryId":-1,"handling":"UNRECOVERABLE","category":"EXCEPTION","exceptionConditionId":-1,"presentationMessage":"TravelNow.com cannot service this request.","verboseMessage":"Exception Caught: null"},"customerSessionId":"8ab1d482-f968-49d2-a429-a1cbab748fe5"}}
So i will get that error repeatedly. Please help me how i can find the right data.
Your problem here is that you are parsing the parameters in the url they need to be given in the body see thetop of this page: http://developer.ean.com/docs/book-reservation/examples/rest-reservation/
Not sure how you do this in PHP but you can use -d on the command line
I am trying to remote login using curl php. I can login to site but session not maintaining in curl response. I have triedsession_write_close() as how to maintain session in cURL in php? but its not going to work. I am using cookies too but nothing.
Here is what i tried:
$ch = curl_init();
$params['ror_csrf_token'] = $hiddenValue;
$params['n'] = '';
$params['email'] = 'xxx.xxx#evontech.com';
$params['password'] = 'xx';
$params['remember_me'] = 'on';
$form_action_url = 'http://www.xxxxxxxx.com/go/login';
$postData = '';
foreach($params as $k => $v)
{
$postData .= $k.'='.$v.'&';
}
$postData = rtrim($postData, '&');
print_r($postData);
$theaders[] = "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
$theaders[] = "Content-Type: application/x-www-form-urlencoded";
$theaders[] = "Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7";
$theaders[] = "Accept-Encoding: gzip,deflate,sdch";
$theaders[] = "Accept-Language: en-US,en;q=0.8";
$theaders[] = "Cache-Control: max-age=0";
//$theaders[] = "Connection: keep-alive";
//$theaders[] = "Content-Length: 119";
curl_setopt($ch, CURLOPT_URL,$form_action_url);
//curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 100);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);//follow redirection
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookies.txt'); // set cookie file to given file
//curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookies.txt'); // set same file as cookie jar
//curl_setopt($ch, CURLOPT_COOKIE, 'cookies.txt');
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36');
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "POST");
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $postData);
curl_setopt($ch, CURLOPT_HTTPHEADER, $theaders);
curl_setopt($ch, CURLOPT_HEADER, false);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
//curl_setopt($ch, CURLOPT_NOBODY, false);
//curl_setopt($ch, CURLOPT_REFERER, "http://www.ripoffreport.com/go/login");
//curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
//curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
echo $content = curl_exec($ch);
$headers = curl_getinfo($ch);
$errors = curl_error($ch);
echo "<pre>";
print_r($headers);
curl_close ($ch);
Note: I have googled a lot since 10 days, but nothing seems to work for me.
You have to set both CURLOPT_COOKIEJAR and CURLOPT_COOKIEFILE options to the same absolute path value ('cookies.txt' is a relative path).
This is necessary in order to enable cookies auto-handling (and therefore, session maintaining) within redirects series which the script will have.
Also you shouldn't set CURLOPT_CUSTOMREQUEST and CURLOPT_POST options together, only one of them (CURLOPT_POST in your case).
So the script should have the following lines:
curl_setopt($ch, CURLOPT_COOKIEFILE, dirname(__FILE__).'/cookies.txt');
curl_setopt($ch, CURLOPT_COOKIEJAR, dirname(__FILE__).'/cookies.txt');
BTW: session_write_close() doesn't affect to CURL requests.
I am trying to perform a cURL request (either directly from shell or via PHP) that will return a URL's content essentially the same as the request made through a browser (minus any cookies/logins, etc).
A basic cURL request for www.google.com will return what appears to be the Japanese version of Google Search with some character encoding issues.
Testing with the options including setting a standard User Agent and follow location still does not result in what I assumed would be a very similar request to my browser. Is there a set of flags I should be using to closely imitate a browser request?
The code below is currently used for testing, but even with cookies being stored Google assumes the location is Japan (google.co.jp).
$header = array(
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language: en-us,en;q=0.5",
"Connection: keep-alive",
"Cache-Control: no-cache",
"Content-Type: application/x-www-form-urlencoded; charset=UTF-8",
"Pragma: no-cache",
);
$useragent = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)';
$ch = curl_init();
curl_setopt($ch, CURLOPT_VERBOSE, 0);
curl_setopt($ch, CURLOPT_URL, $request);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_COOKIEJAR, "my_cookies.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "my_cookies.txt");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
$data = curl_exec($ch);
curl_close($ch);
$header = array(
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language: en-us,en;q=0.5",
"Connection: keep-alive",
"Cache-Control: no-cache",
"Content-Type: application/x-www-form-urlencoded; charset=UTF-8",
"Pragma: no-cache",
);
$useragent = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)';
$ch = curl_init();
curl_setopt($ch, CURLOPT_VERBOSE, 0);
curl_setopt($ch, CURLOPT_URL, $request);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_HTTPHEADER, $header);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_COOKIEJAR, "my_cookies.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "my_cookies.txt");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_PROXY, 'PROXY_IP_HERE:PROXY_PORT'); // Use a proxy located in USA
$data = curl_exec($ch);
curl_close($ch);
I'm trying to use CURL with PHP to login to Vimeo.com, Vimeo login.
To provide the data for CURL to use (cookie and field data), I'm using a browser extension to read the field data off of the webpage and get the cookies. I'm then passing that data through to my server and am trying to login using curl.
I'm quite positive that the browser extension part works correctly (gets the correct data) because I can verify what it's passing with what it should be passing, and it matches correctly.
Additionally, I've used this on other sites as well, and it has no problem logging in, but on vimeo the exec returns false.
Any thoughts?
function curlpage(){
$ch = curl_init();
$url = $this->input->post('url');
$data = $this->input->post('data');
$cookie = $this->input->post('cookie');
$method = $this->input->post('method');
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_POSTFIELDS,$data);
if(strtolower($method)=="put"){
curl_setopt($ch, CURLOPT_PUT, 1);
}
else{
curl_setopt($ch, CURLOPT_PUT, 0);
}
if(strtolower($method)=="get"){
curl_setopt($ch, CURLOPT_HTTPGET, 1);
}
else{
curl_setopt($ch, CURLOPT_HTTPGET, 0);
}
if(strtolower($method)=="post"){
curl_setopt($ch, CURLOPT_POST, 1);
}
else{
curl_setopt($ch, CURLOPT_POST, 0);
}
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_UNRESTRICTED_AUTH, true);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_CAINFO, getcwd().'/certificates/BuiltinObjectToken-EquifaxSecureCA.crt');
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_COOKIE, $cookie);
curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/20100101 Firefox/17.0");
curl_setopt($ch, CURLOPT_REFERER, $url);
$error = curl_error($ch);
$url=curl_getinfo($ch, CURLINFO_EFFECTIVE_URL);
if(!preg_match('/^http(s)?:\/\//', $url)){
$url = 'http://' . $url;
}
$host = parse_url($url, PHP_URL_HOST);
$page = curl_exec($ch);
curl_close($ch);
return array('page'=>$page, 'url'=>$host, 'error'=>$error);
}
Here's a sample of the data I'm sending to the above function on my server (with a bogus email an password and altered cookies):
data=action%253Dlogin%2526service%253Dvimeo%2526email%253Dhou%2540fah.com%2526password%253Dudwt%2526token%253D6b2fc081bcdf02b1f58a390d6a3f8b83
cookie=__utma%3D18392654.1284111214.1456668252.1456678435.1456181183.3%3B__utmb%3D18302654.2.10.1454681883%3B__utmc%3D18232154%3B__utmz%3D17202654.1456675435.2.2.utmcsr%3Dgoogle%7Cutmccn%3D(organic)%7Cutmcmd%3Dorganic%7Cutmctr%3D(not%2520provided)%3B
method=POST
url=http%3A%2F%2Fvimeo.com%2Flog_in
$ret = customSendDataByCurl("https://vimeo.com/log_in");
preg_match("/xsrft: \'(.*)\',/i",$ret,$token);
$token = $token[1];
echo "$token <hr>";
$cookie = '(copy from your browser using tamper data)... xsrft='.$token;
$headers = array(
"Referer: https://vimeo.com/log_in",
"User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0",
"Accept: application/json",
"Accept-Language: en-US,en;q=0.5",
"X-Requested-With: XMLHttpRequest",
"X-Request: JSON",
"Content-Type: application/x-www-form-urlencoded; charset=utf-8",
);
$ret = customSendDataByCurl("https://vimeo.com/log_in?action=warm", "POSTDATA=email=(email url encoded)&token=".$token, $headers, $cookie);
$fields = array(
"action" => "login",
"service" => "vimeo",
"email" => "(email)",
"password" => "(pass)",
"token" => $token,
);
$headers = array(
"Referer: https://vimeo.com/log_in",
"User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0",
"Content-Type: application/x-www-form-urlencoded; charset=utf-8",
);
$ret = customSendDataByCurl("https://vimeo.com/log_in", http_build_query($fields), $headers, $cookie);
$ret = customSendDataByCurl("https://vimeo.com/stats/video/84142281/totals/export:csv", http_build_query($fields), $headers, $cookie);
var_export($ret);
function customSendDataByCurl($agateway, $apostfields=null, $headers=array(), $cookie="") {
if(is_array($apostfields)) $apostfields = http_build_query($apostfields);
$cookiesFile = 'cookies.txt';
$ch = curl_init($agateway);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt ($ch, CURLOPT_SSL_VERIFYHOST, FALSE);
curl_setopt( $ch, CURLOPT_COOKIESESSION, true );
if ($headers) curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt( $ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.3; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0' );
curl_setopt($ch,CURLOPT_COOKIEJAR,$cookiesFile);
curl_setopt($ch,CURLOPT_COOKIEFILE,$cookiesFile);
curl_setopt($ch, CURLOPT_COOKIE, $cookie);
if(!empty($apostfields)) curl_setopt($ch, CURLOPT_POSTFIELDS, $apostfields);
$response = curl_exec($ch);
if(!$response) $response="CURL #".curl_errno($ch).": ".curl_error($ch);
return $response;
}
Vimeo, I've actually found to be a weird case for websites. They don't set all of their cookies up front, but rather set certain cookies needed to login when the form is submitted. So one of my problems was that I was not submitting all of the correct cookie information.
My second problem was that I was not correctly encoding my data as it was being sent.
All said and done, I now have got it to work!