PHP file_get_contents/curl - getting different result than browser - php

I'm trying to get content of this page: http://www.nytimes.com/2014/01/26/us/politics/rand-pauls-mixed-inheritance.html?hp&_r=0
I tried file_get_contents and curl solution but all gives me a Login page of NYTimes and I have no idea why.
Tried these file_get_contents()/curl getting unexpected page, PHP file_get_contents() behaves differently to browser, file_get_content get the wrong web
Is there any solution? Thanks
EDIT:
//this is the curl code I use
$cookieJar = dirname(__FILE__) . '/cookie.txt';
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookieJar);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookieJar);
curl_setopt($ch, CURLOPT_URL, $link);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12');
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
$data = curl_exec($ch);
curl_close($ch);

try to test it using saving cookies to same directory where the script resides first
so set the cookies path like that
$cookie = "cookie.txt";
this code works with me and i got the page
<?php
function curl_get_contents($url)
{
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
$get_page = curl_get_contents("http://www.nytimes.com/2014/01/26/us/politics/rand-pauls-mixed-inheritance.html?hp&_r=1");
echo $get_page;
?>

I think you need cURL to allow cookies to be saved. Try adding these lines to the cURL setup. For me this worked:
$cookie = dirname(__FILE__) . "\cookie.txt";
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);

Use Live HTTP Headers firefox plugin to check what is going on during page access. There can be redirections, cookie set etc. And then try to implement this behaviour with php curl (note: set user-agent as and other client headers the same as browser)

Related

I can not make the cookie is written with curl and php

I'm in this new medium curl but I search the internet for a solution and can not find it. I'm trying to fill a remote form using curl and send data by post. the problem is that the external website has some security measures. One of those is that I need to complete the form to get the value that was generated and keep the cookie. external code page reads:
document.getElementById('sell_session').value = readCookie('classified_session');
My code is this:
$cookie_file = "/home/reelonhe/public_html//temp/cookie.txt";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,'http://www.olx.com.ar/posting.php?categ_id=857');
curl_setopt($ch , CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:10.0.2) Gecko/20100101 Firefox/10.0.2');
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Accept-Language: es-es,en"));
curl_setopt($ch, CURLOPT_SSLVERSION,3);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
$result1 = curl_exec($ch);
$error = curl_error($ch);
$contents = curl_exec($ch);
$httpcode = curl_getinfo($ch,CURLINFO_HTTP_CODE);
curl_close($ch);
echo $error;
I tried the absolute path of the cookie with relative path. etc folder and nothing has permission to read and write. Do not know what else to do.
In your cookie file path
$cookie_file = "/home/reelonhe/public_html//temp/cookie.txt";
There seems an extra '/' before temp folder.
It should be
$cookie_file = "/home/reelonhe/public_html/temp/cookie.txt";
I am not sure it will solve your problem.

PHP cURL to login to facebook

I am attempting to login to Facebook using curl, but everything I have tried has ended up in Facebook saying, "Cookies are not enabled on your browser. Please enable cookies in your browser preferences to continue."
$login_email = 'email';
$login_pass = 'password';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'https://www.facebook.com/login.php');
curl_setopt($ch, CURLOPT_POSTFIELDS,'email='.urlencode($login_email).'&pass='.urlencode($login_pass).'&login=Login');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookies.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookies.txt");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3");
curl_setopt($ch, CURLOPT_REFERER, "http://www.facebook.com");
$page = curl_exec($ch) or die(curl_error($ch));
echo $page;
The cookie file 'cookies.txt' exists, and has 644 permissions.
I have also attempted to use multiple of the snippets online, but they all give the same error. I cannot continue with my current project until I get this working and I am able to navigate Facebook using curl as well. Any help is appreciated.
Thanks in advance.
This may help:
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2);
Check this answer:
Post to a Facebook user's wall with cURL PHP
I had the same problem and I fixed it by adding the following:
curl_setopt($s, CURLOPT_COOKIESESSION, false);
This must be used on curl cookie :
curl_setopt($ch, CURLOPT_COOKIEFILE, getcwd () . '/mirazmac_cookie.txt' );
curl_setopt($ch, CURLOPT_COOKIEJAR, getcwd () . '/mirazmac_cookie.txt' );
Better to use use facebook login sdk
Because Facebook constantly makes changes to their source code.

Can't get url using JSON script parsed with file_get_contents

I have this link I want to parse some information in it or just save it in a file...
can't do it without this simple code:
Example:
<?php
$myFile = 'test.txt';
$get= file_get_contents("http://www.ticketmaster.com/json/resale?command=get_resale_listings&event_id=0C004B290BF2D95F");
file_put_contents($myFile, $get); ?>
The output is:
{"version":1.1,"error":{"invalid":{"cookies":true}},"command":"get_resale_listings"}
I tried many other things like fopen or include did not work either. I don't understand because when I put the url in the browser it shows exactly ALL the code (google chrome) OR even better ask me to save it as a file (explorer). Looks like a browser cookies or something that doesn't load on my localhost ??
thanks for your tips.
You need to access that url with CURL.
The server checks if the client has cookies enabled. Using file_get_content() You do not send any information about client (browser).
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://www.ticketmaster.com/json/resale?command=get_resale_listings&event_id=0C004B290BF2D95F');
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_COOKIEJAR, "my_cookies.txt");
curl_setopt($ch, CURLOPT_COOKIEFILE, "my_cookies.txt");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3");
curl_exec($ch);

login form requiring loading the page first php curl

I posted this as php/curl but am open to any working solution.
example.com/login.asp has a hidden value inside the login form:
input type="hidden" name="security" value="123456789abcdef"
I tried to use curl to get this extra security value and include it to another curl call however the value changed after the first curl. I have read a related post, which suggests using php file_get_contents but it didn't work with the specific website.
Current php curl looks like this:
function curling ($websitehttps,$postfields,$cookie,$ref,$follow) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $websitehttps);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HTTPHEADER, array('Content-Type: application/x-www-form-urlencoded', 'Connection: Close'));
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1");
curl_setopt($ch, CURLOPT_TIMEOUT, 60);
curl_setopt($ch, CURLOPT_MAXREDIRS, 10);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
if ($cookie != "") {
curl_setopt($ch, CURLOPT_COOKIE,$cookie);
}
if ($postfields != "") {
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS,$postfields);
}
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, $follow);
curl_setopt($ch, CURLOPT_AUTOREFERER,TRUE);
curl_setopt($ch, CURLOPT_REFERER, $ref);
curl_setopt($ch, CURLOPT_FAILONERROR, TRUE);
$data = curl_exec($ch);
curl_close($ch);
return $data;
}
I am required to use the extra security code in post fields ($postfields) which should look like something similar to this:
ref=https%3A%2F%2Fexample.com%2F&security=123456789abcdef
Is there a way to do this?
Adding some extra lines to two separate curl sessions solved the problem.
Lines added to the first curl session:
curl_setopt ($ch, CURLOPT_COOKIEJAR, '/tmp/cookie.txt');
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
These additions created a cookie file in tmp folder.
Lines added to the second curl session:
curl_setopt ($ch, CURLOPT_COOKIEFILE, '/tmp/cookie.txt');
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
And here, used the information inside the cookie file to get the same security code on login page.
The solution described at another website may also work. In my case, server settings did not let me to use it.

How to retrieve captcha and save session with PHP cURL?

UPDATE: SOLVED
Hi all, i've got it, just save cookie
to temp file, and resubmit form with
curl and set cookies with previous
temp file :) thanks all for respond :)
This my working code
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url_register);
curl_setopt($ch, CURLOPT_USERAGENT, $this->useragent);
curl_setopt($ch, CURLOPT_COOKIEJAR, $this->cookie);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
$out['result'] = curl_exec($ch);
$out['error'] = curl_error($ch);
$out['info'] = curl_getinfo($ch);
curl_close($ch);
And for next curl just use CURLOPT_COOKIEFILE like this
/* fetch captcha url with existed cookie */
$ch = curl_init($captcha_url);
curl_setopt($ch, CURLOPT_USERAGENT, $this->useragent);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_COOKIEFILE, $this->cookie);
curl_setopt($ch, CURLOPT_FILE, $fp);
$out2['result'] = curl_exec($ch);
$out2['error'] = curl_error($ch);
$out2['info'] = curl_getinfo($ch);
curl_close($ch);
Hi all,
i'm create some script to submit content via php curl. first fetch session and captcha, and user must submit captcha to final submit.
the problem is i can't get captcha, i've try with this code and preg_match to get image tag and return it
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.2) Gecko/20070219 Firefox/2.0.0.2');
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_setopt($ch, CURLOPT_COOKIE, 1);
curl_setopt($ch, CURLOPT_COOKIEJAR, "1");
curl_setopt($ch, CURLOPT_COOKIEFILE, "1");
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
$result = curl_exec($ch);
curl_close($ch);
But no luck, page i'm trying to submit is http://abadijayaiklan.co.cc/pasang-iklan/.
I hope someone can help me out :)
Thanks and regards
From the php manual page on curl_setopt, CURLOPT_COOKIEFILE and CURLOPT_COOKIEJAR should both specify a filename. You have them set to '1' (which may be valid, but is that what you intended?)

Categories