I have been experimenting with curl for accessing the PayPal payment authorisation site using PHP.
e.g.
...
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $nvp);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$res = curl_exec ($ch);
preg_match_all('/Set-Cookie: .*/', $res, $cookieMatches);
foreach ($cookieMatches[0] as $cookieMatch)
header($cookieMatch);
preg_match('/Location: .*/', $res, $locMatches);
header($locMatches[0]);
header('Vary: Accept-Encoding');
header('Strict-Transport-Security: max-age=500');
header('Transfer-Encoding: chunked');
header('Content-Type: text/html');
The principle being simply to reflect the original redirect (I am sure there is a simpler way to do this). However, the response from PayPal seems to indicate some kind of cookie error.
My hunch is that the cookie has been linked to the originating machine in some way. Can anyone confirm this, or am I just missing something obvious!
The CURL has built-in support for cookies (as you know). But it's been tricky. I haven't managed cookies to work until I declared option
curl_setopt ($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
Third parameter is a name of the file storing cookies - preferably in temp folder. Maybe you should just try this approach.
With this the redirects work "automatically".
curl_setopt ($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
//SAVE THE COOKIES
curl_setopt ($ch, CURLOPT_COOKIEFILE, 'cookie.txt');
USE THE COOKIES
curl_setopt ($ch, CURLOPT_FOLLOWLOCATION, 1');
// Follow Where the location will take you, maybe you catch the issue.
Since it's working on browser it has to work using CURL, unless they are using javascript to set cookies.
even if they are using cookies depending on IP address, try to start the session from beginning using curl so they set your server ip address with generated cookies.
Related
I am trying to grab the meta data from a news article on the NY Times website, specifically http://www.nytimes.com/2014/06/25/us/politics/thad-cochran-chris-mcdaniel-mississippi-senate-primary.html
Whenever I try however I am getting redirects from the sight because my "browser" does not accept cookies. I have enabled the curl options to save cookies and tried following the accepted answers in a few other StackOverflow questions (here, here, and here) and while the answer worked on those websites it doesn't seem to work on the nytimes site.
My current php curl function looks like this:
function get_extra_meta_tags_curl($url) {
$ckfile = tempnam("/public_html/commentarium/", "cookies.txt");
$ch = curl_init($main_url);
curl_setopt($ch, CURLOPT_COOKIEJAR, $ckfile);
curl_setopt($ch, CURLOPT_COOKIEFILE, $ckfile);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$output = curl_exec($ch);
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_COOKIEJAR, $ckfile);
curl_setopt($ch, CURLOPT_COOKIEFILE, $ckfile);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$output = curl_exec($ch);
curl_close($ch);
echo $output;
}
The problem appears to be that when I request the URL, nytimes.com checks if the browser accepts cookies. I checks a couple of times before redirecting to the login page with a REFUSE_COOKIE_ERROR. Instead of posting the full redirect list here you can see it on my test page here along with the raw html that the final redirect returns and what my current get_extra_meta_tags_curl function is returning under CURL test
Thanks for any help!
You enable cookies auto-handling in wrong manner. CURLOPT_COOKIEJAR only enables cookies saving (storing), but you need also enable cookies loading and passing them with request (by CURLOPT_COOKIEFILE option). Otherwise cookies auto-handling won't work and you will experienced mentioned "Browser does not accept cookies" problem.
So you have to set both CURLOPT_COOKIEJAR and CURLOPT_COOKIEFILE options to the same value ($ckfile) at each CURL request:
...
curl_setopt($ch, CURLOPT_COOKIEJAR, $ckfile);
curl_setopt($ch, CURLOPT_COOKIEFILE, $ckfile);
...
I have a php proxy server to which email and password are posted on the beginning (login). I then use
$ch = curl_init($url);
curl_setopt ($ch, CURLOPT_POST, 1);
curl_setopt ($ch, CURLOPT_POSTFIELDS, $myvars);
curl_setopt( $ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt ($ch, CURLOPT_COOKIEJAR, $cookieJar);
curl_setopt( $ch, CURLOPT_HEADER, 1);
curl_setopt( $ch, CURLOPT_RETURNTRANSFER, 1);
to post forward to some API that expects certain parameters. As you can see I save a cookie in a cookie jar file.
I can then use this cookie file to call any other requests to proxy -> API and successfully get the response. Everything works just fine. I use
curl_setopt ($ch, CURLOPT_COOKIEFILE, $cookieJar);
to make other requests after user successfully signed in.
The problem is that only one user can login(and call other requests) at the time, because there is only one cookie jar file. I could probably generate unique cookie files on proxy and access them somehow with each new request by each user. But that is a load on the server and definitely not a good idea.
So what I would like to do is to save a cookie that is received into variable instead of a file and then send this to user...
This didn't work for me unfortunately; I can probably manage to write my own regex but I am wondering if there is a possibility to directly save a cookie into variable with curl or do I have to parse headers manually? (I want to be able to feed CURLOPT_COOKIEFILE with cookie in variable rathen than cookie in file)
Lets try this with a single curl handle($ch):
Making my first request:
$url= "http://www.google.com";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt ($ch, CURLOPT_COOKIEJAR, '-'); // <-- see here
$result = curl_exec($ch);
// remember i didn't close the curl yet!
Now make another curl request with the same handle:
$url= "http://www.google.com";
curl_setopt($ch, CURLOPT_URL,$url);
$result = curl_exec($ch);
// if you are done, you can close it.
curl_close($ch);
In this example, I have used the - for the cookiejar. Which means it will not use any file. So during second curl request, it will use the cookiejar of previous call.
One problem: It will print the cookie jar values into std-output.
I've done scraping for lots of sites but one in particular isn't saving it's cookies to my cookie file. Any ideas?
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_TIMEOUT,8200);
curl_setopt($ch,CURLOPT_TIMEOUT_MS,8200);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT ,8200);
$cookie_file = "cookies/zapper.txt";
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 0);
if ($fields) {curl_setopt($ch,CURLOPT_POST, count($fields)); }
if ($fields) {curl_setopt($ch,CURLOPT_POSTFIELDS, $fields_string); }
This is the first site that I've done that doesn't respond to my cookie saves. All others use the same code and work perfectly. I've even emulated the post of their forms and faked the header in case it was checking [those.
The site I'm trying to mimic an add to cart process for is http://zapper.co.uk/
Read a possible solution directly from the php.net site about curl_setopt. It's a workaround to get Cookie content from the header output. Seems to be a cool alternative.
Also, you can get surprising results modifiying some of your rules at curl_setop. Sometimes we use more options than needed.
I also recommend you to echo the whole $ch content (It will print page like the browser does). Sometimes you get a detailed error not present at headers seeing the live result content.
I've got NTLM (Active Directory) based service, and I need to write a PHP application. Normally, users are logging in to website with Activre Directory credentials, and it's ok.
But what I want to do, is to let them type in their credentials to PHP-written site, which in next step will use cURL to authenticate users to that Active Directory based site where they normally log in.
And this part is hard. I need then to keep session of users that through PHP cURL script authenticated to Active Directory based site in order to use them again later
(CRON querying site to determine that it has changed and automatically do some operations when this happens, which normally user has do manually).
In order to NOT store their credentials to authenticate again when this change happens, I somehow need to store NTLM session in PHP cURL site to every user that authenticated to
that site through this PHP cURL site.
My question is: Is that even possible?
Thanks in advance.
#Willem Mulder
The code you've posted actually does cookie-storing, but that is not my point becouse I've already done that (sorry for not writing it before). What I got so far is:
$cookie_file_path = dirname(__FILE__) . '/cookies.txt';
$ch = curl_init();
//==========================================================================
curl_setopt($ch, CURLOPT_USERPWD, $username. ':' . $password);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_FAILONERROR, 0);
curl_setopt($ch, CURLOPT_MAXREDIRS, 100);
//==========================================================================
$ret = curl_exec($ch);
By using options CURLOPT_COOKIEFILE and CURLOPT_COOKIEJAR, cURL does the cookie storing in local file "cookies.txt". The problem is, that when I comment CURLOPT_USERPWD option (after authenticating and storing cookie, so theoretically I have session), I cannot authorize to website. Perhaps it reinitializes NTLM Handshake authorisation and is expecting username and password, which I don't want to store.
I want to store session info only, to provide service this session info and omit second authentication, but cURL seems to not take this data from cookie file, and REWRITES it with not relevant data send to me from service as response to NOT AUTHRORISED access request.
Well, yes you could
$ch = curl_init('http://www.google.com/');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// Get headers too with this line
curl_setopt($ch, CURLOPT_HEADER, 1);
$result = curl_exec($ch);
// Get cookie
preg_match('/^Set-Cookie:\s*([^;]*)/mi', $result, $m);
var_dump(parse_url($m[1]));
// And then of course store it somewhere :-)
As seen here how to get the cookies from a php curl into a variable
<?php
$ebay_user_id = "id"; // Please set your Ebay ID
$ebay_user_password = "password"; // Please set your Ebay Password
$cookie_file_path = dirname(__FILE__).'/cookie.txt'; // Please set your Cookie File path
$LOGINURL = "http://signin.ebay.com/aw-cgi/eBayISAPI.dll?SignIn";
$agent = "Mozilla/4.0 (compatible;)";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$LOGINURL);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
$result = curl_exec ($ch);
curl_close ($ch);
$LOGINURL = "http://signin.ebay.com/aw-cgi/eBayISAPI.dll";
$POSTFIELDS = 'MfcISAPICommand=SignInWelcome&siteid=0&co_partnerId=2&UsingSSL=0&ru=&pp=&pa1=&pa2=&pa3=&i1=-1&pageType=-1&userid='. $ebay_user_id .'&pass='. $ebay_user_password;
$reffer = "http://signin.ebay.com/aw-cgi/eBayISAPI.dll?SignIn";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$LOGINURL);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS,$POSTFIELDS);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_REFERER, $reffer);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
$result = curl_exec ($ch);
curl_close ($ch);
print $result; ?>
I'm really new player on cURL...
I have this code now using in login into ebay.
The problem for now is the cookies it told me that it was blocked by something.
The message it shows: Your web browser settings are blocking cookies.
I use firefox for test and tried other browser also got the same issues.
I have confirmed that my browser setting are accepted for the cookies access.
Also, I have checked there has conntent inside the cookies.txt file, so that mean the cookies.txt can be access correctly.
So....What is the problem for this issue? The code I used are correct?
Thanks everyone for help.
Try modifying the agent to something similar;
'Mozilla/5.0 (Windows NT 6.1; rv:15.0) Gecko/20100101 Firefox/15.0.1'
Edit: actually I believe the problem is you need to query the signin page first,
first visit "http://signin.ebay.com/aw-cgi/eBayISAPI.dll?SignIn"
this will set the cookies, then sign in as you have.
you can try it in a browser, navigate to the eBay sign in page,
clear your cookies and then signin.
You will get the browser not supporting cookies error.
You need to understand something and that is that doing a HTTP request with curl through php has nothing to do with your browser. The website you are accessing doesn't care what browser you use to run the php script. The actual request is done by your server, not by your browser.
On the other hand, if eBay engineers are smart they'd block this, you probably aren's supposed to do things like this, that's what the Ebay API's are for.
And a little tip, use a HTTP Client library, doing things like this in plain cURL is a pita and gives some very bad and unreadable code.
Check https://github.com/guzzle/guzzle for example.