Curl: can't save cookies in cookiejar - php

I'm trying to login to kiala website. Kiala is a "shipping" compagny and I like to get a new shipping token with php. Since there is no api for doing this I tried with curl. Now I don't have much experience with curl and I can't make curl saving the cookies to the jar. I've tried many things but now I get to the point that I want to rip my hair off. I need these cookies to make further requests.
I've made a dummy account for testing
website: http://www.kialaverzendservice.be/
login form: http://www.kialaverzendservice.be/login.required.action?os_destination=%2Fsender%2Fstart.action
email: kialatest#mailinator.com
pwd: test123
I get the following header
HTTP/1.1 200 OK Date: Thu, 10 Oct 2013 09:46:37 GMT Server:
Apache-Coyote/1.1 Content-Type: text/html;charset=ISO-8859-15 Vary:
Accept-Encoding Set-Cookie:
berkano-seraph-login=eGV5ZcKEZXhlwoZlwoJkfGJ5Y31jemTCgGJ7YsKGYsKCYnhifmLCgmLChmN6YsKGY3dmwoNiwoZjwoNjeGh+YnhjfWJ5Yn1mwoFmwoFm;
Expires=Fri, 10-Oct-2014 09:46:37 GMT; Path=/ Set-Cookie:
kiala-c2c-language=nl; Expires=Tue, 28-Oct-2081 13:00:44 GMT; Path=/
Transfer-Encoding: chunked
My simplified php code: I'm able to login but the cookies in the header are not set in my cookiejar-file? btw I'm on localhost (wamp), but I don't think it matters.
loginToKiala();
function loginToKiala(){
$url = 'http://kialaverzendservice.be/sender/start.action';
//POST vars
$fields = array(
'os_username' => urlencode('kialatest#mailinator.com'),
'os_password' => urlencode('test123'),
'os_cookie'=>urlencode('true')//remember me
);
//url-ify the data for the POST
$fields_string='';
foreach($fields as $key=>$value) { $fields_string .= $key.'='.$value.'&'; }
rtrim($fields_string, '&');
$ch=curl_init();
$cookie_file = './cookies.txt';
if (! file_exists($cookie_file) || ! is_writable($cookie_file))
{
echo 'Cookie file missing or not writable.';
exit;
}
//set the url, number of POST vars, POST data
curl_setopt($ch,CURLOPT_URL, $url);
curl_setopt($ch,CURLOPT_POST, count($fields));
curl_setopt($ch,CURLOPT_POSTFIELDS, $fields_string);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file);
//curl_setopt($ch,CURLOPT_AUTOREFERER, true);
//curl_setopt($ch, CURLOPT_REFERER, 'http://www.kialaverzendservice.be/');//set referer for first request
//curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1");
curl_setopt($ch, CURLOPT_HEADER, 1);
//curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_exec ($ch); // execute the curl command
curl_close ($ch);
unset($ch);
}
any help is appreciated!

First of all you're not using CURLOPT_POST correctly. cURL is expecting TRUE, FALSE, 1 or 0. The function you have there, count($fields) will (probably) return a number higher than one.
Do this instead:
curl_setopt( $ch, CURLOPT_POST, 1 );
At least, if I understand correctly that you want to use POST instead of GET.

Related

How do I have session available in PHP application which makes HTTP requests to java servlets using cURL (Session is started in servlet)? [duplicate]

I'm trying to connect to an API, authenticate a user and then view the user details. This is accomplished by first accessing the login endpoint at
http://api.example.com/login/<username>/<password>
to log in and then the following to view user details:
http://api.example.com/user/
This all works in a web browser. However, once I try to use Curl, the login works fine, but when attempting to view user details, I get back a 401, unauthorized error. I believe this is because Curl isn't saving the session cookies properly? Can someone point out why it isn't working and how to fix it? I've tried searching stack exchange, however, none of the solutions I've tried have worked for my situation. The code I'm using to curl the endpoints is shown below. Thanks!
define("COOKIE_FILE", "cookie.txt");
// Login the user
$ch = curl_init('http://api.example.com/login/joe/smith');
curl_setopt ($ch, CURLOPT_COOKIEJAR, COOKIE_FILE);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, true);
echo curl_exec ($ch);
// Read the session saved in the cookie file
echo "<br/><br/>";
$file = fopen("cookie.txt", 'r');
echo fread($file, 100000000);
echo "<br/><br/>";
// Get the users details
$ch = curl_init('http://api.example.com/user');
curl_setopt ($ch, CURLOPT_COOKIEJAR, COOKIE_FILE);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, true);
echo curl_exec ($ch);
This code will output:
HTTP/1.1 200 OK Date: Mon, 22 Oct 2012 21:23:57 GMT Server: LiteSpeed Connection: close X-Powered-By: PHP/5.3.14 Set-Cookie: cfapi=f481129c9616b8f69cc36afe16466545; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Content-Type: application/json X-Powered-By: CFWAPI 0.1a Content-Length: 46 {"status":200,"msg":"Successfully Logged In."}
# Netscape HTTP Cookie File # http://curl.haxx.se/rfc/cookie_spec.html # This file was generated by libcurl! Edit at your own risk. api.example.com FALSE / FALSE 0 cfapi 94f63b07ccf7e34358c1c922341c020f
HTTP/1.1 401 Unauthorized Date: Mon, 22 Oct 2012 21:23:57 GMT Server: LiteSpeed Connection: close X-Powered-By: PHP/5.3.14 Set-Cookie: cfapi=a8eb015a7c423dde95aa01579c4729a4; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Content-Type: application/json X-Powered-By: CFWAPI 0.1a Content-Length: 49 {"status":401, "msg":"You need to login first!"}
You also need to set the option CURLOPT_COOKIEFILE.
The manual describes this as
The name of the file containing the cookie data. The cookie file can
be in Netscape format, or just plain HTTP-style headers dumped into a
file. If the name is an empty string, no cookies are loaded, but
cookie handling is still enabled.
Since you are using the cookie jar you end up saving the cookies when the requests finish, but since the CURLOPT_COOKIEFILE is not given, cURL isn't sending any of the saved cookies on subsequent requests.
You have correctly used "CURLOPT_COOKIEJAR" (writing) but you also need to set "CURLOPT_COOKIEFILE" (reading)
curl_setopt ($ch, CURLOPT_COOKIEJAR, COOKIE_FILE);
curl_setopt ($ch, CURLOPT_COOKIEFILE, COOKIE_FILE);
This is how you do CURL with sessions
//initial request with login data
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://www.example.com/login.php');
curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/32.0.1700.107 Chrome/32.0.1700.107 Safari/537.36');
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, "username=XXXXX&password=XXXXX");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie-name'); //could be empty, but cause problems on some hosts
curl_setopt($ch, CURLOPT_COOKIEFILE, '/var/www/ip4.x/file/tmp'); //could be empty, but cause problems on some hosts
$answer = curl_exec($ch);
if (curl_error($ch)) {
echo curl_error($ch);
}
//another request preserving the session
curl_setopt($ch, CURLOPT_URL, 'http://www.example.com/profile');
curl_setopt($ch, CURLOPT_POST, false);
curl_setopt($ch, CURLOPT_POSTFIELDS, "");
$answer = curl_exec($ch);
if (curl_error($ch)) {
echo curl_error($ch);
}
I've seen this on ImpressPages
Yup, often called a 'cookie jar' Google should provide many examples:
http://devzone.zend.com/16/php-101-part-10-a-session-in-the-cookie-jar/
http://curl.haxx.se/libcurl/php/examples/cookiejar.html <- good example IMHO
Copying that last one here so it does not go away...
Login to on one page and then get another page passing all cookies from the first page along Written by Mitchell
<?php
/*
This script is an example of using curl in php to log into on one page and
then get another page passing all cookies from the first page along with you.
If this script was a bit more advanced it might trick the server into
thinking its netscape and even pass a fake referer, yo look like it surfed
from a local page.
*/
$ch = curl_init();
curl_setopt($ch, CURLOPT_COOKIEJAR, "/tmp/cookieFileName");
curl_setopt($ch, CURLOPT_URL,"http://www.myterminal.com/checkpwd.asp");
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, "UserID=username&password=passwd");
ob_start(); // prevent any output
curl_exec ($ch); // execute the curl command
ob_end_clean(); // stop preventing output
curl_close ($ch);
unset($ch);
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_COOKIEFILE, "/tmp/cookieFileName");
curl_setopt($ch, CURLOPT_URL,"http://www.myterminal.com/list.asp");
$buf2 = curl_exec ($ch);
curl_close ($ch);
echo "<PRE>".htmlentities($buf2);
?>

Enable SSL connection for https in cURL PHP (header blank)

I am writing a cURL script to access the current days interest rate from the Fannie Mae website which is https. I havent been able to get past the CURLOPT_SSL_VERIFYPEER, true); option.
No username or password is required, however I need SSL verification turned on.
Testing on XAMPP dev server.
I have downloaded the .crt and .pem certs from the website using FF and saved them in the same source dir and pointed to both using CURLOPT_CAINFO, no luck
I downloaded the latest cacert.pem file from http://curl.haxx.se/ca/cacert.pem and pointed to that as well using CURLOPT_CAINFO, no luck.
If I turn CURLOPT_SSL_VERIFYPEER, to false I can retrieve the header (see below), however when I set it to true there is no header.
Tried about 7-8 solutions found by searching on here along with reading the php documention on cURL and trying several workarounds listed there, no luck.
I need to be able to retrieve the header and eventually the body using CURLOPT_SSL_VERIFYPEER, true
Any help is appreciated.
<?php
// script is designed to access an https site and retrieve the last table showing the most recent 90 day commitment for the Fannie Mae 30 year fixed rate mortgage. Site is designed to work with cookies and has a valid SSL cert.
//turn error reporting on
error_reporting(E_ALL); ini_set("display_errors", 1);
// cookie file name/location
$cookie_file_path = "cookies.txt";
// verify if cookie file is accessible and writable
if (! file_exists($cookie_file_path) || ! is_writable($cookie_file_path))
{
echo 'Cookie file missing or not writable.';
exit;
}
// url connection
$url = "https://www.fanniemae.com/content/datagrid/hist_net_yields/cur30.html";
// Initiate connection
$ch = curl_init();
// Set cURL and other options
curl_setopt($ch, CURLOPT_URL, $url); // set url
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6"); // set browser/user agent
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); // automatically follow Location: headers (ie redirects)
curl_setopt($ch, CURLOPT_AUTOREFERER, 1); // auto set the referer in the event of a redirect
curl_setopt($ch, CURLOPT_MAXREDIRS, 5); // make sure we dont get stuck in a loop
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 10); // 10s timeout time for cURL connection
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); // allow https verification if true
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 2); // check common name and verify with host name
curl_setopt($ch, CURLOPT_SSLVERSION,3); // verify ssl version 2 or 3
curl_setopt($ch, CURLOPT_CAINFO, getcwd() . "VeriSignClass3PublicPrimaryCertificationAuthority-G5.pem"); // allow ssl cert direct comparison
curl_setopt($ch, CURLOPT_HEADERFUNCTION, 'read_header'); // get header
curl_setopt($ch, CURLOPT_NOBODY, true); // exclude body
curl_setopt($ch, CURLOPT_COOKIESESSION, TRUE); // set new cookie session
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path); // file to save cookies in
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path); // file to read cookies in
// grab URL and pass it to the browser
curl_exec($ch);
// close cURL connection, save cookie file, free up system resources
curl_close($ch);
// show header
function read_header($ch, $string) {
print "Received header: $string";
return strlen($string);
}
?>
This is the header that is received if CURLOPT_SSL_VERIFYPEER is set to false, blank if true
Received header: HTTP/1.1 200 OK
Received header: Date: Thu, 19 Sep 2013 00:40:16 GMT
Received header: Server: Apache
Received header: Set-Cookie: JSESSIONID=4297C1E1760A836F691FE821FBF8B805.cportal-cl01; Path=/; Secure; HttpOnly
Received header: Cache-Control: no-store
Received header: Expires: Wed, 31 Dec 1969 23:59:59 GMT
Received header: Pragma: no-cache
Received header: X-FRAME-OPTIONS: SAMEORIGIN
Received header: Content-Language: en-US
Received header: Content-Length: 9344
Received header: Content-Type: text/html;charset=ISO-8859-1
Received header:
You're excluding the body by using curl_setopt($ch, CURLOPT_NOBODY, true);. And I don't think you need to install certificate on your machine. The following few lines will give you everything.
$url = 'https://www.fanniemae.com/content/datagrid/hist_net_yields/cur30.html';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url); // set url
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.6) Gecko/20070725 Firefox/2.0.0.6"); // set browser/user agent
curl_setopt($ch, CURLOPT_HEADERFUNCTION, 'read_header'); // get header
curl_exec($ch);
function read_header($ch, $string) {
print "Received header: $string";
return strlen($string);
}

curl post from localhost

I'm trying to simulate http post from FireFox's LiveHTTP Replay by using curl. I believe the remote site has some sort of validation that checks where the request is coming from. If the request is coming from their own domain, then its fine. When i try to run a php curl script, i can see from the Live HTTP header that i'm making a GET request rather than a POST. Besides that, the Host, which is expected to be www.aliexpress.com has automatically changed to localhost.
If i use the Live HTTP Replay, it is working fine. So i copied the header data and tried to implement with curl but to no avail. Eg:
http://www.aliexpress.com/cross-domain/shoppingcart/index.html
POST /cross-domain/shoppingcart/index.html HTTP/1.1
Host: www.aliexpress.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:22.0) Gecko/20100101 Firefox/22.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
X-Requested-With: XMLHttpRequest
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
Referer: http://www.aliexpress.com/store/product/Wireless-N-Wifi-Repeater-802-11N-B-G-Network-Router-Range-Expander-300M-2dBi-Antennas-US/701252_523523529.html
Cookie: ali_apache_id=113.210.130.113.1374818286515.884332.4; ali_apache_track=mt=1|ms=|mid=my1023002521; xman_f=NkltWLLI3tebQbeQzQLiNBd2/KPKX0D81t0DghVMEl/frYuA+aVHnWGevMXWTEPqdLRqlKLbExYQkL61WPSt7Tr0LrdqOLLGM8yY5cBFOvY79qV9R5iTGSd44oPoKZruCpupEK9UBNSiOIf7Go1TN1AiM0ArpkHYTZ4rigCwLp5l2IEPYmFC8UzRnLivCFmLxbDuEewB52ulEop1Y9xtdEr88bjnwci1PldcvTxCmVDiOnm6rRfbnVfMAWaSWIkqQrnOEfwq2B4B/OER9K9IH7EHAMadb9IiOdMo3yavyt4DGWquCAq1izTtU8GE2mRmvi+PZ8WmR+PNOM3zYU4eaWM7uEevjmV2S7kTtlElmJGqxaT5RpSLcxiRxxbYJToejY36QxDf0MIIKTaaJTacVg==; aep_history=product_selloffer%5E%0Aproduct_selloffer%09709591781%091035163509%09523523529; __utma=3375712.263559759.1374818300.1375458795.1375606693.4; __utmz=3375712.1374818300.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); ali_beacon_id=113.210.130.113.1374818286515.884332.4; cna=+/14CsYcs08CAXGC0nFA22WC; xman_us_f=x_locale=en_US&no_popup_today=n&x_user=MY|dexter|wong|ifm|142465266&last_popup_time=1374818565653; u_info=qunLQLajxN+hFYWqPKiksew8tjAelFSu8cj+oG6e7nY=; intl_common_forever=wWZJ4jwSXakD7oylp5nnW9Nwmq8zgJYiqow0UyNV6PEUhc4f0KJghQ==; CoreID6=n; JSESSIONID=F86BD98D5E0CF42A7BE875F648DFA640; ali_apache_tracktmp=W_signed=Y; acs_usuc_t=acs_rt=9af4ec13bf134eb298193f9ac69395dc; acs_t=/iJST0zcbQeKUIQrTj1tDurMpZLQIdanO+zAZzyDlS+blTn+Rwd4skbiIdbQbEDO; __utmc=3375712; xman_t=rr/A0xwWzcNjpVptbDP061VCJ0dhjFwJPMn/JxOmi4eSjWlXq98nD8HBxnqOtR8ZIUClJqCqI39uwIkL6/R0WYQtBiqtFb8R0KGmzoiHDZ4R6dMhSZeEC5Am8y6iywMSG5My1MlUAhuWQI6/EPBlSYOWa8V/3IiNJnCOUd/Wm4DWQt6YHxS12kJbrUZxu2M7HeOquFa8Ga+yB/P0DT0Z9EhRum3S3uBC2+rFkh50z+91raLJiWJ0PV9NqHup3sPpAstiWlmem8QfBps0tFSx7tZn9WkllmyNJsTUYWO0cuxr0gpjWPU72Bb6fsroRovgRZ6xeqDah+WT94rnU2jrRybsL+7JDXmYPYC0GOTHjsSsloHSyGTvoD+FNyS3jGQPoP8KL7NXi+Dq+FrAqOETg3OH2oJp/h7nH5CWcsdojLHTngkABhNnB0ky/YRS8dV0s0oukEDPt+iXVEjQBBsIjAmtVX2fYx8KGRiRNiff/4rehQ4GDZzk2kdfJHItnUSk694SnpAgB6PrkNpGvu8adLjy8W6GuXk2XzujhsCSNkQ+3/uNpEbqoAimkCW+6KjJujJCPYIGevineVzSjMih7eDWpP/5TbgWtyhWKv3F5QbKZzibUq6w/YnerorvCHNPcssWgl0lswk=; __utmb=3375712.4.10.1375606693; xman_us_t=x_lid=my1023002521&sign=y&x_user=RmoP5to3fHwR+VNOC9lIAD7BpyTVa0YBflCR3S4eFIU=&need_popup=y
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Content-Length: 93
productId=523523529&standard=&quantity=7&country=MY&company=CPAM&cartfrom=main_store&skuAttr=
I've tried to implement the raw header(above) with the following:
<?php
// create a new cURL resource
$fields = array(
'productId' => 523523529,
'standard' => '',
'quantity' => 8,
'country' => 'MY',
'company' => 'CPAM',
'cartfrom' => 'main_store',
'skuAttr' => ''
);
foreach($fields as $key=>$value) { $fields_string .= $key.'='.$value.'&'; }
rtrim($fields_string, '&');
echo $fields_string . "<br/>";
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Host: www.aliexpress.com",
"Content-Type: application/x-www-form-urlencoded",
"Content-length: ". "93",
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language: en-US,en;q=0.5",
"Accept-Encoding: gzip, deflate",
"X-Requested-With: XMLHttpRequest")); //proceeding with the login.
curl_setopt($ch, CURLOPT_URL, urlencode("http://www.aliexpress.com/cross-domain/shoppingcart/index.html"));
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)");
//The encoded url below is referring to the login form for aliexpress.com
curl_setopt($ch, CURLOPT_REFERER, "http%3A%2F%2Fwww.aliexpress.com%2Fstore%2Fproduct%2FDual-sim-I9300-S3-MTK6589-quad-core-android-phone-1G-RAM-4G-ROM-4-7-inch%2F901666_1035163509.html%3FpromotionId%3D210526801");//This tells the server where were you directed from.
curl_setopt($ch,CURLOPT_POST, count($fields));
curl_setopt($ch,CURLOPT_POSTFIELDS, $fields_string);
//curl_setopt($ch, CURLOPT_COOKIESESSION, true);//indicates that this is a new session, i assume this forces the server to assign a new session?
//curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);//follows the redirection that is supplied by the server
curl_setopt($ch, CURLOPT_HEADER, true);
//curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);//THIS IS VERY IMPORTANT! This one of the most common option that is used because this simply means that
//the response from the server is returned as a string rather than output directly.
curl_setopt($ch, CURLOPT_UNRESTRICTED_AUTH, true);//This means to keep sending the login information(username and password) when there is a redirection
$str = curl_exec($ch);
// close cURL resource, and free up system resources
curl_close($ch);
?>
Thanks for your help!
Regards,
Dexter
try setting CURLOPT_POST to true
curl_setopt($ch, CURLOPT_POST, 1);
I think you should modify your code a bit:
remove the urlencode when setting the value of CURLOPT_URL
remove the Host and Content-length header from CURLOPT_HTTPHEADER
use http_build_query to build your $fields_string
For debugging purpose, I set the CURLOPT_RETURNTRANSFER to be true and var dump the respond.
My working code
$fields = array(
'productId' => 523523529,
'standard' => '',
'quantity' => 8,
'country' => 'MY',
'company' => 'CPAM',
'cartfrom' => 'main_store',
'skuAttr' => ''
);
$fields_string = http_build_query($fields);
echo $fields_string . "<br/>";
$ch = curl_init();
// set URL and other appropriate options
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Content-Type: application/x-www-form-urlencoded",
"Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Accept-Language: en-US,en;q=0.5",
"Accept-Encoding: gzip, deflate",
"X-Requested-With: XMLHttpRequest")); //proceeding with the login.
curl_setopt($ch, CURLOPT_URL, "http://www.aliexpress.com/cross-domain/shoppingcart/index.html");
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)");
//The encoded url below is referring to the login form for aliexpress.com
curl_setopt($ch, CURLOPT_REFERER, "http://www.aliexpress.com/store/product/Dual-sim-I9300-S3-MTK6589-quad-core-android-phone-1G-RAM-4G-ROM-4-7-inch/901666_1035163509.html?promotionId=210526801");//This tells the server where were you directed from.
curl_setopt($ch,CURLOPT_POST, count($fields));
curl_setopt($ch,CURLOPT_POSTFIELDS, $fields_string);
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);//THIS IS VERY IMPORTANT! This one of the most common option that is used because this simply means that
//the response from the server is returned as a string rather than output directly.
curl_setopt($ch, CURLOPT_UNRESTRICTED_AUTH, true);//This means to keep sending the login information(username and password) when there is a redirection
$str = curl_exec($ch);
var_dump($str);
var_dump(curl_error($ch));
// close cURL resource, and free up system resources
curl_close($ch);
And the respond
HTTP/1.1 200 OK
Date: Sun, 04 Aug 2013 11:51:31 GMT
Server: Apache
P3P: CP="CAO PSA OUR"
Content-Language: en-US
Vary: Accept-Encoding,User-Agent
Content-Encoding: gzip
X-XSS-protection: 1;mode=block
Content-Length: 56
Content-Type: plain/text;charset=utf-8
Set-Cookie: ali_apache_id=1.54.42.221.1375617091161.869918.6; path=/; domain=.aliexpress.com; expires=Wed, 30-Nov-2084 01:01:01 GMT
Set-Cookie: JSESSIONID=6EB1295945C27F8A2F788587D4C0E0A7; Path=/
Set-Cookie: ali_apache_track=; Domain=.aliexpress.com; Expires=Fri, 22-Aug-2081 15:05:38 GMT; Path=/
Set-Cookie: ali_apache_tracktmp=; Domain=.aliexpress.com; Path=/
Set-Cookie: acs_usuc_t=acs_rt=8fdfad47f53b46d489d0a905a5a9fb7c; Domain=.aliexpress.com; Path=/
Set-Cookie: xman_t=ZwO1ZDjGpaou2015+mejeWnS90vHjsN3YIDxbrXYOz/mbbJeIZM3q7Pw6ZGTygK2; Domain=.aliexpress.com; Path=/; HttpOnly
Set-Cookie: acs_t=2nqPb5i+QB7aDai5FXRM12xDJghxP4qjmcwPjwaXQ4SI6eV7eGpxjRGNjukEXuEW; Domain=.aliexpress.com; Path=/; HttpOnly
Set-Cookie: xman_f=MC/MUpjkYCKP+PRcAK43k9eQrTR+PE1rldMoChEUHVVlAUcYwh10BKJ0lxWlsPe4p+pYIPC/Vy4wIHJK8fiy4koUaF68CAolRC6UH7q0nmU5HcqWzgyjnA==; Domain=.aliexpress.com; Expires=Fri, 22-Aug-2081 15:05:38 GMT; Path=/; HttpOnly

Keeping session alive with Curl and PHP

I'm trying to connect to an API, authenticate a user and then view the user details. This is accomplished by first accessing the login endpoint at
http://api.example.com/login/<username>/<password>
to log in and then the following to view user details:
http://api.example.com/user/
This all works in a web browser. However, once I try to use Curl, the login works fine, but when attempting to view user details, I get back a 401, unauthorized error. I believe this is because Curl isn't saving the session cookies properly? Can someone point out why it isn't working and how to fix it? I've tried searching stack exchange, however, none of the solutions I've tried have worked for my situation. The code I'm using to curl the endpoints is shown below. Thanks!
define("COOKIE_FILE", "cookie.txt");
// Login the user
$ch = curl_init('http://api.example.com/login/joe/smith');
curl_setopt ($ch, CURLOPT_COOKIEJAR, COOKIE_FILE);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, true);
echo curl_exec ($ch);
// Read the session saved in the cookie file
echo "<br/><br/>";
$file = fopen("cookie.txt", 'r');
echo fread($file, 100000000);
echo "<br/><br/>";
// Get the users details
$ch = curl_init('http://api.example.com/user');
curl_setopt ($ch, CURLOPT_COOKIEJAR, COOKIE_FILE);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_HEADER, true);
echo curl_exec ($ch);
This code will output:
HTTP/1.1 200 OK Date: Mon, 22 Oct 2012 21:23:57 GMT Server: LiteSpeed Connection: close X-Powered-By: PHP/5.3.14 Set-Cookie: cfapi=f481129c9616b8f69cc36afe16466545; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Content-Type: application/json X-Powered-By: CFWAPI 0.1a Content-Length: 46 {"status":200,"msg":"Successfully Logged In."}
# Netscape HTTP Cookie File # http://curl.haxx.se/rfc/cookie_spec.html # This file was generated by libcurl! Edit at your own risk. api.example.com FALSE / FALSE 0 cfapi 94f63b07ccf7e34358c1c922341c020f
HTTP/1.1 401 Unauthorized Date: Mon, 22 Oct 2012 21:23:57 GMT Server: LiteSpeed Connection: close X-Powered-By: PHP/5.3.14 Set-Cookie: cfapi=a8eb015a7c423dde95aa01579c4729a4; path=/ Expires: Thu, 19 Nov 1981 08:52:00 GMT Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Pragma: no-cache Content-Type: application/json X-Powered-By: CFWAPI 0.1a Content-Length: 49 {"status":401, "msg":"You need to login first!"}
You also need to set the option CURLOPT_COOKIEFILE.
The manual describes this as
The name of the file containing the cookie data. The cookie file can
be in Netscape format, or just plain HTTP-style headers dumped into a
file. If the name is an empty string, no cookies are loaded, but
cookie handling is still enabled.
Since you are using the cookie jar you end up saving the cookies when the requests finish, but since the CURLOPT_COOKIEFILE is not given, cURL isn't sending any of the saved cookies on subsequent requests.
You have correctly used "CURLOPT_COOKIEJAR" (writing) but you also need to set "CURLOPT_COOKIEFILE" (reading)
curl_setopt ($ch, CURLOPT_COOKIEJAR, COOKIE_FILE);
curl_setopt ($ch, CURLOPT_COOKIEFILE, COOKIE_FILE);
This is how you do CURL with sessions
//initial request with login data
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://www.example.com/login.php');
curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Ubuntu Chromium/32.0.1700.107 Chrome/32.0.1700.107 Safari/537.36');
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, "username=XXXXX&password=XXXXX");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_COOKIESESSION, true);
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie-name'); //could be empty, but cause problems on some hosts
curl_setopt($ch, CURLOPT_COOKIEFILE, '/var/www/ip4.x/file/tmp'); //could be empty, but cause problems on some hosts
$answer = curl_exec($ch);
if (curl_error($ch)) {
echo curl_error($ch);
}
//another request preserving the session
curl_setopt($ch, CURLOPT_URL, 'http://www.example.com/profile');
curl_setopt($ch, CURLOPT_POST, false);
curl_setopt($ch, CURLOPT_POSTFIELDS, "");
$answer = curl_exec($ch);
if (curl_error($ch)) {
echo curl_error($ch);
}
I've seen this on ImpressPages
Yup, often called a 'cookie jar' Google should provide many examples:
http://devzone.zend.com/16/php-101-part-10-a-session-in-the-cookie-jar/
http://curl.haxx.se/libcurl/php/examples/cookiejar.html <- good example IMHO
Copying that last one here so it does not go away...
Login to on one page and then get another page passing all cookies from the first page along Written by Mitchell
<?php
/*
This script is an example of using curl in php to log into on one page and
then get another page passing all cookies from the first page along with you.
If this script was a bit more advanced it might trick the server into
thinking its netscape and even pass a fake referer, yo look like it surfed
from a local page.
*/
$ch = curl_init();
curl_setopt($ch, CURLOPT_COOKIEJAR, "/tmp/cookieFileName");
curl_setopt($ch, CURLOPT_URL,"http://www.myterminal.com/checkpwd.asp");
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, "UserID=username&password=passwd");
ob_start(); // prevent any output
curl_exec ($ch); // execute the curl command
ob_end_clean(); // stop preventing output
curl_close ($ch);
unset($ch);
$ch = curl_init();
curl_setopt($ch, CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_COOKIEFILE, "/tmp/cookieFileName");
curl_setopt($ch, CURLOPT_URL,"http://www.myterminal.com/list.asp");
$buf2 = curl_exec ($ch);
curl_close ($ch);
echo "<PRE>".htmlentities($buf2);
?>

cURL login problem

Using cURL to scrape a secure (i.e. login) page, and I'm at my wits' end. I managed to successfully scrape two sites with little or no problems, and now I just can't log into this one. cURL gets all the pages I ask it to, but they're all not logged in, which doesn't help. So maybe someone could spot a mistake I've missed?
The code is:
$url_to = 'http://fastorder.newrock.es/store2009/index.php/customer/account/loginPost/';
$url_from = 'http://fastorder.newrock.es/store2009/index.php/customer/account/login/';
$url_get = 'http://fastorder.newrock.es/store2009/index.php/';
$name_pass = 'login%5Busername%5D=*****&login%5Bpassword%5D=*****&send=';
function login($link,$user,$from) {
$fp = fopen("cookie.txt", "w");
fclose($fp);
$log = curl_init();
curl_setopt($log, CURLOPT_REFERER, $from);
curl_setopt($log, CURLOPT_URL, $link);
curl_setopt($log, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($log, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($log, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6");
curl_setopt($log, CURLOPT_TIMEOUT, 40);
curl_setopt($log, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($log, CURLOPT_HEADER, TRUE);
curl_setopt($log, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($log, CURLOPT_POST, TRUE);
curl_setopt($log, CURLOPT_POSTFIELDS, $user);
$data = curl_exec($log);
curl_close($log);
}
login($url_to,$name_pass,$url_from);
function get($url) {
$get = curl_init();
curl_setopt($get, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($get, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($get, CURLOPT_URL, $url);
return curl_exec ($get);
curl_close ($get);
}
$html = get($url_get);
echo $html;
This is the (more or less) same script that worked on the other two sites, and it manages to log in fine. What threw me off in the start are the codes in the $name_pass. Turns out the site has named name and password input fields as login[username] and login[password]. Why the hell for, I've no idea, but I've tried sending it both with codes and with brackets, and nothing helped.
Live HTTP Headers is giving me the following for the page:
http://fastorder.newrock.es/store2009/index.php/customer/account/loginPost/
POST /store2009/index.php/customer/account/loginPost/ HTTP/1.1
Host: fastorder.newrock.es
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 115
Connection: keep-alive
Referer: http://fastorder.newrock.es/store2009/index.php/customer/account/login/
Cookie: frontend=6tjul97q4mvn0046ier0k79li8
Content-Type: application/x-www-form-urlencoded
Content-Length: 81
login%5Busername%5D=*****&login%5Bpassword%5D=*****&send=
HTTP/1.1 302 Found
Date: Fri, 26 Feb 2010 12:29:19 GMT
Server: Apache/2.0.63 (CentOS)
X-Powered-By: PHP/5.2.10
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Location: http://fastorder.newrock.es/store2009/index.php/customer/account/
Content-Length: 0
Connection: close
Content-Type: text/html; charset=UTF-8
I've tried to copy everything I could to the cURL script, thinking there's some obscure way of blocking the scrip from logging in. But right now I'm totally stuck and I've got no idea what to do next. And I've dug through a lot of tutorials, and they all give advices that worked like a charm for the first two sites.
Halp?
It may be this:
login%5Busername%5D=*****&login%5Bpassword%5D=*****&send=
I'm no curl guru, but your script seems to be OK, so maybe you should not escape the characters.
I would do local tests with curl and this kind of login forms. Maybe you can debug what's wrong from there. If I'm right, there will be empty fields.
Suggestion: Use Fiddler (www.fiddler2.com) to diff the request traffic, CURL vs your browser.
There is something broken with that store's registration/login. The activation email said to just login to activate the account. I've tried logging in multiple times but I get the error "This account is not activated." everytime I try to login.
Below is a quick change that prints the returned login page.
$url_to = 'http://fastorder.newrock.es/store2009/index.php/customer/account/loginPost/';
$url_from = 'http://fastorder.newrock.es/store2009/index.php/customer/account/login/';
$url_get = 'http://fastorder.newrock.es/store2009/index.php/';
$name_pass = 'login%5Busername%5D=*****&login%5Bpassword%5D=*****&send=';
function login($link,$user,$from) {
$fp = fopen("cookie.txt", "w");
fclose($fp);
$log = curl_init();
curl_setopt($log, CURLOPT_REFERER, $from);
curl_setopt($log, CURLOPT_URL, $link);
curl_setopt($log, CURLOPT_COOKIEJAR, "cookie.txt");
curl_setopt($log, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($log, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6");
curl_setopt($log, CURLOPT_TIMEOUT, 40);
curl_setopt($log, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($log, CURLOPT_HEADER, TRUE);
curl_setopt($log, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($log, CURLOPT_POST, TRUE);
curl_setopt($log, CURLOPT_POSTFIELDS, $user);
$data = curl_exec($log);
curl_close($log);
return $data;
}
echo login($url_to,$name_pass,$url_from);
function get($url) {
$get = curl_init();
curl_setopt($get, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($get, CURLOPT_COOKIEFILE, "cookie.txt");
curl_setopt($get, CURLOPT_URL, $url);
return curl_exec ($get);
curl_close ($get);
}
$html = get($url_get);
echo $html;
Edit:
Is the cookies data is being written to the cookies file (cookie.txt)? If not...
Check the file permissions, make sure its writable.
A bug in earlier versions of php5 caused the cookies file option to be ignored.
Details on the bug are here: http://bugs.php.net/bug.php?id=33475
Solution: Add unset($log) after curl_close($log);
Its hard to debug this script w/o being able to test it.

Categories