So a little trivia first..
There is written in ASP.NET website, which uses NTLM protocol to authenticate users that want to log in. It's perfectly ok when they normally use it, they type in website URL, they provide their credentials, authenticate and maintain session in web browser.
What I want to do, is create PHP website that will act as bot. It is my companys internal website and I am approved to do so. The problem I run into, is managing session. Users will be able to type in their credentials in my PHP website, and my PHP website will authenticate them to target site, using cURL.
The code I got so far is:
$cookie_file_path = dirname(__FILE__) . '/cookies.txt';
$ch = curl_init();
//==============================================================
curl_setopt($ch, CURLOPT_USERPWD, $username. ':' . $password);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_FAILONERROR, 0);
curl_setopt($ch, CURLOPT_MAXREDIRS, 100);
//=============================================================
$ret = curl_exec($ch);
Above code logs in to target website by cURL (which manages NTLM handshake, as it seems), and fetches websites content. It also stores Session ID that is sent back in cookie file.
What I'm trying to do next, is comment the CURLOPT_USERPWD option, in hope that this script will use session ID stored in cookie file to authenticate previously logged in user in second execution of this script. It could get rid of user credentials and do not store it anywhere that way, becouse it is not safe to store it in manually created session, database, or anywhere else.
I need this becouse bot will be using CRON to periodically check if website status has changed and perform some user actions as reaction to this. But to do this, user first must be authenticated, and his username and password must not be stored anywhere, so I have to use session information estabilished when he initially logged in.
CURL seems to NOT DO THIS. When I execute script second time with commented CURLOPT_USERPWD option, it does not use stored cookie to keep beeing authenticated. Instead, it REWRITES cookie file with not relevant data send to me from service as response to NOT AUTHRORISED access request.
My questions are:
Why cURL doesnt use stored session information to keep beeing authenticated?
Is there any way to maintain this session with cURL and NTLM protocol based website?
Thanks in advance.
A few Month ago I had a similar problem then you. I tried to get a connection to a navision soap api. Navision use the ntlm authentication. The problem is that curl doesn't native support ntlm so you have to do it yourself.
A blog post that helped me a lot in this situation was the following:
http://rabaix.net/en/articles/2008/03/13/using-soap-php-with-ntlm-authentication
** Edit
Sorry i misread you question.
You problem is simple.
Just receive the header from a request with this line
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
You can then get from the result of curl_exec function, the Set-Cookie header.
preg_match('/^Set-Cookie:\s*([^;]*)/mi', $ret, $match);
$cookie = parse_url($match[0]);
Now you can store it somewhere, and use it on the 2ten request.
I have the same problem and i solved it using curl_setopt($ch, CURLOPT_COOKIEFILE, ""); line of code. The string should be exactly empty.
Related
I'll try to explain the issue, I don't know if it is doable...
If you login here (1st link):
https://profile.ea.com/
And then go here (2nd link):
https://accounts.ea.com/connect/auth?client_id=sparta-companion-web&response_type=code&prompt=none&redirect_uri=nucleus:rest
You will find a code like this only if you are logged (token):
{"code":"QUORAL0aEYq2RjJGJwFEIddI99wM_FaZ_FgktceQ"}
That token is what I need to make some (not documented) API calls work from my web app (cURL with PHP).
I'm trying to emulate what I do when I execute a login but it seems to fail every time and I can't understand why...
This is my 1st call for the 1st link:
<?php
$username = urlencode('myaccount#mail.test');
$password = 'Mysecretpassword';
$event = 'submit';
$loginUrl = 'https://profile.ea.com/';
//init curl
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $loginUrl);
curl_setopt($ch, CURLOPT_POST, 1);
//curl_setopt($ch, CURLOPT_VERBOSE, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, 'email='.$username.'&password='.$password.'&_eventId='.$event);
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookie.txt');
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$firstlogin = curl_exec($ch);
//var_dump($firstlogin);
//login done?
Then the token part:
//get the token
curl_setopt($ch, CURLOPT_URL, 'https://accounts.ea.com/connect/auth?client_id=sparta-companion-web&response_type=code&prompt=none&redirect_uri=nucleus:rest');
//execute the request
$token = curl_exec($ch);
var_dump($token);die;
The main issue is the 1st part:
The profile.ea link seems to do a redirect to a unique url, maybe the CURLOPT_FOLLOWLOCATION is not enough?
The data needed to login is an array with 'email', 'password' and '_eventId' but I can't find any other required field.
What I'm doing wrong? Why the login is not working? How can I debug what is not working?
Probably the issue is that when you hit first url with code $firstlogin = curl_exec($ch);. It essentially does not mean that you got logged in for every request now. The next url hit needs to know, who you are before sending you the data, and they are possibly using cookies for this identification. Try simulating it in a browser. Probably, with the first URL request, there are some cookies returned after logging in, which are then forwarded with the next request.
You need to replicate cookies with your curl too. Try to extract any cookies being set with login cURL with the code at this link.
Then forward the cookies with your cURL request using curl_setopt($ch, CURLOPT_HTTPHEADER, array("Cookie: test=cookie"));
If above doesn't work, try replicating the same scenerio at browser and find out all the cookies in browser. Then replicate cookies already set and the ones being set by the login cURL request. This should probably work.
you're doing lots of things wrong,
you don't urlencode $username, but you need to. you don't urlencode $password, but you need to. you don't urlencode $event, but you need to. you don't use, nor try to fetch, the csrf token (called execution) prior to sending the login request, that won't work. you try to login without a pre-existing cookie session, that won't work, the cookie session and the csrf token are tied together, if 1 of them are missing/incorrect, your login won't be successful, and your code fetches neither of them. you're also missing a lot of login post parameters, including phoneNumber, passwordForPhone, _rememberMe, and several others, add them all.
I'm trying to record data from Philippine Stock Exchange website. I have found that they have an endpoint which is http://www.pse.com.ph/stockMarket/companyInfo.html?method=fetchHeaderData&company=29&security=146
I can clearly access it using any browsers except when I go into incognito mode where I'm being shown with a content saying Access Denied and it never stops loading. When I try to access it using PHP I'm quite sure that what is happening is the same as the later.
I'm trying to access it using PHP to no avail, here are the attempts I tried:
file_get_contents
cURL with user agent
cURL with temporary cookies
Tried all in localhost and in live server.
Code:
$c = tempnam ("/tmp", "CURLCOOKIE");
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://www.pse.com.ph/stockMarket/companyInfo.html");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_COOKIESESSION, true );
curl_setopt($ch, CURLOPT_COOKIEJAR, $c);
curl_setopt($ch, CURLOPT_COOKIEFILE, $c);
curl_setopt($ch, CURLOPT_POSTFIELDS, "method=fetchHeaderData&ajax=true&company=29&security=146");
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER["HTTP_USER_AGENT"]);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
var_dump(curl_exec($ch));
curl_close ($ch);
I don't have any clear idea on why and how does this happen. Can someone explain to me why it happens and what are the possible solutions (PHP only if possible)
I have reviewed other developer's approach on this API (They all implemented it using Java) and it is just a simple POST request and it is done. I have not verified though if their code is still working. I can't post links to their repository (limited).
SOLUTIONS:
Problem 1. Can't access API
$posts = array(
"method"=>"fetchHeaderData",
"ajax"=>"true",
"company"=>29,
"security"=>146
);
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://www.pse.com.ph/stockMarket/companyInfo.html");
curl_setopt($ch, CURLOPT_POSTFIELDS,$posts);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
var_dump(curl_exec($ch));
curl_close ($ch);
It seems I have two different problems. I can now access and use the API using the code above. No need for other options. Turning the post data into array fixed the problem.
Problem 2. Access Denied
On the problem about the Access Denied, it is cookie related. Answered below by #Wayne.
Unfortunately, I can't accept two answers.
Try this solution. convert your post data in array then pass this array in CURLOPT_POSTFIELDS
$posts = array(
"method"=>"fetchHeaderData",
"ajax"=>"true",
"company"=>29,
"security"=>146
);
$c = tempnam ("/tmp", "CURLCOOKIE");
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "http://www.pse.com.ph/stockMarket/companyInfo.html");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_COOKIESESSION, true );
curl_setopt($ch, CURLOPT_COOKIEJAR, $c);
curl_setopt($ch, CURLOPT_COOKIEFILE, $c);
curl_setopt($ch, CURLOPT_POSTFIELDS,$posts);
curl_setopt($ch, CURLOPT_USERAGENT, $_SERVER["HTTP_USER_AGENT"]);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
var_dump(curl_exec($ch));
curl_close ($ch);
It is because they have their server setup to stop you from doing that. They are securing the data with a cookie.
Cookie details
When you visit the site http://www.pse.com.ph/stockMarket/companyInfo.html it gives you a cookie as it knows you are a human visitor.
In your browser tools enter
document.cookie
to see your cookie. It will provide you an individual the data because you have the cookie.
Remove the cookie
document.cookie = "JSESSIONID=; expires=Thu, 01 Jan 1970 00:00:00 UTC; path=/;";
and visit
http://www.pse.com.ph/stockMarket/companyInfo.html?method=fetchHeaderData&company=29&security=146
without going to get a cookie http://www.pse.com.ph/stockMarket/companyInfo.html first you will get the 403 (Forbidden)
Also they do not have jsonp with a callback so an ajax request will violate the cross domain security. Requests for the JSON must be from pages that originate from their domain or an approved domain.
Why would they do that.
Likely their licence to the information does not allow them to give it to other websites, or they need/want to get paid to provide the information to other websites. Or they have terms of use for the information.
Where can you get the data ... data wants to be free
I don't see anyplace on their site http://www.pse.com.ph where they have API information and how to request permission to access it.
Programable web has been the number one source for finding APIs, they have 96 stock APIs listed ... Obviously I can not just copy their data and past it here, but one of these API may work for you?
I am trying to use a button on my php web-application to launch a logged-in session on another website. In other words I want my application to:
open a new tab/window (achieved)
go to another website + login or
(alternatively) collect the session data needed for the target site to consider the current browser logged in.
This is achieved (in an incomplete manner with the following code:
$ch = curl_init();
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie);
curl_setopt($ch, CURLOPT_URL,$url);
curl_setopt($ch, CURLOPT_USERAGENT, $agent);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS,$postdata);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION,1);
curl_setopt($ch, CURLOPT_REFERER, $url);
$result = curl_exec ($ch);
curl_close ($ch);
print $result;
This successfully visualise the "logged-in" page of the remote site but whenever I click on any of the functionalities of such remote site I get and obvious 404. This is because I am just printing the output of the successful login via cURL and my browser is not dealing with the remote application on the target website. E.g. my address bar says I am in local.dev/loggedin.php instead of being at secure.targetsite.com/loggein.php.
This maybe helpful: Once logged-in via the browser, the target website sets a session cookie that allows the session to survive for a certain amount of time so that may also be useful. Can my web-application just fetch and store the session data from the auth procedure carried out by curl and use it to login?
This might not be possible to be done via cURL..
I was thinking of just parsing the response header for the cookie and use php setcookie() but it does not work: I get bounced by the remote app as if I was never logged in.
Please be patient, I am not an expert in the use of curl.
I have done that for a few of my own applications, but it should work for almost anything that can be logged in via an html form submission. You can't use curl for this because it is running on your web server (whether that is on your local machine or in the cloud somewhere is irrelevant) and not actually being run by your browser. Your PHP application needs to open a new tab/window with a page that includes an HTML that includes all necessary fields, method="get" or "post" as appropriate, and action="the destination login URL". Then just add an automatic form submission - e.g., with jQuery $('#form_id').submit() on page load.
I've got NTLM (Active Directory) based service, and I need to write a PHP application. Normally, users are logging in to website with Activre Directory credentials, and it's ok.
But what I want to do, is to let them type in their credentials to PHP-written site, which in next step will use cURL to authenticate users to that Active Directory based site where they normally log in.
And this part is hard. I need then to keep session of users that through PHP cURL script authenticated to Active Directory based site in order to use them again later
(CRON querying site to determine that it has changed and automatically do some operations when this happens, which normally user has do manually).
In order to NOT store their credentials to authenticate again when this change happens, I somehow need to store NTLM session in PHP cURL site to every user that authenticated to
that site through this PHP cURL site.
My question is: Is that even possible?
Thanks in advance.
#Willem Mulder
The code you've posted actually does cookie-storing, but that is not my point becouse I've already done that (sorry for not writing it before). What I got so far is:
$cookie_file_path = dirname(__FILE__) . '/cookies.txt';
$ch = curl_init();
//==========================================================================
curl_setopt($ch, CURLOPT_USERPWD, $username. ':' . $password);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, 0);
curl_setopt($ch, CURLOPT_USERAGENT, $user_agent);
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, 0);
curl_setopt($ch, CURLOPT_HTTPAUTH, CURLAUTH_ANY);
curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie_file_path);
curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie_file_path);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLINFO_HEADER_OUT, true);
curl_setopt($ch, CURLOPT_FAILONERROR, 0);
curl_setopt($ch, CURLOPT_MAXREDIRS, 100);
//==========================================================================
$ret = curl_exec($ch);
By using options CURLOPT_COOKIEFILE and CURLOPT_COOKIEJAR, cURL does the cookie storing in local file "cookies.txt". The problem is, that when I comment CURLOPT_USERPWD option (after authenticating and storing cookie, so theoretically I have session), I cannot authorize to website. Perhaps it reinitializes NTLM Handshake authorisation and is expecting username and password, which I don't want to store.
I want to store session info only, to provide service this session info and omit second authentication, but cURL seems to not take this data from cookie file, and REWRITES it with not relevant data send to me from service as response to NOT AUTHRORISED access request.
Well, yes you could
$ch = curl_init('http://www.google.com/');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// Get headers too with this line
curl_setopt($ch, CURLOPT_HEADER, 1);
$result = curl_exec($ch);
// Get cookie
preg_match('/^Set-Cookie:\s*([^;]*)/mi', $result, $m);
var_dump(parse_url($m[1]));
// And then of course store it somewhere :-)
As seen here how to get the cookies from a php curl into a variable
I'm using a web-service from a provider who is being a little too helpful in anticipating my needs. They have given me a HTML snippet to paste on my website, for users to click on to trigger their services. I'd prefer to script this process, so I've got a php script which posts a cURL request to the same url, as appropriate. However, this provider is keeping tabs on my session, and interprets each new request as an update of the first one, rather than each being a unique request.
I've contacted the provider regarding my issue, and they've gone so far as to inform me that their system is working as intended, and that it's impossible for me to avoid using the same ASP.NET session for each subsequent cURL request. While my favored option would be to switch to a different vendor, that doesn't appear to be an option right now. Is there a reliable way to get a new ASP.NET session with each cURL request?
I've tried the following set of CURLOPT's, to no avail:
//initialize curl
$ch = curl_init($url);
//build a string out of the post_vars
$post_str = http_build_query($post_vars);
//set the necessary curl options
curl_setopt($ch, CURLOPT_TIMEOUT, 30);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_FAILONERROR, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, $post_str);
curl_setopt($ch, CURLOPT_FAILONERROR, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_COOKIESESSION, 1);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, 1);
curl_setopt($ch, CURLOPT_FORBID_REUSE, 1);
curl_setopt($ch, CURLOPT_USERAGENT, "UZ_".uniqid());
curl_setopt($ch, CURLOPT_REFERER, CURRENT_SITE_URL."index.php?newsession=".uniqid());
curl_setopt($ch, CURLOPT_HTTPHEADER, array("Pragma: no-cache", "Cache-Control: no-cache"));
//execute the call to the backend script, retrieve the results
$xmlstr = curl_exec($ch);
If cURL isn't helping much, why not try other methods to call the services from your script, like php's file() function, or file_get_contents().
If you see do not see any difference at all, then the service provider might be using your ip to track your requests. Try using some proxy for a test.
Normal Asp.net session is tracked by a cookie called ASP.NET_SessionId. This cookie is sent within the response to your first request. So as long as your curl requests don't send back this asp.net cookie, each of your requests will have no connection to each other. Use the curl -c option to see what cookies are flying in-between you and them. Overriding this cookie with a cookie file should work if you confirm that it is normal asp.net session being used here.
It is quite poor for a service to use session (http has much cleaner ways of maintaining state which ReST exploits) so I wouldn't completely rule out the vendor switch option.
Well given the options you are using, it seems you have covered your basics. Can you find out how their sessions are setup?
If you know how they setup a session, IE what they use (if it is IP or what not) and then you can figure out a work around. Another option is trying to set the cookies in a different cookie file:
CURLOPT_COOKIEFILE - The name of the file containing the cookie data. The cookie file can be in Netscape format, or just plain HTTP-style headers dumped into a file.
But if all they do is check cookies your current code should work. If you can figure out what the cookie's name is, you can pass a custom cookie that is blank with the request to see if that works. But if you can get information out of them on how their session's work, that would be best.
use these two line to handle the session:
curl_setopt($ch, CURLOPT_COOKIEJAR, "path/to/cookies.txt"); // cookies.txt should be writable
curl_setopt($ch, CURLOPT_COOKIEFILE, "path/to/cookies.txt");