I would like to know if it's possible to add a cookie with name, value, domain, path, secure, http only and expiry before exec the curl.
I'm looking for it and what I found was only some ways to set the name and value of the cookie. And I also found a lot of ways to add it by using a file, but I would like to add the cookie without the file.
Another question related to the topic:
If I init the curl to make a GET request and then without close the curl I make a POST. Is it possible to use the cookies that the GET request has received to make the POST (without file)?
You can do this using the CURLOPT_COOKIE cookie option with curl_setopt. Example:
<?php
curl_setopt($ch, CURLOPT_COOKIE, "<cookie-name>=<cookie-value>; Domain=<domain-value>; Path=<path-value>; Secure; HttpOnly; Expires=<date>");
more on how to format the cookie header can be found on the man docs https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Set-Cookie
For your second question you simply need to set the CURLOPT_COOKIEFILE to an empty string to enable cookie
<?php
curl_setopt($curl, CURLOPT_COOKIEFILE, "");
Related
I am trying to get content of the page: I use google dev tool (network) and use "copy as curl" which gives me:
curl 'http://www.example.com/default.aspx/GetAnnonces' -H 'Cookie: `ASP.NET_SessionId=eolrcogrk1owhmpbsogwd0mf; EPC_alerte=;`
This works fine for a while, I guess beacuse of the session life period.
My question is:
Where the SessionId "eolrcogrk1owhmpbsogwd0mf" comes from and how to generate it so I can access the page any time ?
It comes from the Set-Cookie HTTP response header of the page you visited. If you're trying to use cURL in PHP it will automatically handle cookies for you and you can set CURLOPT_COOKIEJAR with curl_setopt to retain cookies even after the request is complete.
If you just want to see the response headers you could also use curl_setopt($handle, CURLOPT_HEADER, true) and look at the Set-Cookie response headers. Though there's no practical reason for doing this for most typical use cases since cURL will just handle the cookies for you like your browser would.
I need to authenticate my user through a curl script
session_start();
$_POST["username"]= "user";
$_POST["password"]= "password";
$ch = curl_init();
$url = 'signin.php';
curl_setopt($ch,CURLOPT_URL, $url);
curl_setopt($ch,CURLOPT_POST, count($_POST));
curl_setopt($ch,CURLOPT_POSTFIELDS, $_POST);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
$result = json_decode(curl_exec($ch),true);
curl_close($ch);
The signin.php makes another curl call to an api, I made sure that signin.php returns all required information, sets all required session variables, returns an array:
echo json_encode(array(
'success' => true,
'ALLSESSION' => $_SESSION,
'error'=> ""
));
the ALLSESSION is returning the correct session variables, but they are not accessible directly, I mean I cant use $_SESSION["userid"], its not existent in the array of sessions.
How to preserve the session between the 2 pages?
Thanks
The problem is that the client is not remembering/transmitting the PHP session id.
When an HTTP client makes a request to a php script (via an HTTP server), it must include the session id in the request if it wishes to continue a previously started session. This can be done either in the HTTP headers as a cookie or as a URL parameter (named PHPSESSID by default).
If you do not want to use PHP's default session variable name, or if you want to use a POST variable instead of a URL parameter, then you can use any request variable or URL parameter you wish (whether it be GET, POST, or COOKIE), but then you will need to manually interpret this variable on the server-side.
Here are three solutions, in order of most recommended to least recommended.
Turn on cookie support in cUrl or
Pass the session id as a URL parameter or
Pass the session id as a request variable (post/cookie) or a URL parameter that does not use the name expected by PHP, and then manually start the session on the server-side using that session id.
Solution #1: Turn on cookie support in cUrl
PHP uses the session id in the cookie to reload your session data each time you make a request from that client.
In this case, the client is cUrl. You need to setup your cUrl request to allow/use cookies.
This is done by setting the CURLOPT_COOKIEJAR and CURLOPT_COOKIEFILE options.
session_start();
$_POST["username"]= "user";
$_POST["password"]= "password";
$ch = curl_init();
$url = 'signin.php';
//Name of a file to store cookie data in.
//If the file does not exist, it will be created.
//cUrl (or your web server) needs to have write permissions to the folder.
$cookieFile = "/some/writable/folder/filename";
curl_setopt($ch,CURLOPT_URL, $url);
curl_setopt($ch,CURLOPT_POST, count($_POST));
curl_setopt($ch,CURLOPT_POSTFIELDS, $_POST);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
//Tell cUrl about the cookie file
curl_setopt($ch,CURLOPT_COOKIEJAR, $cookieFile); //tell cUrl where to write cookie data
curl_setopt($ch,CURLOPT_COOKIEFILE, $cookieFile); //tell cUrl where to read cookie data from
$result = json_decode(curl_exec($ch),true);
curl_close($ch);
Any subsequent cUrl calls that use $cookieFile for CURLOPT_COOKIEJAR and CURLOPT_COOKIEFILE will have the same session data as prior calls.
Solution #2: Pass the session id in the URL query string using the expected parameter name (PHPSESSID by default, but this can be changed)
You can append the session id to all urls like this:
somepage.php?PHPSESSID=sessionidgoeshere
"PHPSESSID" is the variable name that is used by default in PHP. If the server is setup to use a non-default name, then you would need to use that variable name instead.
With solution #2, you will still need to store the session id on the client-side somehow.
Solution #3: Pass the session id as a request variable or a URL parameter and then manually start the session on the server-side using that session id.
This solution is not recommended for normal situations. Unlike the previous solutions, this one requires changes to the server-side script as well as the client-side (cUrl). This solution is only useful if you specifically want to send the session id as something other than a URL parameter or cookie, or if you want to use a variable name other than the name that the server is expecting.
Place the following code in your server-side PHP that is handling the request, prior to starting the session:
session_id($_POST[<param_name>]); or session_id($_GET[<param_name>]); or session_id($_COOKIE[<param_name>]);
I suggest using Solution #1 unless you have a compelling reason not to.
Also, PHP doesn't care whether the request is a GET or a POST or any other HTTP request method. Regardless of the HTTP request method, if the session id is passed as a URL parameter or in a cookie, then the related session will persist on the server-side.
I was wondering if someone knew the equivalent of doing (from terminal):
curl --cookie "session_id=12345" http://www.example.com
Using CURL in php. I would prefer to do it without using a cookies.txt file by just doing the php curl calls by passing a cookie key/value pair. Please let me know if this makes sense, otherwise I can clarify further. I'm using this to connect to an API that requires sending a session variable via a cookie.
MORE CLARIFICATION:
The spec specifies this...
"The first thing that has to be done is to login. The response has a session id in it. This should be stored and used for subsequent calls. This should be added as a cookie, session_id, for further calls into the API."
You want CURLOPT_COOKIE as specified in the curl_setops page.
$ch = curl_init('http://www.example.com');
curl_setopt($ch, CURLOPT_COOKIE, 'session_id=12345');
curl_exec($ch);
curl_close($ch);
For multiple cookies, separate with a semicolon and a space:
curl_setopt($ch, CURLOPT_COOKIE, 'session_id=12345; fruit=apple');
You may be looking for the following flags:
CURLOPT_COOKIESESSION
And:
CURLOPT_COOKIE
CURLOPT_COOKIEFILE
CURLOPT_COOKIEJAR
Good evening!
I've an script in PHP which makes a CURL call to a remote host loggin page.
After loggin in and keeping the session via cookiejar opt and cookiefile opt, I use the same CURL connection handler to loggin in on to the immediatly next page wich needs an upload.
When it's done, I got the full session parameters and I can call any page I want from the site, but IN CURL!
The idea, is that this script wich uses CURL, needs to finally be redirected to one of those pages in the remote host using the CURL session, but this is not possible, because from curl you can not show the results as a redirected page.
So I've tried alot of options. None of em works at all.
Schema:
PHP script on a local server.
Call to domain.com/loggin.php (creates curl ch)
Keep curl session on cookie.txt file.
Call to domain.com/loggin_2.php with the same ch (non closed last one).
Full logged in on the remote site.
Back to the PHP script. Need to redirect to domain.com/index.php, wich needs Session variables filled in with the full login process.
What to do then?
1) After having full loggin in, read cookies.txt file to get PHPSESSID.
Then tried to use setcookie(), or via header("Set-cookie: ...") and immediatly after, using header("Location: domain.com/index.php").
Doesn't work.
2) Tried same thing via ajax call and finally document.cookie = ...
Doesn't work.
3) Adding a third cURL call to a file in my remote host wich prints a JSONED $_SESSION.
Getting it on my PHP script, decoding it and loaded on my local session via foreach on any array value (foreach()...$_SESSION[$c] = $v).
Added a session_start() before this foreach. And immediatly after, a header("Loaction: domain.com/index.php").
Doesn't work.
4) Added a session_write_close() before the header("Loaction: domain.com/index.php").
Doesn't work.
So I don't really know how to use the CURL session.
I've tried to manually fix the PHPSESSID via Web Developer Firefox plugin. And I wrote down the curl generated session id. It perfeclty works. So, It should be possible to fix it via scripting on my php script! But I can't!
Give me a hand, please!
Thanks!
I may have gotten lost a bit, but I think I understand.
You can use
CURLOPT_HEADER for some debugging (will contain current redirected page info)
and CURLOPT_FOLLOWLOCATION like so:
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, 'http://domain.com/login.php');
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
I also use
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
to return as a string, which is much more useful for debugging, or parsing.
URL1: https://duapp3.drexel.edu/webtms_du/
URL2: https://duapp3.drexel.edu/webtms_du/Colleges.asp?Term=201125&univ=DREX
URL3: https://duapp3.drexel.edu/webtms_du/Courses.asp?SubjCode=CS&CollCode=E&univ=DREX
As a personal programming project, I want to scrape my University's course catalog and provide it as a RESTful API.
However, I'm running into the following issue.
The page that I need to scrape is URL3. But URL3 only returns meaningful information after I visit URL2 (it sets the term there Colleges.asp?Term=201125), but URL2 can only be visited after visiting URL1.
I tried monitoring the HTTP data going to and fro using Fiddler and I don't think they are using cookies. Closing the browser instantly resets everything, so I suspect they are using Session.
How can I scrape URL 3? I tried, programatically, visiting URLs 1 and 2 first, and then doing file_get_contents(url3) but that doesn't work (probably because it registers as three different sessions.
A session needs a mechanism to identify you as well. Popular methods include: cookies, session id in URL.
A curl -v on URL 1 reveals a session cookie is indeed being set.
Set-Cookie: ASPSESSIONIDASBRRCCS=LKLLPGGDFBGGNFJBKKHMPCDA; path=/
You need to send this cookie back to the server on any subsequent requests to keep your session alive.
If you want to use file_get_contents, you need to manually create a context for it with stream_context_create for to include cookies with the request.
An alternative (which I would personally prefer) would be to use curl functions conveniently provided by PHP. (It can even take care of the cookie traffic for you!) But that's just my preference.
Edit:
Here's a working example to scrape the path in your question.
$scrape = array(
"https://duapp3.drexel.edu/webtms_du/",
"https://duapp3.drexel.edu/webtms_du/Colleges.asp?Term=201125&univ=DREX",
"https://duapp3.drexel.edu/webtms_du/Courses.asp?SubjCode=CS&CollCode=E&univ=DREX"
);
$data = '';
$ch = curl_init();
// Set cookie jar to temporary file, because, even if we don't need them,
// it seems curl does not store the cookies anywhere otherwise or include
// them in subsequent requests
curl_setopt($ch, CURLOPT_COOKIEJAR, tempnam(sys_get_temp_dir(), 'curl'));
// We don't want direct output by curl
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
// Then run along the scrape path
foreach ($scrape as $url) {
curl_setopt($ch, CURLOPT_URL, $url);
$data = curl_exec($ch);
}
curl_close($ch);
echo $data;