Here is the problem we have been facing for the past few weeks.
1/ Our setup
PHP 5.4 + MySQL
2 dedicated servers, load-balanced
Sessions are replicated between the 2 servers using memcached
3 applications running on these servers :
One custom-developped application, using default php session settings
Another custom-developped application, using different session settings (cookie name, path)
One Wordpress CMS
2/ The problem
The problem occurs on our first application.
Some of our users reported that they sometimes get disconnected after a few minutes (when the session is setup to last 3 hours). It can happen to them several time in the same day, then no disconnection for a few days, but the problem always comes back.
So far the fraction of users impacted is small, but I would like to solve this before it "spreads" to other users.
The problem seems to occur in different places of the application, though we have identified 3 scenarii where most of the errors occur :
Some involve submitting a form ($_SESSION variable is modified)
Other simply involve opening a popup page, with no modification of the session data
We have tried to reproduce the different scenarii described by the users : sometimes we have been able to, but most of the time we don't have any problem, which makes it hard to debug.
Other notes :
The problem is recent, this application had been running for years without any problem.
It doesn't seem to be related to our server load, because the problem still occured during the summer break when our trafic was low
It only affects one session/users at a time: all the other users logged in at the same time don't experience this problem
The problem occured on all the different browsers (IE, Firefox, Chrome)
3/ Technical analysis
When a disconnect occurs, the user is redirected to a page "Your session has expired or you don't have the right to view". When this page is loaded, we get a technical email with a dump of the $_SESSION variable.
When a session expires the normal way, the email we get shows that the $_SESSION variable is empty (normal behavior).
When an unexpected disconnect occurs, what is interesting is that the $_SESSION is not entirely empty : out of the ~20 elements the array contained, only one is left (always the same).
So this would mean the session is not expired, but not enough data is left to "identify" the user, hence the "no rights" page displayed. As a confirmation when this occurs, we can check in memcached that this session still holds some data.
These are the potential problem causes we have identified so far, and what we have done to rule them out :
Memcached indicates between 70 et 80% freespace, so we don't think it is the problem.
We removed Memcached and went back to using a NFS shared directory for session files: the problem actually got worse. This would point to an applicative bug, because NFS being slower to write data, session loss would occur more often.
We have browsed all the different forums (including SO) talking about PHP session data loss, and reviewed our code accordingly. The code base is big, but we have used automated tools and scripts to avoid missing a file.
session_start() is called at the beginning of each page.
exit() is called after each header("Location...")
register_globals is Off
We have tested the possible interractions between our 2 other applications and the problematic one, though they don't share any code, database or session handling. Nothing identified there.
We have analyzed our access logs around the times of the disconnections, to check for behavior patterns : no luck here either.
So we have no idea what causes this problem, as it seems to occur randomly, so my questions are :
The problem could come from our code: did we miss anything to check ? This solutions seems unlikely as the code works most of the time for all our users, but I am still considering it.
The problem could come from another application/process that would "empty" part of the session variable array. We have also reviewed the code from the other applications, but didn't find anything that could cause this.
And if another process is doing this, why would it only empty some sessions and not all of them ?
Thanks for your help.
I don't think you'll get a definitive answer to your question. There are too many probable causes and you haven't shown any code.
Still, my guess is that you have memcached.sess_locking turned Off, or if you have a custom session implementation - that it doesn't implement locking at all.
Eventually, this leads to a race condition between two simultaneous HTTP requests.
My guess is based on the often seen bad advice to turn off locks or free them as soon as possible, in order to achieve higher performance.
If this problem "suddenly" occurred, check what has changed. Did you do any work on the application? If so check committed code (you talked about automated tools so I expect there to be a repository which would allow for accurate finding of code changes).
Did you change anything on the server? Like upgrade software, upgrade/change hardware, make changes to the other two applications ?
One thing that popped to mind, did you check the drives you use for caching? It could be a corrupted part of the file system. Which would explain the random user part.
I couple of things I always to is:
Try to determine the moment of first occurrence as accurate as possible. At my work this occasionally triggers someone saying "oh yeah that might have to do with when I changed/updated/created this or that" so this might help. On the other hand it can sometimes takes days, weeks or more before something gets noticed so start expanding that time-frame if nothing comes up.
You have already a couple of scenario, find the common factor in these. If they don't share any code, stop looking there. If they DO share code search there. Of course sharing (part of) it here might allow us to help you search.
Do an organised search. I usually do the main application check when I am the one working most on the application (or even better when I created it). A colleague will check surrounding applications that might have influence on it. In your case those 2 other applications. Finally our sysadmin will check for newly installed or updated software on the server(s) and he will also check with our network guys if anything changed hardware wise or network related (for other people this could be the hosting provider).
It could be as simple as a WordPress plugin that uses sessions and calls either session_name() or session_id() with a different value, overlapping your custom applications with default session settings.
Since WordPress itself does not use sessions, plugins are often written from the perspective of having free rein with sessions. I just did a search on a WordPress test site and found sessions used in a gallery plugin, a plugin for putting a background image on the page, a shopping cart plugin, and a plugin I was writing that needed to carry an uploaded file from one admin page to another.
Related
We have 2x pfSense FW's in HA, behind that, 2x Zen Load Balancers in Master/Slave Cluster, behind those, 3x Front End web stack servers running NGinx, PHP-FPM, PHP-APC. In that same network segment, there are 2x MySQL DB Servers in Master/Slave replication.
PHP sessions on the front ends should be "cleaned up" after 1440 seconds:
session.gc_maxlifetime = 1440
.
Cookies are expired when the users browser closes:
session.cookie_lifetime = 0
Today, we were alerted by an end user that they logged in (PHP based login form on the website), but were authenticated as a completely different user. This is inconvenient to say the least.
The ZLB's are set to use Hash: Sticky Client. They should stick users to a single Front End (FE) for the duration of their session. The only reason I can think of this happening is that two of the FE's generated the same PHP Session ID, and then somehow the user was unlucky enough to be directed to that other FE by the LB's.
My questions are plentiful, but for now, I only have a few:
Could I perhaps set a different SESSID name per front end server? Would this stop the FE's generating session ID's that were the same? This would at least then result in the user getting logged out rather than logged in again as a different user!
We sync the site data using lsyncd and a whole bunch of inotifywatch processes, but we do not sync the /var/lib/php directories that contain the sessions. I deliberately didn't do this... I'm now thinking perhaps I should be syncing that. lsyncd will be able to duplicate session files across all 3 front ends within about 10seconds of the sessions being modified. Good idea as a temporary fix?
Lastly, I know full well that the client should be using the DB to store sessions. This would completely eradicate it being able to duplicate the session ID's. But right now, they are unwilling to prioritise that in the development time-line.
Ideas very much welcome as I'm struggling to see an easy way out, even as a temporary measure. I cant let another client get logged in as a different user. It's a massive no-no.
Thanks!!
Judging by your question you are somewhat confused by the problem - and its not clear exactly what problem you are trying to fix.
Today, we were alerted by an end user that they logged in (PHP based login form on the website), but were authenticated as a completely different user
There's potentially several things happening here.
Cookies are expired when the users browser closes:
Not so. Depending on how the browser is configured, most will retain session cookies across restarts. Since this is controlled at the client, its not something you can do much about.
PHP sessions on the front ends should be "cleaned up" after 1440 seconds
The magic word here is "after" - garbage collection is triggered on a random basis. Session files can persist for much longer and the default handler will happily retrieve and unserialize session data after the TTL has expired.
Do you control the application code? (if not, your post is off-topic here). If so, then its possible you have session fixation and hijack vulnerabilities in your code (but that's based on the description provided by the user - which is typically imprecise and misleading).
Its also possible that content is being cached somewhere in the stack inappropriately.
You didn't say if the site is running on HTTP, HTTPS or mixed, and if HTTPS is involved, where the SSL is terminated. These are key to understanding where the issue may have arisen.
Your next steps are to ensure that:
you have logout functionality in your code which destroys the session data and changes the session id
that you change the session id on authentication
That your session based scripts are returning appropriate caching information (including a Varies: Cookie header)
It is highly improbable that 2 systems would generate the same session id around the same time.
Really you want to get away from using sticky sessions. It's not hard.
You've got 2 layers at your front end that are adding no functional or performance value, and since you are using sticky sessions, effectively no capacity or resillience value!!! Whoever sold you this is laughing all the way to the bank.
I was working on a web app and thought it would be awesome to give the user a tooltip or side note on a new feature that has been added.
This would be similar to the way Google or Facebook displays their tips about updates.
I had an idea of setting a cookie (or session variable) to say that the tip has been viewed and not display it again, but this would be running an unnecessary if statement on every page load.
Here is what I have so far:
if ($_COOKIE['displayTip'] != 'no') {
echo "tip";
}
Are there better ways of doing this, or is one if every page load worth it?
A single if - especially if it is not checking a database - is no problem. Don't worry about it, you can waste more CPU time with other things. If it adds functionality its fine.
One thing you could do is to have the logic on in the Browser (JavaScript) only. This has the advantage you do not have to generate two different pages. This might be beneficial for caching. In JS you can also access cookies to make that seen flag persistent.
I'm using a CMS-framework that initates session_start() upon page creation, however 90% of the site visitors are guests without the need for sessions, resulting in an awful lot of empty session-variables at server.
What's the best practice here? IF logged in, I need to know at an early level so I guess it means is both:
Postpone session_start() until it's actually needed
Keep the session_start() at an early stage, but make it conditional based on existance of cookie PHPSESS
Or is there a better fix, unknown to I?
Many large sites postpone the session initialization until it's actually needed, e.g. on cart pages and the user profile screen of each user.
In theory this isn't that complicated, if you were using a config file which would be required within all pages you could simply swap out a different config file for the pages that don't require any user recognition.
You're using a CMS-framework so perhaps you're somewhat limited within it. If you can differentiate the page creation process, using session_start(); in one case and not in the other, then this shouldn't be that big of a deal. Keeping the logged in users logged on once returned to the other area of the system (the index file etc) would not work though. Of course you could be using local_storage to aid you with that but relying on JavaScript only isn't very reliable.
The easiest way would probably be to spit the system into two areas, one which doesn't use any sessions (index, other information files, etc) so the process behind those page creations would be different, i.e. not using sessions.
Perhaps you could have a session class like mentioned but it would most likely conflict with other previously outputs causing the sessions to fail, but if you could flush the other output then this could possibly work, but it's kind of a hack in my opinion and it wouldn't really solve the previously mentioned problem.
Have you considered changing the lifetime of the sessions? The default setting is 24 minutes but would reducing it to 15 change anything? Is 24 minutes really that big of a deal for you? Maybe the settings in your environment make them be even longer. Are there other aspects of the system that might be the actual performance issue? Are you hosting the system yourself or do those empty sessions really not matter?
If you're expecting a lot of guest traffic without the need for sessions, then don't use it unless you absolutely need it. What I typically do is create a class for session, and add the session_start to the constructor of the class. Then, when I need sessions, I can simply call the session class within an underlying global.class.
There's a bug, which we can not replicate, which involves users in one specific region of our enterprise customers swapping. For example, a user logs in as themselves on the login page, and when arriving at the home, they are another user.
It seems like accidental session hijacking, here are the clues:
cakephp security is set to low (this only means the cookie doesn't
rewrite every page load, and the the cookie does not do a user agent
check )
our cookie is set to not care about subdomains (.example.com instead of example.com)
enterprises users areredirected using a 302 if they login to the wrong area (should we use 303?)
there was a 301 accidentally sent out, but users are able to replicate
all the affected users are behind a single router, sharing internet via Sprint MPLS
all the affected users may be using computers issued by the customer
their IT claim there is no proxy cache, and no remote VPN access, yet they claim to be able to replicate the issue from home computers and off the network.
Since we can not replicate the issue in any way, we can only assume that the issue is specific to their network.
How can we prove that their network/computers are causing the session swapping? Or, what configuration on our end could be causing this, when no other users experience this issue?
[edits/updates]
Responding to some direction provided by comment - our traffic is not large enough to send duplicate IDs. (the statistically probability is too low to see what we've seen the customer replicate ).
see also:
Zend Framework Session swapping issue
why is php generating the same session ids everytime in test environment (WAMP)?
Update:
We use FCGI, and apparrently mod_php is required to understand x_forwarded_for
What's wrong with this function call?
This may be a problem with improper session invalidation in the log out. please ensure that all the variables in the session are properly terminated or explicitly null terminate every object in the session and then invalidate the session.
The second reason may be the use of variables check for static variables in your code. improper use of static variables may also cause this intermittent issue.
Use logger to log session id mapped to the user ids that can narrow down your problem and help you understand what exactly happening.
Invalidating the existing session in login action and creating a new session and copying content to the new session will help a lot.
First, do not assume the customer is at fault. It may an issue on their side or yours. Do not make an assumption as to which before testing.
Regardless of who's fault it is, the burden is on you to fix or help fix it.
First, having one user become another is often the result of a Session ID problem. The security level you have set in Cake does not regenerate the Session ID for every request.
I would start by logging the $session->id() as a user both inside and outside your local network. Then compare to see if the session id is ever the same or ever an empty string. One fix for this is to generate a unique id for each user.
If the Session ID is unique for each instance, you may want to test it under load.
The point is to test first and make conclusions based upon findings, not speculation.
I've mentioned it on here before, but still have the problem, so have added some extra info :)
We have a local intranet site that everyone on the network uses, maybe 5% (or even less) of the users that use the site have problems where the session isn't stored properly.
I've tried defining a path manually and whereas most users sessions are created and remain just fine, on the affected machines it seems that it either forgets the cookies are there, or it can't read them, and then goes on to create a new cookie almost every time you refresh a page.
Things to note are...
happens on both the Zend login screen and our systems, so for once it's not my dodgy code!
it only affects <5% of users
it only happens when using IE
it only happens with Windows XP or Vista - Windows 2000 works fine!
it happens to those users on any XP/Vista machine, so I can use my bosses PC and it's fine, but if he uses mine it doesn't work again...
I've tried messing with the security settings in IE too, changing the cookie security to allow all cookies/sessions, but no luck on that either unfortunately. :(
I've tried nettool but it didn't really help much unfortunately, since it just showed that the new cookies/sessions were being created, but didn't say why.
I've also tried checking the date and times are on the same on the server as they are on the workstation, and yup everything is set fine.
The name of the server is 'gc-hr01' - not sure if that should affect anything like this.
Any help would be amazing, really stuck on this.
Thanks!
The session cookie might get refused because of insufficient validity and/or privacy information. Try to make those as specific as possible, for example:
session_set_cookie_params(0, '/path/to/your/php-application', '.www.example.com', false, true);
I know this is an old thread, but in case anyone else happens upon this, the cause of this problem is likely the hyphen in the machine name. Apparently IE7 won't store cookies for domain names with a - or _ because they technically aren't valid. Certainly would be nice to get an error message instead of silently dropping the cookie, wouldn't it?
Could it be that 5% of the users have user information that, when retrieved from the cookie, disrupts the proper reading/decoding/parsing of the cookie server-side?