I have a web tier in AWS running Nginx+PHP-fpm using memcache on ElastiCache for sessions. Over the last 6 months or so we've been experiencing a very strange issue where every so often perhaps 6 weeks or so the ElastiCache node runs out of memory and starts evicting keys which leads to some users being loosing session, being logged out and of course frustrated and loosing their place in the app.
I've tried several things. One being leveraging the php-memcached module in ini:
session.save_handler = memcached
session.save_path = "<aws elasticache dns:port>"
And yes I verified that the save_path url I'm actually using is correct and receiving network connections. I've also verified through CloudWatch metrics that the cache node is indeed receiving network connections and data.
This configuration did not work, so I replaced it with a Zend framework session manager and save handler. I verified through phpinfo() that session.save_handler was set to user and also verified that the browser is getting the right cookie that I configured in Zend session.
Still, we're having the same problem as illustrated in the following CloudWatch screenshot:
The vertical spikes in memory are I believe due to memcache clearing expired keys which seems to happen every 24 hours. The very last (far right) spike is where I rebooted the node. The strange thing is that everytime it clears keys, it doesn't clear enough. We end up with an ultimately downward trend in available memory which at some point causes memory to run out and memcache to start evicting keys.
I'm at a loss as to what could be the problem and what to try next in an effort to debug. Any thoughts? Thanks!
This isn't a bug, just how Memcached is supposed to work. By the very nature of being a cache, the data should be (relatively) ephemeral. If you're current node doesn't have enough memory to support all the values you are trying to store it has no choice but to evict keys. If you're only storing sessions and you're filling up an entire cache instance, you're best option would be to upgrade the size of your cache node (thats a lot of sessions!), or in AWSs case, add another node.
If you're storing other data on the cache node as well, set intelligent expiration times for those items so they expire and free space up periodically.
Update: I'll also add, if you're comfortable using cookies, having a time-limited cookie to recreate dropped sessions is a nice fill-in as well. Basic "Keep my logged in" code should suffice
Related
I have configured my session save handler to work with memcache instead of file based because it caused a lock while having more than one open tab. I've done this in php.ini (session.save_handler = memcache)
Because my memcache is full, I believe these items are being deleted through the LRU algorithm. How do u suggest to overcome this? Is there any simple save-to-disk option, or set to memcache to not delete these unless expired?
Indeed, memcached will delete items in order to be able to store new ones. And it has no mechanism to persist items. You can set it to not delete items unless expired, but if you do that and the cache fills up with unexpired items, then it will not accept any new items, which means that no-one will be able to start a new session (or modify an existing one).
Bottom line: memcached is a cache, not a database. Use it only to store data you can recreate when needed. (Also, see the memcached FAQ entry Everyone does it!.)
The amount of memory that memcached can use is set when the memcached server starts up. You could arrange for it to use more memory, if you have more memory on the server, in which case it will probably keep sessions around for longer. But that's just delaying the inevitable. (Also, as the article linked to from that FAQ entry points out, since memcached is not in any way failure resistant, you cannot restart the memcached server without losing all of your saved sessions.)
Setup is a Linux box running apache with apc. TwoLevels-Cache is planned as apc + file. Does Zend two level cache save everything to second level even the data kept in first level?
I was wondering what happens if I restart my server. Will the data in the first level apc cache be lost or can it be retrieved from second level file cache?
From the look of the code, the answer is "it depends". Data will be stored in both caches if the priority is high enough and the fast cache is not full or filling fast. Otherwise it will only be stored in the 'slow' cache.
However I'm sure that the APC cache would be empty after a reboot. There wouldn't be any sensible way for PHP to detect this and warm that cache back up for you, and no real reason for it to. Your app should not rely on cache data being present, and should regenerate it if it is missing.
I'm working with a couple of Web Servers behind a Load Balancer and I can enable Sticky Sessions to hold a user to the one specific Web Servers - this will work.
I have been reading about PHP Sessions & MemCache. I must say what I've read is a touch confusing as some pages say its a good idea and others the opposite.
Questions:
is it possible to keep php sessions in memcache?
is it better to use sticky sessions over memcache?
what are the problems with php sessions in memcache - note: I can get enough cache (amazon so its expandable).
1: YES. And I strongly recommend storing PHP sessions in Memcached. Here's why:
Memcached is great for storing small chunks of data that are frequently accessed by the database and filesystem.
Memcached was designed specifically for sessions. It was originally the brainchild of the lead developer of livejournal.com and later used to also cache the content of users' posts. The benefit was immediate: most of the action was taking place in memory. Page load times greatly improved.
Thankfully, PHP and Apache have an easy implementation to handle sessions with Memcached. Simply install with a few shell commands
example for Debian:
sudo apt-get -t stable install php7.4-memcached
and
change your php.ini settings to something similar to:
(taken from https://www.php.net/manual/en/memcached.sessions.php)
session.save_handler = memcached
; change server:port to fit your needs...
session.save_path = "localhost:11211"
The key is the session.save_path
It will no longer point to a relative file path on your server.
APC was mentioned - APC for the caching of .php files used by the program. APC and Memcached will reduce IO significantly and leave Apache/Nginx free to server resources, such as images, faster.
2: No
3: The fundamental disadvantage of using Memcached is data volatility
Session data is not persistent in Memcached. So if and when the server crashes, all data in memory is lost. Everyone will have to log in again.
And then you have memory consumption...
Remember: the sessions are stored in the memory. If your website handles a large number of concurrent users, you may have to shell out a little extra money for a larger memory allocation.
1. Yes, it is possible to keep PHP sessions in memcached.
The memcache extension even comes with a session handler that takes very little configuration to get up and running. http://php.net/manual/en/memcached.sessions.php
2. Memcache/Sticky Sessions
I don't really know which is "better". I feel this is going to be one of those "it depends" answers. It likely depends on your reasons for load balancing. If a small number of users cause lots of load each, or if it's a large number causing a small load each.
3. Cons of Memcache
There are probably 2 main cons to using memcache for sessions storage.
Firstly, it is volatile. This means, if one of your memcached instances is restarted/crashes etc. any sessions stored in that instance are lost. While if they were using traditional file based sessions, they will be still there when the server returns.
Secondly and probably more relevant, memcached doesn't guarantee persistance, it is only meant to be a cache. Data can be purged from memcached at any time, for any reason. While, in reality, the only reasons data should be purged is if the cache is nearing its size limits. The least recently accessed data will be expelled. Again, this might not be an issue, as the user is probably gone if their session is stale, but it depends on your needs.
If you want to use "memcacheD" extension not "memcache" (there are two different extensions) for session control, you should pay attention to modify php.ini.
Most web resources from Google is based on memcache because it's earlier version than memcacheD. They will say as following:
session.save_handler = memcache
session.save_path = "tcp://localhost:11211"
But it's not valid when it comes to memcacheD.
You should modify php.ini like that:
session.save_handler = memcached
session.save_path = "localhost:11211"
There is no protocol indentifier.
From: http://php.net/manual/en/memcached.sessions.php#99646
As my point of view its not recommended storing sessions in Memcached.If a session disappears, often the user is logged out,If a portion of a cache disappears or either due to a hardware crash it should not cause your users noticable pain.According to the memcached site, “memcached is a high-performance, distributed memory object caching system, generic in nature, but intended for use in speeding up dynamic web applications by alleviating database load.” So while developing your application, remember that you must have a fall-back mechanism to retrieve the data once it is not found in the Memcached server.
We have a problem with PHP session when APC is enabled on our server.
The app works great without APC. However, since we enabled APC, the sessions seems to be getting mixed up when the server experiences heavy load, i.e. users are randomly logging on as another. Everything reverts back to normal once we disabled APC. We can't seem to find anyone with the same problem, except a related problem with these guys (set-cookie was being cached in MS ASP): http://msdn.microsoft.com/en-us/magazine/cc163577.aspx#S2
Anyone else has similar experience? Can you recommend any suggestions?
PS: We have all our sessions handled by files in php.ini. We are also running apache2.
We're having a similar issue here. APC is only a primary suspect at this point because it's been difficult to reproduce.
We're using Zend Framework w/ session management and the theory is that Zend's code is cached in APC and when the system is under severe load the code is using a previously cached SID instead of the current one.
Our safeguard is to save the session ID inside the session data and compare the values when the session is retrieved. If the IDs are different we destroy the session and exit.
well please verify that apc really mixes up data...
the only possible i can think of when this could happen is when it gets full and does a stackoverflow.
please check the usage and maby increase the cache size.
There is a problem that crops up with APC where custom session handling objects get destroyed before the session data is saved. It could be unrelated to your problem, but an explicit call to session_write_close before the regular PHP shutdown fixes that one.
Your problem sounds a bit different though, but I can confirm that issues do exist.
Just to complete the post, we had success in reducing this problem with three things:
1) We made extra checks to make sure that the IP is matched to the session ID, and logout the user otherwise. We could then use this to track how often the problem occurs.
2) We switched to XCache and immediately saw lower number of confused session IDs. However, under very heavy load the problem rears its ugly head again.
3) We then double the memory for Xcache in the php config (xcache.size and xcache.var_size) and now the problem is gone.
So we suspect either APC or Xcache running out of memory was the problem. We're still waiting to see if this is a permanent solution.
I had some thoughts back ago about using memcached for session storage, but came to the conclusion that it wouldn't be sufficient in the event of one or more of the servers in the memcached pool were about to go down.
A hybrid version is to save the main database (mySQL) from load caused by reads would be to work out a function that tries to fetch the data from the cache pool, and if that fails gets it from the database.
After putting some more thought into it, I started to think about using APC cache for session related data. If our web server would go down, sessions would be lost either way, so storing them in a local APC or a localhost memcached server maybe isn't that bad?
What's your experiences?
Generally, session data is something which should be treated as volatile in any situation. The user can always choose to eliminate the cookie themselves at any point (if you are using cookies, of course). For this reason, I see no problem with using memcached for session data.
For me, I'd just keep it simple - no need for a DB fallback unless you absolutely must never lose the user's session in the event of a memcached server failure. As I said at the beginning, I always treat sessions as purely volatile in any case and don't really store anything of any significance in them.
That's my two cents anyways.