I have a Symfony2 website that I'm testing in production. I went ahead and cleared its cache because I've made and will probably make more modifications, however there is a small problem:
While the cache is being cleared and say, afterwards I want to warm it up, someone that accesses the website rebuilds the cache. That creates a small problem as the cache is being built, but not completely, while half of it gets deleted because the deletion is still in progress.
What happens afterwards is, the cache is built, but only a part of it. Symfony thinks that the cache is built entirely, and runs without trying to build it anymore, but it runs on a half-built cache. The deletion process is a bit long (~15 sec), so in this timeframe nobody must try and create the cache by accessing the website.
Either that, or the cache is completely built, it overwrites the old cache, and the system treats these new files as old ones, deletes part of them and some others remain. Not entirely sure, I'm not sure how to check this.
For instance, one of the errors that I'd get is
The directory "D:\xampp\htdocs\med-app\app\app\cache\dev/jms_diextra/metadata" does not exist.
If I wouldn't use that bundle I'd get another cache problem from Doctrine. This appears at every website access until I delete the cache again WITHOUT anyone accessing the website. it completely blocks access to the website and makes it non-functional.
Also, what about the warmup? That takes a while, too. What if someone accesses the website while the cache is being warmed up? Doesn't that create a conflict, too?
How to handle this problem? Do I need to close the apache service, clear and warm cache and then restart apache? How is this handled with a website in production?
EDIT
Something interesting that I have discovered. The bug occurs when I delete the cache/prod folder. If I delete the contents of the folder without deleting the folder itself, it seems the bug does not occur. I wonder why.
Usually it is good practice to lock the website into maintenance mode if you're performing updates, or clearing the cache for any other reason in the production. Sometimes web hosting services have this option to handle this for you, or there is a nice bundle for handling maintenance easily from the command line.
This way you can safely delete the cache and be sure no-one visits the page and rebuilds the cache incorrectly.
Usually if you have to clear the Symfony cache it means you're updating to a new version - so not only are you having to clear the cache, but you're probably having to dump assets and perform other tasks. In this case what I've done in the past that has worked very well is to treat each production release as its own version n its own folder - so when you install a new version you do it unconnected from the webserver, and then just change your webserver to point to the new version when you are done. The added benefit is if you mess something up and have to perform a rollback, you just immediately link back to the previous version.
For example, say your Apache config has DocumentRoot always points to a specific location:
DocumentRoot /var/www/mysite/web
You would make that root a symlink to your latest version:
/var/www/mysite/web -> /var/www/versions/1.0/web
Now say you have version 1.1 of your site to install. You simply install it to /var/www/versions/1.1 - put the code there, install your assets, update the cache, etc. Then simply change the symlink:
/var/www/mysite/web -> /var/www/versions/1.1/web
Now if the site crashes horribly you can simply point the symlink back. The benefit here is that there is no downtime to your site and it's easy to rollback if you made a mistake. To automate this I use a bash script that installs a new version and updates the symlinks with a series of commands connected via && so if one step of the install fails, the whole install fails and you're not stuck between version limbo.
Granted there are probably better ways to do all of the above or ways to automate it further, but the point is if you're changing production you'll want to perform the Symfony installation/setup without letting users interfere with that.
Related
I'm recently getting back into programming after taking a break for quite some time. I'm currently trying to rebuild my CakePhp 2.X apps for CakePhp 4.X. I'm developing in a local environment using Bitnami WAMP stack.
The issue is that development is very slow because changes do not seem to be taking effect immediately. I have tried disabling all caching by using
Cache::disable();
I've tried placing this line in various places
config/boostrap.php
config/app.php
config/app_local.php
src/application.php
How this is impacting me: I'll make an update, for instance, to a model table file or controller file. I'll go refresh my site to preview the change and either there is no update or there might be an error. To fix the error, I try to undo the changes I made. I go back to my browser and hard refresh the page. I continue to see the same error for 10+ minutes. This often leads me to wanting to undo previous steps but I know that those previous steps didn't cause the issue and it was only the most recent change that caused it. It's making it difficult to keep track of what changes are causing issues and what solutions are working. Even something as simple as updating my navigation element (templates/elements/nav.php) to add a new link does not show on the page when I refresh. I've also tried clearing my browser cache (I use Chrome).
Did you try to clear all keys? You can do it with:
// Will clear all keys.
Cache::clear();
Cache::disable() should work also.
You can also delete the contents of /tmp/cache/ if the caching is set on File.
Maybe the problem is Bitnami WAMP itself. Try to disable the server cache:
https://docs.bitnami.com/installer/infrastructure/wamp/administration/disable-cache/
If you are developing on top of an AMP Stack or customizing any
Bitnami Stack, your files (like JavaScript files) may be cached by the
server and even you modify them your changes will not appear to be
applied.
In order to disable the cache in the server and let the files be
served each time, disable PageSpeed for Apache and OPCache for PHP,
enabled by default
I am working on changing a Drupal 7 site and have run into strange behavior where an old version of a file I've changed keeps re-appearing. I've flushed caches via the admin interface as well as truncating the cache_ tables.
On my staging server (which I have access to), things work fine. On our production server (which I do not have SSH access to and cannot easily get access to), they do not and I have limited ability to debug, so I have to guess. I suspect there is some Drupal or Apache setting that is causing these old files to be shown because the filesystem has identical contents. The behavior is almost as if Drupal will look for any file named the same (even if it is in the wrong directory) and show that.
In my case, I have all my files under /var/www/html (standard LAMP setup). At one point, I tar cfz the entire thing and kept that at /var/www/html/archive.tgz (not removing it by mistake). So now I'm wondering if somehow Drupal is reading the contents of that archive and using the old file. Sounds crazy, but has anyone run into something like that?
The other possibility is that my cache cleaning is still limited in some way. Outside of truncating cache_ tables in the database, is there any way to forcibly remove all cached entries? Any insight into this mystery would be appreciated.
Obviously, your production server runs some additional caching proxy like Varnish. You need to clear cache there as well.
I've built a CMS (using the Codeigniter PHP framework) that we use for all our clients. I'm constantly tweaking it, and it gets hard to keep track of which clients have which version. We really want everyone to always have the latest version.
I've written it in a way so that updates and upgrades generally only involve uploading the new version via FTP, and deleting the old one - I just don't touch the /uploads or /themes directories (everything specific to the site is either there or in the database). Everything is a module, and each module has it's own version number (as well as the core CMS), as well as an install and uninstall script for each version, but I have to manually FTP the files first, then run the module's install script from the control panel. I wrote and will continue to write everything personally, so I have complete control over the code.
What I'd like is to be able to upgrade the core CMS and individual modules from the control panel of the CMS itself. This is a "CMS for Dummies", so asking people to FTP or do anything remotely technical is out of the question. I'm envisioning something like a message popping up on login, or in the list of installed modules, like "New version available".
I'm confident that I can sort out most of the technical details once I get this going, but I'm not sure which direction to take. I can think of ways to attempt this with cURL (to authenticate and pull source files from somewhere on our server) and PHP's native filesystem functions like unlink(), file_put_contents(), etc. to preform the actual updates to files or stuff the "old" CMS in a backup directory and set up the new one, but even as I'm writing this post - it sounds like a recipe for disaster.
I don't use git/github or anything, but I have the feeling something like that could help? How should (or shouldn't) I approach this?
Theres a bunch of ways to do this but the least complicated is just to have Git installedo n your client servers and set up a cron job that runs a git pull origin master every now and then. If your application uses Migrations it should be easy as hell to do.
You can do this as it sounds like you are in full control of your clients. For something like PyroCMS or PancakeApp that doesn't work because anyone can have it on any server and we have to be a little smarter. We just download a ZIP which contains all changed files and a list of deleted files, which means the file system is updated nicely.
We have a list of installations which we can ping with a HTTP request so the system knows to run the download, or the click can hit "Upgrade" when they log in.
You can use Git from your CMS: Glip. The cron would be a url on your own system, without installing Git.
#Obsidian Wouldn't a DNS poisoning attack also compromise most methods being mentioned in this thread?
Additionally SSH could be compromised by a man in the middle attack as well.
While total paranoia is a good thing when dealing with security, Wordpress being a GPL codebase would make it easy to detect an unauthorized code change in your code if such an attack did occur, so resolution would be easy.
SSH and Git does sound like a good solution, but what is the intended audience?
Have you taken a look at how WordPress does it?
That would seem to do what you want.
Check this page for a description of how it works.
http://tech.ipstenu.org/2011/how-the-wordpress-upgrade-works/
Does anyone have a great system, or any ideas, for doing as the title says?
I want to switch production version of web app-- written in PHP and served by Apache-- from release 1234 to release 1235, but before that happens, have all files already in the opcode cache (APC). Then after the switch, remove the old cache entries for files from release 1234.
As far as I can think of there are three easy ways of atomically switching from one version to the next.
Have a symbolic link, for example /live, that is always the document root but is changed to point from one version to the next.
Similarly, have a directory /live that is always the document root, but use
mv live oldversion && mv newversion live
to switch to new version.
Edit apache configuration to change the document root to newversion, then restart apache.
I think it is preferable not to have to do 3, but I can't think of anyway to precompile all php files AND use 1 or 2 to switch release.
So can someone either convince me its okay to rely on option 3, or tell me how to work with 1 or 2, or reveal some other option I am not thinking of?
Exactly why do you want to do this?
When I switch to a new release, I just stop my Apache, replace the PHP files on the server via checkout from my VCS and restart Apache. It mostly takes less than 30 seconds to pull this off, it's even scriptable. All that is executed at a time when there are few or no users. After that, the opcode cache is empty and will fill itself as soon as someone uses the files.
Maybe the first few hits will not have optimal performance but that should not be so bad. On my production systems no one ever noticed this short performance drop. And there are some really heavy load systems among them.
You can even place a simple "upgrading ... be back soon" screen on your server while replacing the files to avoid any disturbing error messages to your users.
Then after the switch, remove the old
cache entries for files from release
1234.
You can only clear the entire cache or none at all. It's not a selective process. This may change your approach to the problem. If precompilation is paramount you may be forced to bring down the server during the switchover.
We've recently enabled APC on our servers, and occasionally when we publish new code or changes we discover that the source files that were changed start throwing errors that aren't reflected in the code, usually parse errors describing a token that doesn't exist. We have verified this by running php -l on the files the error logs say are affected. Usually a republish fixes the problem. We're using PHP 5.2.0 and APC 3.01.9. My question is, has anyone else experienced this problem, or does anyone recognize what our problem is? If so, how did you fix it or how could we fix it?
Edit: I should probably add in some details about our publishing process. The content is being pushed to the production servers via rsync from a staging server. We enabled apc.stat_ctime because it said this helps things run smoother with rsync. apc.write_lock is on by default and we haven't disabled it. Ditto for apc.file_update_protection.
Sounds like a part-published file is being read and cached as broken. apc.file_update_protection is designed to help stop this.
in php.ini: apc.file_update_protection integer
apc.file_update_protection setting
puts a delay on caching brand new
files. The default is 2 seconds which
means that if the modification
timestamp (mtime) on a file shows that
it is less than 2 seconds old when it
is accessed, it will not be cached.
The unfortunate person who accessed
this half-written file will still see
weirdness, but at least it won't
persist.
Following the question being edited: One reason I don't see these kinds of problems is that I push a whole new copy of the site (with SVN export). Only after that is fully completed does it become visable to Apache/Mod_php (see my answer How to get started deploying PHP applications from a subversion repository? )
The other thing that may happen of course, is that if you are updating in place, you may be updating files that depend on others that have not yet been uploaded. Rsync can only guarantee atomic updates for individual files, not the entire collection that is being changed/uploaded. Another reason I think to upload the site en-mass, and only then put into use.
It sounds like APC isn't preforming or getting the correct file stat info. You could check it to make sure the APC configuration apc.stat is set correctly. Another thing you could do it force the cache to clear with apc_clear_cache() when you publish new code.
Never saw that before, even if i'm a huge user of APC.
Maybe try to trigger a script that empty the APC opcode everytime you send new code on the server ?
When you get a file with a parse error, back it up, then repubish. Take that same file that now works and do a diff against the file with the parse error.
ctime means creation time. You will want to manually flush your entire cache every time you do updates.
You can easily do this, by putting the apc.php script somewhere on your server. This script gives you cache statistics, and will allow you to drop the cache altogether.
The script comes with APC.
Hopet his helps,
Evert
This is probably happening because there's a mismatch between your code, and the cached versions of the code.
For example, APC has a cached version of User.php, but you made changes to User.php or to the data that User uses. The cached version is still running even after your deploy, because it hasn't expired yet.
If you clear your APC cache entries when you deploy, this issue should disappear.