Okay, might sounds little like a troll but it's not. Since I started to programming with symfony, I encountered very very very weird bugs. Like 3 times in a month. That was always related to my caching files and every time, I spent hours to finally figure it out it was coming from him.
I'm working on a project with cryptography, and when my dear symfony start to forget the keys between two encryption with the same key : I've start to freaked out about the future of my web application.
(the real bug is to weird that I can't really explain it)
I am going to store sensitive data and I can't imagine some of this things append in production and all I can say is since I've cleared AND remove my cache folder, the bug disappear. What a trusting behaviour !
So, are those weird bug only related to development environment due to the large amount of file update ?
What is your context to tell you that you need to clear your cache in development environment ?
Should I deactivate all kind of cache for the production environment to guarantee that will not happen again ?
Thank you guys.
Think 3 time, before you decide to disable cache on prod. It's probably a bad idea.
On dev if you modify bundles, some config, etc, sometimes Symfony fails to refresh cache, and to be sure that errors you receive are real you should delete whole app cache and clear memcached/redis if you use it.
On production you probably will/should create new directory with clean fresh version of your app with clean cache and then you replace your old revision with new one (usually by changing the symlink). Thats why if your app works on dev with clean cache it should be ok on prod env.
Related
I am using Laravel 5 (to be specific, "laravel/framework" version is "v5.0.27"), with session driver = 'file'.
I'm developing on Windows 7 64 bit machine.
I noticed that sometimes (once a week or so) I get unexpectedly and randomly logged out. Sometimes this happens even immediately after I log in. I have added log messages to my auth logic code, but the log code was not triggered. Laravel behaved as if it has completely lost the session file.
Another, more serious issue, was that sometimes after debugging sessions (using xdebug and Netbeans) Laravel started losing also other files - .env settings, some debugbar JS files etc. The error log had messages like:
[2015-07-08 13:05:31] local.ERROR: exception 'ErrorException' with message 'mcrypt_encrypt(): Key of size 7 not supported by this algorithm. Only keys of sizes 16, 24 or 32 supported' in D:\myproject\vendor\laravel\framework\src\Illuminate\Encryption\Encrypter.php:81
[2015-07-08 13:05:31] local.ERROR: exception 'PDOException' with message 'SQLSTATE[HY000] [1044] Access denied for user ''#'localhost' to database 'forge'' in D:\myproject\vendor\laravel\framework\src\Illuminate\Database\Connectors\Connector.php:47
This clearly signals that .env file was not read by Laravel, so it is using default settings:
'database' => env('DB_DATABASE', 'forge'),
'key' => env('APP_KEY', 'somekey'),
Losing files happened rarely, maybe once a month or so, and it always happened after debugging sessions. I always had to restart Apache to make it work again.
To stress-test the system and reproduce the issues reliably, I used a quick hack in my Angular controller:
setInterval(function(){
$scope.getGridPagedDataAsync();
}, 500);
It is just a basic data request from Angular to Laravel.
And that was it - now I could reproduce the session losing and .env losing in 3 minutes or less.
I have developed AJAX-intensive web applications earlier on the same PC with the same Apache+PHP, but without Laravel, without .env, and I hadn't noticed such issues before.
While debugging through code, I found out that Laravel is not using PHP built-in sessions at all, but has implemented their own files-based session. Obviously, it does not provide the same reliability as default PHP sessions and I'm not sure why.
Of course, in real life scenarios my app won't be that AJAX-intensive, but in my experiences on some occasions it is enough with just two simultaneous AJAX requests to lose the session.
I have seen some related bug reports on Laravel for various session issues. I haven't yet seen anything about dot-env, though, but it seems suffering from the same issue.
My guess is that Laravel does not use file locks and waiting, thus if a file cannot be read for some reason (maybe locked by some parallel process or Apache) then Laravel just gives up and returns whatever it can.
Is there any good solution to this? Maybe it is specific to Windows and the problems will go away on a Linux machine?
Curious, why Laravel (or Symfony) developers haven't fixed their session file driver yet. I know that locking/waiting would slow it down, but it would be great to at least have some option to turn on "reliable sessions".
Meanwhile I'll try to step through Laravel code and see if I can invent some "quick&dirty" fix, but it would be much better to have some reliable and "best practices" solution.
Update about .env
The issue turned to be not related to locking files. I found the Laravel bug report for .env issue, which lead me to a linked report for Dotenv project which, in turn, says that it is a core PHP issue. What disturbs me is that Dotenv devs say that Dotenv was never meant to be used for production, but Laravel seems to rely upon Dotenv.
In https://github.com/laravel/framework/pull/8187 there seems to be a solution which should work in one direction but some commenter says that in their case the issue was the opposite. Someone called crynobone gave a clever code snippet to try:
$value = array_get($_ENV, $key, getenv($key));
There appeared another suggestion to use "makeMutable()" on both Dotenv and Laravel Githubs, but commenters report that this might break tests.
So I tried the crynobone's code but it did not work for me. While debugging, I found out that in my case when things break down for concurrent requests, the $key cannot be found nor in getenv(), nor in $_ENV and not even in $_SERVER.
The only thing that worked (quick&dirty experminet) was to add:
static::$cached[$name] = $value;
to Dotenv class and then in helpers.php env() method I see that:
Dotenv::$cached[$key]
is always good, even when $_ENV and getenv both give nothing.
Although Dotenv was not meant for production, I don't want to change our deployment and configuration workflow.
Next I'll have to investigate the session issues...
Addendum
Related Laravel bug reports (some even from version 4. and it seems, not fixed):
https://github.com/laravel/framework/issues/4576
https://github.com/laravel/framework/issues/5416
https://github.com/laravel/framework/issues/8172
and an old article which sheds some light on what's going on (at least with session issues):
http://thwartedefforts.org/2006/11/11/race-conditions-with-ajax-and-php-sessions/
After two days of intensive debugging I have some workarounds which might be useful to others:
Here is my patch for Dotenv 1.1.0 and Laravel 5.0.27 to fix .env issues:
https://gist.github.com/progmars/db5b8e2331e8723dd637
And here is my workaround patch to make session issues much less frequent (or fix them completely, if you don't write to session yourself on every request):
https://gist.github.com/progmars/960b848170ff4ecd580a
I tested them with Laravel 5.0.27 and Dotenv 1.1.0.
Also recently recreated patches for Laravel 5.1.1 and Dotenv 1.1.1:
https://gist.github.com/progmars/e750f46c8c12e21a10ea
https://gist.github.com/progmars/76598c982179bc335ebb
Make sure you add
'metadata_update_threshold' => 1,
to your config/session.php for this patch to become effective.
All the patches should be applied to vendor folder each time it gets recreated.
Also you might want to separate the session patch out because you update session.php just once, but the other parts of the patch should be applied to vendor folder each time it gets recreated before deployment.
Be warned: "It works on my machine". I really hope Laravel and Dotenv developers will come up with something better, but meanwhile I can live with these fixes.
My personal opinion that using .env to configure Laravel is a bad decision. Having .php files that contained key:value style of configuration was much better.
However, the problem you are experiencing is not PHP's fault, nor Apache's - it's most likely Windows issue.
A few other things: Apache contains a module that allows PHP binary to be integrated into Apache's process or thread, called mod_php - the issue with this is that PHP is not only slow, but getting another binary integrated into an existing one is super tricky and things might be missed. PHP also must be built with thread-safety in this case. If it's not, then weird bugs can (and will) occur.
To circumvent the problem of tricky integration of one program into another, we can avoid this completely and we can have .php served over FastCGI protocol. This means that the web server (Apache or Nginx) will take the HTTP request and pass it to another "web" server. In our case, this will be PHP FastCGI Process Manager or PHP-FPM.
PHP-FPM is preferred way of serving .php pages - not only because it's faster (much, much faster than integrating via mod_php), but you can easily scale your HTTP frontend and have multiple machines serve .php pages, allowing you to easily horizontally scale your HTTP frontend.
However, PHP-FPM is something called a supervisor process and it relies on process control. As far as I'm aware, Windows do not support process control in the way *nix does, therefore php-fpm is unavailable for Windows (in case I am wrong here, please correct me).
What does all of this mean for you? It means that you should use software that's designed to play nicely with what you want to do.
This is the logic that should be followed:
A web server accepts HTTP requests (Apache or Nginx)
Web server validates the request, parses the raw HTTP request, determines whether the request is too big and if everything goes well in this case, it proxies the request to php-fpm.
php-fpm processes the request (in your case it boots up Laravel) and returns the HTML which the web server shows to the user
Now, this process while great, comes with a few issues and one huge problem here is how PHP deals with sessions. A default PHP session is a file stored somewhere on the server. This means that if you have 2 physical machines serving your php-fpm, you're going to have problems with sessions. This is where Laravel does something great - it lets you use encrypted cookie based sessions. It comes with limitations (you can't store resources in those sessions and you have a size limit), but a correctly built app wouldn't store too much info in a session in the first place. There are, of course, multiple ways of dealing with sessions, but in my opinion the encrypted cookie is super, super trivial to use and powerful. When such a cookie is used, it's the client who carries the session information and any machine that contains decryption key can read this session, which means that you can easily scale your setup to multiple servers - all they have to do is have access to same decryption key (it's the APP_KEY in the .env). Basically you need to copy the same Laravel installation to machines that you wish to serve your project.
The way I would deal with the issue that you have while developing is the following:
Use a virtual machine (let's say Oracle Virtualbox)
Install Ubuntu 14.04
Map a network drive to your Windows host machine (using Samba)
Use your preferred way of editing PHP files, but they would be stored on the mapped drive
Boot an nginx or Apache, along with php-fpm on the VM to serve your project
Now what you gain via this process is this: you don't pollute your Windows machine with a program that listens on ports 80 / 443, when you're done working you can just shut the VM down without losing work, you can also easily simulate how your website would behave on an actual production machine and you wouldn't have surprises such as "it works on my dev machine but it doesn't work on my production machine" because you'd have the same software for both purposes.
These are my opinions, they are not all cold-facts and what I wrote here should be taken with a grain of salt. If you think what I wrote might help you, then please try to approach the problem that way. If not, well, no hard feelings and I wish you good luck with your projects.
I've noticed while developing my first site, that the smallest changes in database columns, connection options, and various other components cause the website to fail until I correct the problem which may or may not require a lot of time (woe is me). I'm wondering what steps I can take now in order to prevent these headaches, as I've delayed the website launch in order to keep upgrading. Once I'm done implementing the changes I would like to see, I know I won't truly be done, but at some point I have to move on to the next stage.
Yes, I know there is probably no one good solution, and ultimately a self-correcting design is more trouble than its worth at this point. But if any
grey beards have any tips that they could offer based on their own experiences working with WebDev, particularly with LAMP stacks, I would greatly appreciate it.
Specifically, I would like to know what to look out for when modifying databases and website code after customer information is in active use, in order to prevent errors, and how to roll out the changes.
EDIT 1:
Yes, so the answer seems to be that I need to copy the live site to my testing environment. I'm looking going to some of the already suggested development solutions. Regular backups are crucial, but I can just see inserting new columns and modifying queries as a cause for mis-ordered tables and such. "That's where being a good programmer and testing diligently comes in handy", someone in the corner said. As I look into the proposed solutions, I welcome all others in the meantime. A real-time copy of the 'live-site' would be nice to create on the fly during testing.
The above answers are all very valid and in the end, they represent your target solution.
In the meantime, you may already do a lot for your website, even with a gradual migration to those practices.
In order to do so, I suggest you to install PHPUnit (or whatever Unit comes with the web languages you use). There are also "graphical" versions of it, like VisualPHPUnit, if that's more of your taste.
These tools are not the permanent solution. You should actually aim adding them to your permanent solution, that is setting up development server etc.
However, even as interim solution they help you reach a fairly stable degree of quality for your software components and avoid 80-90% of the surprises that come with coding on a live server.
You can develop your code in a separate directory and test it before you move it into production. You can create mock objects which your code under test may freely interact with, without fear of repercussions. Your tests may load their own alternate configuration so they work on a second, copy database.
Moving even further, you may include your website into tests itself. There are several applications like Selenium that allow you both to automate and test your production website, so that you can reliably know that your latest changes did not negatively affect your website semantics.
In short, while you should certainly aim at getting a proper development environment running, you can do something very good even today, with few hours of study.
Start using some (maybe simplified) sort of release management:
Maintain a development environment. Either locally, or in a second .htaccess-protected folder online. Make it use it´s own db.
Test every change in this dev env. Once you are satisfied, move them to productive env.
Use git or svn (svn might be simpler to learn, but opinions vary. Checkout "tortoise") to save a snapshot ("commit") of every change you make. That way you can diff throu your latest commits if anything goes wrong.
Maintain regular backups.
I'm new to Symfony2 and I must admit I'm enjoying it. I started playing with the SonataAdmin but soon a major doubt popped out:
is it normal it takes almost 3 seconds to load a listing page (using an empty database)?? I know in production I should go for APC or memcache to speed up things but it looks strange to me it takes so much time.
I'm developing using a virtualmachine with Turnkey lamp (1GB ram).
My PC is fairly new: Intel i3 8Gb ram.
Please tell me what you think/experience.
Thanks.
In development environment, it is hard to measure performance because the framework and the bundles sometimes need to parse a lot of configuration files, introspect objects and perform time consuming task and cache the output.
In production, a lot of stuff is done upfront, i.e. when you are deploying to your web server. The work upfront is done to avoid parsing files, do time consuming task, etc. This is the reason why in production, you almost can't change anything without running php app/console clear:cache again after the modification have been done. Even changing a single Twig template requires a cache clear to update the output presented to the end user.
I did not test this bundle personally, but an admin generator bundle needs to check a lot of properties and objects to perform his task properly. This is indeed time consuming but this is required only in development mode. When in production, this introspection process is not necessary and the information is probably cached somewhere. This should give way better performance in production environment than in development environment.
Bottom line, I don't think this bundle suffer from a performance problem but this depends on your needs and objectives. The only thing I can be sure of: test it in production mode to see the speed it will give you in the end. Clear you cache for production mode and use app.php instead of app_dev.php. Also, check the documentation on performance that can be found on symfony.com.
php app/console cache:clear --env=prod --no-debug
Hope this helps.
Regards,
Matt
I've checked other threads about this subject, but nothing really answers my exact question. I'm currently developing a site that updates data on a live web server (usually photos) multiple times daily using a cron and some external data feeds. Additionally, we're about to allow user-submitted data. Now, I want to start using svn because the project is getting big enough where it's starting to get relatively complicated and people are making edits to our code on the live server. I've installed svn for apache and all of the subdomains mentioned below are set up on one server using vhosts. We haven't started using this workflow yet, so if I could get some advice before we start it would be greatly appreciated.
Okay, so I've set up a repository under a subdomain (http://svn.website.com), and checked it out to a development subdomain (http://dev.website.com). I also plan to check out a copy to the live environment (http://www.website.com). The idea is to first develop on the dev subdomain, commit when it's ready to go live, and then svn up to the live environment. I've read the pros and cons between exporting and updating, and updating seems best for this project because we often will commit small tweaks/code changes and exporting seems like overkill. I have also setup the appropriate directives to avoid .svn directory access. Does it sound like I'm on the right track with this? Anything I should be aware of?
If the following is a good way to take care of everything, then we can move onto the second question. The user-submitted data (mainly photos) are being uploaded to the live environment. I have to somehow manage this between the live and dev environment so we can develop with up-to-date user-submitted files. I don't really know the best way to do this...I feel like there's a serious flaw if I were to svn add and svn ci every time a file is submitted. What is the best way to accomplish this?
Sounds like you're growing!
First, I'd question as to whether SVN is really the best choice. My guess is that you know it already, have worked with it, as have others on your team, and you want to stick with it. That's fine, but just an FYI I've used SVN and Git and I found Git generally easier to use, and I would argue that it is becoming more and more commonly used by application developers.
As for your workflow, I used to go through an almost identical process with a large distributed team and it worked fine. Depending on the amount of traffic you get, I might suggest a different production deployment strategy. When you're ready for a new production push, clone into a new directory, labeled perhaps by date or milestone names, and create a new virtual host for that directory. That way, you get a couple benefits:
You don't run the risk of having someone access your application and requesting a certain file which is in the process of being updated
You can easily revert back to an earlier milestone by simply changing the active production virtual host. Bugs and problems make it out of development regardless of how thoroughly you test, and sometimes they're serious enough to warrant just reverting back to a previous codebase. Of course, schema changes would add complexity to this, but if we're just talking about code this should avoid frantic commit reverts at 2am.
This was suggested by Rasmus Lerdorf at a conference I attended when he was posed nearly the same question you're asking.
As for ensuring developers are working with the most recent fileset, do you really want to do this? If you're processing a serious quantity of images, that's going to take up a lot of hard disk space. Sure with a CDN or other mass storage no big deal, but do you want your developers to have to download 1000 new photos every time they svn up?
If you're dead set on doing this, I'd suggest adding some code within your app for automatically doing an svn add, svn ci when svn status has a certain result (ie, this is a script you could call with cron, or every time new media gets added). You could also check into using svn hooks (git has these as well) to do things like notify you that the repo has been updated.
Hope this helps. I'd strongly discourage including all of the files users upload unless you have a really, really strong reason for doing so. The complexity that's gonna add is gonna be a pain = )
Good luck and welcome to StackOverflow!
I am currently in the process of learning the CodeIgniter PHP framework. At the current moment, I am looking to have a DEV environment and a PRODUCTION environment. Coming from a purely C and JAVA background, I am used to having everything locally (with version control), but since I will have the PRODUCTION side on a website, I was wondering a couple of different things:
Is it better to keep the DEV environment locally?
What would be a good (and easy) way when making changes on the DEV side to get them up to the PRODUCTION side (assuming the DEV env is local)?
Is it possible (and if so, how?) to setup CodeIgniter to have a DEV & PROD environment in the same codespace (say on the server but with different databases tables)?
When using Version Control on an application I would create in the Framework, is it recommended to only have the files I create for my application, or add the whole Framework (to keep the code consistent with version of the Framework)?
I appreciate any thoughts or suggestions you have in advance.
Thanks!
I don't use CodeIgniter, so I may not be able to answer all your questions ; but, still, here a few pointers :
Development environment : I like when each developper has it's own environment ; and it's generally easier / faster when it's on his machine
still, if your development machines are running windows, and you production server will be running Linux, it could bring some problems (there are a few differences between those two, like case-sensitivity in files names)
in this case, provided you have a powerful enough computer, using VMWare or VirtualBox to run a minimalist Virtual Machine (with Linux+Apache+PHP+MySQL on it ; the code source is on it too, exported via a samba share) might be a good solution -- that's what I've been doing few more than half a year, and it works pretty well
Pushing changes from dev to prod :
one solution is to log on the production server and do an "svn update" ; but I don't really like that (if some files have been modified directly on the production server -- yes, this sometimes happens), you may encounter conflicts and the like ; and that's definitly not fun at all, as it may break the site ^^
one solutioon I like more (takes more time to deploy, but if you deploy only, say, once a week, it's quite OK -- and definitly more secure) is to use "svn export" on one of the dev machines, create a tar/zip/whatever archive, and upload it to the prod server. Then, you extract it to a NEW directory ; and when it's done, you change a symbolic link to point your root directory to that new directory. Nice thing is : you keep the old sources on the production server, and if there's a catastrophic problem with what you deployed, you just have one symbolic link to change back to revert to the previous version -- and that can sometime save the day ^^
oh, and, as a sidenote : you should write a script to do that automatically for you : it will avoid messing with ine step one day, when doing it manually (and will help for the day when the guy that always does that is in holidays ^^ )
About using one instance of the sources for both environments : even if you plan on doing this only for the framework, I wouldn't recommend it :
it means having dev and prod on the same machine, which is bad : what if some still in development script becomes mad and does bad things ? What if one developpers types some kinf of "rm -Rf" in the wrong directory ?
you thought about different databases tables (I'd rather go with different databases ; with different users, to avoid anyone doing any bad request on the wrong database !), but that's not the only thing : what about temporary files ? cache, for instance ?
I'd really prefer having to fully separated instances ; even if it means have the sources / the framework twice on the machine -- and I'd really really recommend having two different machines !
About having the framework on SVN :
The more things you have on SVN, the easiest it is to get a new development environment set up : ideally, just one "svn checkout", and there's a new environment ready for the new developper who just joined your team ; coupled with a Virtal Machine you can give him (copied from another's machine), you can have developpers ready to work on your project in a couple of dozen minutes -- which is nice ;-)
Still, having the Framework in SVN is a PITA to update it : you have to delete it, replace it, re-commit it, ...
Using svn:externals (if you can -- depends on your setup / your framework), pointing to the framework's SVN server itself, might be a good thing to always be up to date (don't necessarily point to HEAD ; using a tag or a branch might be good enough for you).
Hope these few notes help...
Have fun !
1) I agree with Pascal MARTIN - its best for everyone to have their own local dev environment; that way they can play without stepping on each others toes. It may mean, then, that you want to have some type of test or staging environment where team members (and project stakeholders) can see the integrated, in-progress code.
2, 3) More generally, it sounds like you're asking how to automate/deploy to one or more environments. There are several commercial and open source options for doing this. We've recently started to use Capistrano (http://www.capify.org) and have been really happy with the results. It's a ruby tool, and written using ruby-on-rails-isms. If you're not familiar with those (I wasn't) it takes a little bit of reading and Googling to figure it out. However, at its heart is simply a means to define and run scripts on remote servers. Those scripts can be used on any type of deployment (we use PHP, for example). Two great things about Capistrano that address your question:
It knows about version control; whether you use SVN, git, or others, it knows (several ways) of pulling down the latest code from the repository and doing whatever is needed to update remote server(s).
It does transactions, so if something goes wrong with the build/deploy process, it can automatically rollback to a previous version
4) That's probably the simplist model; simply download a codeigniter installation and write your code in the applications/ directory. This could be hassle someday if you want to upgrade CI to take advantage of some new hot feature. You should be able to get around this by defining an svn:external link to the codeigniter, so that when they update, it gets rolled into your code as well. See: http://svnbook.red-bean.com/nightly/en/svn.advanced.externals.html for more info...