I am in the midst of setting up the development environment (PHP/MySQL) for my start-up. We use three sets of servers:
LIVE - the servers which provide the actual application
TEST - providing a testing version before it is actually released
DEV - the development servers
The development servers run SVN with each developer checking out their local copy. At the end of each day completed fixes are checked in and then we use Hudson to automate our build process and then transfer it over to TEST. We then check the application still functions correctly using a tester and then if everything is fine move it to LIVE. I am happy with this process but I do have two questions:
How would you recommend we do local testing - as each developer adds new pages or changes functionality I want them to be able to test what they are doing. Would you just setup local Apache and a local database and have them test locally on their own machine?
How would you recommend dealing with data layer changes?
Is there anything else you would recommend doing to really make our development process as easy and efficient as possible?
Thanks in advance
+1 to each developer running her own setup, complete with Apache and database.
Keep the database schema under version control.
Possibly you could keep (maybe in a separate repository) a small but representative set of data, in a test database. Each morning you check out the latest copy of this test database, and start hacking. When you change schemas, update your test data repository accordingly.
Anyone doing development SHOULD have their own local environment. I use Mac so I run MAMP so that I can have my own LAMP environment local and independent of any other environment. This will also allow me to know that nobody else is changing / working on the same components I am and removes any possible confusion. If you are a windows user, there are also easy to install local versions of the LAMP stack such as XAMP, etc. If you are running Linux as your desktop, you will most likely already know how to install LAMP for the flavor of Linux you are running.
The database schema version is a great idea. It is what we use as well. In addition to the schema under version control, we add a schema version table to the schema and keep it updated so we can quickly tell what version is in production/qa/dev when we need to compare.
As for the data layer changes there are two things I would recommend.
Always create your migration path, forward and backward. This means that when you have the schema you want to put on production to upgrade an existing schema, you should always make it part of the release. A clear concise process for ALTERing the tables. By the same token, you need to have a working and tested ROLLBACK version as well in case something goes wrong.
What I have found helpful is using a backup of production to load on my local (or QA/DEV) so that I have the most up-to-date data / schema to play with without affecting production. If you are not performing regular backups of production, maybe now is a good time to implement a policy. Then you will kill two birds with one stone. You will have backups for any outage and a useful live schema with data you can load to test with on another machine. This will also lend itself to raising any possible issues with schema changes as the data will be matching production. So if it works locally (and on DEV/QA), it reduces the risk of something going wrong in production.
Related
I am an avid webdev hobbyist and freelance, up until now I simply edit the website live (put a maintenance message up while its being made), now all my projects up until now have also been very small.
eg I make a site, show em, take money and go, I've never had to work on a site after it's gone live.
Now my new project is pretty big and I know I will have to edit it after its gone live and maybe have a small team of devs (atm just me)
So how do people professionally handle this? I know I will need a prefix-amp app cos i run an apache server, I've also heared that people use github for versioning, but I'm not really sure because apparently its not svn?
Thanks
ps. I have a windows 7 pc, so no mac apps please
up until now i simply edit the website live
Terrible in my book ;)
so how do people professionally handle this?
First you need to setup a development server (it would be best to keep it as close as possible to the expected live environments). On this server you would install all the software you need.
You may also want to setup a staging server.
i know i will need a prefix-amp app
I hope you are not talking about those one click installers. If you would do it professionally you should install everything yourself that way you can set it up the way you need it.
ive also heared that people use github for versioning, but im not really sure because apparently its not svn?
GitHub is just a website. What you are looking for is git or svn for versioning. You could also setup a git or svn server locally instead of using services like GitHub. Basically what versioning is is that when somebody makes a change to the code he/she would need commit the changes. This way it is easy to keep track of changes in the codebase (like what was changed, when was it changed and by whom).
Local XAMP-stack (LAMP, or WAMP) for development
intranet-system for test and maybe staging
Of course the live system
Versioncontrol, I prefer git. Of course you can use SVN too, but... lets say: It's SVN.
Make changes local, test this changes local
everythings fine: Push it into the "master" vcs-repository
New version ready (or it's "sunday-night-release-time")? Push all that stuff on test/stage
Everythings fine there too: Push it into the live system
Thats very shortened of course, but it should give you an idea.
The tool where you manage your software version is not that important. Use Git, or SVN or whatever, the one you like most. But use _one_.
Equally important is that you run the "page" on two sites, a test and a live system, strictly apart. Both systems have to be very close in their layout, all changes must first be done in the test system, be verified and then done in the same manner in the live system. Do not allow changes only to be made to the live system ('cause it's just a small change'). No exceptions.
Then think about deployment: how will you transfer changed files to the target system ? You need routines for this, that run once started and don't forget a step in between.
Firstly you need some kind of versioning system: either SVN or Git. GitHub is simply an online service that provides managed Git repositories. Secondly you need a development server.
If it were just you doing development, you could host both of these on your local desktop PC, but since other developers are going to be joining, you need a remote server. If you don't want to be running a server out of your home, the best option is a VPS (virtual private server) on which you can install Git, Apache, etc. and anything else you need.
As for development software, take your pick- there are loads of options. A common choice is the NetBeans IDE and TortoiseGit combo. You use NetBeans to develop your code on, automatically uploading to your development server, then you TortoiseGit to commit and sync changes.
Only when you're ready to go live do you copy the code from the dev server to the production server.
I am working on a multi-site Magento implementation and I need suggestions.
I am working on a dual core 1.8ghz machine at work with 3GB ram and have found it to be very slow when using MAMP for my personal development environment, so I have set up a virtual machine using CentOS that mimics the actual staging and production server.
The problem is I have to do my changes, commit them to SVN and then update the repository on my virtual server which is using a crap load of production time. This Magento implementation is a multi site/store so it uses a lot of RAM and I need a stable file system with speed.
I am on a Windows XP Pro machine using WAMP normally but like I said I have found that to be slow. Is there any way I could develop it on my machine without it slowing down every other process?
I have my IDE (Aptana) open and most the time Thunderbird open as well as MySQL Workbench so I have a lot of programs running and the Virtual Machine dedicates those resources, but like I said it it tedious to continuously update the repository.
Can any Magento or expert developers weigh in? I really want to plan ahead for when I get more projects like this, and I would love to hear what some of you other developers do.
Thanks,
Darren
You don't specify that the virtual box is remote, but I'm going to make that assumption here (since a virtual box on your local host would by definition be slower than the machine it is hosted on).
I would suggest connecting to your higher-spec machine remotely to work with the files and testing on that environment. Basically, use it as a development instance and make changes remotely. Then, check in those changes as necessary.
I am very much a Linux developer so your toolchain is a bit outside of my expertise, but you should be able to mount the remote drive as a local network drive, or your IDE may even support mounting it over SSH (which would probably be a bit faster).
Hope that helps!
Thanks,
Joe
EDIT: I should say that, if your virtual instance is not remote, that changes my answer a little bit. Go rent a virtual dedicated machine with some reasonable stats, set it up for Magento, and then do go the above. The monthly cost of a virtual dedicated machine is quite low compared to any hardware technology upgrades you might buy, and it gives you a safe place to break things. The downside to this workflow is that you must be online to accomplish anything.
I develop locally on a Mac using MAMP (and in the past Zend Server). The IDE of my choice is Netbeans. I highly suggest you checkout (no pun intended) Git for doing development work.
I have a server that I use for production and staging. The production and staging have two different databases. A Git central repo also resides on this server.
Essentially, I do my modules and dev work locally on my Mac. I try upgrades and such, then use Git to commit them and push them to the central repo on the server. Once there, I merge the changes into the staging server and test, test, test. If everything is cool, I then pull (merge) the changes into production.
This has served my purposes quite well and I have not had a botched upgrade or anything... if something goes wrong you just rollback the changes in Git. You did make a dump of the DB before upgrading correct :)
As you get more sophisticated you can start using phing to do some automated roll out of your code, but that is another story all together.
Best of luck!
I have a question related to phing. Do I use it on my local development machine to deploy an application or do I call it via ssh from production machine? I'm not sure where to do the steps from Eran Galperin (What is your preferred PHP deployment strategy). I'm a beginner with deployment scripts. So be forgiving :)
Marco
You'll usually have two deploy scripts - One that you run locally, which ssh's in to the remote server and then kicks off the remote script.
Your local script might do an svn export, zip the contents and push them (scp) to the remote server first. The remote script would do stuff like migrating, restarting services etc.
The basic strategy is that you create a build (well, basically a copy of your web application that you can put online) on your local machine, and then deploy it on the server.
You run it on your local machine and the PHPing will connect to production server and deploy the website. :)
Phing can be applied in many ways.
If you don't have directives on how to deploy, the best way is usually first to generate a "test build" from your local copy, check out if it is working as intended and then generate the productive build.
In some cases where you have to follow certain rules in your development enviroment you can easily adapt Phing to meet those demands.
I developed a phing deploy script and I've used it in both of the ways you've described. As long as the phing library and dependencies are installed locally or on the remote machine you can do it in both ways, and it's actually a benefit to be able to run it from any of your environments.
Let me elaborate on my experience, but keep in mind that every person lives within the microcosm of the needs and requirements of their own projects/studios so it's possible that what works for me, may not work for other people.
I think deployment is usually so important for a business that it needs to happen in the exact same way every time. When there's a multitude of people doing the deployments, then it's easy for bad habits or procedures to diverge and that's when mistakes can happen. Automated deployments with phng are handy because if everyone is using the same procedure, then there's much lower potential for mistakes. Phing makes it easy to follow strict procedures because all the developers have to do is to run a simple command like "phing deploy" or "phing build" or what have you.
Now going back to your original question, it's also helpful for these procedures to be adhered to on local development machines as well as on staging, testing, production servers because just like people, machines can have small ideosyncracies that differ from machine to machine. With phing, you can develop a script that will have the same results on any of your enterprise machines (depending on the strength and intent of your code but phing lives in a format that makes it easy to meet that ideal). So if you run your phing build from your development box, ideally it has the same effect if you run it from your testing box or your production server. If you can run the deployment the same way on any machine, you can feel confident that any problems that arise after deployment were not a result of your deploy procedures (as they will not differ from when you deployed to your testing or local dev box).
It's really nice to be able to deploy my code to my local machine with the exact same procedure that I use to deploy it to production.
As far as what is the best to do, I say, why not all of the above? If you can run it locally, then you can run it remotely. If the deploy procedures are strong, then the result will be the same.
You can do it both ways. You can run the tool locally as you develop to run tests and just generally ensure that your code is in good condition. The server automation (if you choose to do that) can also run tests, generate documentation, and deploy changes to test or production setups.
Doing it locally reduces the chance that the process will get stalled on the server setup, while the CI works on a regular basis to ensure nothing has been broken via mistake.
I've always just FTPed files down from sites, edited them and put them back up when creating sites, but feel it's worth learning to do things properly.
I've just commited everything to a SVN repo, and have tried sshing into the server and checking out a tagged build, as well as updating that build using switch.
All good, but it's a lot lot slower than my current process.
What's the best way to set something like this up? Most of my time is just bug fixes or small changes rather than large rewrites, so I'm frequently updating things.
You don't necessarily need to use SVN to deploy the files to the server. Keep using FTP for that and just use SVN for revision history.
You should look at installing rsync to upload changes to your server.
Rsync is great because it compares your local copy of the repo to the copy that's currently on the server and then only sends files that have changed.
This saves you having to remember every file that you changed and selecting them manually to FTP, or having to upload your whole local copy to the server again (and leaving FTP to do the comparisons).
Rsync also lets you exclude files/folder (i.e. .svn/ folders) when syncing between your servers.
I'd recommend you keep using Subversion to track all changes, even bug fixes. When you wish to deploy to your production server, you should use SSH and call svn update. This process can be automated using Capistrano, meaning that you can sit at your local box and call cap deploy -- Capistrano will SSH into your server and perform the Subversion update. Saves a lot of tedious manual labor.
For quick updates I just run svn update from the server.
Sometimes for really really quick updates I edit the files using vim and commit them from the server.
It's not very proper, but quick and quite reliable.
If you want to do this properly, you should definitely look into setting up a local SVN repository. I would also highly recommend setting up a continuous integration (CI) server such as cruise control, which would automatically run any tests against your PHP code when ever you check in to svn. Your CI server could also be used to publish your files via FTP to your host at the click of a button, once it has passed the tests.
Although this sounds like a lot of work, it really isn't and the benefits of a smooth deployment process will more than pay for itself in the long run.
For my projects, I usually have a repo. On my laptop is a working copy, and the live website is a working copy. I make my changes on the local copy, using my local webserver. When everything is tested and ready to go, I commit the changes, then I ssh into the remote server and svn update.
I also keep a folder in this repository which contains sql files of any changes I've made to the database structure, labelled according to their revision number. For instance, when I commit Revision 74 and it has a couple extra columns in one of the tables, included in the commit will be dbupdates/rev74.sql. That way, after I do my svn update, all I just have to run my sql file (mysql db_name -p -u username < dbupdates/rev74.sql) and I'm good to go.
If you want to get real funky with it, you could use a build script to get the current version from SVN, then compile your PHP code, then on a successful build, automatically push the changes to your server.
This will help in debugging and may make your code run faster. Also, getting into the build habit has really improved my coding over just pushing the PHP straight to the server and debugging via Firefox.
The benefits of source control reveal themselves as the complexity of the project and number of developers increase. If you are working directly on a remote server, and are only making quick patches most of the time, source control might not be worth the effort to you.
Preferably, you should be working from a local working copy of the repository (meaning you should also set up a local server). Working against a remote server using SVN as the only means to update it would slow you down quite considerably.
Having said that, working with SVN (or any other source control) will yield many benefits in the long run - you have a complete history of changes, you can always be sure the server is up-to-date (if you ran update) and if you add more developers to the project you can avoid costly source overwrites from each other.
What I do at work, is use FTP to upload changes to a test server. Then when I am finished with the section of the site that I was working on, I commit the changes and update both. Sometimes, if I am working on something and I change a lot of files in different directories, I commit it and update the test server. But I don't update the production server. But I am the only programmer here, I wouldn't recommend committing possibally buggy code if there is more than one programmer.
I use ZendStudio for Eclipse (currently version 6.1). And I use SVN to keep my source codes available. Initially I thought the process was somewhat slow due to commit process (and entering commit comment) and wait until it stops.
However after learning that Ctrl+Alt+C to Commit and check 'Always run in Background', the process doesn't slow at all.
Plus, I do run everything locally, then only SSH after a while.
I did a post-commit hook to automatically update my web. It´s fast but you can make mistakes.
IF on a *nix server AND you have the appropriate SSH access AND you have space to keep multiple copies of the website, THEN the single most useful versioning technique I have found is to use a symbolic link to point to the "current" version of the website. (You can still use SVN to version source code -- this is a way to easily/instantly switch between versions of the website on the server.)
Set up the webserver to point to /whatever.com as the root of the website.
Have a folder like /website/r1v00 to which you FTP the website files, then create a symlink called "whatever.com" that points to /website/r1v00
When you have an updated version of the website, create another folder called /website/r1v001, FTP all the files for the updated site, then change the symlink for "whatever.com" to now point to /website/r1v01. If there are any problems with the new site, you can back it out instantly by simply pointing the "whatever.com" symlink back to /website/r1v00
Of course, you can/should set up scripts to automate the creation and switching of the symlink. In my case, I have an "admin" page written in PHP that lists all the available versions, and allows me to switch to any of them. This technique has saved my bacon several times...!
Obviously this does not address any issues with versioning database schemas or database content.
We have a PHP project that we would like to version control. Right now there are three of us working on a development version of the project which resides in an external folder to which all of our Eclipse IDEs are linked, and thus no version control.
What is the right way and the best way to version control this?
We have an SVN set up, but we just need to find a good way to check in and out that allows us to test on the development server. Any ideas?
We were in a similar situation, and here's what we ended up doing:
Set up two branches -- the release and development branch.
For the development branch, include a post-commit hook that deploys the repository to the dev server, so you can test.
Once you're ready, you merge your changes into the release branch. I'd also suggest putting in a post-commit hook for deployment there.
You can also set up individual development servers for each of the team members, on their workstations. I find that it speeds things up a bit, although you do have some more setup time.
We had to use a single development server, because we were using a proprietary CMS and ran into licensing issues. So our post-commit hook was a simple FTP bot.
Here is what we do:
Each dev has a VM that is configured like our integration server
The integration server has space for Trunk, each user, and a few slots for branches
The production server
Hooks are in Subversion to e-mail when commits are made
At the beginning of a project, the user makes a branch and checks it out on their personal VM as well as grabs a clean copy of the database. They do their work, committing as they go.
Once they have finished everything in their own personal space they log into the integration server and check out their branch, run their tests, etc. When all that passes their branch is merged into Trunk.
Trunk is rebuilt, the full suite of tests are run, and if all is good it gets the big ol' stamp of approval, tagged in SVN, and promoted to Production at the end of the night.
If at any point a commit by someone else is made, we get an e-mail and can merge those changes into our individual branches.
Beanstalk has built-in post-commit hooks for deploying to development, staging, and production servers.
One way to use subversion for PHP development is too setup a repository for one or all three developers, and use this repository, more as a syncing tool, than true version control.
You could,
Make a repo
Add your entire PHP document structure of your project
Checkout a copy of this repo into the correct spot on your dev server
Use an svn hook, that activates on commit
This hook, will automatically update the contents of the dev sever, whenever anybody on the team checks in any code.
Hook resides in:
svn_dir/repo_name/hooks/post-commit
And could look like:
/usr/bin/svn up /path_to/webroot --username svn_user --password svn_pass
That will update your working copy on the dev server to the latest check in.
What about something distributed? You can start for example with Mercurial, try different workflows, and see which one fits you the best.
Each of you could run it locally, or on your own dev server (or even the same one with a different port...).
One possible way (there are probably better ways):
Each of you should have your own checked out version of the project.
Have a local copy of the server on your computer and test it there throughout the day. Then at the end of each day (or whenever), you merge together whatever you are ready to test, and you check it out onto the dev server and test it.
Another tool you can use for the builds is TeamCity which is free for 20 build configurations (enough for most small companies/projects.) This way you can run your tests as well as schedule builds.