I'm trying to get into the mindset of CI and have been playing with Phing this weekend. It all seems straight forward enough to use and have many examples going already.
However something that still puzzles me is how people actually use it. That is, I'm not looking for what tests you do, but instead a suggested work flow using Phing, at what stage do you activate it, at what stage in the development cycle is it actioned.
For example, we have several websites, currently we edit the source locally and on save upload to the live site (I know how bad this is...), we do some quick testing and make sure the code works as planned. If so we commit to the repos and carry on. If not, we can roll back or edit undo and resave. Whilst this now seems crazy the simplicity has worked well for us.
We now have a small team however, so I'm trying to push Phing into this process, to get all the added benefits of the linting/sniffing/mess detecting e.t.c. however I can't figure the best order of events.
Would you suggest :
Edit the code locally.
On save upload the file to a test site.
Test the site on the staging server.
All being well, commit the changes to the repos. Then run phing.
Assess output of Phing, updating code as necessary, re-save, re-commit, re-run phing.
Assuming Phing passes, as I'm running phing on another server, do a svn export and start the deployment process.
The above seems a bit long winded to me. Is it because it looks like I'm trying to merge a test deployment with a live deployment that's confusing me?
Also it would seem a bit backwards to commit, then run Phing, then have to edit and possibly re-commit before trying again.
Hence would it make more sense to :
Edit the code locally.
Save, action a test deployment build with Phing.
Make sure the code has passed all the code checks e.t.c.
Using Phing make sure code is copied to the staging server.
Test the site on the staging server.
All being well, commit the changes to the repos.
Then live deployment build with Phing.
The problem with the above is, let's say I just wanted to correct the spelling of a word hard-coded into an HTML page, seems overkill?
Finally, how do people set up their servers, do you have one server for the live site, one for the staging, and one to host Phing (and any CI software on)?
The point of tools like Phing is automation. So, to answer this "what stage do you activate it, at what stage in the development cycle is it actioned", I would say you should bring it in as soon as you feel you will gain a benefit from using it.
For example, if you have a process which takes multiple commands to complete, there would be a benefit of using Phing (or even just a shell script) to automate the steps, especially if there's more than one person who needs to do it or it's especially error-prone.
So going with that, you should use phing to make your life easier, not harder. Any task which involves more than one shell command, or always involves typing the same command with lots of hard to remember parameters, is usually something you could/should use phing to automate.
Considering the first list of steps you mention, it is indeed a bit longwinded. Instead, you should use phing earlier, to automate the steps:
If you have automated tests, use phing to run them
Commit to source control
Use phing to deploy the code
Use phing to do the svn export or any other manual steps
So basically I would do pretty much what you suggested in your second list.
You probably should make phing commands for the separate steps, and make phing commands for running commonly run together commands in one shot (eg. tests and then deploy).
This would allow you to skip phases if you feel it's necessary, such as in the example you gave about changing just some text.
A typical approach to servers is having the live site on its own server, and then having the staging/testing server mirror it as closely as possible. For CI or other utilities, you can usually host them on the staging server, proven they do not interfere with the main application you are developing.
Related
I would like to have some input on how a professional development setup with the following requirements might look like.
several PHP-developers (say PHP)
each developer belongs to one group
each group has one team-leader who delegates tasks
each developer works on one Windows 7 machine
and developes either with NetBeans or Eclipse
each developer 'owns' one virtual test-server where he can run the code
the VCS in use is SVN
there is a staging server where the product is ultimately tested before it gets released/deployed
I gave some specific technology to not be too abstract and b/c I also would be interested in concrete suggestions for plug-ins etc.
There are several questions coming to my mind in that setup.
1) So every developer will work on
personal branch.
2) This branch is checked out in a working copy.
Now ... this working copy is edited locally on the PC with the dev's IDE and executed/tested on the server.
What would be in that case the best/usual way to do that? I mean - how do you get your edited code on the server without causing too much overhead?
Would the dev have the code on his local disk at all? Or would it be better to have the IDE write on the remote virtual server through a tunnel or via a specific protocol?
3) Every day a dev will commit his work into his personal branch which resides in a central repository.
Is there a best practice on where the repository is supposed to be located? A seperate server?
4) Then after a dev finished his task either s/he or the team-leader merges the new code into the respective main-branch or trunk.
The most confusing part is about what I wrote between 2) and 3). Because so far I only worked with a local server. For example a VM with a server running a code which is located in a shared folder so I will be able to edit it directly. I'm not sure how to bridge the gap efficiently when the server is now actually remote. Efficiently means not having to upload manually via FTP for example.
Also external sources or book recommendations are welcome.
edit
My question/s is/are aiming at a quasi-standard / best-practice. I think this is pretty much a standard development scenario so there must be a 'usual' solution.
edit 2
Okay ... so lets try with a picture:
V is the virtual test-server for one or more developers D. C and C' are the two code-versions. They should be kept as identical as possible.
Two solutions come to my mind:
1 : Edit C, then upload it to C', then execute C', then commit C.
2 : No C existant. Just C' which is edited through some tunnel technology and executed and committed.
My guts tell me both solutions are semi-optimal. So what would be "professional" / most efficient / fastest / most convenient / most friction-less / least error-prone / best practice / industry standard?
Any questions?
Maybe its not of great help but GIT sounds like a perfect fit to your problems, i recommend to take a look to the GIT features. And if you got time check Linus Torvalds him self talking ablout GIT. http://www.youtube.com/watch?v=4XpnKHJAok8
The standard procedure as you describe is more or less the same. I also you this approach for my team. It can also be called staged application development.
Here is how I am doing it, I use a remote SVN host (ex: assembla.com, unfuddle.com) to store all my codes. My team members store the information there on these remote svn servers. You can also buy an VPS and setup SVN there and user the same approach.
Best practices is to test locally and commit and commit as many times as you can but every commit must solve a problem or include a significant segment that adds any new feature.
Once the commit is done by everyone the lead developer then can login to the staging server via SSH using tools like PuTTY. First time the lead developer has to checkout the code into the folder where the codes are to be located. During this phase file conflict may arise if multiple developers edits same segment of a file. The lead developer should then resolve the code first and then proceed with the checkout. Once checked out, there onwards the lead developer will only need to do a svn update on the staging server to make the code up to date.
Basic idea is to get the code working on local setup then commit and update the staging for testing the application on a simulated scenario and then commit it to the live site.
There are a lot of if's and but's here which will need me to write a chapter on :) but in short this is the zest.
Tools (you can use to work under this setup):
- Tortoise SVN Manager
- PuTTy
- NetBeans
hope it helps :)
I don't like working with personal branches. I worked with ClearCase for almost 15 years and even though ClearCase probably handles personal branching better than most, it was still a big pain. Even worse, personal branches encourages people to not commit their work until the last minute -- usually a day or two before a major release.
For that reason, and to force developers to stay on track with each other, I highly recommend everyone working together on a single branch (or on the trunk) as much as possible. I keep telling developers to take small bites when they make changes.
What you sound like you need is a way to automate the deployment. That is, I make changes on my local machine, and with a single command, I make sure that the server has a duplicate copy of the code. You also want the deployment to be efficient. If you change a single 2 kilobyte file of a 2 gigabyte, 10,000 file deployment, you only want to copy over that one file, not 10,000 gigabytes. For that, I would recommend you write a deployment script in Ant.
Your developers can modify files, then deploy those files via an Ant script. The developers don't have to remember what files they had updated because Ant will automatically handle that. In fact, Ant can even modify files to make sure they contain the right environment information as they get copied over. And, of course, Ant can rearrange the files if the setup on the server is different from the setup in the source repository. And both Netbeans and Eclipse can execute Ant scripts right in the IDE.
So:
Have your developers modify code on their local machine.
Run an Ant script to make sure the server and the local machine are in sync.
Test on the server.
Then, check in their changes once they're happy with the results on the server.
Someone mentioned a Continuous Build System like Jenkins. That actually would be a good idea anyway even though it doesn't solve this particular issue. Jenkins could have its own server and database. Then when you commit your code, Jenkins would update the server and run automated tests. Jenkins can then create a report. It all gets displayed on Jenkin's webpage. Plus, you can archive your deployments on Jenkins, so if you tell someone to test "Build #20", they can simply pull it off of Jenkins where its easy to find.
I'm sure everyone has different ways of doing things but here are my thoughts.
"Best Practice" is probably "Continous Integration" ie each developer doesn't have their own branch but checks in to a common development branch. This forces them to handle conflicts and coordinate with each other early and often to avoid the lead developer from managing a huge train wreck merge later down the road. Take a look at cruisecontrol if you really want to go that route.
The best way is if they have a local apache web server and full php stack. You can use the Zend_Server community edition to get up and running on windows fast. Most standard php code will run just fine on both Windows and Linux, but if you are doing lots of file manipulation or cron job or cli stuff, or need memecache, etc you'll run into incompatabilities. If thats the case and the Linux only stuff is going to bite you use VMWARE or VirtualBox to run local linux instances and install the IDE inside those and just make sure they have goobs of RAM to deal with it.
Each developer needs to run a syncronize inside of Eclipse, basically an svn update, deal with any conflicts with the other developers right then and there, do local testing and commit their changes.
I setup a post_commit hook on the svn server that calls and /autobuild.php on my web server. autobuild.php runs svn update and gets the latest code changes as well as does any chown or chmod file permissions stuff and resets any server specific config files config.php. Its a little tricky to get it setup so that the apache user can run svn update, but once you do your beta/testing server always has the latest committed code. CruseControl, and several others can also help you do this sort of thing and add unit testing, etc
Now your Lead Developer still has a job to do merging the Development Branch into the Production One, testing on the dev server, and reviewing the commits of the others and deciding how and when to push out a release, but your not putting the burden on him of resolving every conflict and merging every change.
Your developers are not ftping files or ssh remoting into servers, they just work locally in their IDE and interact with each other through svn (and email, phone, chat, etc) updating to get the new code and commiting as they finish things.
I don't see any good coming out of having a seperate branch for each developer using SVN. Merging those branches might work in Git but with SVN your lead developer will be hating life very quickly with that type of setup.
Unfortunately, I've never had a senior developper or mentor to show me some of the best practices. So I develop sites (php/mysql) on my Windows machine with WAMP, I test in hidden (password restricted) folders on the production server and finally move sites to production folder.
I would like to have a more fluid/practical/error-proof setup so that from development > test > production, there is no hiccups.
The important points/questions are (you probably can come up with a lot more):
Ease of use
Easy to dev/test modifications after site is live (to avoid tests on production site)
No server difference between local/test/prod (error reporting, apache setting, etc)
Avoid problem with DB differences (ex: if columns were added, how do you add them to prod DB.?)
Do you skip the test environment or do dev AND test on the same.?
etc...
How do you guys develop PHP/MySQL websites.?
Do you use SVN.? Do you use IDEs.? Do you use VMs.?
Thanks.
This is a kind of frequent question - this is why most seasoned devs do not reply on - and generally end up in a flame war with torrid opinions. So, be careful about that.
But you seem to be an nice guy intending to get on the right paths, seeking for some really productive paths. And I recognize a little about myself on this a few years ago.
OK, first thing to keep in mind is: do not blind follow anyone on anything. Anyone can claim to be a great master, but you can find at least 10.000 guys far way better and completely anonymous. So, for anything you hear about do the following: listen, test, and take your own conclusions. If there is just one golden rule this is it. Everything else is crappy until your own conclusions appear. You are your final judge.
That said, let me begin for the one of the most current question: IDE. What you should use? You should use the one you can produce more and makes you more comfortable. Netbeans, Eclipse, VIM, Notepad++, Notepad, gedit, kate, quanta plus.... You have many options, and each person has it's own opinion. Test what you think interesting and go ahead with the one you choose.
This is true also for any methodology, framework or tool. Use, learn, and get critic about it. Stick with the one which makes you more comfortable and productive.
Same thing for developing environment. Does not matter that much if you develop on Windows, Mac or Linux. The important is get the resources you need available. The resources you need can and generally do change from one project to another.
So the best environment to develop a certain project is one that reflects the real environment where the production will run. What if you develop with PHP 5.3 OOP resources and at the end you get on a PHP 5.1? That's the point. The final environment is who tells you what is the best environment to develop, not the inverse.
For testing, you should trace a strategy. I'm talking about that as a 5 years Test Team Lead inside IBM. This because there are a LOT of testing you can perform, but not all can be really interesting to the current project.
First decide, according to project needs, what you are going to test. Security, performance, UI display, UI effects, error handlings, load and balance, usability, accessibility...
Take notes of what you are going to test (what, when, where, success criteria), and make a report of success and failures.
As I said before, the project needs is what guides you on every step. Testing is not different. If you just need to check the display on different browsers, feel free to use different machines, or VM's.
Generally this is sufficient. But if the project requires performance or load testing, then you will need specific load testing softwares. I will not get deep in this subject as it is very extensive.
It takes some time to find a ideal process and tools match, and after achieve that, you will always discover a new tool to test or a process to make you save a little time. This is IT.
Here's my recommendations:
have a dev environment purely for development. keep a staging and/or a live environment based on the resources you have at your disposal. the staging environment is where you test and ensure there are no serious issue(s) with your application. the live environment is basically your production setup. in fact, the staging and live should ALWAYS be the same. It is useful to reproduce issue(s) on the staging and do a bit of troubleshooting without modifying the code. Bear in mind, this also holds true for any associated databases.
Use SVN or some form of version control. This way you will have the ability to fall back to any stable version of the application if someday the world falls apart!
If you are using Linux environments you can write simple scripts to synchronize the setup with your latest (STABLE) development environment. Ideally, you do your development and conduct unit tests to ensure everything works as per design. Run a script and the staging environment is updated with the latest codebase. Conduct functional tests on staging and ensure that everything works as per specs. Run another script and your latest changes are moved into live/production environment.
My development process is still a bit rough, and I am looking forward to the answers also.
What I do for large projects is setup a git repo on my linux desktop and my windows desktop. I will test locally if possible. As components as finishinged I will push my changes to the centrally hosted git repo (private git hub account usually), or pull them to dev (i setup dev as a repo and pull from ssh). All MySQL updates are stored in update files, and I use netbeans for development (although I have used eclipse and others, netbeans just works for me).
I think you hit on all the important points. Personally, I
run the same OS and server software on the production server and my development system. Same versions of PHP, Python, MySQL, Django, etc.
I don't change DB structure often. I set up the DB tables on the dev. system, then use mysqldump to produce the table creation SQL. I install it on the sever using mysql <name_of_sql.file. When I do make changes, I back up the DB and then just do it through the command line interface. For PHP, I use Doctrine just for the table structure/migration support.
I write everything in Kate (Linux), Komodo Editor(Mac), or Notepad++ (Windows). I don't like IDEs very much, I much prefer to-the-point text editors.
I upload files to a staging dir, and check them with diff before copying them to the actual location.
This isn't the most sophisticated set up, but it's worked quite well for me. I'm the only dev working on the projects I'm involved with, which probably explains a lot. The most important part, that isn't really just personal preference, is the first one - match your dev/test system as closely to the production system as possible, including the OS.
I am a lone developer most of my time, working on a number of big, mainly PHP-based projects. I want to professionalize and automate how changes to the code base are handled, and create a Continuous Integration process that makes the transition to work in a team possible without having to make fundamental changes.
What I am doing right now is, I have a local test environment for every project; I use SVN for each project; changes are tested locally, and then transferred to the on-line version, usually via FTP. API documentation is generated manually from the source code; Unit tests are something I am getting into slowly, and it's not yet part of my daily routine.
The "build cycle" I am envisioning would do the following:
A changeset gets checked into SVN after having been tested locally.
I start the build process. The SVN HEAD revision gets checked out, modified if necessary, and made ready for upload.
API Documentation gets generated automatically - if I haven't set it up in detail yet, using a default template, scanning the whole code base.
The new revision is deployed to the remote location via FTP (Including some directory renaming, chmodding, importing databases, and the likes.) This is something I already like phing for very much, but I'm open for alternatives of course.
Unit tests residing in a predefined location are run. I am informed about their failure or success using E-Mail, RSS or (preferably) HTML output that I can grab and put into a web page.
(optionally) a end-user "changelog" text file in a pre-defined location gets updated with a pre-defined part of the commit message ("It is now possible to filter for both "foo" and "bar" at the same time). This message is not necessarily identical with the SVN commit message, which probably contains much more internal information.
Stuff like code metrics, code style checking and so on are not my primary focus right now, but on the long run, they certainly will. Solutions that bring this out-of-the-box are very kindly looked upon.
I am looking for
Feedback and experiences from people who are or were in a similar situation, and have successfully implemented a solution for this
Especially, good step-by-step tutorials and walkthroughs on how to set this up
Solutions that provide as much automation as possible, for example by creating a skeleton API, test cases and so on for each new project.
and also
Product recommendations. What I know so far is phing/ant for building, and phpUnderControl or Hudson for the reporting part. I like them all as far as I can see, but I have of course no detailed experience with them.
I am swamped with work, so I have a strong inclination towards simple solutions. On the other hand, if a feature is missing, I'll cry about it being too limited. :) Point-and-click solutions are welcome, too. I am also to commercial product recommendations that can work with PHP projects.
My setup
I am working on Windows locally (7, to be exact) and most client projects are run on a LAMP stack, often on shared hosting (= no remote SSH).
I am looking for solutions that I can run in my own environment. I am ready to set up a Linux VM for this, no problem. Hosted solutions are interesting for me only if they provide all of the aspects described, or are flexible enough to interact with the other parts of the process.
Bounty
I am accepting the answer that I feel will give me the most mileage. There is a lot of excellent input here, I wish I could accept more than one answer. Thanks everyone!
I've been through buildbot, CruiseControl.net, CruiseControl and Hudson. All though I really liked CruiseControl*, it was just too much of a hassle with really complex dependency cases. buildbot is not easy to set up, but it's got a nice aura (I just like python, that's all). But hudson won over the former three because:
It's just easy to set up
It's easy to customize
It looks good and got nice overview functionality
It got point-and-click updates, for itself and all installed plugins. This is a really nice feature, that I appreciate more and more
Caveat: I only ever used linux as base for the above mentioned build servers (CC.net ran on mono), but they should all - according to the docs - run cross-platform.
Setting up a hudson server
Prerequisites:
Java (1.5 will serve you just fine)
Read access to the subversion server (I have a separate account for the hudson user)
From here, it's just:
java -jar hudson.war
This will run a small server instance right off your console, and you should be able to browse the installation at your http://localhost:8080, if you don't have anything else running on that port in advance (you can specify another port by passing the --httpPort=ANOTHER_HTTP_PORT option to the above command) and everything went well in the 'installation' process.
If you go to the available plugins directory (http://localhost:8080/pluginManager/available), you'll find plugins for supporting your above mentioned tasks (subversion support is installed per default).
If that has whet you appetite, you should install a java application server, such as tomcat or jetty. Installation instructions are available for all major application servers
Update: Kohsuke Kawaguchi has constructed a windows service installer for hudson
Setting up a project in hudson
The links in the following walk-through assumes a running instance of hudson located at http://localhost:8080
Select new Job (http://localhost:8080/view/All/newJob) from the menu on the left
Give the job a name and tick Build a free-style software project on the list
Pressing 'ok' will take you to the configuration page of the job. All the options have a little question mark besides them. Pressing this will bring up a help text regarding the option.
Under the option group 'Source Code Management' you would be using Subversion. Hudson accepts both url access as well as local module access
Under the option group 'Build Triggers', you would use 'Poll SCM'. The syntax used here is that of cron, so polling the subversion repository every 5 minutes would be */5 * * * *
The process of building the project is specified under the option group 'Build'. If you already have an ant build file with all the targets you need, you're in luck. Just choose 'Invoke ant' and write the name of the target. The option group supports maven and shell commands as well out of the box, but there is also a plugin available for phing.
Tick off additional build actions in 'Post Build Actions', such as e-mail notifications or archiving of build artefacts.
For setting up processes for which hudson have no plugins, you can either call them directly through a shell script from within the build setup, or you could write you own plugin
Pitfalls:
If you have it produce build artefacts, remember to have hudson clean up after itself in regular intervals.
If you have more than 20 projects set up, consider not displaying their build status as the default main page on hudson
Good luck!
The term you are looking for is "continous integration."
Here is an example of someone who uses GIT + phpundercontrol: http://maff.ailoo.net/2009/09/continuous-integration-phpundercontrol-git/
CruiseControl (which is a CI server), can use Hosted SVN/GIT as a source. So you can even use it with GitHub or Beanstalk or something else.
Then you can integrate that with the following kind of software:
PHPUnit
php-codesniffer
phpdocumentor
PHP Gcov
PHPXref
Yasca
etc.
You could also try this hosted CI: http://www.php-ci.net/hosting/create-project
Keep in mind though, that those tools need custom support if you integrate them yourself.
Have you also thought about project management and patch management?
You can use Redmine for project management. It has integrated continuous integration support, but only as client side (not as CI server).
Try using a hosted SVN/GIT/etc. solution, because they will cover your backups and keep their servers running, so you can focus on development.
For a tutorial on how to setup Hudson, see: http://toptopic.wordpress.com/2009/02/26/php-and-hudson/
I use Atlassian's Bamboo continous integration server for my main PHP project (along with their other products such as fisheye (repository browsing), jira (issue tracker) and clover (code coverage)).
It supports SVN and now supports Git and it has a great user interface. It is available for linux, windows and mac and can run standalone on its own tomcat server which is great for people (like me) who does not like to take days to setup their tools). Although it may look expensive, being a lone developer myself I purchased the starter kit license for 10$ (10$ by software). This is great for small teams and it is worth the look.
PHPTesting PHPCI This is nice, continuous integration server built in php.
Plus, its free and open source. :)
it has number of plugins..
PHPCI includes integration plugins for:
Atoum
Behat
Campfire
Codeception
Composer
Email
Grunt
IRC
PHP
Lint
MySQL
PDepend
PostgreSQL
PHP Code Sniffer
PHP Copy/Paste Detector
PHP Spec
PHP Unit
Shell Commands
Tar / Zip
I am mostly a sys admin but sometimes I code PHP as well. As a side project I created some scripts that will make it simple and painless to set up a full blown PHP CI environment using Jenkins. It also runs a sample project for you so you can see how each build step is configured.
If you want to try it out all you need is a Debian/Ubuntu box and shell access.
http://yauh.de/articles/379/setting-up-a-ci-environment-for-php-projects-using-jenkins-ci
Update To add some content to my answer:
You can simply set up a Jenkins CI for PHP using Ansible. Since v1.4 it supports roles which you can download from their galaxy.ansibleworks.com community site and it will do the heavy lifting for you. It is called jenkins-php.
I would suggest using Jenkins http://jenkins-ci.org/ it's free and it's open source.
It's pretty straight forward to setup, works on multiple platforms and integrates well with other continuous integration tools like SonarQube (+ SQUALE) to measure technical debt and Thucydides for testing automation.
I would highly suggest using GIT or GIT Hub for version control instead of SVN. From my point of view it's just a better version control system that will help you scale your development efforts later.
Since you're working mostly with PHP project there are some other tools you can use.
PHPUnit - For unit testing
PHP CodeSniffer - Check for coding standards
PHP Depend - Shows your PHP code dependencies
XDEBUG - For performance testing
All of these tools and be triggered with a Jenkins job and helps with the quality and performance of your code.
Good luck and Enjoy!
I do not use many of the products, or even types of products that you use, but I will give you my experience.
I run a TEST environment in parrallel with my PROD environment. I have no local testing per se. If it is too hard to get soemthing up into a real TEST environment, then I fix my build process. I don't see the point in testing locally, as the environments are different. UPDATE: The only thing I do locally is run "php -l" before I upload anything. Stops the stupid mistakes.
The build process works with whatever is in the current workspace, which includes uncommitted code. This is not everyone's cup of tea, but I am going to TEST very often. Everything gets committed before going to PROD.
Part of my build process (similar to yours) creates two META files. One contains the last (typically) 100 changes and also gives me the current changelist number. The shows me what changes are installed. The other contains the CLIENTSPEC (in Perforce terms) which shows me exactly what branches were used in this build. Together these give me reproducible builds.
I do not build straight to the target environment, but to a staging area on the server. I use SSH so this makes sense. This gives me a few advantages. Most importantly it avoids dying half way through a large upload. It also gives me a place to store META files, and all the build files are automatically archived (so I can go straight back to any build). The script also logs the update (so there is an entry in the log stream and I can see pre- and post-) and kicks all daemons (I use daemontools so "svc -t"). All of these are better off on the target machine.
One other issue is DB changes. I keep a master script of the DB schema, which I update every time the schema changes. Each of the changes also go into a changes.sql script, which is uploaded with the build to the staging area. The script is run as part of the install script.
I've recently begun the same kind of process, and am using Beanstalk for svn hosting.
There are two nifty features in the paid accounts (start at $15pm i think):
deployment allows the user to create ftp targets for staging and production servers, which can be deployed at the click of a button (inc specifying a revision and branch)
webhooks allow the user to set up a url that is called on each commit/deploy, passing across things like revision number, description and user. This could be used to update docs, run unit tests and update changelogs.
I'm sure there are other hosted or self-hosting svn servers with these two features, but beanstalk is the one i have experience of and it's working very, very well
There's also an API, which I imagine could be used to integrate deployment further in to your process.
Consider fazend.com, a free hosted CI platform, which automates configuration and installation procedures. You don't need to setup version control, bug tracking, CI server, test environment, etc. Everything is done on-demand.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I'm beginning a new project in PHP and I'd love to get some feedback from other developers on their preferred strategy for PHP deployment. I'd love to automate things a bit so that once changes are committed they can be quickly migrated to a development or production server.
I have experience with deployments using Capistrano with Ruby as well as some basic shell scripting.
Before I dive head first on my own it would be great to hear how others have approached this in their projects.
Further information
Currently developers work on local installations of the site and commit changes to a subversion repository. Initial deployments are made by exporting a tagged release from svn and uploading that to the server.
Additional changes are typically made piecemeal by manually uploading changed files.
For PHP, SVN with Phing build scripts are the way to go. Phing is similar to ANT but is written in PHP, which makes it much easier for PHP developers to modify for their needs.
Our deployment routine is as follows:
Everyone develops on the same local server at work, every developer has a checkout on his machine back home as well.
Commits trigger a post-commit hook which updates a staging server.
Tests are ran on staging server, if they pass - continue.
Phing build script is ran:
Takes down production server, switching the domain to an "Under construction" page
Runs SVN update on production checkout
Runs schema deltas script
Runs tests
If tests fail - run rollback script
If tests pass, server routes back to production checkout
There's also phpUnderControl, which is a Continuous Integration server. I didn't find it very useful for web projects to be honest.
I'm currently deploying PHP using Git. A simple git push production is all that's needed to update my production server with the latest copy from Git. It's easy and fast because Git's smart enough to only send the diffs and not the whole project over again. It also helps keep a redundant copy of the repository on the web server in case of hardware failure on my end (though I also push to GitHub to be safe).
We use Webistrano, a web frontend for Capistrano, and are very happy with it.
Webistrano allows multi-stage, multi-environment deployments from SVN, GIT and others. It has built-in rollback support, support for separate server roles such as web, db, app, etc., and deploys in parallel. It allows you to override config parameters on multiple levels, such as per stage, and logs the results of every deploy, optionally mailing it.
Even though Capistrano and Webistrano are Ruby applications, the syntax of the deployment 'recipes' is easy and powerful enough to understand for any PHP programmer. Originally Capistrano was built for Ruby on Rails projects, but easily accommodates PHP projects.
Once configured it is even easy enough to be used by non-programmers, such as testers deploying a staging version.
To provide the fastest deploy possible we installed the fast_remote_cache method, which updates a svn working-copy cache on the remote server, and then hardlinks the result.
I use Apache Ant to deploy to different targets (dev, QA and live). Ant is designed to work for Java deployment, but it provides a pretty useful general case solution for deploying arbitrary files.
The syntax of the build.xml file is pretty easy to learn - you define different targets and their dependencies which run when you call the ant program on the command line.
For example, I have targets for dev, QA and live, each of which depends on the cvsbuild target which checks out the latest head revision from our CVS server, copies the appropriate files to the build directory (using the fileset tag), and then rsyncs the build directory to the appropriate server. There are a few quirks to learn, and the learning curve is not totally flat, but I've been doing it this way for years with no trouble so I'd recommend it for your situation, though I'm curious what other answers I'll see on this thread.
I do stuff manually using Git. One repository for development, which gets git push --mirror'ed to a public repo, and the live server is a third repo pulled from that. This part I suppose is the same as your own setup.
The big difference is that I use branches for nearly every change I'm working on (I've got about 5 right now), and tend to flip back and forth between them. The master branch doesn't get changed directly except for merging other branches.
I run the live server direct from the master branch, and when I'm finished with another branch and ready to merge it, flip the server to that branch for a while. If it breaks, putting it back to master takes seconds. If it works, it gets merged into master and the live code gets updated. I suppose an analogy of this in SVN would be having two working copies and pointing to the live one via a symlink.
I know Phing has been mentioned a few times now, but I've had great luck with phpUnderControl. For us we
Check out individual copies of branches to local machines
Branches are tested and then merged into Trunk
Commits to Trunk are automatically built by phpUnderControl, runs tests and builds all documentation, applies database deltas
Trunk gets run through quality testing and then merged into our Stable branch
Again, phpUnderControl automatically builds Stable, runs tests, and generates documenation and updates database
When we're ready to push to production we run a rsync script that backs up Production, updates the database, and then pushes the files up. The rsync command is invoked by hand so that we make sure someone is watching the promotion.
an alternative to home-made deployment scripts is to deploy to a platform-as-a-service which abstracts away a lot of that work for you. A PaaS will typically offer its own code deployment tool, as well as scaling, fault-tolerance (eg. not going down when hardware fails), and usually a great toolkit for monitoring, log checking etc. There's also the benefit of deploying to a known good configuration which will be kept up-to-date over time (one less headache for you).
The PaaS I would recommend is dotCloud, in addition to PHP (see their PHP quickstart) it can also deploy MySQL, MongoDB and a whole bunch of additional services. It also has nice goodies like zero-downtime deployment, instant rollback, full support for SSL and websocket, etc. And there's a free tier which is always nice :)
Of course I'm slightly biased since I work there! Other options worth checking out in addition to dotCloud are Pagodabox and Orchestra (now part of Engine Yard).
Hope this helps!
Solomon
That you automatically and blindly take changes from a repository to production servers sounds dangerous. What if your committed code contains a regression bug, so your production application gets glitchy?
But, if you want a Continuous Integration system for PHP, I guess Phing is the best choice for PHP. I haven't tested it myself, though, as I do stuff the manual way of e.g. scp.
I am way late to the party, but I thought I would share our methods. We use Phing with Phingistrano, which provides Capistrano-like functionality to Phing via pre-built build files. It is very cool, but only works if you use Git at the moment.
I have a working copy of an SVN release branch on the server. Updating the site (when there aren't schema changes) is as easy as issuing an SVN update command. I don't even have to take the site offline.
Phing is probably your best bet, if you can stand the pain of xml configuration files. The Symfony framework has its own port of rake (pake), which works quite well, but is rather tightly coupled to the rest of Symfony (Though you could probably separate them).
Another option is to use Capistrano. Obviously it doesn't integrate as well with PHP, as it does with Ruby, but you can still use it for a lot of stuff.
Lastly, you can always write shell scripts. So far, that's what I have done.
http://controltier.org/wiki/Main_Page
we are going to use it for multi-server deployments & maintenance.
One year late but...
In my case, deployment is not automatic. I find it dangerous to deploy code and run database-migration scripts automatically.
Instead, subversion hooks are used to deploy only to testing/staging server. Code is deployed to production at the end of an iteration, after having run tests and made sure things will work. For the deployment itself, I use a custom-made Makefile that uses rsync for transferring files. The Makefile may also run the migration scripts on the remote server, pause/resume web and database servers.
At my work myself and my team have developed a Phing oriented replacement for capistrano's deploy and we've also incorporated some of the goodies available in phing like PHPUnit testing, phpcs and PHPDocumentor. We've made it a git repo that can be added to a project as a submodule in git and it works very well. I've attached it to a handful of projects and it's modular enough that it's easy to make it work with any project on any of our several environments (staging, testing, production, etc...).
With the phing build scripts you can run them from the command line manually, and I've also had success automating the build/deploy routines with Hudson and now Jenkins ci.
I can't post any links now because the repo isn't public yet, but I've been told we're going to open source it sometimes soon, so please feel free to contact me if you're interested or if you have any questions on automating your deployment with phing and git.
I guess SVN deploy way is not very good. Because:
You need to open the SVN access for the whole world
have many .svn in the production web servers
I think Phing to produce a branch + combine all the js/css + replace stage config + ssh upload to all www servers is better way.
ssh to 10 www server and svn up is also trouble.
We currently use a hand-rolled setup and configuration script and a hand-rolled continuous integration script to build and deploy our application. I am looking at formalizing this somewhat with a third party system designed for these purposes.
I have looked into Phing before, and I get that it's basically like Ant. But, my Ant experience is somewhat limited so that doesn't help me much. (Most of the Java work I have done was just deployed as a jar file).
I have looked into Cruise Control before, and I understand that phpUnderControl is a plug-in for CC. But, Phing says it also works with CC. So I am not clear on the overlap here. Do I need both Phing and phpUnderControl to work with CruiseControl, or are they mutually exlclusive?
What I need exactly is something that can:
Check out source from SVN
Install the database from SQL file
Generate some local configuration files from a series of templates and an ini file
Run all of our unit tests (currently ST, but easy to convert to PHPUnit) and send an email to the dev team if any tests break (with a stack trace of course)
Generate API documentation for the application and put it somewhere
Run a test coverage report
Now, we have just about all of this in one form or another. But, it'd be nice to have it all automated and bundled together in one process.
phing is pretty much ant written in PHP where phpUnderControl adds support for PHP projects to CruiseControl and uses phing or ant on the backend to parse the build.xml file and run commands.
I just set up CruiseControl and phpUnderControl and it's been working great. It checks out my SVN, runs it through phpDocumentor, PHP_CodeSniffer, and PHPUnit whenever we do a check in. Since it's all based off of the build.xml file you can run just about any software you want through it.
I'm sure lots of people will say this by the time I've typed this but...
I know it's not PHP but we're finding Capistrano just the job for this kind of thing. It really is an excellent piece of software.
We've been using Phing, and the cost to set it up has been very low; it's really easy to learn even if you don't know ANT. I've had very bad experiences with CruiseControl (instability - going down randomly) - so I like the simplicity of Phing. Plus, it's easily extensible using PHP (in case you have a custom task that they don't support out of the box).