I have worked within a web development company where we had our local machines, a staging server and a a number of production servers. We worked on macs in perl and used svn to commit to stage, and perl scripts to load to production servers. Now I am working on my own project and would like to find good practices for web development when using shared web hosting and not working from a unix based environment (with all the magic I could do with perl / bash scripting / cron jobs etc)
So my question is given my conditions, which are:
I am using a single standard shared web hosting from an external provider (with ssh access)
I am working with at least one other person and intended to use SVN for source control
I am developing php/mysql under Windows (but using linux is a possibility)
What setup do you suggest for testing, deployment, migration of code/data? I have a xampp server installed on my local machine, but was unsure which methods use to migrate data etc under windows.
I have some PHP personnal-projects on shared-hosting ; here are a couple of thoughts, from what I'm doing on one of those (the one that is the most active, and needs some at least semi-automated synchronization way) :
A few words about my setup :
Some time ago, I had everything on SVN ; now, I'm using bazaar ; but the idea is exactly the same (except, with bazaar, I have local history and all that)
I have an ssh access to the production server, like you do
I work on Linux exclusivly (so, what I do might not be as easy with windows)
Now, How I work :
Everything that has te be on the production server (source-code, images, ...) is commited to SVN/bazarr/whatever
I work locally, with Apache/PHP/MySQL (I use a dump of the production DB that I import locally once in a while)
I am the only one working on that project ; it would probably be OK for a small team of 2/3 developpers, but not more.
What I did before :
I had some PHP script that checked the SVN server for modification between "last revision pushed to production" and HEAD
I'm guessing this homemade PHP script looks like the Perl script you are currently usng ^^
That script built a list of directories/files to upload to production
And uploaded those via FTP access
This was not very satisfying (there were bugs in my script, I suppose ; I never took time to correct those) ; and forced me to remember the revision number of the time I last pushed to production (well, it was automatically stored in a file by the script, so not that hard ^^ )
What I do now :
When switching to bazaar, I didn't want to rewrite that script, which didn't work very well anyway
I have dropped the script totally
As I have ssh access to the production server, I use rsync to synchronise from my development machine to the production server, when what I have locally is considered stable/production-ready.
A couple of notes about that way of doing things :
I don't have a staging server : my local setup is close enough to the production's one
Not having a staging server is OK for a simple project with one or two developpers
If I had a staging server, I'd probably go with :
do an "svn update" on it when you want to stage
when it is OK, launch the rsync command from the staging server (which will ba at the latest "stable" revision, so OK to be pushed to production)
With a bigger project, with more developpers, I would probably not go with that kind of setup ; but I find it quite OK for a (not too big) personnal project.
The only thing "special" here, which might be "linux-oriented" is using rsync ; a quick search seems to indicate there is a rsync executable that can be installed on windows : http://www.itefix.no/i2/node/10650
I've never tried it, though.
As a sidenote, here's what my rsync command looks like :
rsync --checksum \
--ignore-times \
--human-readable \
--progress \
--itemize-changes \
--archive \
--recursive \
--update \
--verbose \
--executability \
--delay-updates \
--compress --skip-compress=gz/zip/z/rpm/deb/iso/bz2/t[gb]z/7z/mp[34]/mov/avi/ogg/jpg/jpeg/png/gif \
--exclude-from=/SOME_LOCAL_PATH/ignore-rsync.txt \
/LOCAL_PATH/ \
USER#HOST:/REMOTE_PATH/
I'm using private/public keys mecanism, so rsync doesn't ask for a password, btw.
And, of course, I generally use the same command in "dry-run" mode first, to see what is going to be synchorised, with the option "--dry-run"
And the ignore-rsync.txt contains a list of files that I don't want to be pushed to production :
.svn
cache/cbfeed/*
cache/cbtpl/*
cache/dcstaticcache/*
cache/delicious.cache.html
cache/versions/*
Here, I just prevent cache directories to be pushed to production -- seems logical to not send those, as production data is not the same as development data.
(I'm just noticing there's still the ".svn" in this file... I could remove it, as I don't use SVN anymore for that project ^^ )
Hope this helps a bit...
Regarding SVN, I would suggest you go with a dedicated SVN host like beanstalk or use the same server machine to run an SVN server so both developers can work off it.
In the latter case, your deployment script would simply move the bits to a staging web folder (accessible via beta.mysite.com) and then another deployment script could move that to the live web directory. Deploying directly to the live site is obviously not a good idea.
If you decide to go with a dedicated host or want to deploy from your machine to the server, use rsync. This is also my current setup. RSync does differential syncs (over SSH) so it's fast and it was built for just this sort of stuff.
As you grow you can start using build tools with unit tests and whatnot. This leaves only the data sync issue.
I only sync data from remote -> local and use a DOS batch file that does this over SSH using mysqldump. Cygwin is useful for Windows machines but you can skip it. The SQL import script also runs a one line query to update some cells such as hostname and web root for local deployment.
Once you have this setup, you can focus on just writing code and remote deployment or local sync and deployement becomes a one click process.
One option is to use a dedicated framework for the task. Capistrano fits very well with scripting languages such as php. It's based on Ruby, but if you do a search, you should be able to find instructions on how to use it for deploying php applications.
Related
I'm just trying to find an easier way to deploy a site I'm working on. I'm working alone with a test a production server and right now deployment means copying a subset of the files and database data onto my computer and uploading it to the prod site. I'm sure there's a simple synchronization tool out there but so far I've had no luck in finding anything.
What I'd really like is an application I can run locally (on windows) or something I could install on my server for let me have a one-click deployment. Any suggestions?
Thanks!
godwin
Edit
I have decided for now to go with GoodSync and Toad. Thanks for the suggestions.
man scp
SCP(1) BSD General Commands Manual SCP(1)
NAME
scp - secure copy (remote file copy program)
SYNOPSIS
scp [-1246BCpqrv] [-c cipher] [-F ssh_config] [-i identity_file] [-l limit] [-o ssh_option] [-P port] [-S program] [[user#]host1:]file1
[...] [[user#]host2:]file2
DESCRIPTION
scp copies files between hosts on a network. It uses ssh(1) for data transfer, and uses the same authentication and provides the same
security as ssh(1). Unlike rcp(1), scp will ask for passwords or passphrases if they are needed for authentication.
Any file name may contain a host and user specification to indicate that the file is to be copied to/from that host. Copies between two
remote hosts are permitted.
When copying a source file to a target file which already exists, scp will replace the contents of the target file (keeping the inode).
If the target file does not yet exist, an empty file with the target file name is created, then filled with the source file contents. No
attempt is made at "near-atomic" transfer using temporary files.
The options are as follows:
-1 Forces scp to use protocol 1.
-2 Forces scp to use protocol 2.
...
I use GoodSync http://www.goodsync.com/ for this sort of thing. It's really good. Runs on windows, can sync between any combination of local files (S)FTP, windows, linux network shares etc.
Then create a scheduled task/cronjob to run an export of the database into the syncronised folder and have one do an import at the other end. Obviously this process is one way.
http://www.phing.info/docs/guide/stable/
PHing is an automated build system made for PHP. Works with GIT, SVN, PHPUnit, etc...
You basically set up XML files that give PHing instructions on what to do. Allows you to run test suites along with build creation, build multiple varied versions at a time, copy files as well as db, and a bunch of other cool features.
Also, it's open source and platform independent.
What are you using for source control? Some tools like Git and SVN have ready-made methods for this sort of thing. See here for a quick Git solution.
I would second the advice about Git/SVN, but would put in a strong plug for Git via GitHub. Use GitHub as your "central" Git repository. Your local Git repository will push to GitHub, and your production server will pull from GitHub.
There is some overhead to learning Git/GitHub, but really, in the situation you've described (a single engineer and two servers), Git isn't any more complicated then SVN (or CVS or anything else).
We use an FTP Synchronizer, which seems to work pretty well. I don't know offhand of any good free ones.
Example: http://www.ftpsynchronizer.com/
Depends on what type of server you are running, but you could run SVN (Subversion). There is a plugin for Eclipse, Aptana, and Zend Studio if you use that to develop.
Essentially you could have a development repository that sits on the server. You would pull your code down to your local environment and commit it back after changes. Then you can setup another repository that is your live data or production thats linked back to your Development repository.
When you want to update the live data, you just update it so if any trouble happens you can roll back that code without having to roll back your development code. Once you get good at all that you can start branching and tagging your projects.
I personally use both SVN and Git, but I prefer Git because it works so much better. Though if you are using Windows, the command line tools just aren't the same as linux.
I have a question related to phing. Do I use it on my local development machine to deploy an application or do I call it via ssh from production machine? I'm not sure where to do the steps from Eran Galperin (What is your preferred PHP deployment strategy). I'm a beginner with deployment scripts. So be forgiving :)
Marco
You'll usually have two deploy scripts - One that you run locally, which ssh's in to the remote server and then kicks off the remote script.
Your local script might do an svn export, zip the contents and push them (scp) to the remote server first. The remote script would do stuff like migrating, restarting services etc.
The basic strategy is that you create a build (well, basically a copy of your web application that you can put online) on your local machine, and then deploy it on the server.
You run it on your local machine and the PHPing will connect to production server and deploy the website. :)
Phing can be applied in many ways.
If you don't have directives on how to deploy, the best way is usually first to generate a "test build" from your local copy, check out if it is working as intended and then generate the productive build.
In some cases where you have to follow certain rules in your development enviroment you can easily adapt Phing to meet those demands.
I developed a phing deploy script and I've used it in both of the ways you've described. As long as the phing library and dependencies are installed locally or on the remote machine you can do it in both ways, and it's actually a benefit to be able to run it from any of your environments.
Let me elaborate on my experience, but keep in mind that every person lives within the microcosm of the needs and requirements of their own projects/studios so it's possible that what works for me, may not work for other people.
I think deployment is usually so important for a business that it needs to happen in the exact same way every time. When there's a multitude of people doing the deployments, then it's easy for bad habits or procedures to diverge and that's when mistakes can happen. Automated deployments with phng are handy because if everyone is using the same procedure, then there's much lower potential for mistakes. Phing makes it easy to follow strict procedures because all the developers have to do is to run a simple command like "phing deploy" or "phing build" or what have you.
Now going back to your original question, it's also helpful for these procedures to be adhered to on local development machines as well as on staging, testing, production servers because just like people, machines can have small ideosyncracies that differ from machine to machine. With phing, you can develop a script that will have the same results on any of your enterprise machines (depending on the strength and intent of your code but phing lives in a format that makes it easy to meet that ideal). So if you run your phing build from your development box, ideally it has the same effect if you run it from your testing box or your production server. If you can run the deployment the same way on any machine, you can feel confident that any problems that arise after deployment were not a result of your deploy procedures (as they will not differ from when you deployed to your testing or local dev box).
It's really nice to be able to deploy my code to my local machine with the exact same procedure that I use to deploy it to production.
As far as what is the best to do, I say, why not all of the above? If you can run it locally, then you can run it remotely. If the deploy procedures are strong, then the result will be the same.
You can do it both ways. You can run the tool locally as you develop to run tests and just generally ensure that your code is in good condition. The server automation (if you choose to do that) can also run tests, generate documentation, and deploy changes to test or production setups.
Doing it locally reduces the chance that the process will get stalled on the server setup, while the CI works on a regular basis to ensure nothing has been broken via mistake.
So we are pushing to create good processes in our office. I work in a web shop that has been doing web sites for over a decade. And we don't use version control. I know! It's bad, not my fault. I'm the guy with a SoftE background pushing for this at a minimum.
The tech lead has been looking into it. We all use Mac workstations and mostly use Coda for editing since it is a great IDE. It has SVN support built in but expects it to work on local files. We're trying to explore mounting the web directory as a local network drive with an SFTP tool.
We are a LAMP shop, BTW.
I am wondering what the model is here. I think we have typically would checkout the whole site to our local machine where we have apache running and then test it there? This isn't how we work yet, we do everything on the server. We've looked at checking things in and out, but some files are owned by apache and the ownerships change when I check them in, because I'm not apache.
I just want to know a way to do this that works given my circumstances. Would be nice to not have to run apache locally.
You might want to checkout the Coda mailing list and ask there. Lots of Coda enthusiasts there with specific experience.
If you don't want to have to run locally could make Apache on your server run a copy of the site for every developer, on a different port per person, and then mount those web-roots to the local macs and make that the working directory. If you're a small shop that's not hard to manage. I find that pretty easy to set up and saves a lot of resources on the local machines. The one-site-per-person helps to avoid conflicts with multiple people working on files at the same time.
What I'd additionally recommend is to have a script that gets the latest changes from SVN and deploys the entire site to the production server when you're ready. You could have that script change permissions on appropriate files/folders as needed to be owned by Apache. The idea once you're using source control is to never manually edit the production files -- you should have something that deploys it from SVN for you.
A few notes:
Take a look at MacFuse / MacFusion (the latter is the application, the former is the library behind it) to mount remote directories via SSH / FTP as local ones.
Allow your developers to check out into their local environment (with their own LAMP stack if they're savvy), or look into a shared dev environment with individual jails. This way your developers can run their own LAMP stack (which you could deploy for them on the machine) without interfering with others.
The idea being, let them use a workflow that works best for them, to minimize the pain in adapting to this change (if change management might be an issue!)
Just as an example, we have a shared dev server where jails are created with a single command for new developers. They have a full LAMP stack ready to go, and we can upgrade and re-deploy jails easily to keep software up to date. Developers have individual control to add custom settings / extensions if they need it for work, while the sys admins have the ability to reset everything when someone accidently breaks their environment :)
Those who prefer not to use jails, and are able to, manage their own local environments (typically through Macports or MAMP).
I've heard the phrase "deploying applications" which sounds much better/easier/more reliable than uploading individual changed files to a server, but I don't know where to begin.
I have a Zend Framework application that is under version control (in a Subversion repository). How do I go about "deploying" my application? What should I do if I have an "uploads" directory that I don't want to overwrite?
I host my application through a third party, so I don't know much other than FTP. If any of this involves logging into my server, please explain the process.
Automatic deploy + run of tests to a staging server is known as continuous integration. The idea is that if you check in something that breaks the tests, you would get notified right away. For PHP, you might want to look into Xinc or phpUnderControl
You'd generally not want to automatically deploy to production though. The normal thing to do is to write some scripts that automates the task, but that you still need to manually initiate. You can use frameworks such as Phing or other build-tools for this (A popular choice is Capistrano), but you can also just whisk a few shell-scripts together. Personally I prefer the latter.
The scripts themselves could do different things, depending on your application and setup, but a typical process would be:
ssh to production server. The rest of the commands are run at the production server, through ssh.
run svn export svn://path/to/repository/tags/RELEASE_VERSION /usr/local/application/releases/TIMESTAMP
stop services (Apache, daemons)
run unlink /usr/local/application/current && ln -s /usr/local/application/releases/TIMESTAMP /usr/local/application/current
run ln -s /usr/local/application/var /usr/local/application/releases/TIMESTAMP/var
run /usr/local/application/current/scripts/migrate.php
start services
(Assuming you have your application in /usr/local/application/current)
I wouldn't recommend automatic updating. Just because your unit tests pass doesn't mean your application is 100% working. What if someone checks in a random new feature without any new unit tests, and the feature doesn't work? Your existing unit tests might pass, but the feature could be broken anyway. Your users might see something that's half-done. With automatic deployment from a check-in, you might not notice for a few hours if something made it live that shouldn't have.
Anyhow, it wouldn't be that difficult to get an automatic deployment going if you really wanted. You'd need a post-check-in hook, and really the steps would be:
1) Do an export from the latest check-in
2) Upload export to production server
3) Unpack/config the newly uploaded export
I've always performed the last steps manually. Generally it's as simple as SVN export, zip, upload, unzip, configure, and the last two steps I just alias a couple of bash commands together to perform. Then I swap out the root app directory with the new one, ensuring I keep the old one around as a backup, and it's good to go.
If you're confident in your ability to catch errors before they'd automatically go live, then you could look at automating that procedure. It gives me the jibbly-jibblies though.
At my webdev company we recently started using Webistrano, which is a Web GUI to the popular Capistrano tool.
We wanted an easy to use, fast deployment tool with a centralized interface, accountability (who deployed which version), rollback to previous versions and preferably free. Capistrano is well-known as a deployment tool for Ruby on Rails applications, but not centralized and targeted mainly to Rails apps. Webistrano enhances it with a GUI, accountability, and adds basic support for PHP deployment (use the 'pure file' project type).
Webistrano is itself a Ruby on Rails app, that you install on a development or staging server. You add a project for each of your websites. To each project you add stages, such as Prod and Dev.
Each stage can have different servers to deploy to, and different settings. Write (or modify) a 'recipe', which is a ruby script that tells capistrano what to do. In our case I just used the supplied recipe and added a command to create a symlink to a shared uploads dir, just like you mentioned.
When you click Deploy, Webistrano SSHs into your remote server(s), does an svn checkout of the code, and any other tasks that you require such as database migrations, symlinking or cleanup of previous versions. All this can be tweaked of course, after all, it's simply scripted.
We're very happy with it, but it took me a few days to learn and set up, especially since I wasn't familiar with Ruby and Rails. Still, I can highly recommend it for production use in small and medium companies, since it's proven very reliable, flexible and has saved us many times the initial investment. Not only by speeding up deployments, but also by reducing mistakes/accidents.
This sort of thing is what you would call "Continous Integration". Atlassian Bamboo (cost), Sun Hudson (free) and Cruise Control (free) are all popular options (in order of my preference) and have support to handle PHPUnit output (because PHPUnit support JUnit output).
The deployment stuff can be done with a post build trigger. Like some other people on this thread, I would exercise great caution before doing automated deployments on checkin (and test passing).
check fredistrano, it's a capistrano clone
works great (litle bit confusing installing but after all runs great)
http://code.google.com/p/fredistrano/
To handle uploads, the classic solution is to move the actual directory out of the main webspace, leaving it only for a fresh version to be checked out (as I do in the script below) and then using Apache to 'Alias' it back into place as part of the website.
Alias /uploads /home/user/uploads/
There are less choices to you if you don't have as much control of the server however.
I've got a script I use to deploy a given script to the dev/live sites (they both run on the same server).
#!/bin/sh
REV=2410
REVDIR=$REV.20090602-1027
REPOSITORY=svn+ssh://topbit#svn.example.com/var/svn/website.com/trunk
IMAGES=$REVDIR/php/i
STATIC1=$REVDIR/anothersite.co.uk
svn export --revision $REV $REPOSITORY $REVDIR
mkdir -p $REVDIR/tmp/templates_c
chown -R username: $REVDIR
chmod -R 777 $REVDIR/tmp $REVDIR/php/cache/
chown -R nobody: $REVDIR/tmp $REVDIR/php/cache/ $IMAGES
dos2unix $REVDIR/bin/*sh $REVDIR/bin/*php
chmod 755 $REVDIR/bin/*sh $REVDIR/bin/*php
# chmod -x all the non-directories in images
find $IMAGES -type f -perm -a+x | xargs -r chmod --quiet -x
find $STATIC1 -type f -perm -a+x | xargs -r chmod --quiet -x
ls -l $IMAGES/* | grep -- "-x"
rm dev && ln -s $REVDIR dev
I put the revison number, and date/time which is used for the checked-out directory name. The chmod's in the middle also make sre the permissions on the images are OK as they are also symlinked to our dedicated image server.
The last thing that happens is an old symlink .../website/dev/ is relinked to the newly checked out directory. The Apache config then has a doc-root of .../website/dev/htdocs/
There's also a matching .../website/live/htdocs/ docroot, and again, 'live' is another symlink. This is my other script that will remove the live symlink, and replace it with whatever dev points to.
#!/bin/sh
# remove live, and copy the dir pointed to by dev, to be the live symlink
rm live && cp -d dev live
I'm only pushing a new version of the site every few dats, so you might not want to be using this several times a day (my APC cache wouldn't like more than a few versions of the site around), but for me, I find this to be very much problem-free for my own deployment.
After 3 years, I've learned a bit about deployment best practices. I currently use a tool called Capistrano because it's easy to set up and use, and it nicely handles a lot of defaults.
The basics of an automated deployment process goes like this:
Your code is ready for production, so it is tagged with the version of the release: v1.0.0
Assuming you've already configured your deployment script, you run your script, specifying the tag you just created.
The script SSH's over to your production server which has the following directory structure:
/your-application
/shared/
/logs
/uploads
/releases/
/20120917120000
/20120918120000 <-- latest release of your app
/app
/config
/public
...etc
/current --> symlink to latest release
Your Apache document root should be set to /your-application/current/public
The script creates a new directory in the releases directory with the current datetime. Inside that directory, your code is updated to the tag you specified.
Then the original symlink is removed and a new symlink is created, pointing to the latest release.
Things that need to be kept between releases goes in the shared directory, and symlinks are created to those shared directories.
It depends on your application and how solid the tests are.
Where I work everything gets checked into the repository for review and is then released.
Auto updating out of a repository wouldn't be smart for us, as sometimes we just check in so that other developers can pull a later version and merge there changes in.
To do what you are talking about would need some sort of secondary check in and out to allow for collaboration between developers in the primary check in area. Although I don't know anything about that or if its even possible.
There is also issues with branching and other like features that would need to be handled.
I have to deploy my php/html/css/etc code to multiple servers and i am looking at my options for software that allows easy and secure deployment to multiple servers.
Also helps if it could be tied into my SVN.
Any suggestions?
Capistrano is pretty handy for that. There's a few people using it (1, 2, 3) for deploying PHP code as evidenced by doing a quick search.
Setting up password-less publickey authentication with ssh would allow you to scp your files to any of your servers very quickly (or be automated by a shell script).
Here's a simple tutorial: http://rcsg-gsir.imsb-dsgi.nrc-cnrc.gc.ca/documents/internet/node31.html
If you're running on Redhat or Debian, consider packaging up your code into RPM's or Debs. Then build a yum or dpkg repository and put your packages there. You can then use your system's package management to do upgrades/rollbacks, etc. You can even use puppet to automate the process.
If you want to tie it into subversion, you can create a branch for each new version. Use the commit scripts to build the RPM's when a new branch shows up in a directory.
I'll second Capistrano. It's incredibly powerful and flexible. Our current project uses Capistrano for deploying to different servers as well as multiple servers. We pass two arguments to the cap command:
1) the name of the set of machine specific config options to run and
2) the name of the action to run
ends up looking like this:
cap -f deploy.rb live deploy
or
cap -f deploy.rb dev deploy
Of course the default use case - deploy to lots of machines at once - is a doddle with Capistrano AND you don't need to have Capistrano on the machines you are deploying to. All in all, tasty technology.
I've used Automated Build Studio before for a similar task. It gives you a lot of flexibility in what you can do.
I concur -- set your svn tree up, and use rsync over ssh to copy the tree out to the remote locations. rsync will make it fast and efficient, only copying changes rather than full files.
You want to export your svn tree to some directory, then rsync from there to the remote host's directory tree.
I also forgot to mention that if you use rsync, you can set up rsync to use ssh, so you will only transfer the files that have changed, saving on time and bandwidth.
You can also use kwateeSDCM which is free and allows remote installation via ssh. It also enables you to manage server-specific configuration from a central location and make upgrades seemless.
I had marked a post on how to deploy your websites using Subversion : http://blog.lavablast.com/post/2008/02/I2c-for-one2c-welcome-our-new-revision-control-overlords!.aspx
I found capistrano to be very easy to use once it's setup. The configuration file can be a bit confusing at first for more complicated environments but it soon becomes worthwhile. I deploy to 14 servers on production. I also use multiple environments for deployment to a staging server. One quirk, there's a bug in Ruby that breaks parallel deployment but serially isn't too bad with svn exports.
Capistrano setup is just too complicated. We found that KwateeSDCM was very straightforward to use with a simple web interface and no scripting. We've got our deployment configuration done in no time for Dev and QA configuration on windows and linux servers.