I currently ftp all my files to my website when i do an update (over a slowish adsl connection)
And I want to make things easier, so I just recently started using a hosted svn service, and i thought i could speed things up a bit by doing an svn export of my website directly onto my webserver
i have tried that a few times and it seems to work ok, however it does fetch the entire site everytime which is a bit slow for a 1 file update
so my questions are
is it possible to do an export and only get the changes since the last export (how will this handle deleted files ?)
OR will it be easier to do an svn checkout and svn update it all the time instead of svn export and just hide the .svn folders using apache htaccess
is this a good idea, or is there a better way to publish my website
i am trying to achieve the 1 click deploy type ideal
maybe there are some gotcha's i haven't thought of that someone else has run into
debian/apache/php
I would do an svn checkout, and have done so successfully on a live site for a number of years. You should add mod_rewrite rules to 404 the .svn directories (and files) though.
This is what I'm doing on my host:
For every project I have a structure that looks more less like this:
~/projects/myproj
~/public_html/myproj
First dir is a checkout from SVN, while second one is just svn export.
I have a small bash script
#!/bin/bash
SOURCE="$HOME/projects/"
TARGET="$HOME/public_html/"
for x in `ls $SOURCE`
do
if [ -d $SOURCE$x ]; then
svn update $SOURCE$x
svn export --force $SOURCE$x $TARGET$x
fi
done
Export is done from working copy so it's very fast.
It might not be exactly the answer you are looking for, but, if you have an SSH access to your webserver (it depends on your hosting service ; some "low cost" don't give such kind of access), you can use rsync to "synchronise" the remote site with what you have on your disk.
In the past, I was using something like the idea you are describing (fetching the svn log between last revision pushed to production and HEAD, analysing every lines, and in the end calculating what to send to the server) ; but it was not really a great process ; I now use rsync, and like it way better.
(Here too, you will have to exclude .svn directories, btw)
You can just accept having .svn directories in your website (generally not a problem esp. if you configure it not to permit access to these) - this is the easy option. Alternatively, do what RaYell does, and have two copies of your website on the webserver. One normal checkout outside of the web-directory, and one in your web-directory. When you update, simply export the svn (just a copy with .svn dirs deleted) into the web directory (and you should make sure to first delete old files if you wish to avoid files that have been removed from SVN from remaining on your website).
I do something like this, using robocopy set to mirror the svn checkout while excluding .svn directories, and get both the export and the old-file deletion in one step, thus minimizing downtime if the copy takes long. I'm sure this is easy on unix too, if that's your hosting environment. For example, you can use a local rsync: http://blog.gilluminate.com/2006/12/12/yes-you-can-rsync-between-two-local-directories/
Old topic but since this is what came up in Google during my research I thought I would add to this. I recommend doing an export to the site instead of checking out to it. I like to keep the repositories and the sites separate. I also don't recommend exporting the entire repo to the site each time, specially if only a few file change at a time. What you can do instead is do a diff on the repo to see what's changed from a release to another and only export those files. More info at: http://www.joeyrivera.com/2011/automate-svn-export-to-site-w-bash-script/
Related
Recently I've read this article: http://www.smashingmagazine.com/2009/09/25/svn-strikes-back-a-serious-vulnerability-found/
Developers of many popular sites like apache.org, php.net (http://ru2.php.net/.svn/entries), classmates.com and russian Yandex use SVN, but do not follow the recommendations given by SVN (to use command export).
So, what are the reasons for not using svn export instead of updating the public copy like all they do?
Some people, not including myself, think that to deploy onto production you should just issue an svn up. If you do an export it loses the meta data about the versioning so you can t do that, you have to use another mechanism for tracking which version is where. It is an easy solution, but I think it can make for lazy packaging and also for "fixing in production" as if you do this you can also check back in from production...
From my perspective what I do is lock off/block access to any .svn files on the server (either Apache2 or IIS) this way the hidden folders are not accessible externally, and it allows for version tracking for sites that we use which do not require compiling before rollout
Languages like:
PHP
ASP (not .NET)
PLAIN HTML
COLDFUSION
PDF / IMAGE versioning (if needed, in my case we needed it for updated PDF docs for customers).
So certainly you can use SVN for web development, but you do need to be cautions as you expose your .svn folders to the world if you are not cautious. Otherwise it is a tool you could use to make your job easier and more efficient.
With that said, we simply run an SVN UPDATE on our production to update changed files, and with limited developers working on one piece of code at a time (like I said in my case) we don't get mixups with wrong things getting deployed. PLUS to be safe, always do a SVN CHECK FOR MODIFICATIONS to see what is going to be updated, and hey, if you do make a mistake, roll it back.
With svn export files can never get deleted, only added and modified. This could be issue sometimes.
When the entire website is open-source and available for downloading over a public resource (like PHP's). Protecting the .svn directories so other's can't get the source code is probably not worth the effort over simply doing a svn up.
I am developing (solo web developer) a rather large web based system which needs to run at various different locations. Unfortunately, due to some clients having dialup, we have had to do this and not have a central server for them all. Each client is part of our VPN, and those on dialup/ISDN get dialed on demand from our Cisco router. All clients are accessable within a matter of seconds.
I was wondering what the best way to release an update to all these clients at once would be. Automation would be great as their are 23+ locations to deploy the system to, each of which is used on a very regular basis. Because of this, when deploying, I need to display a 'updating' page so that the clients don't try access the system while the update is partially complete.
Any thoughts on what would be the best solution
EDIT: Found FileSyncTask which allows me to rsync with Phing. Going to use that.
There's also a case here for maintaining a "master" code repository (in SVN, CVS or maybe GIT). This isn't your standard "keep editions of your code in the repo and allow roll backs"... this repo holds your current production code (only). Once an update is ready you check the working updated code into the master repo. All of your servers check the repo on a scheduled bases to see if it's changed, downloading new code if a change is found. That check process could even include turning on the maintenance.php file (that symcbean suggested) before starting the repo download and removing the file once the download is complete.
At the company I work for, we work with huge web-based systems which are both Java and PHP. For all systems we have our development environments and production environments.
This company has over 200 developers, so I guess you can imagine the size of the products we develop.
What we have done is use ANT and RPM build archives for creating deployment packages. This is done quite easily. I haven't done this myself, but might be worth for you to look into.
Because we use Linux systems we can easily deploy RPM packages, the setup scripts within a RPM package can make sure everything gets to the correct place. Also you get a more proper version handling and release process.
Hope this helped you.
Br,
Paul
There's 2 parts to this, lets deal with the simple one first:
I need to display a 'updating' page
If you need to disable the entire site while maintaining transactional integrity, and publishing a message to the users from the server being updated, then the only practical way to do this is via an auto-prepend - this needs to be configured in advance (note - I believe this can be done using a .htaccess file without having to restart the webserver for a new PHP config):
<?php
if (file_exists($_SERVER['DOCUMENT_ROOT'] . '/maintenance.php')) {
include_once($_SERVER['DOCUMENT_ROOT'] . '/maintenance.php');
exit;
}
Then just drop maintenance.php into your webroot and that file will be displayed instead of the expected file. Note that it should probably include a session_start() and auto-refresh to ensure the session is not expired. You might want to extend the above to allow a grace period where POSTs will still be processed e.g. by adding a second php file.
In terms of deploying to remote sites, I'd recommend using rsync over ssh for copying content files - which should be invoked via a controlling script which:
Applies the lock file(s) as shown above
runs rsync to replicate files
runs any database deployment script
removes the lock file(s)
If each site has a different set up then I'd recommend either managing the site specific stuff via a hierarchy of include paths, or even maintaining a comlpete image of each site locally.
C.
At my company we have a group of 8 web developers for our business web site (entirely written in PHP, but that shouldn't matter). Everyone in the group is working on different projects at the same time and whenever they're done with their task, they immediately deploy it (cause business is moving fast these days).
Currently the development happens on one shared server with all developers working on the same code base (using RCS to "lock" files away from others). When deployment is due, the changed files are copied over to a "staging" server and then a sync script uploads the files to our main webserver from where it is distributed over to the other 9 servers.
Quite happily, the web dev team asked us for help in order to improve the process (after us complaining for a while) and now our idea for setting up their dev environment is as follows:
A dev server with virtual directories, so that everybody has their own codebase,
SVN (or any other VCS) to keep track of changes
a central server for testing holding the latest checked in code
The question is now: How do we manage to deploy the changed files on to the server without accidentaly uploading bugs from other projects? My first idea was to simply export the latest revision from the repository, but that would not give full control over the files.
How do you manage such a situation? What kind of deployment scripts do you have in action?
(As a special challenge: the website has organically grown over the last 10 years, so the projects are not split up in small chunks, but files for one specific feature are spread all over the directory tree.)
Cassy - you obviously have a long way to go before you'll get your source code management entirely in order, but it sounds like you are on your way!
Having individual sandboxes will definitely help on things. Next then make sure that the website is ALWAYS just a clean checkout of a particular revision, tag or branch from subversion.
We use git, but we have a similar setup. We tag a particular version with a version number (in git we also get to add a description to the tag; good for release notes!) and then we have a script that anyone with access to "do a release" can run that takes two parameters -- which system is going to be updated (the datacenter and if we're updating the test or the production server) and then the version number (the tag).
The script uses sudo to then run the release script in a shared account. It does a checkout of the relevant version, minimizes javascript and CSS1, pushes the code to the relevant servers for the environment and then restarts what needs to be restarted. The last line of the release script connects to one of the webservers and tails the error log.
On our websites we include an html comment at the bottom of each page with the current server name and the version -- makes it easy to see "What's running right now?"
1 and a bunch of other housekeeping tasks like that...
You should consider using branching and merging for individual projects (on the same codebase), if they make huge changes to the shared codebase.
we usually have a local dev enviroment for testing (meaning, webserver locally) for testing the uncommited code (you don't want to commit non functioning code at all), but that dev enviroment could even be on a separeate server using shared folders.
however, committed code, should be deployed to a staging server for testing before putting it in production.
You can probably use Capistrano even though is more for ruby there are some articles that describe how to use it for PHP
I think Phing can be use with CVS but not with SVN (at least that what I last read)
There are also some project around that mimic Capistrano but written in PHP.
Otherwise there is also a custom made solution :
tag files you want to deploy.
checkout files using the tag in a
specific directory
symlink the directory to the current
document root (easy to rollback to
the previous version)
Naturally check out SVN for the repository, Trac to track things, and Apache Ant to deploy.
The basic process is managing in Subversion, tracking the repositroy and developers in Trac and using Ant deployment scripts to push your site out with the settings needed. Ant allows you to easily deploy a project to a specific location. (Dev/test/prod) etc.
You need to look at:
Continuous Integration
Running unit tests on check-in of code to check it is bug free
Potentially rejecting code if it contains a bug
Having nightly builds
Releasing only the last build that was bug free
You may not get to a perfect solution, especially not at first, but the more you use your chosen solution, the more comfortable everyone will get and be able to make suggestions on improving it.
We check for the stability with ant, every night. And use ant script to deploy. It is very easy to configure and use.
I gave a similar answer yesterday to another question. Basically you can work in branches and integrate before going live.
The biggest thing you will have to get your head round is that you are dealing with changes to files, rather than individual files. Once you have branches there isn't really a current version there are just versions with different changes in.
I am using SVN to manage a copy of my web site. The site runs a typo3 installation, which uses PHP to create the backend.
The problem is, all the stupid .SVN folders show up in the directory listing of PHP. I DO NOT mean the build in apache listing. I mean a directoy listing created by the PHP backend.
So, is there any way to hide special directories from PHP?
[NOTE]
Changing the PHP source code is not an option. Typo3 is too big, and each extensions uses its own code. Would be much more effort than an SVN export script.
Chris
PS: I do not want to setup a svn export --> web_root just to get rid of the files. And I know that I can prevent apache from serving the .SVN directories, I did that. But they still show up in the backend, when browsing the directory tree (which is created by PHP). And they are very annoying...
This is difficult, since you will have to change behavior of something somewhere between the filesystem and Typo3. You have:
Filesystem → Operating System → PHP → Typo3
The files must stay in the filesystem and must stay visible by the operating system, so you can use SVN. Changing Typo3 is not an option for you, and changing PHP has many other major undesirable consequences that you should avoid. So, what you have left is to insert something in between OS→PHP or PHP→Typo3.
The first case is actually possible, depending on what operating system you use, and if you have administrator (root) access. FUSE is part of the Linux kernel, and is also available for many other operating systems. Then, with fuse, you may install a filter like rofs-filtered, that allows you to filter which files and directories are visible in a mounted volume. You use it to create a filesystem that mirrors your SVN checkout directory, filtering the .svn directories.
So, is there any way to hide special directories from PHP?
No.
As long as the user PHP is run under has read access to the directory it will always produce all the files/directories in that directory. There is no way to hide files from certain processes, were this possible writing a root kit to hide from ls and other file system tools would be a lot easier.
The option you would want/need is a way to define files that Typo3 ignores, and have it be system wide and thus used by the extensions as well. You have specified however that you do not want to change the source code, and do not want to do svn export.
You are thus stuck with the .svn directories.
The short answer is "Not easily, simply, or sanely".
Run the website from an export of SVN, not a checkout, instead.
Try this.
<locationmatch "/.svn/">
order allow,deny
deny from all
</locationmatch>
Btw in your loop in PHP you can do a logic check to see if the filename is not ".svn", usually PHP directory tools do that to exclude "." and ".." directories.
The problem is, all the stupid .SVN
folders show up in the directory
listing of PHP. I DO NOT mean the
build in apache listing. I mean a
directoy listing created by the PHP
backend.
What application is doing the directory listing? Have you considered looking into the code of the PHP backend and adding something to prevent the display of the .svn directories?
Just find or write a very simple application that will synchronize your current directory with a new directory that will be exposed to the Web. You could have a service that watches for changes or use something like an rsync with exclusions or what have you. This would be much simpler since, based on another question, you are on Windows.
ther's an extension called np_subversion which will take care of fileadmin changes via subversion. As a nice plus it will hide folders for you
I do not want to setup a svn export --> web_root just to get rid of the files
Are you sure? That’s how SVN is designed: you check code out of SVN to work on it, and export code from SVN to deploy it. If you don’t like that, then SVN probably isn’t the right choice. As gahooa said, maybe switch to Git?
It’s a bit like saying “I want to save my Word document, but I don’t want this stupid .doc file showing up on my computer.” That’s just how the software works.
Sara Golemon's Runkit can do this. You can remap functions like glob(). However, I am not sure if it's a good idea to run it in a production server.
If you don't need the .svn folders, you can just delete them.
find ./ -name ".svn" | xargs rm -f *.svn
I've gotten comfy with SVN, and now I need a way to deploy my code to staging or live servers more easily. I'd also like some method for putting build info in the footer of this site to aid in testing. Site is PHP/MySQL.
First enable keyword substitution for a file where you wish to have the revision info:
svn propset svn:keywords "Rev" file.txt
The add to the file where you want the Revision info stored:
$Rev$
Further readings: http://svnbook.red-bean.com/en/1.4/svn.advanced.props.special.keywords.html
The properties methods will only give you the last revision number of the file you have the property in, not the head revision of the whole repository (a la the Stack Overflow footer). If you are wanting that, you'll need to use svnversion.
I recently started using Capistrano on a project and it superb and very flexible and powerful. I ended up deviating quite far from its normal usage, but it makes one "click" deployment much easier.
If you want to update the version number in a projects AssemblyInfo.cs you may be interested in this article:
CodeProject: Use Subversion Revision numbers in your Visual Studio Projects
If you enable SVN Keywords then every time you check in the project Subversion scans your files for certain "keywords" and replaces the keywords with some information.
For example, At the top of my source files I would create a header contain the following keywords:
'$Author:$
'$Id:$
'$Rev:$
When I check this file into Subversion these keywords are replaced with the following:
'$Author: paulbetteridge $
'$Id: myfile.vb 145 2008-07-16 15:24:29Z paulbetteridge $
'$Rev: 145 $
A script to svn update on an as needed basis.
SVN supports keywords. You can add which keywords you want to expand to the keywords property, and SVN will expand then. See $Id$ or $Rev$, or keywords described in the SVN book.
I'm a fan of using capistrano for pushes. Refer to here.
You could use the SVN $Rev$ property to get the revision number into your footer.
A really simple way to manage this is to setup your app in the following way:
Simply make your deployment app a working copy of your trunk (svn co the project to your /www root) and you run an svn up through an ssh console (ssh user#host.com svn up /path/to/project) when you need to update. You can also rollback with the appropriate checkout mechanisms. This is important: if you do this, add RewriteRules (or equivalent) to your .htaccess (or equivalent) to disallow access to .svn directories. If you can't do the above, run an svn export through ssh instead (so it won't be a 'working copy'), but this will naturally be slower than doing an up.
Also, you can look at what Ruby on Rails does with Capistrano.. it's the same basic concept but supports transactional backups if the update goes wrong in the middle by storing each checkout in a separate folder and symlinking the "latest" to your /www directory.
The keywords stuff will fail in plenty of cases -- like if you've modified the source before deploying, or if you check in from one directory in your project then a different directory in the same project will have different revision numbers. Check the docs carefully to make sure the keywords do what you think they do.
The better way is to use the svnversion program to generate information about your checked out directories at compile or deployment time. Svnversion will show information about the version of ALL of your directories as well as flagging whether or not the source was locally modified.
The method I've come up for my php projects, may not be the best method but after some time searching certainly seems to be, is to do a checkout, run a version check, wipe out the .svn folders, and move on. Here is a portion of shell script I've written:
(first, you need the script the checkout your repo)
# get the svn revision number and create a RELEASE file
svnvers=`svnversion .`
echo "version: $svnvers"
echo "<release><development>0</development><revision>$svnvers</revision></release>" > RELEASE
# remove all .svn directories
find . -name .svn -exec rm -rf {} \;