What is the best way to deal with config / directory / OS differences using SVN to develop and deploy on 2 different platforms?
I am developing in Windows and deploying to Linux.
I'd like to use my home PC to test the files before using commit, I'm using a web framework and you must specify the paths to your system and application files for the framework. Obviously those are different on the 2 machines.
I have the option of using a linux server at home, as a last resort. I really like working with SVN using a GUI right now since I am new to SVN web development.
So what would be the best solution to config differences between my development and live servers?
There has got to be an easier way then changing the file everytime I commit it to the live server, which is really against what SVN is trying to accomplish in the first place.
One option is to keep the configuration files out of the repository - set them up on their own for both your local and server machines, and then don't touch them again unless you have to. (Using the svn-ignore: property can keep you from accidentally adding the config files to the repository later on.)
Another option is to try to use relative paths and then have the config/etc directories be in the same relative location to the working directory, even if they're not in the same absolute locations.
You can set a svn-prop (property) to ignore the config file. Forcing you to create one (manually or generated) specific to every installation.
Related
I use a Linux PC for development purposes. In the PC I develop a few PHP based web sites and projects. Each project belongs to a different client and the majority of those is hosted in different hosting companies.
The directory structure of each project is:
bin
lib
www
logs
conf
For each project I use some standard, developed by me, libraries that require a standard configuration, which is stored in the "conf" directory, like for example, the DB connection data.
For that I use the variable
$configdir=$_SERVER['HOME'].DIRECORY_SEPARATOR."conf
For instance, the backup script use the file $HOME/conf/db.ini in order to retrieve the specific data and backup the DB of the project.
On the production servers, the variable $configdir is (obviously) different for each project. But, on the development server is NOT.
I do not want to use pathinfo, or basename or dirname since I want to use the configuration file path in scripts that run from any possible depth. For example, I would like to use that variable in a script under $HOME/bin/calculations/script.php.
Since, on the development server, $configdir will show the same directory for all projects, I need your ideas on how to keep the same structure, and use the same code in production and development while that variable use the correct "$HOME" for each project.
I'm developing web app using CodeIgniter PHP framework. The server I'm working with does not support any type of source control (i.e. Subversion) unless you go to a higher price tier.
I would still like to put the code under some sort of source control. Does it make sense to do the following:
Install git or SVN on my local machine and develop there
Copy changes from my local machine to development directory on the server (using FileZilla, WinSCP, etc.) and test
Copy changes from development directory to production directory on the server
Does that sound reasonable? Are there better alternatives? Thanks!
If you are using svn locally it's a bit dangerous because then you will need to protect also your computer - I think the best way is to work with the commercial sites offer fully supported svn/git - like http://www.beanstalkapp.com/ or http://www.github.com
You could use source control on your local machine (SVN, Git, etc.) and use an open source tool like Capistrano to deploy the code from your local source control repo to your server via SSH. Or if you're limited to FTP, this blog post has a potential solution.
An advantage of using a tool like Capistrano instead of directly mirroring the files on your local machine to the server via FileZilla or WinSCP is that Capistrano will version your deployed files so that, if you end up breaking something and need to roll back quickly to the previously-deployed version, it can be as easy as changing a symlink to the previous deployment directory.
Does that sound reasonable?
Partially, in p.1. But even in this case I'll suggest to have your repository also at some Repository-Hosting (BitBucket, GitHub, Assembla)
For pp. 2-3: your deploys must to be automatic and non-interactive, thus - you'll have to select another tools (for using in post-commit hook of SCM-of-choice)
Somehow better alternative to 2-3 may be:
Use 2 different branches (DEVEL and PROD) as sources of DEVEL and PROD dir
Post-commit hook, which upload only changed in committed revision files to corresponding dir (NCFTP for FTP, SCP with scenario for ssh)
Main development happens in DEVEL branch
PROD have only mergesets from DEVEL
Workflow is SCM-agnostic and scalable to any reasonable amount of branches and developers
Currently using LAMP stack for my web app. My dev and prod are in the same cloud instance. Now I am getting a new instance and would like to move the dev/test environment to the new instance, separating it from the prod environment.
It used to be a simple Phing script that would do a SVN export into the prod directory (pointed to by my vhost.conf). How do I make a good build process now with the environments separated?
Thinking of transferring the SVN repository to the dev server and then doing a ssh+svn push (is this possible with Phing?)
What's the best/common practice for this type of setup?
More Info:
I'm currently using CodeIgniter for MVC framework, Phing for automated builds for localhost deployment. The web app is also supported by a few CRON scripts written in Java.
Update:
Ended up using Phing + Jenkins. Working well so far!
We use Phing for doing deployments similar to what you have described. We also use Symfony framework for our projects (which is not so much important for this but Symfony supports the concept of different environments so it's a plus).
However we still need to produce different configuration files for database, front controllers etc.
So we ended up having a folder with build.properties that define configuration for different environments (and in our case also for different clients we ship our product to). This folder is linked to the file structure using svn externals (again not necessary).
The Phing build.xml file then accept a property file as a parameter on the command line, takes the values from it and produces all necessary configuration files, controllers and other environment specific files.
We store the configuration in template files and then use copy/filter feature in Phing to replace the placeholders in the templates with the specific values.
The whole task of configuring the given environment can then be as simple as something like this:
phing configure-environment -DpropertyFile=./build_properties/build.properties.prod
In your build file you check if the propertyFile property that specifies the properties file is defined and load the file using <property file="./build_properties/build.properties.prod" override="true" />. Then you just do any magic with the values as you need.
You can still use your svn checkout/update and put all the resulting configuration files into svn ignore (you will have them generated by phing). We actually use additional steps in Phing. Those steps in the end produce a Linux shell installation self-deploy package. This is produced automatically in Jenkins. We then send the package to our clients or the support team can grab the package from Jenkins and they can do the whole deployment just by executing it (we still prefer manual deployments to production servers) or Jenkins can deploy it automatically (for example to test servers).
I'll be happy to write more info if needed.
I recommend using Capistrano (looks like they haven't updated the docs since they moved the site) and railsless-deploy for doing deployment. Eventually, you are probably going to need to add more app boxes and run other tasks as part of your deployment so choosing a framework that will support this can save you a lot of time in the future. I have used capistrano for two PHP deployments (one small and one large) and although its not perfect, it works well. It also handles all of the code checkout / update, moving symlinks into place, and rolling back if something goes wrong.
Once you have capistrano configured, all you have to do is something like:
cap dev deploy
cap prod deploy
Another option that I have explored for doing this is fabric. Although I haven't used it, if I had to deploy a complex app again, I would consider it. The interface is simple and straightforward.
A third option you might take a look at thought its still in the early stages of development is gantry (pardon the self promoting). This is something I have been working on out of frustration with using capistrano to deploy a PHP application in an environment with a lot of moving pieces. Capistrano is great and works well for non PHP application deployments, but you still have to some poking around in the code to understand what is happening and tweak it to suit your needs. This is also why I suggest giving fabric a good look.
I use a similar config now. Lamp + SVN + codeigniter + prd and dev servers.
I run the svn repos on dev. I checkout the repos into the root folder of the dev domain. Then use a post-commit hook to update the root folder everytime any developer commits.
When we are happy and have fully tested the code I ssh into the prd server and rsync the dev root to the prd root.
Heres my solution for the different configs. Outside the root folder I have a config.ini file. I parse the file in my codeigniter constants.php script. This means that the prd and dev server can have separate settings without them ever being in the repos.
If you want help with post-commit, rsync and ini code let me know.
So we are pushing to create good processes in our office. I work in a web shop that has been doing web sites for over a decade. And we don't use version control. I know! It's bad, not my fault. I'm the guy with a SoftE background pushing for this at a minimum.
The tech lead has been looking into it. We all use Mac workstations and mostly use Coda for editing since it is a great IDE. It has SVN support built in but expects it to work on local files. We're trying to explore mounting the web directory as a local network drive with an SFTP tool.
We are a LAMP shop, BTW.
I am wondering what the model is here. I think we have typically would checkout the whole site to our local machine where we have apache running and then test it there? This isn't how we work yet, we do everything on the server. We've looked at checking things in and out, but some files are owned by apache and the ownerships change when I check them in, because I'm not apache.
I just want to know a way to do this that works given my circumstances. Would be nice to not have to run apache locally.
You might want to checkout the Coda mailing list and ask there. Lots of Coda enthusiasts there with specific experience.
If you don't want to have to run locally could make Apache on your server run a copy of the site for every developer, on a different port per person, and then mount those web-roots to the local macs and make that the working directory. If you're a small shop that's not hard to manage. I find that pretty easy to set up and saves a lot of resources on the local machines. The one-site-per-person helps to avoid conflicts with multiple people working on files at the same time.
What I'd additionally recommend is to have a script that gets the latest changes from SVN and deploys the entire site to the production server when you're ready. You could have that script change permissions on appropriate files/folders as needed to be owned by Apache. The idea once you're using source control is to never manually edit the production files -- you should have something that deploys it from SVN for you.
A few notes:
Take a look at MacFuse / MacFusion (the latter is the application, the former is the library behind it) to mount remote directories via SSH / FTP as local ones.
Allow your developers to check out into their local environment (with their own LAMP stack if they're savvy), or look into a shared dev environment with individual jails. This way your developers can run their own LAMP stack (which you could deploy for them on the machine) without interfering with others.
The idea being, let them use a workflow that works best for them, to minimize the pain in adapting to this change (if change management might be an issue!)
Just as an example, we have a shared dev server where jails are created with a single command for new developers. They have a full LAMP stack ready to go, and we can upgrade and re-deploy jails easily to keep software up to date. Developers have individual control to add custom settings / extensions if they need it for work, while the sys admins have the ability to reset everything when someone accidently breaks their environment :)
Those who prefer not to use jails, and are able to, manage their own local environments (typically through Macports or MAMP).
I have created a PHP web-application.
I have 3 environments: DEV, TEST, PROD.
What's a good tool / business practice for me to move my PHP web-application code from DEV to TEST to the PROD environment?
Realizing that my TEST environment still only connects to my TEST database; whereas, I need to PROD environment to connect to my PROD database. So the code is mostly the same, except that I need to change my TEST code once moved into PROD to connect to the PROD database and not TEST database.
I've heard of people taking down Apache in such away that it doesn't allow new connections and once all the existing connections are idle it simply brings down the web server.
Then people manually copy the code and then manually update the config files of the PHP application to also point to the PROD instance.
That seems terribly dangerous.
Does a best practice exists?
1) Separate configuration from the rest of the code. The rest of the code should then be able to run on all 3 locations without modifications. A typical config file could be:
<?
$db = "main_db"; $db_user="web1"; $db_pass = "xyz123";
$site ="example.com";
$htroot = "/var/www/prod/htdocs"
?>
And for the test environment:
<?
$db = "test_db"; $db_user="web1"; $db_pass = "xyz123";
$site ="test.example.com";
$htroot = "/var/www/test/htdocs"
?>
Do not include the configuration files with the passwords in the code base. The code base may be copied through insecure connections or be stored on third party code hosting servers later (see below). And you maybe don't want your passwords on all your backup disks of the code.
You could also create one single config file and use a switch depending on the environment the code runs:
<?
$site = $_SERVER["HTTP_HOST"];
if ($site == "example.com"; ) {
$db = "main_db"; $db_user="web1"; $db_pass = "xyz123";
$htroot = "/var/www/prod/htdocs";
}
if ($site == "test.example.com") {
$db = "test_db"; $db_user="web1"; $db_pass = "xyz123";
$htroot = "/var/www/test/htdocs";
}
?>
But now you could be tempted to put it back into the code base, which is less secure as explained above. And if you do not put it there you have to update 3 files, or use one fixed location per server, and you have to make sure the code finds the file on each server. I personally prefer the one-file-per-site solution from above.
2) You have already "versions". There is one version running on prod now. Give it a unique name and number, which will never change again. You can use that version name when you backup the code and when you refer to the version or when you move it somewhere you name the subdiectory that will contian it after the version.
The version that you will put on prod in the near future is a different version, and if you make changes again this is also a different version.
As a rule of thumb: increase the version number when you move or export the code, when you swap or exchange or upgrade between locations, when you make a demo, and after each feature or milestone and each time when you do a full backup.
Please note that the config files (3, one for prod, test and dev) are NOT part of the versions. So you can "move the versions around" but the not the config files. If you can, put the config files OUTSIDE the tree with the rest of the code, so you do not need to separate them and take care when you move around the versions later. You could move the config one directory "up" and access them from the files like this:
"include ../config.php";
3) If you want to use version control systems, they do a great job but it needs some time to get used to it and if you are in a hurry with your update it is probaby not the right time to start live with it now. But I would for the future recommend to use a latest generation distributed version control system. Distributed means you do not need to setup a server and many more advantages. I will name bazaar, if it is necessary to update over ftp it can do. Please note that a version control system makes exchanging a version very fast, because only the differences between the versions are written. Bazaar has a community and documentation which makes it easy to start. There is also Git, which has the most up to date commercial hosting site: http://github.com. You can view the code online and compare between the versions and there are many more helpful features, even if you are the only coder, but in a group it is even better. The choice between the systems is not easy. I can not recommend CVS, which is outdated. Also SVN is not the latest generation of distributed version control system, I would not recommend to use it if there is not a specific reason, and it will pollute all your subdirs with special subdirs, which can be annoying. For peolple who are used to it and have already code in it it is fine, but for a starter I would say don't.
There is also Mercurial and Darcs among the distributed and open source version control systems. Mercurial also has a great commercial site for collaboration and online code view (http://bitbucket.org).
4) As long as you do not use a version control system, how about using symlinks?
You could have a directory on the server src/versions/ somewhere and put the named versions in there, each one in their own subdirectory. You will only add versions (because a version that exists will not be changed, if you change it it becomes a new version)
You could have src/versions/v001/ src/versions/v002/ src/versions/v003/ or whatever naming scheme you use.
Now here comes the trick: /var/www/prod/htdocs is a symlink to src/versions/v001/
When you upgrade to v002 you just do the following:
shutdown apache
remove the old symlink /var/www/prod/htdocs (at this point the apache webroot is gone!)
create the new symlink /var/www/prod/htdocs being a link to src/versions/v002
start apache
You can also write a script for this with parameters and call it like this:
upgrade-web prod 002
This makes the gap even shorter.
Sometimes you have to do a "fallback", when you find out that the new version has errors in production and you have to go back. This would be easy (because you do not remove the old directories, you just stop apache, delete the symlink and re-create it to the former location, in this case src/versions/v001 )
And if test and dev is on the same server, you can of course also symlink the same directories, so there would be no any for move or copy.
5) If you do it manually without symlinks, why not move instead of copy?
(When the files are not yet on the same server, you can copy them somewhere near, and then start with the migration, so you do not have to stop the server for such a ling time)
If there are several directories on the root level of the project you could move them one at time. Be sure to NOT MOVE the config files. Or find some strategy to bring them bakc. Workflow would be:
Stop apache
Move away all current prod directories and files on the root level except config file(s)
Move all new prod directories and files to the root level except config file(s)
Start apache
6) try to find your perfect individual file and directory layout and perfect workflow. This takes maybe some time and some thinking, but it pays. Do it on a piece of paper until you find the best solution. This could mean that you have to refactor your code and change server config files, but for the future your life is easier when you do administration and upgrades. From my experience: do not wait so long with this step. Your layout shoudl nmake it easy and safe to upgrade. Upgrading is not somehting extraordinary, it is routine and it should be safe and simple to do.
7) If you name your server and workstation environments (operating system of server is linux I guess, but is it a hosted or a root server, do you have ftp acces or also shell (ssh) access, or sftp? where do you develop, on a windows machine, or a mac?) then people can name tools to do the copying and moving. Also interesting: Is the test and dev server the same machine, if not, how they are connected, or they aren't? If not, you would make a 3-way transfer (Copy it to your local workstation and then to the server).
8) Think of file permissions. If you move files around or copy them, maybe the file permissions change, and if the application depends on some of them there should be a way to check and maybe chang. Example: Some applications need writable directories where they put uploaded files or session files or template caching. Other applications do not allow some files for security to be writable.
Use configuration files to determine what database you're connecting to. That is, have a DEV configuration file, a TEST configuration file, and a PROD configuration file. It's generally the best way to avoid costly and frustrating mistakes.
Actually, I don't see any reason why TEST environment should miraculously migrate to PROD without any server shutdowns. TEST production is supposed to be for testing purposes. And even if you are actually TESTING on production server, bring it down (shutdown apache), change one line in your main config file, that is determining what set of minor config files to use) and bring it up again (start apache). This will take not more than 1-3 mins to complete and since you surely not going to do that twelve times a day, you will be fine.
Have your code in a revision control system (I prefer Subversion (svn)). This makes it easy to keep your DEV, TEST and PROD environments in sync, you don't have to keep track of files you modified. Once you are happy with your modifications on DEV, you commit the changes to svn and then run "svn update" on the TEST and eventually after testing on PROD server. Most linux hosting providers have svn client installed or you can install it yourself.
I don't like having a different version of a config file for each site because it requires manually renaming one file and removing the other two. I prefer having DEV, TEST and PROD configurations in the same config file. In the config file I determine which server the code is running on by checking either the hostname or the request url. Then I can have "if" or "switch" statement that would load configuration settings based on which server is currently running the code.
You might also need to sync database structure between your servers. I use sqlyog for this purpose, it has a built-in database structure synchronization tool that compares 2 database structures and prepares SQL to synchronize them.
When we push updates live, it's not often we have to reboot apache. This maybe a side effect of not having high traffic sites (< 1M pagedraws a month).
We have 3 branches, for various stages of development/QA: alpha (bleeding edge but available for viewing by non-developers and testers), beta (somewhat frozen for a particular release, final QA phase) and live (production).
To migrate from one branch to another, we perform a merge between say alpha and beta, commit that merge. Then run a deployment script which updates the branch from SVN on our development machine, then rsync's the code web servers to the beta document root.
As already mentioned by others, each branch can contain it's own config file with appropriate settings to cater for environment differences.
We are in the process of migrating to git to smooth out the branch merging process, which can be a little traumatic in SVN for large projects/releases.