I've stumbled upon the following problem and can't figure out a decent solution. I make websites in PHP for various clients. Like all clients, some will find bugs that need fixing, some will request updates to live sites after a few months and some do both.
When working on updates for clients, I like to preview them to the clients before putting them live.
I've used a couple of different solutions in the past, but none I'm happy with. What I've done so far:
Define VERSIONs and CURRENT_VERSION in the index. Visitors see the approved version. I send the client a specific link that sets a $_SESSION variable, which lets them see the new version. In the code I work with if's and switches to show new things depending on CURRENT_VERSION. This works, keeps all the code in one place and easily allows bugfixing, but riddles the code with stupid if(CURRENT_VERSION >= V2) statements. I also can't use this in CSS files.
Put all files in a "build1" folder, when starting on big updates that need previewing, I copy everything to a "build2" folder. I upload the "build2" folder for the client and password protect it. This works pretty well, however, if I have to bugfix build1 while working on build2, I have to make sure I copy fixes from build1 to build2.
Use a development server (some clients can provide this) this works the best so far, as the dev server is separate from the live site. It works much the same as the second solution, where I have to make a copy of the project, but it feels cleaner to me.
I am however looking for a better way to manage my code, possibly with the use of Git/SVN, but do not know enough about these things if they could help me.
A fairly typical paradigm these days is Development/Staging/Production. You don't need an entire development server for this approach either, VirtualHosts/nginx equivalent will suffice.
I'd suggest that the first thing you need to do is get your projects into Git, the quicker the better!
Disclaimer: This is my workflow, there are many like it, but this one is mine.
Here's an example of my current work flow.
GitHub
A separate repository for every project I've taken on
Development Server
Bare Git Repositories, replicating my GitHub (I'll explain this shortly)
/opt/git-bare
VirtualHosts of all my projects
/var/www/vhosts
Local Machine
I clone my bare repositories, as I need, for quick editing and commits. I'm not worrying about FTP'ing files back and forth or mounting anything locally on to my machine. I find this the BEST way to work on a project. When I am ready to check out some code on my development server, I simply commit and push my work to the bare repository, where I have a post hook that then tells my development VirtualHost to update itself.
This means that within seconds of commiting/pushing my work, from my local machine, I can see it on my development server through my browser.
When I am happy with what I have seen on my development server, I then push my bare repository up to GitHub. Git is a wonderful tool and all my local commits are also available within the logs on GitHub.
Staging
This is a clone, from GitHub, of my master branch on GitHub. This is what I use for showing to clients and getting changes signed off.
Production
My production server is a clone of a tag from GitHub. No matter what I do within my master branch, production will never be effected and should anything ever go wrong with one of my servers I have this tag easily available for rebuilds.
If you have any questions about this, please just fire away.
Before my long answer: if you're looking for a host that would kind of do this for you, stackable supports the concept of multiple 'enviroments'. I'm sure other hosting platforms offer similar features that allow essentially the same thing (AWS Elastic Beanstalk for example), but I don't know of one that is as core to the offering. Note: I don't have any connection to stackable, I'm not even a customer.
Define VERSIONs and CURRENT_VERSION in the index...but riddles the code with stupid if(CURRENT_VERSION >= V2) statements. I also can't use this in CSS files.
If I recall correctly, this is actually similar to how Facebook rolls out changes. You're right, it adds that additional logic; however there's an advantage as you're able to 'preview' the changes to more than a single user (say, all users that are admins, or all users in a specific geographic location).
And of course, the preview uses the same data - which means the user previewing the site will use it like they normally do (instead of odd interaction with contrived data).
While you're right as to the disadvantage, there are cases where this is a useful way to test new features.
Put all files in a "build1" folder, when starting on big updates that need previewing...however, if I have to bugfix build1 while working on build2, I have to make sure I copy fixes from build1 to build2.
Here you're essentially deploying two versions of the project to the same server. In the example you give, you're putting the second copy under the original webroot - but depending on hosting, you could just assign a subdomain and work from two different web roots.
The advantage is similar to the first, as both installs could easily share the same data, and if all requests pass through some kind of front controller you can add logic to only show changes to select users (or use some kind of Basic Auth as you describe).
In this case putting your project into version control (as I see it, git would be better than SVG for this) can make this much easier. On your development system simply switch between branches to work between the existing version and the new version.
If you fix a bug in the old version, you should be able to easily (or more easily than your current workflow) merge that fix into the new version with a few commands. If you fix a bug in the new version which was also in the old version, doing a cherry-pick can allow you to just merge that single change back into the old version.
Deploying your code can be as basic as logging into your web server and doing a git pull, or you could use tools to automate the deployment. Essentially your deploy of the old version would be based on the 'master' branch of your repository (or something similar to that), and the new version would be based off whatever you've called that branch.
Use a development server (some clients can provide this) this works the best so far, as the dev server is separate from the live site. It works much the same as the second solution, where I have to make a copy of the project, but it feels cleaner to me.
As this is very similar to your second method, adding version control will certainly make this easier as well.
There are plenty of resources explaining how to deploy from various version control systems to various hosting platforms, but hopefully this illustrates how that will fit into what you're already doing and make things easier for you.
Related
I've checked other threads about this subject, but nothing really answers my exact question. I'm currently developing a site that updates data on a live web server (usually photos) multiple times daily using a cron and some external data feeds. Additionally, we're about to allow user-submitted data. Now, I want to start using svn because the project is getting big enough where it's starting to get relatively complicated and people are making edits to our code on the live server. I've installed svn for apache and all of the subdomains mentioned below are set up on one server using vhosts. We haven't started using this workflow yet, so if I could get some advice before we start it would be greatly appreciated.
Okay, so I've set up a repository under a subdomain (http://svn.website.com), and checked it out to a development subdomain (http://dev.website.com). I also plan to check out a copy to the live environment (http://www.website.com). The idea is to first develop on the dev subdomain, commit when it's ready to go live, and then svn up to the live environment. I've read the pros and cons between exporting and updating, and updating seems best for this project because we often will commit small tweaks/code changes and exporting seems like overkill. I have also setup the appropriate directives to avoid .svn directory access. Does it sound like I'm on the right track with this? Anything I should be aware of?
If the following is a good way to take care of everything, then we can move onto the second question. The user-submitted data (mainly photos) are being uploaded to the live environment. I have to somehow manage this between the live and dev environment so we can develop with up-to-date user-submitted files. I don't really know the best way to do this...I feel like there's a serious flaw if I were to svn add and svn ci every time a file is submitted. What is the best way to accomplish this?
Sounds like you're growing!
First, I'd question as to whether SVN is really the best choice. My guess is that you know it already, have worked with it, as have others on your team, and you want to stick with it. That's fine, but just an FYI I've used SVN and Git and I found Git generally easier to use, and I would argue that it is becoming more and more commonly used by application developers.
As for your workflow, I used to go through an almost identical process with a large distributed team and it worked fine. Depending on the amount of traffic you get, I might suggest a different production deployment strategy. When you're ready for a new production push, clone into a new directory, labeled perhaps by date or milestone names, and create a new virtual host for that directory. That way, you get a couple benefits:
You don't run the risk of having someone access your application and requesting a certain file which is in the process of being updated
You can easily revert back to an earlier milestone by simply changing the active production virtual host. Bugs and problems make it out of development regardless of how thoroughly you test, and sometimes they're serious enough to warrant just reverting back to a previous codebase. Of course, schema changes would add complexity to this, but if we're just talking about code this should avoid frantic commit reverts at 2am.
This was suggested by Rasmus Lerdorf at a conference I attended when he was posed nearly the same question you're asking.
As for ensuring developers are working with the most recent fileset, do you really want to do this? If you're processing a serious quantity of images, that's going to take up a lot of hard disk space. Sure with a CDN or other mass storage no big deal, but do you want your developers to have to download 1000 new photos every time they svn up?
If you're dead set on doing this, I'd suggest adding some code within your app for automatically doing an svn add, svn ci when svn status has a certain result (ie, this is a script you could call with cron, or every time new media gets added). You could also check into using svn hooks (git has these as well) to do things like notify you that the repo has been updated.
Hope this helps. I'd strongly discourage including all of the files users upload unless you have a really, really strong reason for doing so. The complexity that's gonna add is gonna be a pain = )
Good luck and welcome to StackOverflow!
In the past, I have been developing in a very amateurish fashion, meaning I had a local machine where I developed and tested code and a production machine to which I copied the code when I was done. Recently I modified this slightly to where I developed locally, checked the code into SVN and then updated the production machine through SVN.
Now I would like to start a new project and improve my workflow. Ideally I had the following in mind:
Have one or more local dev environments
Develop and test on local machine(s)
Use SVN (or Git) as code repository
Use a build tool to set up new environments (either dev, staging or production) and deploy code
Since I am not very familiar with this process, I am looking for suggestions on how to best set this idea up and the tools to use, especially when it comes to the build tools. I was looking into Ant and Phing (possibly make), but I am so new to this that I would really like to get some guidance. Are there any good tutorials or books about PHP deployment, especially for beginners? What I am especially interested in are the following topics:
Deployment to different types of servers with different settings (e.g. dev uses different db, db passwords, PHP error reporting than production or staging).
Deployment that automatically pulls code from SVN.
Deployment that temporarily sets a "Maintenance" page for production environment.
Once I mastered the above, maybe even do some testing in the build process.
I know my question might sound quite confused... I admit, I am new to this and might be a little off the target in what I really need. That's why any help is greatly appreciated.
I would suggest making your testing deployment strategy a production-ready install-script -- since you're going to need one of those anyway eventually.
A few tips that may seem obvious to some, but are worth pointing out:
Your config file saved in your VCS should be a template, and should be named differently from the file that will eventually contain the actual settings. E.g. config-dist.php or config-sample.conf or sample/config-mysql.php or something along those lines. Otherwise you will end up accidentally checking in a server-specific configuration file over your template.
For PHP deployment, anticipate that some users will not be able to run server-side scripts through any mechanism other than the web server itself. A PHP-based installer is almost non-negotiable.
You should include a consumer-friendly update mechanism, and for that, wordpress is a great example of a project to emulate. A PHP script can (a) download the latest build, (b) use the ftp functions to update your application's files, and (c) execute an update script which makes the appropriate changes to the database, etc.
For heaven's sake don't do like [redacted] and make your users download and install separate patches for each point release. Have them download the latest (final) release which contains all the updates to date, and applies the correct ALTER TABLE functions in sequence.
Whether the files are deployed via SVN or through FTP, the install/update mechanism should be the same: get the latest files, run the update script. The updater uses the version listed in the PHP script and the version listed in the DB, and uses that knowledge to apply the appropriate DB patches in order. As for how to generate those patches, there are other questions here that you can refer to for more info.
As for the "Maintenance" page, just use the version trick mentioned above to trigger it (compare the version in the DB against the version in the PHP code). It's also useful to be able to mark a site as "down" to the public but make it visible to admins (like Joomla does), which you can trigger through database or filesystem flags.
As for automatically pulling code from SVN, I'd say you're better off with either a cron script or with commit triggers than working that into your application, since it wouldn't be relevant to end users.
This isn't exactly part of your question, but it's relevant:
If you go into distributing code intended for a wide audience, I would advise you to go with building and distributing OpenSSL-signed PHAR packages. You can distribute them over HTTP without a problem, and because they're OpenSSL-signed, you're also mitigating the risk of man-in-the-middle attacks and protecting end-users/customers/clients from someone injecting code if you want to setup an automatic or one-click update.
There's a set of tools I've contributed to in the past that work great for this, but you'll either need PHP 5.3, or you'll need PHP 5.2 with PHAR installed via PECL. https://github.com/koto/phar-util
As far as testing goes, PHPUnit is the de facto standard.
If you are interested in using Git then you should check out this build system from CodeMeme. From what you described it sounds like it would be a good fit. You can add it to any project as a submodule and with the included code you can tailor a build script that will deploy to different multiple servers in multiple environments. It uses Git to build the code for deployment but unfortunately SVN is not supported.
https://github.com/CodeMeme/Phingistrano
I've been reading this site here and there and appears as though you guys have a wonderful community.
As for my background, I am a sophomore at university familiar with SQL, C++, Visual Basic, and some PHP. One of my school projects for the summer term involves building a web application that allows users to log in and schedule specific timeslots over the internet. Typically, I have been the only person working on a project, but in this case I will be part of a group. Since we're all relatively new to working as a team, I would like to set up source control for my group so we're not all working off a shared drive somewhere. Additionally, I would like to make sure that all of us are able to test our changes in some sort of development server that hosts an instance of our website.
My actual question is in regards to the toolset that we should use to achieve this. As a group, we are most familiar with PHP and MySQL so we'll end up using that for the code and database. I have used SVN in the past for my own personal use, but my group members aren't very familiar with source control. We'll probably stick with something simple like Excel for the project management and bug tracking side of things. Ideally, we would like the tools to be free and open source.
How as a group should we manage the construction of the actual application? Are there methods out there that I can use that will allow any one of us to move the files to our development machine and keep track of who did it so we don't end up overwriting each other's changes? If this is not possible, one of us will write some scripts to handle it - but I would like to avoid building basically a separate software application that will only be used to manage our project. Another issue I foresee will be updating the database running on the development machine. Are there any standardised methods that we can use to manage our SQL scripts among the four of us?
I do not expect a really long winded answer here (after all, this is our project!), but any helpful tips would be greatly appreciated. Once I return from holiday I am looking forward to getting started! Thanks!
I recommend your group use source control to synchronize your code. You can either setup your own server or just use a free provider such as github, Google code, or bitbucket.
If you do decide to use one of these sites, a nice feature is that they provide free issue tracking as well, so you can use that instead of Excel.
The best way to manage the SQL scripts is to break them out into separate files and place them under source control as well. You can either create .sql files, or use a tool to manage these changes - for example, have a look at Ruby on Rails' Migrations. This may take some effort to setup, but you'll thank yourself later if you are working on a project of any size...
Draw up a plan for how you would do it if it were just you.
Split the plan up into tasks that take around 3-4 hours to complete. Make sure each task has a measurable objective.
Divy out the tasks. Try to sort them if possible to maximize developer efficiency.
Teach them to use source control. Explain to them that they will use this (maybe not svn, but SOMETHING) in a few years, so they might as well learn how now. Additionally, this will help in every group project they do down the road.
Make a script for building and running your tests. Also script your deployment. This will ensure you have the same mechanism going to live as you do going to test, which increases the number of defects found in testing. (This is as opposed to letting them exist but not found in testing.)
You mentioned updating the development database. It would be entirely reasonable to dump the development database often with a refresh from live. You may want to make 3 environments. Development, staging, and production. The development database would contain fabricated test data. The staging database would have a copy of live (recent to within a few days maybe.) And of course live is live.
Excel works fine as a "bug database." Consider putting it in source control that you manipulate and commit. This will give you a good idea of what happened over time, and you can correct mistakes quicker.
As far as source/version control, I would recommend subversion. There are some GUI tools they might use, or even webDAV to access the SVN. This will allow users to edit files collaboratively and also give you details as to who edited what, when, and why... SVN will also do a pretty good job at merging files that happen to be saved at the same time.
It's not the easiest concept to wrap your head around, but its not very complicated once you get running.
I suggest having everyone read the first chapter from: http://svnbook.red-bean.com/en/1.5/
and they should have a good idea of what's happening.
I am also curious to see what people have to say about the database
How as a group should we manage the construction of the actual application? Are there methods out there that I can use that will allow any one of us to move the files to our development machine and keep track of who did it so we don't end up overwriting each other's changes?
It sounds like you're looking for build management. In the case of PHP, a true "build" is as simple as a collection of source files because the language is interpreted; there is no compilation.
It just so happens that I am one of the developers for BuildMaster, a tool which basically solves every problem you have listed in your question... and it also sounds like it would be free in your case under the Community Edition license. I'll try to address some of your individual pain points and how BuildMaster could be used as a solution.
Source Control
As suggested by others, you must use it. The trick when it comes to deployment is to set up some form of continuous integration so that every time someone checks in, a new "build" is created. In BuildMaster, you can set this up for any source control provider you want.
Issue/Bug Tracking
Excel will work, but it's not an optimal solution. There are plenty of free issue tracking tools you can use to manage your bugs and features. With BuildMaster, you can link your bugs and features list with the application by their release number so you could view them within the tool at any time. It can also modify issue statuses and add descriptions automatically if you want.
Deployments
Using BuildMaster, you can create automated deployment plans for your development environment, e.g.:
Get Latest Source Code
Create Artifact
Copy Files To Development Machine
Deploy Configuration Files
Update Database
The best part is, once you set these up for other environments (glowcoder's point #6), pushing all of your code and database updates is as simple as clicking a button.
Another issue I foresee will be updating the database running on the development machine. Are there any standardised methods that we can use to manage our SQL scripts among the four of us?
Database Updates
Not surprisingly, BuildMaster handles these as well by using the change scripts module. When a member of your team creates a script (e.g. ALTER TABLE ADD [Blah] INT NOT NULL) he can upload it into BuildMaster, then run it on any environment you have created.
The best part is that you can add a step in your automated deployment and never worry about it again. As Justin mentions, you can use .sql files for your object code (stored procedures, views, triggers, etc.) and have those executed on every build since they are essentially code anyway. You can keep those in source control.
Configuration Files
One aspect of all this you may have neglected (but will inevitably run into) is dealing with configuration files. With PHP, you may have an .htaccess file, a php.ini file, a prepend.php, or roll your own custom config file. Since by definition configuration files need to change between your personal machine and the development machine, grabbing them from source control wouldn't necessary work without some bit of hacking a la:
if (DEV) {
// do one thing
}
else if (PROD) {
// do another
}
With BuildMaster, you can templatize your configuration files and associate them with an environment so they can be deployed automatically. It will also maintain a history of changes for you.
Automated Testing
If you want the full ALM effect, you can automatically unit test your code during an automated build, and notify you if anything fails so you know as soon as possible that something is broken.
Apologies for the "long winded" response, but I feel like you're already ahead of the game by observing the problems you might run into in the future and really believe BuildMaster will make all of this deployment stuff simple for your team so you can focus on the fun part, coding!
I am developing a website using CodeIgniter and PHP. Some of my friends suggest I start using Subversion to take care of my organization. Could someone give me a basic explanation of what SVN is without the jargon and also if it is practical for a framework where 5 files are needed to run a single page?
Subversion allows you to store your source in a central location, make changes from multiple places (ie. laptop, desktop), track changes and revert changes to your source tree if needed.
Have you ever made a big 'improvement' and then decided you were going in the wrong direction? With Subversion you can revert to the version of your source before you started heading off in the wrong direction.
Or how about this, you make some subtle changes to the codebase, but don't notice that you introduced a new bug until much later. You can use subversion to revert to previous code versions, helping you track down the change that introduced the bug.
It is more valuable when sharing the source with multiple developers, but even for my single developer projects I find it very handy to have all my source and changes in a Subversion repository.
If you combine it with Trac, you can then keep track of bugs/features, milestones, and use the Trac Wiki to document your project.
Every single project (even with only one developer) should be under source control, no matter what framework or language you are using.
Using simple words, Subversion will keep a history of your code files, should you like to revert them to a previous state or restore a copy in case of a disk crash (SVN must be on another computer in this case). It will also help you see easily the differences between two versions of a file.
For more information, just read TortoiseSVN user's guide, it has a pretty good overview of Subversion.
Here's some good info : Chapter 2. Basic Concepts
SVN is a version control system. It is used as a central repository for all of your code.
A significant strength of SVN is that it uses a Copy-Modify-Merge work flow model versus a Lock-Modify-Unlock model. The idea behind this is that each developer checks out or copies (copy) their own version of the code, works on it (modify), checks it back in and reconciles any changes (merge) that may conflict with other work another developer has done.
This is really handy as if you wanted to work on a piece of your code, you won't have to worry about the file being locked because someone else is working on it.
If anything, developers should use SVN to catalogue all revisions of their code and revert back if needed.
SVN is a version control system - which means that it stores various previous versions of a given file, and allows many people to work on the same file, and in the end put all the changes together. It also helps you go back to a previous version if necessary.
You should use that in your project if there are a lot of people working on the project. But even if you are the only one working, it will be easier to manage the project if it is on version control.
It's a way to keep control of versions. You could think of it as some kind of incremental backup. You send your code to the SVN server and it saves only the differences from the last version. This way, you can go back to an earlier version if you need it. It also allows you to work with more people and merge differences with ease. I say, you should use it. It's easy and useful.
Irrespective of what software you produce and even if you are on your own, SVN has to be used. I depend on it every day. Its the one software tool I could not live without. If you can install it off-site it gives you backup as well.
Simple things like being able to do a diff on the previous version of a file when you are making changes, can be a great help.
Use Toortoise under windows for the client side. It integrates into Windows and is just great to use.
I would like to setup two environments for my new website written in PHP. One - to develop new versions and test them. And second production where my actual stable version of website will be available.
Website in PHP will consist of many PHP and other files (JS, images, and so on). So I think how to prepare this environment in best way to make it easy to do source control, fast copy website from development environment to production environment and to make development version available for people on the web so they will be able to see actual work progress and suggest changes or report bugs.
Could you please give me some advice where to go from this starting point? Are there books about this (from practical point of view?) or do you have experience or tips what to watch out and what is important to make this process easy and good for me and other people involved in developing new project?
For starters use the following three:
SVN - this will give you source control and allow you to track changes. You may want to get GUIs on top of this (Tortoise is a popular one) to ease the learning curve.
RSYNC - this will allow you to streamline your syncing between local and remote site with a single command. RSYNC uses a diff engine to sync which means that incremental syncs happen in a matter of seconds. During intense programming, I will sometimes sync 4-5 times in one hour pushing out little changes real fast just because I can so easily.
MySQLDump - This will allow you to import/export data from your production site. I usually do this once a week to get production data on my local servers which not only gives me a local backup but also lets me toy around with production data on a local test environment.
Those three alone will save you a lot of time in the long run and allow you to scale. Later on you can look into an automated build tools, unit testing frameworks, xml documentation framework and the like to build some serious products.
I work with a setup like this, so I can give you some tips on how to do this. I've been doing this for a while now, working out the kinks here and there, and feel like this is a setup I can honestly say is pretty darn productive.
Small note: I work on OSX, so the specific applications used might be a bit different from you if you're a linux/windows user.
I run a production 'server' on my Mac, using MAMP (www.mamp.info) to easily supply me with an Apache server with PHP and MySQL. You could use a similar tool such as XAMPP or install everything manually, it's really up to you.
Then I have my live servers, where my websites and customer websites are hosted. For each new website project (let's take abc.com as an example) I create a subdomain called staging.abc.com, on which I do my testing. It's always a good thing to test things on the exact same hard- and software before actually going live.
I use Subversion (or in short, SVN) for my versioning needs, with the added bonus that I can easily add 'hooks' to automatically update my online production server whenever I send my newly updated version to the SVN server. SVN also allows you to easily work with more than one person on the same project. For more information on SVN and how to use it, I suggest the great (and free) online book found here: http://svnbook.red-bean.com/
So in short: I work locally with MAMP providing me with a local 'working' server. After that, I test online on a staging.abc.com location to see if everything works well, and to possibly allow others to see the project (in case you want your client to see what's going on, for example), and after that I actually publish the project by putting it on the actual domain.
There are many more things that can be done to optimize your workflow, but this should get you started.
Hope this helps!
-Dave
I prefer to have development occur on the developers local box if possible. If other developers are involved, you probably want to setup your version control such that both the database schema, javascript, css, and the php code can be checked out and setup on a developer's personal box pretty easily (assuming they have the correct LAMP/WAMP setup)
I've also seen it where people maintain a test website on a server where active development occurs. I would avoid this for active development, but use this for black-box testing of the latest checked in code (the latest build).
Once your test website checks out, then its a matter of exporting the code from your version control to the location the live website is. With svn, you can really just do an update of the live code with svn update specifying a revision or tag that indicates the current live version.
I would further recommend keeping some settings, like db access/username/pass, in a separate included file that is not version controlled. Keep this elsewhere, let developers plug in the access rights to their local database on their PC. On your server, plug in everything you need to access the database there. This should be really trivial code (defining a few variables) so not having it version controlled shouldn't be a big deal. If you like, you could version control a templated version, but I wouldn't put the real database info into version control.
Here is a pretty good starting place if you want to use MAMP or WAMP to develop locally and then push that to github and then update your live site from github:
http://www.mybringback.com/bringers/14509/git-local-and-shared-server-development-environment-with-ssh-setup/
Hope that helps.