This is more or less related to project management and also with every developer. How you guys handle this situation when you have developed many features on development site and all are tested by client and ready to go live.
These features have some code in common files ie. One PHP file have the code for one feature as well as one other feature.
But client will ask you to upload only 2 feature out of 10 or 15. Files are common if you upload that file directly will leads to error problems because they have code for other features. If you upload all updated files then all feature will be live.
A possible way is go back and comment out that feature which is not needed live for now from common files. But there is possiblities to forgot to comment anywhere else.
This is also not a good way and at last client will say what happen everything was tested on development server and why these bugs and errors are introduced on live server.
This will reduce the faith on developers.
I faced this problem many times and could not found any good way to avoid these issues. So I am thinking that you guys also facing or faced this problem.
I am thinking versioning system can help here.
How you guys are handling this?
Could you share ideas?
The situation you are describing is impossible to manage sanely. I don't believe it would be possible to make this situation work, but the real question is why would you want to?
There are a number of issues with the scenario you describe, but the core issue is really this. You are testing one thing, and deploying another. You acknowledge in your question the interconnected nature of changes. In reality it is even more difficult than you describe. You simply cannot know how a system will behave when you try and deploy parts of a tested solution. Why test it at all?
The only sensible solution I can see is to have a sandbox environment where new features are demonstrated. However keep your test enviornment only for testing stuff that will go live. So in your example the one or two features are in test, ready to be signed off for prod, and the other featues are locked in the sandbox.
This leads to the next problem, which is managing your source code. I don't see any sane strategy for managing the arbirtrary inclusion of features from a code base. Even under the mostflexible system I know, Perforce, any branching straegy would require awful resolves on merges as you try to move stuff in and out.
I have seen this happen, and believe me it gets very ugly.
I suggest you come up with a better solution. Talk to your client and change the way things are done. It will be better for you, and in the long run better for them.
A solution could be to use cheap version branching as provided by VCS such as Git or Mercurial. The project would consist in many feature branches used to develop said features and build branches where feature branches would be merged and adhoc fixing would take place. When a build branch is ready for test, it is tested, fixed if needed and then the build branch is shipped to production platform.
When features have been validated, the build branch can be merged into remaining feature branches so the branches under development can integrate the "official" changes.
To sum up, the application is custom built from existing feature branches as needed.
One reasonably sane way to manage this on the code level is to isolate each feature into a plugin. Then you can add/remove features on-demand by simply enabling or disabling corresponding plugins.
But this solution has certain costs:
Time to develop and test plugin engine for your app
You need to test every plugin configuration (set of enabled plugins and their versions) that is going to be deployed. Otherwise there's a risk that this specific set is not compatible and end users would be first to see resulting crash, or data loss, or some other horror
Additional time to wirte plugins the way that they're minimally dependent on each other.
It's usually worth it only if you have many clients with different needs. In your case, I'd recommend explaining cost of separately enabling features to your client to see if they really need it this hard. Most likely, they don't
I've read that you can host multiple drupal sites, while they use the same core files(so not needing to copy a few megabytes for each site). I wanted to ask if there is an automated tool that can create a new site, while let you choose a template and then connecting it to the drupal system?
Are there tools like that(with a web layout)?
I would really like to get a few pointers as to how, lets say a company for building websites, will be able to use an automated system to build sites easily. I also understand that with drupal you have alot of manuver to edit your own code, when lets say you want some future in one of the sites. Is it pure php/html or in order to do that you have to delve into core Drupal futures? Also what are the chances that somebody already did it before and you can use this module?
Last, if a company wants to move to a Drupal system (web development company), how much of a transformation is it? Should they be Drupal core experts in order to not lose themself? Or they can keep a drupal base while still using the regular html/php? I really appreciate any leads.
Thanks.
*the questions is also intended to Joomla.
To answer your first question, the Aegir project is a system whereby you can use Drupal to create and manage Drupal sites. That includes installing from install profiles--which are sort of like site templates--or a distribution (Drupal installations pre-packaged with modules). The downside is that installation is fairly involved, more so than just Drupal itself. There's a lot of documentation on the Drupal groups site for Aegir. For a straight multi-site install, there's some documentation on the subject, but the install instructions with the software come with help that you should consult first.
As for your second question, the answer is (unfortunately) "it depends". Knowledge of PHP, especially "the Drupal way", plus integration with the community, are huge plusses. If you intend to join the community, immediately sign up both yourself and all developers an account on Drupal.org and, if you find solutions to bugs or other problems, providing back is a sign of goodwill, and it usually pays back dividends (one example: you submit a patch, it gets included in a module, and then the community maintains it for you). Developers need not be experts with Drupal core, but they need to be pretty comfortable with learning the API and knowing how to create sites for clients in general. First start with requirements gathering, then see how it fits into the Drupal way of doing things. If it doesn't fit, then use the right tool.
That's a tip of the iceberg view from the developer's point of view (as opposed to the businessman's point of view). There are plenty of companies that do only Drupal and there are plenty of companies where Drupal is one tool they use out of many.
Often after a Drupal (6.x) site is launched, I have people starting to sign up and enter their own content. Whenever there is need for an upgrade, the database on production is copied to dev and then the development is done on dev, later get pushed to staging for client's approval.
When the site is eventually ready to go live, there is a problem. Production server has the latest user inputted content, dev and staging have the latest functionality. Simply overwriting the database on production won't work. What I usually do is to write down what has been done to dev and than follow the steps to go though the implementations again on production. As the system grows bigger, one single mistake on production may cause lost of business. I can't shutdown the site for several hours. I can't tell how many people are using the site at a given time, even so it's impossible to wait for a time where nobody is on the site to make the upgrade.
Has anyone have any good idea?
Thanks in advance.
There are two concepts you need to look into: The first is "Exportables" which is generally a way of exporting all the configuration of a given module. The second is "Features" (terribly named, yes) which is a way of grouping a set of Exportables into a given changeset for version control, updating, deployment, rollback, etc.
For clarification, many modules implement their own "Exportables" methodology what I linked to above was the Exportables module. Here's a wider strategy for it - http://www.sthlmconnection.se/tips-and-tweaks/exportable-configuration-your-drupal-module-ctools
It's the million dollar question: How to transfer code, configuration and content between different Drupal sites? In Drupal, code is stored in files (or at least it should be) while configuration and content are usually in the database.
Taking your code from one server to another isn't that hard, and code has another advantage: it's easy to store and manage in a version control system like SVN or GIT. That's why most solutions focus on taking stuff out of the database and putting it into code.
Already mentioned by CaseySoftware, the Features module is what you need to store configuration in code. Features has a stable release since a couple of weeks and the community seems to agree that Features is the way forward.
Moving content between sites is a little harder, because content can be added or changed on dev, staging and production simultaneously. Exportables is an attempt to solve that, but it's not the only one. Make sure you also check out Deploy and the Features-based UUID Features Integration modules. None of those modules is stable yet and time will tell which one is the best solution.
The deeper I get with Drupal the more I have to make changes to code within other people's modules. These are usually small changes, and so far it doesn't make sense to rewrite the module's functionality for my own needs.
I'm trying not to make any modifications to Drupal core since that just seems like asking for trouble.
But sooner than later, I'm going to need to update these modules with new releases and then repatch my changes back in. How do you stay organized when making these changes, upgrading modules, and re-applying your changes back?
I'm taking as many notes as I can but the spiderweb is growing around me!
Another key is to maintain explicit .patch files for any changes you make, and include documentation with them. If you have to upgrade to a new version of a module, install the clean copy and re-apply the patch. If it doesn't re-apply cleanly, you know you've got a problem.
That's where maintianing your own source tree in SVN/git/etc can be handy.
Do you change the module in a way other people may benefit?
Then send the patch to the module maintainer, so everybody benefits and you don't have the trouble to stay organised.
There is a good discussion in this question.
I particularly like Nick Sergeants article describing a method where you checkout the Drupal sources via CVS and then control your changes and the CVS records using SVN.
You only have the two already mentioned Options.
1.) If your changes can benefit the community then give back and hopefully it will become Part of the next release.
2.) If your changes are to specific and fit only your needs than your only chance to stay organized is setting up your own VCS (e.g. Subversion) and learn to use it properly.. ;)
There are some good infos on using Subversion with drupal on the Drupal.org site.
There is also a new Module available that allow you to capture some features into your own modules (e.g. features (http://drupal.org/project/features)). I have not much experience with it but maybe this is also a solution for you.
The way that Eaton describes above tends to be how we handle the situation for relatively large sites. If you keep a running log of patches, you can always re-create changes.
One of the aspects that seems to be missed in a lot of the "submit back to the community" posts is that just because you submit it back to the community doesn't mean it'll ever get applied. The way we handle this scenario is by keeping an explicit patch file with an indication of what D.O. issue it's related to. In the case where the patch is eventually integrated, you can remove your patch and pat yourself on the back. In the scenario where your patch is never accepted, at least you still have your log.
The book Leveraging Drupal: Getting Your Site Done Right (Wrox Programmer to Programmer) describes workflow and developing sites with cvs/svn from the beginning.
It's only $32 free shipping at amazon right now (as low as $24 used but remember $4 shipping you might as well get it new. I recommend checking your library for it, if they don't have it try inter-library loan. But this book describes exactly what you are seeking, step by step.
From my experience, one of the bigger problems we come across during our webdevelopment process is keeping different setups updated and secure across different servers.
My company has it's own CMS which is currently installed across 100+ servers. At the moment, we use a hack-ish FTP-based approach, combined with upgrade scripts at specific locations to upgrade all of our CMS setups. Efficiently managing these setups becomes increasingly difficult and risky when there are several custom modules involved.
What is the best way to keep multiple setups of a web application secure and up-to-date?
How do you do it?
Are there any specific tips regarding modularity in applications, in order to maintain flexibility towards our clients, but still being able to efficiently manage multiple "branches" of an application?
Some contextual information: we mainly develop on the LAMP-stack. One of the main factors that helps us sell our CMS is that we can plugin pretty much anything our client wants. This can very from 10 to to 10.000 lines of custom code.
A lot of custom work consists of very small pieces of code; managing all these small pieces of code in Subversion seems quite tedious and inefficient to me (since we deliver around 2 websites every week, this would result in a lot of branches).
If there is something I am overlooking, I'd love to hear it from you.
Thanks in advance.
Roundup: first of all, thanks for all of your answers. All of these are really helpful.
I will most likely use a SVN-based approach, which makes benlumley's solution closest to what I will use. Since the answer to this question might differ in other usecases, I will accept the answer with the most votes at the end of the run.
Please examine the answers and vote for the ones that you think have the most added value.
I think using a version control system and "branching" the part of the codes that you have to modify could turn out to be the best approach in terms of robustness and efficiency.
A distributed version system could be best suited to your needs, since it would allow you to update your "core" features seamlessly on different "branches" while keeping some changes local if need be.
Edit: I'm pretty sure that keeping all that up to date with a distributed version system would be far less tedious than what you seem to expect : you can keep the changes you are sure you're never going to need elsewhere local, and the distributed aspect means each of your deployed application is actually independent from the others and only the fix you mean to propagate will propagate.
If customizing your application involves changing many little pieces of code, this may be a sign that your application's design is flawed. Your application should have a set of stable core code, extensibility points for custom libraries to plug into, the ability to change appearance using templates, and the ability to change behavior and install plugins using configuration files. In this way, you don't need a separate SVN branch for every client. Rather, keep the core code and extension plugin libraries in source control as normal. In another repository, create a folder for each client and keep all their templates and configuration files there.
For now, creating SVN branches may be the only solution that helps you keep your sanity. In your current state, it's almost inevitable that you'll make a mistake and mess up a client's site. At least with branches you are guaranteed to have a stable code base for each client. The only gotcha with SVN branches is if you move or rename a file in a branch, it's impossible to merge that change back down to the trunk (you'd have to do it manually).
Good luck!
EDIT: For an example of a well-designed application using all the principles I outlined above, see Magento E-Commerce. Magento is the most powerful, extensible and easy to customize web application I've worked with so far.
I may be wrong, but it seems to me what Aron is after is not version control. Versioning is great, and I'm sure they're using it already, but for managing updates on hundreds of customized installations, you need something else.
I'm thinking something along the lines of a purpose-built package system. You'll want every version of a module to keep track of its individual dependencies and 'guaranteed compatibilities', and use this information to automatically update only the 'safe' modules.
E.g. let's say you've built a new version 3 of your 'Wiki' module. You want to propagate the new version to all the servers running your application, but you've made changes to one of the interfaces within the Wiki module since version 2. Now, for all default installations, that is no problem, but it would break installations with custom extensions on top of the old interface. A well-planned package system would take care of this.
To address the security question, you should look into using digital signatures on your patches. There are lots of good libraries available for public-key-based signatures, so just go with whatever seems to be the standard for your chosen platform.
Not sure whether someone's said this, there are a lot of long responses here, and I've not read them all.
I think a better approach to your version control would be to have your CMS sat on its own in its own repository and each project in its own. (or, all of these could be subfolders within one repo i guess)
You can then use its trunk (or a specific branch/tag if you prefer) as an svn:external in each project that requires it. This way, any updates you make to the CMS can be committed back to its repository, and will be pulled into other projects as and when they are svn updated (or the external is svn:switch 'ed).
As part of making this easier, you will need to make sure the CMS and the custom functionality sit in different folders, so that svn externals works properly.
IE:
project
project/cms <-- cms here, via svn external
project/lib <-- custom bits here
project/www <-- folder to point apache/iis at
(you could have cms and lib under the www folder if needed)
This will let you branch/tag each project as you wish. You can also switch the svn:external location on a per branch/tag basis.
In terms of getting changes live, I'd suggest that you immediately get rid of ftp and use rsync or svn checkout/exports. Both work well, the choice is up to you.
I've got most experience with the rsync route, rsyncing an svn export to the server. If you go down this route, write some shell scripts, and you can create a test shell script to show you the files it will upload without uploading them as well, using the -n flag. I generally use a pair of scripts for each environment - one a test, and one to actually do it.
Shared key authentication so you don't need a password to send uploads up may also be useful, depending on how secure the server to be given the access is.
You could also maintain another shell script for doing bulk upgrades, which simply calls the relevant shell script for each project you want to upgrade.
Have you looked at Drupal? No, not to deploy and replace what you have, but to see how they handle customizations and site-specific modules?
Basically, there's a "sites" folder which has a directory for every site you're hosting. Within each folder is a separate settings.php which allows you to specify a different database. Finally, you can (optionally) have "themes" and "modules" folders within sites.
This allows you to do site-specific customizations of particular modules and limit certain modules to those sites. As a result, you end up with a site that the vast majority of everything is perfectly identical and only the differences get duplicated. Combine that with the way it handles upgrades and updates and you might have a viable model.
Build into the code a self-updating process.
It will check for updates and run them when/where/how you have configured it for the client.
You will have to create some sort of a list of modules (custom or not) that need to be tested with the new build prior to roll-out. When deploying an update you will have to ensure these are tested and integrated correctly. Hopefully your design can handle this.
Updates are ideally a few key steps.
a) Backup so you can back out. You should be able to back out
the entire update at any time. So,
that means creating a local archive
of the application and database
first.
b) Update Monitoring Process - Have the CMS system phone home to look for a new build.
c) Schedule Update on availability - Chances are you don't want the update to run the second it is available. This means you will have to create a cron/agent of some kind to do the system update automatically in the middle of the night. You can also consider client requirements to update on weekends, or on specific days. You can also stagger rolling out your updates so you don't update 1000 clients in 1 day and get tech support hell. Staggered roll-out of some kind might be beneficial for you.
d) Add maintenance mode to update the site -- Kick the site into maintenance mode.
e) SVN checkout or downloadable packages -- ideally you can deploy via svn checkout, and if not, setup your server to deliver svn generated packages into an archive that can be deployed on client sites.
f) Deploy DB Scripts - Backup the databases, update them, populate them
g) Update site code - All this work for one step.
h) Run some tests on it. If your code has self-tests built in, it would be ideal.
Here's what I do...
Client-specific include path
Shared, common code is in shared/current_version/lib/
Site specific code is in clients/foo.com/lib
The include path is set to include from the clients/foo.com/lib, and then share/lib
The whole thing is in a version control system
This ensures that the code uses shared files wherever possible, but if I need to override a particular class or file for some reason, I can write a client specific version in their folder.
Alias common files
My virtual host configuration will contain a line like
Alias /common <path>/shared/current_version/public_html/common
Which allows common UI elements, icons, etc to be shared across projects
Tag the common code with each site release
After each site release, I tag the common code by creating a branch to effectively freeze that point in time. This allows me to deploy /shared/version_xyz/ to the live server. Then I can have a virtual host use a particular version of the common files, or leave it pointing at the current_version if I want it to pick up the latest updates.
Have you looked at tools such as Puppet (for system administration incl. app deployment) or Capistrano (deployment of apps in RubyOnRails but not limited to these)?
One option would be to set up a read-only version control system (Subversion). You could integrate access to the repository into your CMS and invoke the updates through a menu, or automatically if you do not want the user to have a choice about an update (could be critical). Using a version control system would also allow you to keep different branches easily
As people have already mentioned that using version control (I prefer Subversion due to functionality) and branching would be the best option. Another open source software available on sourceforge called cruisecontrol. Its amazing, you configure cruisecontrol with subversion in sach a way that any code modification or new code added in serversion, Cruise control will know automatically and will do build for you. It will save your hell of time.
I have done the same way in my company. we have four projects and have to deploy that project on different servers. I have setup cruiseconrol in such a way that any modification in code base triggers automatic build. and another script will deploy that build on the server. your are good to go.
If you use a LAMP stack I would definitely turn the solutions files into a package of your distribution and use it for propagate changes. I recommend for that matter Redhat/Fedora because of RPM and it's what I have experience on. Anyway you can use any Debian based distribution too.
Sometime ago I made a LAMP solution for managing an ISP hosting servers. They had multiple servers to take care of web hosting and I needed a way to deploy the changes of my manager, because every machine was self-contained and had a online manager. I made a RPM package containing the solution files (php mostly) and some deploying scripts that runned with the RPM.
For automated updating we had our own RPM repository set on every server in yum.conf. I set an crontab job to update the servers daily with the latest RPMs from that trusted repository.
Trustiness can be achieve too because you can use trust settings in the RPM packages, like signing them with your public key file and accepting only signed packages.
Hm could it be an idea to add configuration files? You wrote that a lot of small script are doing something. Now if you'd build them into the sources and steered them with configuration files shouldn't that "ease" that?
On the other hand having branches for every customer looks like an exponential growth to me. And how would you "know" which areas you've done something and do not forget to "make" changes in all other branches also. That looks quite ugly to me.
It seems a combination of revision controls, configuration options and/or deployment receipts seems to be a "good" idea.....
With that many variations on your core software, I think you really need a version control system to stay on top of pushing updates from the trunk to the individual client sites.
So if you think Subversion would be tedious, you've got a good sense for what the pain points will be... Personally, I wouldn't recommend Subversion for this, since it's not really that good at managing & tracking branches. Although benlumley's suggestion to use externals for your core software is a good one, this breaks down if you need to tweak the core code for your client sites.
Look into Git for version control, it's built for branching, and it's fast.
Check out Capistrano for managing your deployments. It's a ruby script, often used with Rails, but it can be used for all sorts of file management on remote servers, even non-ruby sites. It can get the content to the remote end through various stragegies including ftp, scp, rsync, as well as automatically checking out the latest version from your repository. The nice features it provides include callback hooks for every step of the deploy process (e.g. so you can copy your site-specific configuration files which might not be in version control), and a release log system--done through symlinks--so you can quickly roll back to a previous release in case of trouble.
I'd recommend a config file with the list of branches and their hosted location, then run through that with a script that checks out each branch in turn and uploads the latest changes. This could be cron'd to do nightly updates automatically.