I had a question regarding Magento: -
Currently I have a client whose online site and catalog was being built on magento, however they are wanting to divide up their site to use a different platform for the catalog management and consumer purchasing experience, while keeping the rest of their corporate site intact on magento.
Since they were using enterprise before, the idea is to move to Community to use for their corporate site, however I'm wondering if this is feasible to do from a migration standpoint. Is there any way to port across their CMS pages (catalog isn't important) from their old enterprise version to a fresh community version, and have it remain intact?
Enterprise has lots of additional features in place. Also its URL table are modified. So you need to check if these features are used in design.If so you need to remove all those plus you might have to move some data or populate some tables.
Its possible to move back from enterprise to community.But is not a cake walk.
Joomla 1.5 will reach the end of its life in a short term and many site are being upgrade to a 1.7 or 2.5 version. We are trying to figure out how we can upgrade our sites. Unfortunately the developers of Joomla, who are doing a great job, haven't kept backwards compatibility high on their requirementslist.
We know there are many resources describing how to migrate a Joomla site to version X from version 1.5. But in our company we have about 120 Joomla sites. With all the migration steps that have to be done to the templates, custom written code and the third party modules we use this would be a hell of a job to migrate. So we are looking into methods and techniques that would make our (upgrade) job easier.
I can't imagine we are the only one with this problem so I am looking for more information on migrating these sites on a large scale. We can't be the only one who are struggling with this.
To give some detail, for upgrading of the minor versions we used the Vendor branches technique which worked awesome. In short, in our SVN repository we have a folder containing the current Joomla release. In the same repository we have a folder containing our own Joomla version with some custom code adjustments. Every project is based on that custom version. With the use of version branching we could easily update all our projects to the latest Joomla version.
For the major upgrade this technique won't be suitable. For instance we expect that some projects won't be upgraded to the new Joomla version for compability issues.
A way to solve this for the 2.5 branch could be to create two new folders with the 2.5 release of Joomla and our own customized 2.5 version. Each migrated project then would be branched of the 2.5 customized version. The migration process would be tedious and for sure be a manual drill.
We are afraid that we have to do this for every major release of Joomla so this won't be a real solution.
A solution we are thinking of is using phar and composer to create the project. If we succesfully can create a joomla phar as library and put custom development in an other phar, upgrading should be as simple as replacing the phar. Third party modules should be put into a phar archive also for easy updating. If modules don't support this, we are going to phar it ourself.
Of course we know that Joomla has a new, integrated update mechanism. We are looking into this mechanism but doubt we can use it since we have some custom patches to core code or module functionality.
To summarize this post, we have two challenges we'd love to get some feedback of.
How would you sggest upgrading 120+ sites to the latest release of Joomla
How do you manage Joomla updates if you have a large number of Joomla sites to maintain
The bad news is that there is no automated upgrade path from Joomla 1.5 to 2.5, as the changes are so drastic that they are almost like night and day. The template changes are such that you may have to rewrite them from scratch. Do not forget that 2.5 does a number of things differently too so you may also face a learning curve.
My suggestion would be to have a tiered migration plan and only migrate the sites that you need to or can justify the costs of the migration as the components, modules and plugins you use.
When doing so you need to watch the release schedule which provides a Long Term Release every 18 months each of which will most probably break backward compatibility from the previous versions, so you will end up with sites at 1.5, 2.5, 3.x etc
I believe that phar can be used in order to distribute a new upgraded version - but it will not help you in the upgrade process itself.
My (painful) experience with a migration from 1.5 to 1.7 taught me that not only the code changes were dramatic but also the DB changes (structure!), ACL implementation etc etc. The template will probably be the least of your problems.
My question back to you is, why do you want to upgrade ALL the websites ? if a specific website needs tools/plugins that are available only on higher versions of Joomla then I guess it's a good enough reason. But to upgrade all the websites will be, like you anticipate, a project from hell...
I hire tommrow a new developer, since now i worked alone, now i need to do some enviorment to developing and do a stage - online step
what is the leading tools (even if need to pay somthing) to do that?
i saw webenabled.. so far..
You'll need a some sort of version control system (VCS) for your project code. Since Drupal.org now use Git which is pretty good and awesome, you should too. There are several hosting solution for Git, the most popular seems to be GitHub.
In your code repository, I recommend not to put the whole site directory but only your own custom code. Regardless the used VCS, here is what I put in my code repository
A .make file used to download Drupal core, contrib modules and contrib themes and apply patches (if required)
a module folder with only the custom modules
a themes folder with only the custom themes
A build script to
run drush make on the .make file to download Drupal core and contribs to a (VCS ignored) dist folder
copy the modules folder to dist/sites/all/modules/custom
copy the themes folder to to dist/sites/all/themes/custom
This to
properly track changes to your project custom code
properly track used core and contribs versions (in the .make file)
prevent core or contribs hack but allow patching when required (Drush Make requires the applied patches to be available at a publicly accessible HTTP address)
For the build script, I use Phing but any scripting languages (ant, bash, php, ruby, etc.) could be used. With some additional work, the build script can also be used to run automated test (SimpleTest) and code validation (php -l and Coder Review). In the end, the build script produce and update dist folder ready for deployment.
For multi developpers project, I try to have as much configurations as possible exported into code instead of working at the database level to store. Mainly by using exportables through the Features module and by having a project specific profile to define and update non-exportable configurations through its hook_install and hook_update_N implementations. See The Development -> Staging -> Production Workflow Problem in Drupal and the Code driven development: using Features effectively in Drupal 6 and 7 presentation.
There are a few options for this, there is deployment module that is alpha but apparently works good. Then there is plain old svn ( or even rsync ). That get the job done pretty fast, and give you the added bonus of source code management but you need to transfer databases manually.
Last but not least, and the most powerful method of the 3 mentionned, is drush.
Whatever you chose depends on the time you are willing to invest in this step, because short-term they all involve a little more time than just copying a site in another folder but the task would be automated once you do it, so long-term you can easily repeat the deployment and this is where these tools will make you save time.
Good-luck!
I'm looking to upgrade a themed/custom Magento from 1.3.x to Magento 1.9 Enterprise. So far, after multiple attempts at upgrading, I have failed.
After the first upgrade, I uploaded the new Magento in a clean environment, copied the database to a dev database. Using this, the upgrade occurred with two errors: It appears Magento upgrades only support 1.4+ currently, and previous mysql upgrade scripts were not included. After the "install" of the upgrade, I couldn't access wither the admin, or the frontend, and there was no errors to tell me what gives.
Scrapping that idea, I tried a clean install: It worked fine. Then I tried importing all the products from a CSV export. Worked OK, but custom attributes such as images, sizes, etc. didn't transfer over. I have over 900 product, and entering everything manual would be a pain, and unfeasible. Scrapping that idea.
Now I'm at various upgrade configurations, upgrading from Magento 1.3. I'm going to try and upgrade 1.3 to 1.4, and then 1.4 to Enterprise, but has anyone performed such an upgrade successfully before and might be able to provide hints?
Thanks,
Bryon
Byron, I feel your pain. I struggled with an upgrade from 1.3 to 1.4 a month ago.
Try the technique mentioned here: http://www.webshopapps.com/blog/2010/02/upgrading-magento-to-version-1-4-keeping-it-simple/ In the end it worked for me.
The thing that is sort of counter intuitive is the deletion of the database. I kept trying to skip that step, and that's what stymied me for a while. You have to delete the database and reload the data (it does something to the key constraints). In the end I was able to upgrade to 1.4 without manually moving anything.
You should go the route 1.3 to 1.4 , 1.4 to enterprise and switch to default skin while doing so. Skin/templates needs special attention later as the dom is quite different. Merging 1.3 templates to enterprise dom will take ~ 2-4 days experienced slicer who knows how to use diff tools
my usual workflow for this is:
add all three magento versions to git and tag by version , use your own magento installation as base and ignore your template folders and local/community extensions that are not installed by default
on your web directory , checkout your base version
git pull 1.4 to your installation and visit the website to get the upgrades
git pull enterprise to your installation and visit the website to get the upgrades
doing it in such order you also get rid of removed files that magento has removed from each version and you also get all changes and new files.
Magento Enterprise Edition Upgrade Procedure for 1.9 to 1.9.1
Generally all Magento upgrades work by running the updated code with the old database. The differences will be detected and incorporated automatically on the next page request. Magento keeps track of every module's version number for this reason. This is not advised with this upgrade if you have custom code.
Disclaimer – if you have a lot of customization, the upgrade will break the system; it is best to do this on a new (temporary) site, compare, bug fix, then test, then cross browser test.
Your general approach:
Close production server Backup all
DBs and Magento installation Turn
off all your custom extensions and
themes
Delete from HDD: core Magento modules, their layouts, all standard themes and cache.
Get 1.9.1 EE, copy it into a fresh DB installation, then place custom code over the top.
File compare between OTB 1.9.0 and 1.9.1. Pay special attention to a list of core controllers which have been overridden and compare the difference between these controller in version 1.9.0. and 1.9.1.
Here is a list of known problematical issues which will cause rework in our custom code:
1) Google Analytics (does not work in
1.9.0 and to fix it, many changes are required to our custom code)
2) Flat
Category
3) Searching by Attribute –
(xml fix)
4) iFrame problem in CMS
pages
5) Missing admin custom tabs
(compare before and after)
6) Home
page enterprise_home has to be
renamed! (this is an example of a
hidden pitfall undocumented and
represents a warning to you to factor
in time for such problems)
7) Check Mage/Community for new modules which
override modules which we need.
8) Anything which extends the customer
entity should be rigorously tested.
9) JavaScript – be careful - the
actual js templates may be the same,
but the blocks and modules which call
them may have subtle changes!
10) Custom Product Imports – do a test
product import on 1.9.1 using dataflow
method and see
what db fields are needed then add them into the procededural code for your custom code.
Check release notes documentation and update for your theme, whether it supports EE 1.9. Turn it on if it supports, otherwise you'll need another theme.
Check release notes documentation and updates for all your custom extensions - whether they support 1.9.1 Turn them on - one by one.
You will have problems upgrading all core DB data if it's made automatically, check which fields are missing/changed and add them.
Cross Browser testing - problems with your custom theme, and you'll need to check your custom extensions and upgrade their template files, skin css and DB data to fit 1.9.1.
Testing is the biggest task, walk through the application, notice errors and warnings, fix them.
From my experience, one of the bigger problems we come across during our webdevelopment process is keeping different setups updated and secure across different servers.
My company has it's own CMS which is currently installed across 100+ servers. At the moment, we use a hack-ish FTP-based approach, combined with upgrade scripts at specific locations to upgrade all of our CMS setups. Efficiently managing these setups becomes increasingly difficult and risky when there are several custom modules involved.
What is the best way to keep multiple setups of a web application secure and up-to-date?
How do you do it?
Are there any specific tips regarding modularity in applications, in order to maintain flexibility towards our clients, but still being able to efficiently manage multiple "branches" of an application?
Some contextual information: we mainly develop on the LAMP-stack. One of the main factors that helps us sell our CMS is that we can plugin pretty much anything our client wants. This can very from 10 to to 10.000 lines of custom code.
A lot of custom work consists of very small pieces of code; managing all these small pieces of code in Subversion seems quite tedious and inefficient to me (since we deliver around 2 websites every week, this would result in a lot of branches).
If there is something I am overlooking, I'd love to hear it from you.
Thanks in advance.
Roundup: first of all, thanks for all of your answers. All of these are really helpful.
I will most likely use a SVN-based approach, which makes benlumley's solution closest to what I will use. Since the answer to this question might differ in other usecases, I will accept the answer with the most votes at the end of the run.
Please examine the answers and vote for the ones that you think have the most added value.
I think using a version control system and "branching" the part of the codes that you have to modify could turn out to be the best approach in terms of robustness and efficiency.
A distributed version system could be best suited to your needs, since it would allow you to update your "core" features seamlessly on different "branches" while keeping some changes local if need be.
Edit: I'm pretty sure that keeping all that up to date with a distributed version system would be far less tedious than what you seem to expect : you can keep the changes you are sure you're never going to need elsewhere local, and the distributed aspect means each of your deployed application is actually independent from the others and only the fix you mean to propagate will propagate.
If customizing your application involves changing many little pieces of code, this may be a sign that your application's design is flawed. Your application should have a set of stable core code, extensibility points for custom libraries to plug into, the ability to change appearance using templates, and the ability to change behavior and install plugins using configuration files. In this way, you don't need a separate SVN branch for every client. Rather, keep the core code and extension plugin libraries in source control as normal. In another repository, create a folder for each client and keep all their templates and configuration files there.
For now, creating SVN branches may be the only solution that helps you keep your sanity. In your current state, it's almost inevitable that you'll make a mistake and mess up a client's site. At least with branches you are guaranteed to have a stable code base for each client. The only gotcha with SVN branches is if you move or rename a file in a branch, it's impossible to merge that change back down to the trunk (you'd have to do it manually).
Good luck!
EDIT: For an example of a well-designed application using all the principles I outlined above, see Magento E-Commerce. Magento is the most powerful, extensible and easy to customize web application I've worked with so far.
I may be wrong, but it seems to me what Aron is after is not version control. Versioning is great, and I'm sure they're using it already, but for managing updates on hundreds of customized installations, you need something else.
I'm thinking something along the lines of a purpose-built package system. You'll want every version of a module to keep track of its individual dependencies and 'guaranteed compatibilities', and use this information to automatically update only the 'safe' modules.
E.g. let's say you've built a new version 3 of your 'Wiki' module. You want to propagate the new version to all the servers running your application, but you've made changes to one of the interfaces within the Wiki module since version 2. Now, for all default installations, that is no problem, but it would break installations with custom extensions on top of the old interface. A well-planned package system would take care of this.
To address the security question, you should look into using digital signatures on your patches. There are lots of good libraries available for public-key-based signatures, so just go with whatever seems to be the standard for your chosen platform.
Not sure whether someone's said this, there are a lot of long responses here, and I've not read them all.
I think a better approach to your version control would be to have your CMS sat on its own in its own repository and each project in its own. (or, all of these could be subfolders within one repo i guess)
You can then use its trunk (or a specific branch/tag if you prefer) as an svn:external in each project that requires it. This way, any updates you make to the CMS can be committed back to its repository, and will be pulled into other projects as and when they are svn updated (or the external is svn:switch 'ed).
As part of making this easier, you will need to make sure the CMS and the custom functionality sit in different folders, so that svn externals works properly.
IE:
project
project/cms <-- cms here, via svn external
project/lib <-- custom bits here
project/www <-- folder to point apache/iis at
(you could have cms and lib under the www folder if needed)
This will let you branch/tag each project as you wish. You can also switch the svn:external location on a per branch/tag basis.
In terms of getting changes live, I'd suggest that you immediately get rid of ftp and use rsync or svn checkout/exports. Both work well, the choice is up to you.
I've got most experience with the rsync route, rsyncing an svn export to the server. If you go down this route, write some shell scripts, and you can create a test shell script to show you the files it will upload without uploading them as well, using the -n flag. I generally use a pair of scripts for each environment - one a test, and one to actually do it.
Shared key authentication so you don't need a password to send uploads up may also be useful, depending on how secure the server to be given the access is.
You could also maintain another shell script for doing bulk upgrades, which simply calls the relevant shell script for each project you want to upgrade.
Have you looked at Drupal? No, not to deploy and replace what you have, but to see how they handle customizations and site-specific modules?
Basically, there's a "sites" folder which has a directory for every site you're hosting. Within each folder is a separate settings.php which allows you to specify a different database. Finally, you can (optionally) have "themes" and "modules" folders within sites.
This allows you to do site-specific customizations of particular modules and limit certain modules to those sites. As a result, you end up with a site that the vast majority of everything is perfectly identical and only the differences get duplicated. Combine that with the way it handles upgrades and updates and you might have a viable model.
Build into the code a self-updating process.
It will check for updates and run them when/where/how you have configured it for the client.
You will have to create some sort of a list of modules (custom or not) that need to be tested with the new build prior to roll-out. When deploying an update you will have to ensure these are tested and integrated correctly. Hopefully your design can handle this.
Updates are ideally a few key steps.
a) Backup so you can back out. You should be able to back out
the entire update at any time. So,
that means creating a local archive
of the application and database
first.
b) Update Monitoring Process - Have the CMS system phone home to look for a new build.
c) Schedule Update on availability - Chances are you don't want the update to run the second it is available. This means you will have to create a cron/agent of some kind to do the system update automatically in the middle of the night. You can also consider client requirements to update on weekends, or on specific days. You can also stagger rolling out your updates so you don't update 1000 clients in 1 day and get tech support hell. Staggered roll-out of some kind might be beneficial for you.
d) Add maintenance mode to update the site -- Kick the site into maintenance mode.
e) SVN checkout or downloadable packages -- ideally you can deploy via svn checkout, and if not, setup your server to deliver svn generated packages into an archive that can be deployed on client sites.
f) Deploy DB Scripts - Backup the databases, update them, populate them
g) Update site code - All this work for one step.
h) Run some tests on it. If your code has self-tests built in, it would be ideal.
Here's what I do...
Client-specific include path
Shared, common code is in shared/current_version/lib/
Site specific code is in clients/foo.com/lib
The include path is set to include from the clients/foo.com/lib, and then share/lib
The whole thing is in a version control system
This ensures that the code uses shared files wherever possible, but if I need to override a particular class or file for some reason, I can write a client specific version in their folder.
Alias common files
My virtual host configuration will contain a line like
Alias /common <path>/shared/current_version/public_html/common
Which allows common UI elements, icons, etc to be shared across projects
Tag the common code with each site release
After each site release, I tag the common code by creating a branch to effectively freeze that point in time. This allows me to deploy /shared/version_xyz/ to the live server. Then I can have a virtual host use a particular version of the common files, or leave it pointing at the current_version if I want it to pick up the latest updates.
Have you looked at tools such as Puppet (for system administration incl. app deployment) or Capistrano (deployment of apps in RubyOnRails but not limited to these)?
One option would be to set up a read-only version control system (Subversion). You could integrate access to the repository into your CMS and invoke the updates through a menu, or automatically if you do not want the user to have a choice about an update (could be critical). Using a version control system would also allow you to keep different branches easily
As people have already mentioned that using version control (I prefer Subversion due to functionality) and branching would be the best option. Another open source software available on sourceforge called cruisecontrol. Its amazing, you configure cruisecontrol with subversion in sach a way that any code modification or new code added in serversion, Cruise control will know automatically and will do build for you. It will save your hell of time.
I have done the same way in my company. we have four projects and have to deploy that project on different servers. I have setup cruiseconrol in such a way that any modification in code base triggers automatic build. and another script will deploy that build on the server. your are good to go.
If you use a LAMP stack I would definitely turn the solutions files into a package of your distribution and use it for propagate changes. I recommend for that matter Redhat/Fedora because of RPM and it's what I have experience on. Anyway you can use any Debian based distribution too.
Sometime ago I made a LAMP solution for managing an ISP hosting servers. They had multiple servers to take care of web hosting and I needed a way to deploy the changes of my manager, because every machine was self-contained and had a online manager. I made a RPM package containing the solution files (php mostly) and some deploying scripts that runned with the RPM.
For automated updating we had our own RPM repository set on every server in yum.conf. I set an crontab job to update the servers daily with the latest RPMs from that trusted repository.
Trustiness can be achieve too because you can use trust settings in the RPM packages, like signing them with your public key file and accepting only signed packages.
Hm could it be an idea to add configuration files? You wrote that a lot of small script are doing something. Now if you'd build them into the sources and steered them with configuration files shouldn't that "ease" that?
On the other hand having branches for every customer looks like an exponential growth to me. And how would you "know" which areas you've done something and do not forget to "make" changes in all other branches also. That looks quite ugly to me.
It seems a combination of revision controls, configuration options and/or deployment receipts seems to be a "good" idea.....
With that many variations on your core software, I think you really need a version control system to stay on top of pushing updates from the trunk to the individual client sites.
So if you think Subversion would be tedious, you've got a good sense for what the pain points will be... Personally, I wouldn't recommend Subversion for this, since it's not really that good at managing & tracking branches. Although benlumley's suggestion to use externals for your core software is a good one, this breaks down if you need to tweak the core code for your client sites.
Look into Git for version control, it's built for branching, and it's fast.
Check out Capistrano for managing your deployments. It's a ruby script, often used with Rails, but it can be used for all sorts of file management on remote servers, even non-ruby sites. It can get the content to the remote end through various stragegies including ftp, scp, rsync, as well as automatically checking out the latest version from your repository. The nice features it provides include callback hooks for every step of the deploy process (e.g. so you can copy your site-specific configuration files which might not be in version control), and a release log system--done through symlinks--so you can quickly roll back to a previous release in case of trouble.
I'd recommend a config file with the list of branches and their hosted location, then run through that with a script that checks out each branch in turn and uploads the latest changes. This could be cron'd to do nightly updates automatically.