We have three servers with the same WordPress site - a Live version, a Beta version, and a Kiosk version (a limited version of the site used for an iPad kiosk). The Beta site is deployed from a beta branch on GitHub, while the Live and Kiosk sites are both deployed from the master branch. Any differences in functionality between the sites are controlled by environment variables, beyond that they share the same code.
The wp-content/uploads folder is shared between all three versions of the site, using a mounted Amazon EFS filesystem (couldn't use S3 due to plugin conflicts). Here's where the problem comes in.
The Live and Kiosk servers share the same RDS database, so their media libraries are always in sync. However, the Beta server uses its own separate database for testing purposes - and since WP media library objects are stored in the DB, this leads to the following issues:
When a file is uploaded to the Beta site, it isn't automatically added to the media library on the Live/Kiosk sites (and vice-versa)
When a file is removed from the shared upload folder, it isn't automatically removed from the media library, leading to broken/empty media library entries.
There's nothing necessarily wrong with this; it's just how Wordpress works. I've been circumventing the issue by using the "Add From Server" plugin which allows me to manually keep the media libraries in sync - but it would be nice to automate this.
So I think either of the following solutions would work:
A method to keep the "media" table of the two databases in sync, while keeping the rest of the tables separate
A method to automatically update the media library on all 3 servers according to the content of the wp-content/uploads folder, which is also smart enough to ignore thumbnail versions (the "Add From Server" plugin unfortunately doesn't do this, so when I tried I ended up with countless "thumbnails of thumbnails" if that makes sense)
Has anyone worked with a similar environment before? Any ideas regarding either of my proposed solutions, or possibly a better solution?
As for the dev/staging environments I usually use rsync in combination with crontab or Jenkins and symlink the copied media folder to wp-content/uploads everytime I build the staging with a dump of the production db.
This will also give you the perk of automatically having backups of your db and media folder on a different server.
I'd generally advice against updating the live db by altering a beta/dev environment since this usually defies the purpose of that kind of environment.
Regarding keeping kiosk and production in sync, the plugin WPSiteSync looks promising. I haven't tried it in production yet but it should fit your requirements.
Related
Scenario:
My team manages multiple Joomla websites for our clients. While we manage the development/hosting of the sites, the clients do all of the content updates (creating articles, content, uploading images etc...)
We run these websites in the following standard configuration where we have
A development server
A staging server
A production server
The client makes all of the content updates to the production server (on a daily basis). The other two servers are used primarily for new development and testing.
We currently are using, BitBucket as our SVN for these websites (we are just starting out with this). Currently all files pertaining to the website are stored in the repo.
The Problem
Based on our current setup, if a developer makes changes to the dev environment, and that change set is then pushed to the production environment, we end up overwriting all of the content updates that our clients have made in the production environment.
My Question
How do we successfully utilize a source control system, and maintain the flexibility to allow our clients to continue to make updates directly on the production server, without forcing them to make content changes on dev, staging, and then production?
While you briefly described your workflow, there are some things to be considered. There is no general rule, but look into the following suggestions:
Put into version control JUST the extensions you have developed (template, components, plugins etc.). The customizations are anyway the only things you add to Joomla. Hopefully no core hacks. Alternatively if you really want to version the whole installation, you should at least ignore media folders that are changed by the clients / you. I see no need to put the whole Joomla site under version control.
I imagine your clients are not actually changing PHP scripts, just media files and database entries. You should only push to production code and / or database schema changes.
If you are relying on a commit - push to FTP feature, or manually pushing files, I would suggest looking into building a distributable version of your changes, in a form of a package that can be deployed via the Extension manager. Building packages can be done in one click with a tool like Phing. For example if you make some changes to the template, create a new template version, create the package and update first the staging server / testing server and if all goes well, the production.
Some things shouldn't be in version control.
In general, source code should be versioned and data should not. I'm not familiar with Joomla, but any kind of "uploaded content" directory should be in the ignore file for your version control system. That way you can deploy changes to the software without worrying about overwriting data.
Of course, your data should be backed up regularly, but that's not what revision control is for.
If you have
a staging server where you test the website changes (layout, new functionality, new css)
a production server where the user publishes new content
you need partial database updates along with file synchronization.
The database is pretty hard as the assets table may be affected both by configuration changes on the staging server and by new content on the production server; for this and any other shared tables, we address the issue by making sure the ids don't conflict right after the update leaving a sufficient gap.
Although - to quote most other answers - revision control is not for data nor for the database, it is indeed very nice, especially with pre and post-commit hooks to perform the required database actions; however, any scripting/publishing tools going from rsync-rdiff to phing to ant - maven will do
I have been approached to develop some front-end components (CSS/JS & templates) for an existing Typo3 website. The client already has a development server set up, with Typo3 & extensions installed, and a copy of their database. I have SSH & FTP access to this development server. I am looking for advice on the fastest & most practical way to begin work on the templates.
My level of experience is that I have done front-end work with Typo3 before, but always in a team with an existing version control, build & deployment workflow. In this case, I would be responsible for setting up my own workflow and tools.
I have spent some time reading through version control, build & deployment-related questions on Stack Overflow, but many of them assume:
A team with multiple developers
A long-running project with major changes to functionality, in which it makes sense to invest considerable time up-front to the build process
It is 2010.
An existing workflow (e.g existing development environments) into which an additional tool such as git will be added.
In my case, I will be the only developer working on this, and I have been hired only to make some layout updates. I have no existing development environment for Typo3, and would need to set this up. My work on this project is not intended to run for more than a couple of weeks. I would like to find a solution somewhere in between "edit files directly on the development server" (very bad practise) and "set up a fully-featured PHP application deployment service using Magellanes" (probably good practise, but a steep learning curve for me and a large investment of time).
Ideally, I want to wind up in a situation where I have:
A local development environment on my Mac with Typo3 installed where I can preview & test code changes
Version control such as git on my local system
A way to push my changes to the development site
Can anyone share with me tools or workflow suggestions for how to get to that point as quickly as possible?
My environment is similar to yours, and this is my typical setup:
Version control
The website is version controlled with git. There may or may not be a repository to regularly push to, but my local version is always included in the backups. The following things are excluded from version control:
Files in typo3temp, but not the directories (Put the .gitignore at the end of the post into a directory to keep it, but not the files in it)
The server specific configuration
deprecation logs
IDE files
...
To include the TYPO3 core, there are three variants:
Just add it to the repository
Add the core git repository as git submodule
Add symlinks to the core into the repository.
The latter may be problematic if the directory structure is different in both environments, and the first one clutters the repository, so when possible, go with the second one.
TYPO3 configuration
The TYPO3 configuration is split into three files:
Localconfiguration.php - version controlled, not manually edited (it is constantly overwritten by TYPO3).
AdditionalConfiguration.php - version controlled, manually edited.
ServerspecificConfiguration.php - not version controlled, manually edited. Contains database credentials, colorspaces for imagemagick when different on localhost and remote host, defines caching backends to use and similar stuff.
The additional configuration file includes the server specific file. Both use the \TYPO3\CMS\Core\Configuration\ConfigurationManagerConfigurationManager::setLocalConfigurationValueByPath('DB/host', 'localhost');-syntax for settings to make this work.
Deployment
To deploy the site, I have used two things:
git ftp - this is useful if there is only FTP access.
rsync - This is the better option.
To automate the deployment (and other stuff like building CSS from LESS or SASS), a task runner like grunt.js or ant is useful. Make sure you exclude the server specific configuration and typo3temp from synchronization.
Building such a setup from scratch does not take that much time, maybe 1 or 2 hours, but less then a day. This may of course differ depending on your experience.
Here is the .gitignore mentioned above:
*
!.gitignore
!*/
I am about to use WordPress as CMS with a lot of customization, but my question is, how should I sync development to production?
Assuming WordPress is not used, a typical development cycle is to develop the pages locally, and create the DB scripts. Then when everything is ready, it is posted to the site. And then again, more db and code changes, then only the changes are updated/applied, and so on.
Now, with WordPress, you get all the great features (one example is blogging, comments, almost ready CMS ...etc). However deployment is a pain! Ideally, I would rather keep my typical development cycle described above. Therefore I am going to be implementing WordPress locally (from wordpress.org) and then pushing the changes to my production server.
So, assuming that I create a new page in WordPress locally (I will never create pages on the server, all locally, I will find a way to disable wp-admin for the server), a number of files are created. This is not a problem so far. HOWEVER, if I want to add content to that newly created page, that content is saved to my local database. Even though that content is a database change, it is considered (from my point of view) a new change that should be pushed to server rather than add that content via the live server (because that content is considered static, it is not a blog post or a comment, it is a static page).
Now, that new page content is saved to the DB, and therefore, the DB will have changes done on my local machine that I should push to the server along with the files that I will FTP to the server.
My questions are:
Is this approach correct? If not, what do you suggest
What is the best way to find database diffs? What is a tool to use? Does MySQL Workbench provide something like that? I intend to use that tool to find diffs and then generate an update script for the DB. The reason for this question is I normally make the changes myself, and I know how to track them, but now, those DB changes are generated by WordPress and I need to reverse engineer them to find out which changes are made.
Assuming I got step 2 to work, is there anything in that script that should be modified? Such as server names? Does WordPress hard-code server names for example?
So to summarize and give you more information about my development environment, I use XAMPP for development on Windows 7, with PHP and MySQL setup. I also use Mercurial for source control. As mentioned above, I will use WordPress as part of the solution and I intend to use it to help me create a CMS solution. I will use it locally for page generation, and disable that feature for online (keeping online for blog posts and similar entries only). I am doing that so as to keep things in-sync. If I create a page locally, some data is saved to the DB. Now, how do I sync/upload?
Thanks.
OK, after further investigation, here is what I concluded.
All theme development should be version-controlled
All plugin development should be version-controlled
Content of pages and posts are not part of the development porcess, this is contect and should only be backed up.
This way, you do not need to worry about DB changes ...etc.
Hope this helps everyone.
You might use a Version Control System? What OS is the development on, e.g. Win or Linux? And what is the production OS? I use http://serverpress.com for my testing environment though there are others, WAMP, LAMP, etc.
We are a small team developing PHP applications in a LAN. Both on Mac and PC.
Individual developers check out and edit source code to their own machine, on which Apache is running. Local testing is then done over localhost.
For the DB, the application connects to a common MySQL installation, on a dedicated machine in the LAN. This works quite well because we rarely make (destructive) changes to the DB schema. This means that all the individual applications running access the same test data.
But uploaded files remain a problem: they are only uploaded to the dev's local machine, although a reference to them is stored in the central DB. This means that the other team members may be shown a broken link for a user uploaded image, that physically only exists on one devs local machine.
The ideal solution would be to have the entire persistance layer on a central machine. Any ideas on how best to achieve this?
Map a network folder or use a service like dropbox or similar. A local db is nice to have though and doesn't take up too much resources.
Basically you want some sort of shared filesystem. There are lots of options: a samba share, an NFS-mounted directory, Dropbox (or a similar service), etc. I would suggest looking into the available options to see what suits your infrastructure best.
i have dealt with this once and what i did was to use our NAS as the storage of everything. we developed our website on the NAS itself over FTP. it was like cloud development, IDE only on our side, every file we edited, image uploaded and so on was on the NAS. the website itself was also running on the NAS (since the NAS has the ability to be a server and have mySQL)
the NAS was turned from a network storage - to an actual local server.
By how I understood your question, i assumed you only needed to share uploaded resource (like images) and not develop in the same app at once.
First I suggest that you DON'T develop on the production machine at all.
Now, here are 2 ways you can do it though:
Modular Development:
you develop independently. You don't touch each other's code. you develop features separately from each other. that way, you won't be stepping into each other's shoes. this also promotes "loose coupling" which in le man's terms, "when one feature breaks, the other's won't"
you should check out this video on how you can break down your development into "modules". This is in JS though, but the architecture can still apply.
Version Controlled Development
Break your development into 3 layers:
Production (aka Stable) is the code that is public. You don't develop or touch code here. You only publish the code only when it is tested thoroughly. Also, this is NOT the public server. this is just the public code. however, what lives here is the actual replica of the public site.
Testing (aka Beta) is where you test your developed code. This system is for testing purposes only. You don't touch the code here either. You are just here to find bugs on your own. It's your "Quality Assurance Layer". This layer is also where your codes merge (discussed later)
Development (aka Alpha) is where you touch your code. Here, you share your code, test it, break it, try new features as well as fix the bugs you found in Testing
as you can see, you don't break your systems due to overwriting, or broken links etc.
Now, your development strategy. Use a version system like GIT (distributed) or SVN(central) and create 3 branches according to the ones above. For this example, this uses a distributed approach (i prefer it)
Assign a "maintainer/ring master" in your group who consolidates your work and publishes it to testing. What this maintainer does is to collect your "finished" developed code and puts it into his testing branch. anyone can then clone his testing branch to your testing branch to test your code. Whatever bug they/you find, you refine in the development and submit it to him again. only after that feature is quality assured, then the maintainer can publish it to the stable where he clones it to the public server.
After all that's done and when you have moved on, you just clone the stable branch to your development branch and you start anew. Now you have a fresh canvas to play with.Overwriting is handled by the version control system, and the maintainer. you need not worry about that.
as for resources, you would not want to bog down your local development system with arriving resources from the public server. version control systems also have "ignore lists" to prevent you from cloning some resources. clone only what's necessary. if you are developing a weather widget, you only need images for weather widget. you don't need images from the other widgets (unless neccessary)
Is there a good option for having more than one person developing a Wordpress application with a testing site.
The biggest hurdle that I have encountered are path issues when developing locally and integrating to a testing environment.
Does anyone have a good process for maintaining developer environment(s), keeping working content and links, and the code is maintained in source control?
To clarify, I would like to develop locally, and have a testing environment, and avoid path issues. I am open to other solutions, or ideas.
It comes down to three main concepts
Development environment should be as close to production as possible.
Use Source Control!
Automated Deployment Scripts Take out as much human error when deploying.
The development enviornment/process I prefer looks like this.
Dev/Local
SVN To Check Out Code Locally
Virtual Box running Ubuntu as a Solution for LAMP Environment or XAMPP
Deploy Scripts (Script Automation e.g. NAnt/Ant) For Staging / QA
Module Development
Theme Development
Etc
QA
Initial Content Setup / QA
After Initial Development Staging is Used Less
Production
Live Content Entry
Blogging Etc
As for path issues after initial content development path issues become less relevant, as most content is performed live. If a backup of production is used to create a dev site, SQL scripts and manually changes can be used as necessary. Also switching to using a Virtual Box solution helps ensure everything is in root. But the answer by FractalizeR does help.
May be I don't get the problem in a whole, but what is the problem of putting the whole source code of WordPress into version control, check it out to a single test server for tests?
If you have problems with site names, force your developers to check out to their machines and store it under www.yourwpdomain.local (mind the .local part). They can use DNS or simple hosts file to resolve .local domain address into 127.0.0.1. Apache setup is pretty straightforward.
The rule is Separation of Concerns. You should not place your compiled components like DLLs, EXEs, or data of a system in source control. Wordpress is included in this context for two reasons: 1. the site is stored in the database. 2. The database and the site had better be on a nightly backup schedule. Would you ever need to restore the db or WP from Git? Heck no! Additionally, any WP customization should be placed in child themes, that will be included in source control. The parent theme? NO WAY! Never customize the parent theme in WP. If you are making changes to your base Wordpress site and\or parent theme, then you are at risk of losing your customization when Wordpress or the theme is updated.