We are a small team developing PHP applications in a LAN. Both on Mac and PC.
Individual developers check out and edit source code to their own machine, on which Apache is running. Local testing is then done over localhost.
For the DB, the application connects to a common MySQL installation, on a dedicated machine in the LAN. This works quite well because we rarely make (destructive) changes to the DB schema. This means that all the individual applications running access the same test data.
But uploaded files remain a problem: they are only uploaded to the dev's local machine, although a reference to them is stored in the central DB. This means that the other team members may be shown a broken link for a user uploaded image, that physically only exists on one devs local machine.
The ideal solution would be to have the entire persistance layer on a central machine. Any ideas on how best to achieve this?
Map a network folder or use a service like dropbox or similar. A local db is nice to have though and doesn't take up too much resources.
Basically you want some sort of shared filesystem. There are lots of options: a samba share, an NFS-mounted directory, Dropbox (or a similar service), etc. I would suggest looking into the available options to see what suits your infrastructure best.
i have dealt with this once and what i did was to use our NAS as the storage of everything. we developed our website on the NAS itself over FTP. it was like cloud development, IDE only on our side, every file we edited, image uploaded and so on was on the NAS. the website itself was also running on the NAS (since the NAS has the ability to be a server and have mySQL)
the NAS was turned from a network storage - to an actual local server.
By how I understood your question, i assumed you only needed to share uploaded resource (like images) and not develop in the same app at once.
First I suggest that you DON'T develop on the production machine at all.
Now, here are 2 ways you can do it though:
Modular Development:
you develop independently. You don't touch each other's code. you develop features separately from each other. that way, you won't be stepping into each other's shoes. this also promotes "loose coupling" which in le man's terms, "when one feature breaks, the other's won't"
you should check out this video on how you can break down your development into "modules". This is in JS though, but the architecture can still apply.
Version Controlled Development
Break your development into 3 layers:
Production (aka Stable) is the code that is public. You don't develop or touch code here. You only publish the code only when it is tested thoroughly. Also, this is NOT the public server. this is just the public code. however, what lives here is the actual replica of the public site.
Testing (aka Beta) is where you test your developed code. This system is for testing purposes only. You don't touch the code here either. You are just here to find bugs on your own. It's your "Quality Assurance Layer". This layer is also where your codes merge (discussed later)
Development (aka Alpha) is where you touch your code. Here, you share your code, test it, break it, try new features as well as fix the bugs you found in Testing
as you can see, you don't break your systems due to overwriting, or broken links etc.
Now, your development strategy. Use a version system like GIT (distributed) or SVN(central) and create 3 branches according to the ones above. For this example, this uses a distributed approach (i prefer it)
Assign a "maintainer/ring master" in your group who consolidates your work and publishes it to testing. What this maintainer does is to collect your "finished" developed code and puts it into his testing branch. anyone can then clone his testing branch to your testing branch to test your code. Whatever bug they/you find, you refine in the development and submit it to him again. only after that feature is quality assured, then the maintainer can publish it to the stable where he clones it to the public server.
After all that's done and when you have moved on, you just clone the stable branch to your development branch and you start anew. Now you have a fresh canvas to play with.Overwriting is handled by the version control system, and the maintainer. you need not worry about that.
as for resources, you would not want to bog down your local development system with arriving resources from the public server. version control systems also have "ignore lists" to prevent you from cloning some resources. clone only what's necessary. if you are developing a weather widget, you only need images for weather widget. you don't need images from the other widgets (unless neccessary)
Related
Scenario:
My team manages multiple Joomla websites for our clients. While we manage the development/hosting of the sites, the clients do all of the content updates (creating articles, content, uploading images etc...)
We run these websites in the following standard configuration where we have
A development server
A staging server
A production server
The client makes all of the content updates to the production server (on a daily basis). The other two servers are used primarily for new development and testing.
We currently are using, BitBucket as our SVN for these websites (we are just starting out with this). Currently all files pertaining to the website are stored in the repo.
The Problem
Based on our current setup, if a developer makes changes to the dev environment, and that change set is then pushed to the production environment, we end up overwriting all of the content updates that our clients have made in the production environment.
My Question
How do we successfully utilize a source control system, and maintain the flexibility to allow our clients to continue to make updates directly on the production server, without forcing them to make content changes on dev, staging, and then production?
While you briefly described your workflow, there are some things to be considered. There is no general rule, but look into the following suggestions:
Put into version control JUST the extensions you have developed (template, components, plugins etc.). The customizations are anyway the only things you add to Joomla. Hopefully no core hacks. Alternatively if you really want to version the whole installation, you should at least ignore media folders that are changed by the clients / you. I see no need to put the whole Joomla site under version control.
I imagine your clients are not actually changing PHP scripts, just media files and database entries. You should only push to production code and / or database schema changes.
If you are relying on a commit - push to FTP feature, or manually pushing files, I would suggest looking into building a distributable version of your changes, in a form of a package that can be deployed via the Extension manager. Building packages can be done in one click with a tool like Phing. For example if you make some changes to the template, create a new template version, create the package and update first the staging server / testing server and if all goes well, the production.
Some things shouldn't be in version control.
In general, source code should be versioned and data should not. I'm not familiar with Joomla, but any kind of "uploaded content" directory should be in the ignore file for your version control system. That way you can deploy changes to the software without worrying about overwriting data.
Of course, your data should be backed up regularly, but that's not what revision control is for.
If you have
a staging server where you test the website changes (layout, new functionality, new css)
a production server where the user publishes new content
you need partial database updates along with file synchronization.
The database is pretty hard as the assets table may be affected both by configuration changes on the staging server and by new content on the production server; for this and any other shared tables, we address the issue by making sure the ids don't conflict right after the update leaving a sufficient gap.
Although - to quote most other answers - revision control is not for data nor for the database, it is indeed very nice, especially with pre and post-commit hooks to perform the required database actions; however, any scripting/publishing tools going from rsync-rdiff to phing to ant - maven will do
I have a PHP/Mysql Desktop server. I also installed PHP/Mysql to my laptop and one of my friends for developing a project. Is there anyway that we can synchronise all the changes (i.e. to the php files as well as database changes whether structural or data wise) we do on our laptops to desktop PC ?
For sure you want to start using Git, possibly with a free private repository on somewhere like www.bitbucket.org. Within your versioning, you can backup a version of your SQL inside of the phpMySQL admin tool, and clone the entire repository every time you want to move it.
There are a few choices
https://github.com/axkibe/lsyncd#readme
will sync real time and uses inotify
and if you want to go real nerd
http://www.drbd.org/
High availability clustering
The best way would to be purchase a VPS from a company like Digital Ocean for $5.00 a month. You can then setup a MySQL database that you can both easily connect to. In addition, you will be able to use it to test and deploy (smaller) projects to the server.
Forgot to address the other part of your question. You will be able to setup git/SVN/CVS on this server as well. There are good tutorials online for this. You can both access and commit your changes to this repository. You could also use a website like GitHub for this version control.
Probably the easiest way (if you just want the files synced as is on each change) would be to use something like Google Drive or a similar service. Though if you want revision history and more advanced tools, you'd probably need some sort of version control repository (git, svn, etc.).
However, I definitely recommend version control if that works for you. And if you go with git, I highly recommend bitbucket as they have free closed source hosting for projects of up to 10 (last time I checked) people.
Along with a version control system (such as SVN, Git, or Mercurial) you can't go wrong with using a tool such as FreeFileSync (free as in freeware) for a quick and easy way to push the changes you make to a project to a centralized location. With your project included in a VCS repository the VCS you choose to use will give you control over how you want your project to be synchronized.
This has helped me a lot over the years and has even made the process faster.
I have been approached to develop some front-end components (CSS/JS & templates) for an existing Typo3 website. The client already has a development server set up, with Typo3 & extensions installed, and a copy of their database. I have SSH & FTP access to this development server. I am looking for advice on the fastest & most practical way to begin work on the templates.
My level of experience is that I have done front-end work with Typo3 before, but always in a team with an existing version control, build & deployment workflow. In this case, I would be responsible for setting up my own workflow and tools.
I have spent some time reading through version control, build & deployment-related questions on Stack Overflow, but many of them assume:
A team with multiple developers
A long-running project with major changes to functionality, in which it makes sense to invest considerable time up-front to the build process
It is 2010.
An existing workflow (e.g existing development environments) into which an additional tool such as git will be added.
In my case, I will be the only developer working on this, and I have been hired only to make some layout updates. I have no existing development environment for Typo3, and would need to set this up. My work on this project is not intended to run for more than a couple of weeks. I would like to find a solution somewhere in between "edit files directly on the development server" (very bad practise) and "set up a fully-featured PHP application deployment service using Magellanes" (probably good practise, but a steep learning curve for me and a large investment of time).
Ideally, I want to wind up in a situation where I have:
A local development environment on my Mac with Typo3 installed where I can preview & test code changes
Version control such as git on my local system
A way to push my changes to the development site
Can anyone share with me tools or workflow suggestions for how to get to that point as quickly as possible?
My environment is similar to yours, and this is my typical setup:
Version control
The website is version controlled with git. There may or may not be a repository to regularly push to, but my local version is always included in the backups. The following things are excluded from version control:
Files in typo3temp, but not the directories (Put the .gitignore at the end of the post into a directory to keep it, but not the files in it)
The server specific configuration
deprecation logs
IDE files
...
To include the TYPO3 core, there are three variants:
Just add it to the repository
Add the core git repository as git submodule
Add symlinks to the core into the repository.
The latter may be problematic if the directory structure is different in both environments, and the first one clutters the repository, so when possible, go with the second one.
TYPO3 configuration
The TYPO3 configuration is split into three files:
Localconfiguration.php - version controlled, not manually edited (it is constantly overwritten by TYPO3).
AdditionalConfiguration.php - version controlled, manually edited.
ServerspecificConfiguration.php - not version controlled, manually edited. Contains database credentials, colorspaces for imagemagick when different on localhost and remote host, defines caching backends to use and similar stuff.
The additional configuration file includes the server specific file. Both use the \TYPO3\CMS\Core\Configuration\ConfigurationManagerConfigurationManager::setLocalConfigurationValueByPath('DB/host', 'localhost');-syntax for settings to make this work.
Deployment
To deploy the site, I have used two things:
git ftp - this is useful if there is only FTP access.
rsync - This is the better option.
To automate the deployment (and other stuff like building CSS from LESS or SASS), a task runner like grunt.js or ant is useful. Make sure you exclude the server specific configuration and typo3temp from synchronization.
Building such a setup from scratch does not take that much time, maybe 1 or 2 hours, but less then a day. This may of course differ depending on your experience.
Here is the .gitignore mentioned above:
*
!.gitignore
!*/
So - let's say I develop a PHP app which I develop in a vagrant box identical to production envrionment. So - as an end result I would have a *.tar.zip file with a code...
How would one organize a deployment into production environment where there are a lot of application servers? I mean - I'm confused how to push code into production synchronously all at once?
More information:
on server code is stored like this:
project
+current_revision ->link to revisions/v[n]
+revisions
+v1
+v2
+v3
...
+data
So when I have to deploy changes I usually run a deploy script that uploads updated tar onto server with ssh, untars into specific dir under revisions, symlinks it into current_revision and restart php-fpm.... This way I can rollback anytime just by symlinking to an older revision.
with multipe servers what bothers me is that not all boxes will be updated at once, ie. technically some glitches might be possible.
If you're looking for a "ready-to-go" answer, you'll need to provide some more info about your setup. For example, if you plan to use git for VCS, you could write a simple shell script that pulls the latest commit and rsyncs with the server(s). Or if you're building on top of Symfony, capifony is a great tool. If your using AWS, there's a provider plugin written by the author of Vagrant that's super easy to use, and you can specify a regex for which machines to bring up or provision.
If instead you're looking for more of a "roadmap", then the considerations that you'll want to take are:
Make building of identical boxes in the remote and local environments as easy as possible, and try to make sure that your provisioning emphasizes idempotence.
Consider your versioning/release structure; what resources will rarely or never change? Include those in a setup function instead of a deploy function, and don't include them in your sync run.
Separate your development and system administration concerns; i.e. do not just package a vagrant box with a *.tar.gz and tie it through config.vm.box_url. The reason for this is that you'd have to repackage every production server with a new box every time you deploy, instead of just changing files on the server, or adding/removing some packages from the server.
Check out some config management tools like Chef and Puppet; even if you don't end up using them, they'll give you an idea of how sysadmin professionals approach this problem.
Lots of ways. If starting from barebones (no cloud infrastructure), I'm a fan of the SVN branch hook. Have a SVN repo for your code. Set up a post-commit hook on it, which checks if anything in /branch/production/ has been changed.
If it has, let the post-commit hook fire all your automated roll-out procedure - and in this case, an easy way to do so is to let all your servers known* to svn export the branch. As simple as that!
(* that's the hard step)
Good day to you all,
I am currently developing a project on Laravel. So far I have always developed online, directly editing my files on the webserver throuh FTP (using PSPad or similar simple editing tools).
What I want to do now (and what i believe most people actually do) is setup a (W)LAMP stack on my local machine and program locally. However it is a little bit unclear to me how to keep my local code (including databases) in sync with the live website. How do you folks do that? I know there's probably lots of ways and tools to do that, but what would be your advice for a best practice? Any advice would be very welcome :)
What many companies do is build offline, then push their edits up to a server using git.
Im no expert on the software so ill describe what you do in a basic form:
My advice would be to create an online repo (repository) to store your project while you edit/update.
There are several git project management systems such as github or bitbucket. I personally use bitbucket
What git does, is when you have built or added what you need offline on local (w)lamp, you then git push them up to your repo or server. The changed files then get merged with the existing on the repo or the server. If you'd like the most recent version of your project you'd simply just git pull them down.
Read the full documentation here to see the wide range of options available when using git
We have a settings array within our platform available as $res::Config.
At runtime, a variable is changed from 'dev' to 'live' after checking the HTTP Host, obviously depending on the IP address.
Within our framework bootstrapping, depending on the value of $res::Config->$env, or the environment set previously as either dev or live, the settings for the database connection are set. You store these settings in the Config array as db_live or db_dev.
However you do it, use an environmental variable to figure out whether you want live or dev, and set up and array of settings accordingly.
We also have sandbox and staging for intermittent development stages.
As for version control, use git or subversion.
Edit: It's also possible that within our vhost file, we setup an environmental variable as either live or dev, and our application reads from this accordingly. I'd suggest this approach :)
There are a number of ways of doing this. But this is a deceptively HUGE question you've asked.
Here is some good practice advice - go and research these items, then have a look at my approach.
Typically you use a precess called version control which allows you to create "versions" or snapshots of your system.
The commonly used "SVN" software is good, but the new (not really any more) kid on the block is GIT, and I personally recommend that.
You can use this system to push the codebase live in a controlled fashion. While the files/upload feature is essentially similar to FTP, it allows you to dump a specific version of your site live.
In environments where there are multiple developers, this is ideal - you can compare/test and work around each other, and version control tends to stop errors between devs.
So - advice part 1: Look up and understand version control, then use it to release CODE to the live environment.
Part 2: I use database dumps and farm them back to my machine to work with.
If the live database needs updating, I can work locally and simply export, then re-import on the live system.
For example: on a recent Moodle project I worked on, to refresh the whole database took seconds... I could push a patch and database update in a few minutes.
However: you should think about maintenance and scheduling... if the site is live and has ongoing data changes then you need to be careful with this. Consider adding a maintenance page.
Advice 2: go research SQL dump/export and importing.
I personally use phpmyadmin to dump and re-import, as it's very convenient.
Advice 3: Working locally then pushing live is MUCH BETTER PRACTICE. You're starting down a much safer and better road than you're on!
Hope that helps... but bear in mind - this is a big subject, so you'll need to research a fair bit.