This question already has answers here:
Should composer.lock be committed to version control?
(10 answers)
Closed 1 year ago.
In an answer to How to get the exact version of included packages in my private repository, I made the statement that composer.lock should not be put under version control for a package. When installing a package, this file is not used after all.
I've peeked in a set of popular repositories, and most of them do not contain a lock file (like Symfony, Laravel, Guzzle, Monolog). On the other hand, the Doctrine repositories contain that file, and I'd like to know if there is any good reason to do so, or to omit the file.
Side note: this is about packages, libraries, however you want to call them. For applications, this is a different thing, as you want to stick to specific versions of each dependency when working together in a team or deploying to other systems. How to handle this different situation is covered in Should composer.lock be committed to version control?, but it does not contain too much arguments for my use case
Since the file is not used in any useful manner when installing the package, as a functionality of the library itself for the end user, it's at least not relevant to the user of the library.
Then the reasoning becomes whether it's useful to the developers of the library to have a locked set of dependencies that they need to perform development tasks, such as specific versions of testing frameworks etc. In those cases the argument can be that the composer.json file fulfils the same role as in a regular application - it locks down the dependencies to those that we know work.
However, there's a caveat here - when developing a library you really want the use case to be the same as what the user of the library experiences when he/she installs it. Considering this it usually makes more sense to lock down the explicit version in composer.json instead of relying on the lock file to provide the same functionality. That makes any CI solution install the correct set (the same as what a user would get) of dependencies when for running the tests. You can however make that process update the lock file locally before running the tests to have multiple test cases - one with the locked dependencies and one with the most recent versions (as the user would get).
Doctrine has made the decision that lock files should be committed for their own reasons, which are perfectly valid - in effect they come down to the tools used for their development workflows:
All Doctrine projects must commit the composer.lock file. Tools like phpstan and phpcs are quite fragile on patch releases and we don't want builds to start failing without us having made any changes to our own code. Whenever a dependency needs to be upgraded, the composer.lock file should be updated locally and the change submitted via pull request.
An argument can be made for both cases; it'll be up to the preferences of the project itself and its developers. I'm leaning towards it not being committed, since that more closely replicates what a user would experience when installing the library. However, there would still be local lock files present for each developer, meaning that what each developer has on their own computer when developing the library could differ. Committing the lock file would make that more similar across the board for all developers, but would require extra care to replicate the experience for users (and then, we're back to our original arguments again..).
My post was not about pure libraries, but a kind of module that has many dependencies on other libraries. The module is part of various applications. If, for example, I run a composer install without composer.lock when deploying my application, I might roll out stands that I have not tested. Therefore I fix the dependencies of my module release on a concrete status and of course commit the composer.lock. Therefore, the comparison to frameworks like Symfony is a bit lagging in my opinion, because nothing is deployed here.
When a team work on a Laravel project, they usually edit route file, the same controllers and views. Resulting in thousands of merging conflicts. And resolving them is time consuming. Is there a way of preventing these conflicts?.
What is git?
Git is a mature, actively maintained open source project. A staggering number of software projects rely on Git for version control, including commercial projects as well as open source. Git is an example of a DVCS (hence Distributed Version Control System). Rather than having only one single place for the full version history of the software as is common in the once-popular version control systems like CVS or Subversion (also known as SVN), in Git, every developer's working copy of the code is also a repository that can contain the full history of all changes.
In addition to being distributed, Git has been designed with performance, security and flexibility in mind.
When to use git?
For example, say a developer, Alice, makes changes to source code, adding a feature for the upcoming 2.0 release, then commits those changes with descriptive messages. She then works on a second feature and commits those changes too. Naturally, these are stored as separate pieces of work in the version history. Alice then switches to the version 1.3 branch of the same software to fix a bug that affects only that older version. The purpose of this is to enable the Alice's team to ship a bug fix release, version 1.3.1, before version 2.0 is ready. Alice can then return to the 2.0 branch to continue working on new features for 2.0 and all of this can occur without any network access and is therefore fast and reliable. She could even do it on an airplane. When she is ready to send all of the individually committed changes to the remote repository, Alice can "push" them in one command.
What are the advantages of git?
Git has a lot of usage and advantage but few are below that may clear to you when and why to use git
Git is a version control system or stagging for open source project. You can control your project by creating version step by step.
If an issue has been aroused after changes it is possible in git to restore a previous version to ignore all those changes and bugs.
We create a project on git as origin master and then make clones in different machines when working as a team project. When we 'push' our changes from different clones on different machines git merge all those changes in one origin master. So we don't need to bother about how to merge changes from different clones in one project folder.
How to resolve merge conflicts in git?
May this be help full to you to understand how to solve easily merges conflicts problem in git. The best tool for this purpose is git-mergetool Run merge conflict resolution tools to resolve merge conflicts in git projects.
Use git mergetool to run one of several merge utilities to resolve merge conflicts. It is typically run after git merges.
If one or more parameters are given, the merge tool program will be run to resolve differences on each file (skipping those without conflicts). Specifying a directory will include all unresolved files in that path. If no names are specified, git mergetool will run the merge tool program on every file with merge conflicts.
Backup in git mergetool git mergetool creates *.orig backup files while resolving merges. These are safe to remove once a file has been merged and its git mergetool session has completed.
Setting the mergetool.keepBackup configuration variable to false causes git mergetool to automatically remove the backup as files are successfully merged.
For further study about git mergetool please see the link:
https://www.git-tower.com/learn/git/ebook/en/command-line/advanced-topics/merge-conflicts
I think this may be helpful to you.
Yes GIT is a good solution for a large project. But you have to deal the route or any common file on which more than one person is working so that the merge conflict are minimum. On any common file make different code block like:
/* Block 1 Start */
/* Block 1 End */
/* Block 2 Start */
/* Block 2 End */
and put all the related functionality inside these blocks, so that the conflict are minimum and when it occurs, it is easy to resolve the conflict as the blocks are divided.
It happens in almost every project when using git , we all face these situations like working on same files which result in merge conflicts but on the other hand we all team members have a synced code at the end of the day. Also as an alternative to avoid multiple merge conflicts in each pull , you can make your branches and then push your code to your respective branches. In this way there will be merge conflict only when you merge these branches to your main branch. This is just my opinion that GIT is the best tool when it comes to project management.
You could also try adding comments section in your file if you don't want to create that many branches.
I'm a total newbie to this Git.
My PHP project files have been added to Git by admin.
Now one new person is going to start working on this project. He will work on one module of this project. So, being a senior developer I've been asked to create a branch for him that will contain only specific files that he will need to start work on the specific module.
So this thing has created so many questions in my mind :
Can I create a special branch for him with only specific/selected files from the project? If yes, how? If no, why?
Now only master branch of project is present. If the new branch of git is created for the new developer and he commits and pushes the changes he made to the git; how will they get merged with the master branch? Do I need to do it manually using third party tool like 'DeployHQ' or anything like or is there any way around.
To keep the things easy for him what I want to do is he should be able to commit, push the changes, those changes would straight away be reflected on server and he should be able to check it by running the pages in a browser. Can I make the this simple and easy as I'm thinking.
In a nutshell I don't want to disclose all of my project files to him and want to keep things easier and simpler for me as well as for him.
Please please please guide me.
Thanks.
The basic building block of GIT version control is project. You can't branch off only some files from the master as it doesn't make any sense in an environment where projects are the single version controlled entities.
You can add or remove files from a branch and later commit to the master with the changes.
Some people refer to the branching model in Git as its “killer
feature” , and it certainly sets Git apart in the VCS community. Why
is it so special? The way Git branches is incredibly lightweight,
making branching operations nearly instantaneous and switching back
and forth between branches generally just as fast. Unlike many other
VCSs, Git encourages a workflow that branches and merges often, even
multiple times in a day. Understanding and mastering this feature
gives you a powerful and unique tool and can literally change the way
that you develop.
We're starting a new project, and we're managing dependencies with Composer. We'll probably build our app on top of Laravel 4. But we'll also create our own library, which we will use for all our next projects, not just this one.
So, we have this terrible doubt: what's the best way to develop a library using composer?
If we list that new library as a dependency, every time we modify it we will have to commit the change to the repository and then call composer update.
That seems terrible!
Is there a better way to do that?
I think there are two ways to handle this, which I use depending on the case:
The library is a pure library, which is standalone, fully tested, and develop it using TDD to ensure that it all works. That way it can be used with the "commit, update" cycle you described just fine I think.
You are developing a plugin or something that must be integrated in something else (application/framework) and testing it standalone is more difficult, or you are developing it very tightly with your application. In this case require the dev-master version of the library so Composer installs it with a git clone (if it was already installed as a tag you will have to rm -rf vendor/your/library to force a reinstall as opposed to an update). You can also force this for tagged releases using the --prefer-source flag. Then once you have a clone in the vendor dir you can very easily work directly in there. If you do work in a team though you will still need to do this commit and then update to make sure the others get the latest version.
The third alternative is to just develop the code in the src/ directory of your application until it is mostly stabilized and then you can extract it as a new package and add it back as a dependency, then fall back on the first two ways I described because it will then be a lot more viable.
If you set the dependency to the repository master branch instead of a packaged distribution file, Composer will check out a working copy into the vendors folder. You can modify this working copy right in the vendors folder, as if it's part of the main project, but then commit it into its own repository. You'll indeed have to make sure to composer update after that to keep the composer.lock file in sync with the development of that library though.
It's still the more convenient way to develop a project in tandem with a dependency.
If you aim to develop a truly awesome library, then you should try to develop it independently of any other software you create.
It should fulfill one exact task only. And this probably is done after some commits, so the initial creation of the library should take only a week or two to come to a stable first version. And this version can be tagged and then used elsewhere.
When tagging, strictly try to follow semantic versioning - that way you can use the library with a version restriction like "~1.0", meaning at least version 1.0, but anything up to 1.9999 is acceptable, as long as it is not 2.0 (which would mean incompatible changes).
And then you really do not need to update any other software when you release a new version of the library. You only need to update if you want to include fixed bugs. Without bugfixes, you can update, but there is no need to do so immediately after the library's new version release.
Composer will take care of all the dependencies you need. The most important thing if you start a new library is to include the composer.json right from the start into the repository.
What if you really want to always include the newest release of the library in every other software you write? I'm not sure you realize the implications this has. It means that you are strictly binding your other software to the most recent library version. Break that version, or introduce a nasty bug, and all your software breaks. So being able to update or not actually is a feature. You will find that all foreign libraries you might use will follow the same release mechanism: They tag a new version if an important bug was fixed, or if a reasonable amount of new features was implemented. They do not wait for you to approve a new version - you have to approve THEIR new version in your software by explicitly updating to the most recent one. And the same should apply to an internal library.
Try to avoid fiddling with "dev-master" solutions mentioned here. They might work, but Composer works best if used with tagged versions. If you have a reasonably stable state of your library, tag it with "0.0.0" and include that version everywhere else instead of "dev-master". And then tag according to semantic version rules.
From my experience, one of the bigger problems we come across during our webdevelopment process is keeping different setups updated and secure across different servers.
My company has it's own CMS which is currently installed across 100+ servers. At the moment, we use a hack-ish FTP-based approach, combined with upgrade scripts at specific locations to upgrade all of our CMS setups. Efficiently managing these setups becomes increasingly difficult and risky when there are several custom modules involved.
What is the best way to keep multiple setups of a web application secure and up-to-date?
How do you do it?
Are there any specific tips regarding modularity in applications, in order to maintain flexibility towards our clients, but still being able to efficiently manage multiple "branches" of an application?
Some contextual information: we mainly develop on the LAMP-stack. One of the main factors that helps us sell our CMS is that we can plugin pretty much anything our client wants. This can very from 10 to to 10.000 lines of custom code.
A lot of custom work consists of very small pieces of code; managing all these small pieces of code in Subversion seems quite tedious and inefficient to me (since we deliver around 2 websites every week, this would result in a lot of branches).
If there is something I am overlooking, I'd love to hear it from you.
Thanks in advance.
Roundup: first of all, thanks for all of your answers. All of these are really helpful.
I will most likely use a SVN-based approach, which makes benlumley's solution closest to what I will use. Since the answer to this question might differ in other usecases, I will accept the answer with the most votes at the end of the run.
Please examine the answers and vote for the ones that you think have the most added value.
I think using a version control system and "branching" the part of the codes that you have to modify could turn out to be the best approach in terms of robustness and efficiency.
A distributed version system could be best suited to your needs, since it would allow you to update your "core" features seamlessly on different "branches" while keeping some changes local if need be.
Edit: I'm pretty sure that keeping all that up to date with a distributed version system would be far less tedious than what you seem to expect : you can keep the changes you are sure you're never going to need elsewhere local, and the distributed aspect means each of your deployed application is actually independent from the others and only the fix you mean to propagate will propagate.
If customizing your application involves changing many little pieces of code, this may be a sign that your application's design is flawed. Your application should have a set of stable core code, extensibility points for custom libraries to plug into, the ability to change appearance using templates, and the ability to change behavior and install plugins using configuration files. In this way, you don't need a separate SVN branch for every client. Rather, keep the core code and extension plugin libraries in source control as normal. In another repository, create a folder for each client and keep all their templates and configuration files there.
For now, creating SVN branches may be the only solution that helps you keep your sanity. In your current state, it's almost inevitable that you'll make a mistake and mess up a client's site. At least with branches you are guaranteed to have a stable code base for each client. The only gotcha with SVN branches is if you move or rename a file in a branch, it's impossible to merge that change back down to the trunk (you'd have to do it manually).
Good luck!
EDIT: For an example of a well-designed application using all the principles I outlined above, see Magento E-Commerce. Magento is the most powerful, extensible and easy to customize web application I've worked with so far.
I may be wrong, but it seems to me what Aron is after is not version control. Versioning is great, and I'm sure they're using it already, but for managing updates on hundreds of customized installations, you need something else.
I'm thinking something along the lines of a purpose-built package system. You'll want every version of a module to keep track of its individual dependencies and 'guaranteed compatibilities', and use this information to automatically update only the 'safe' modules.
E.g. let's say you've built a new version 3 of your 'Wiki' module. You want to propagate the new version to all the servers running your application, but you've made changes to one of the interfaces within the Wiki module since version 2. Now, for all default installations, that is no problem, but it would break installations with custom extensions on top of the old interface. A well-planned package system would take care of this.
To address the security question, you should look into using digital signatures on your patches. There are lots of good libraries available for public-key-based signatures, so just go with whatever seems to be the standard for your chosen platform.
Not sure whether someone's said this, there are a lot of long responses here, and I've not read them all.
I think a better approach to your version control would be to have your CMS sat on its own in its own repository and each project in its own. (or, all of these could be subfolders within one repo i guess)
You can then use its trunk (or a specific branch/tag if you prefer) as an svn:external in each project that requires it. This way, any updates you make to the CMS can be committed back to its repository, and will be pulled into other projects as and when they are svn updated (or the external is svn:switch 'ed).
As part of making this easier, you will need to make sure the CMS and the custom functionality sit in different folders, so that svn externals works properly.
IE:
project
project/cms <-- cms here, via svn external
project/lib <-- custom bits here
project/www <-- folder to point apache/iis at
(you could have cms and lib under the www folder if needed)
This will let you branch/tag each project as you wish. You can also switch the svn:external location on a per branch/tag basis.
In terms of getting changes live, I'd suggest that you immediately get rid of ftp and use rsync or svn checkout/exports. Both work well, the choice is up to you.
I've got most experience with the rsync route, rsyncing an svn export to the server. If you go down this route, write some shell scripts, and you can create a test shell script to show you the files it will upload without uploading them as well, using the -n flag. I generally use a pair of scripts for each environment - one a test, and one to actually do it.
Shared key authentication so you don't need a password to send uploads up may also be useful, depending on how secure the server to be given the access is.
You could also maintain another shell script for doing bulk upgrades, which simply calls the relevant shell script for each project you want to upgrade.
Have you looked at Drupal? No, not to deploy and replace what you have, but to see how they handle customizations and site-specific modules?
Basically, there's a "sites" folder which has a directory for every site you're hosting. Within each folder is a separate settings.php which allows you to specify a different database. Finally, you can (optionally) have "themes" and "modules" folders within sites.
This allows you to do site-specific customizations of particular modules and limit certain modules to those sites. As a result, you end up with a site that the vast majority of everything is perfectly identical and only the differences get duplicated. Combine that with the way it handles upgrades and updates and you might have a viable model.
Build into the code a self-updating process.
It will check for updates and run them when/where/how you have configured it for the client.
You will have to create some sort of a list of modules (custom or not) that need to be tested with the new build prior to roll-out. When deploying an update you will have to ensure these are tested and integrated correctly. Hopefully your design can handle this.
Updates are ideally a few key steps.
a) Backup so you can back out. You should be able to back out
the entire update at any time. So,
that means creating a local archive
of the application and database
first.
b) Update Monitoring Process - Have the CMS system phone home to look for a new build.
c) Schedule Update on availability - Chances are you don't want the update to run the second it is available. This means you will have to create a cron/agent of some kind to do the system update automatically in the middle of the night. You can also consider client requirements to update on weekends, or on specific days. You can also stagger rolling out your updates so you don't update 1000 clients in 1 day and get tech support hell. Staggered roll-out of some kind might be beneficial for you.
d) Add maintenance mode to update the site -- Kick the site into maintenance mode.
e) SVN checkout or downloadable packages -- ideally you can deploy via svn checkout, and if not, setup your server to deliver svn generated packages into an archive that can be deployed on client sites.
f) Deploy DB Scripts - Backup the databases, update them, populate them
g) Update site code - All this work for one step.
h) Run some tests on it. If your code has self-tests built in, it would be ideal.
Here's what I do...
Client-specific include path
Shared, common code is in shared/current_version/lib/
Site specific code is in clients/foo.com/lib
The include path is set to include from the clients/foo.com/lib, and then share/lib
The whole thing is in a version control system
This ensures that the code uses shared files wherever possible, but if I need to override a particular class or file for some reason, I can write a client specific version in their folder.
Alias common files
My virtual host configuration will contain a line like
Alias /common <path>/shared/current_version/public_html/common
Which allows common UI elements, icons, etc to be shared across projects
Tag the common code with each site release
After each site release, I tag the common code by creating a branch to effectively freeze that point in time. This allows me to deploy /shared/version_xyz/ to the live server. Then I can have a virtual host use a particular version of the common files, or leave it pointing at the current_version if I want it to pick up the latest updates.
Have you looked at tools such as Puppet (for system administration incl. app deployment) or Capistrano (deployment of apps in RubyOnRails but not limited to these)?
One option would be to set up a read-only version control system (Subversion). You could integrate access to the repository into your CMS and invoke the updates through a menu, or automatically if you do not want the user to have a choice about an update (could be critical). Using a version control system would also allow you to keep different branches easily
As people have already mentioned that using version control (I prefer Subversion due to functionality) and branching would be the best option. Another open source software available on sourceforge called cruisecontrol. Its amazing, you configure cruisecontrol with subversion in sach a way that any code modification or new code added in serversion, Cruise control will know automatically and will do build for you. It will save your hell of time.
I have done the same way in my company. we have four projects and have to deploy that project on different servers. I have setup cruiseconrol in such a way that any modification in code base triggers automatic build. and another script will deploy that build on the server. your are good to go.
If you use a LAMP stack I would definitely turn the solutions files into a package of your distribution and use it for propagate changes. I recommend for that matter Redhat/Fedora because of RPM and it's what I have experience on. Anyway you can use any Debian based distribution too.
Sometime ago I made a LAMP solution for managing an ISP hosting servers. They had multiple servers to take care of web hosting and I needed a way to deploy the changes of my manager, because every machine was self-contained and had a online manager. I made a RPM package containing the solution files (php mostly) and some deploying scripts that runned with the RPM.
For automated updating we had our own RPM repository set on every server in yum.conf. I set an crontab job to update the servers daily with the latest RPMs from that trusted repository.
Trustiness can be achieve too because you can use trust settings in the RPM packages, like signing them with your public key file and accepting only signed packages.
Hm could it be an idea to add configuration files? You wrote that a lot of small script are doing something. Now if you'd build them into the sources and steered them with configuration files shouldn't that "ease" that?
On the other hand having branches for every customer looks like an exponential growth to me. And how would you "know" which areas you've done something and do not forget to "make" changes in all other branches also. That looks quite ugly to me.
It seems a combination of revision controls, configuration options and/or deployment receipts seems to be a "good" idea.....
With that many variations on your core software, I think you really need a version control system to stay on top of pushing updates from the trunk to the individual client sites.
So if you think Subversion would be tedious, you've got a good sense for what the pain points will be... Personally, I wouldn't recommend Subversion for this, since it's not really that good at managing & tracking branches. Although benlumley's suggestion to use externals for your core software is a good one, this breaks down if you need to tweak the core code for your client sites.
Look into Git for version control, it's built for branching, and it's fast.
Check out Capistrano for managing your deployments. It's a ruby script, often used with Rails, but it can be used for all sorts of file management on remote servers, even non-ruby sites. It can get the content to the remote end through various stragegies including ftp, scp, rsync, as well as automatically checking out the latest version from your repository. The nice features it provides include callback hooks for every step of the deploy process (e.g. so you can copy your site-specific configuration files which might not be in version control), and a release log system--done through symlinks--so you can quickly roll back to a previous release in case of trouble.
I'd recommend a config file with the list of branches and their hosted location, then run through that with a script that checks out each branch in turn and uploads the latest changes. This could be cron'd to do nightly updates automatically.