Here's our situation :
We have 3 different Laravel projects and all 3 projects rely on our Core project.
This Core project is a separate Laravel package hosted on our private repo and is used as a dependency for other projects.
Before, whenever something would change in the Core project we woud just run a composer update ourvendor/ourcorepackage on our servers for each project to pull in the core changes. However as of lately composer seems to suffer from serious memory issues when we try to run the update on our Digital Ocean staging environment with 512 MB Ram. See : https://github.com/composer/composer/issues/1898
The solution I always come across is people saying that you should always run composer install on your production servers. I can relate to that in terms of security because it can be dangerous if you update to a new version of some 3rd party package that can possibly break your code. But in our case we only update our own core package so we know what we're doing but this memory issue forces us to use the composer install method because it is less memory demanding.
So basically this is our current workflow :
When something changes in our core package we need to run a composer
update ourvendor/ourpackage on each project LOCALLY This generates a
composer.lock file
We commit the composer.lock file in our repo
On the servers for each project we run a git pull and run a composer
install. This will only update our core package and runs much faster
and has no memory issues vs composer update
However this solution raises 2 issues :
Since we're working with multiple devs on the same project we sometimes end up having merge conflicts for the composer.lock file when pulling in the changes locally.
Running a git pull on the server gives an error : Your local changes to the following files would be overwritten by merge: composer.lock
Please, commit your changes or stash them before you can merge.
So what am I supposed to do here? Before the pull on the server remove the composer.lock file?
How should we handle the merge conflicts for the composer.lock file?
It's a shame that composer update suffers from memory issues because that method seems much more logical. Just update the package you want and no hassle with the composer.lock file..
Please advice how a correct workflow with GIT and composer should be in our case and how to solve the conflicts above ?
Many thanks for your input
How can you test that a core update (or any other dependency that gets updated) doesn't break things in the projects using it if the developer don't do this step themselves?
That's why the usual workflow is expecting the composer update being run on a development machine having enough RAM (i.e. probably more than 1GB set as memory limit for PHP), and the update should be triggered manually by the developer (and if triggered automatically by a continuous integration build, the memory requirements apply to this machine as well).
There is no way around this memory requirement. A web server with only 512 MB RAM installed might be able to function as a staging server with barely any concurrent users present, but it shouldn't be used to update the Composer dependencies.
Personally I fix the merge conflicts in the composer.lock with a very easy system: Delete the lock file and run composer update. This will update all dependencies to the latest versions that satisfy the version requirements, and create a new working composer.lock file that get's committed during the merge.
I am not afraid to potentially update everything, because either it works as expected, or my tests will catch errors quickly.
I do select the 3rd party packages I use carefully:
they have to tag their versions, preferably using semantic versioning.
I do not use any branches for release versions (the rare occasions that someone used them during development were painful)
they should ship a new major version if they make backwards incompatible changes
the locally developed packages also follow these requirements
This works with around 270 packages served by our local Satis instance (probably also a factor to consider when trying to reduce memory footprint - only the packages known to Composer can end up in memory: Compare the ten thousand packages potentially available on packagist.org with 270 local packages). 60 packages of the 270 are locally developed by 20 developers, and randomly releasing new versions. The update failures in the last 2 years are very rare, and should be handled like other bugs: If a tagged version is detected to be incompatible, we release a bugfix release reverting the change, and tag the original change with a new major release, if the incompatible change is necessary.
So the workflow you ask for is probably like this:
Anytime, any developer should be able to run composer update on their local machine.
They should be able to detect if this breaks things on their local machine.
If nothing is broken, they commit the changes including the composer.lock file to Git
The staging server only runs composer install and will use exactly the versions that the developer used on his machine.
If nothing is broken on staging, that version is ready to be used on production.
Merging an already committed version on another developers machine will likely show merge conflicts with composer.lock.
Resolve conflicts on all other files.
The composer.lock file should be deleted.
From here, the workflow is like above, i.e.:
The developer should be able to run composer update on his local machine.
They should be able to detect if this breaks things on his local machine.
If nothing is broken... and so on.
Another approach (without doing composer update):
Copy your new (and deleted) lines from composer.json into a separate text file.
Use entire remote composer.json and composer.lock files.
During merge conflict mode do:
composer install
For every new package your wrote down in step 1 run composer require vendor/package:version
For every removed package your wrote down in step 1 run composer remove vendor/package
Testing!, Commiting, done!
This method will keep locks from remote branch (maybe master or develop branches), and only updates your new packages.
Sometime composer update can break things.
What I do is.
Discard all of my changes on composer.lock
Merge composer.json
Run composer install so that packages get installed according to composer.lock.
Take one of your package and run composer require vendor/package if you
added that package or composer remove vendor/package if you removed it.
if there are multiple packages it will be installed automatically from composer.json
Related
composer install will install whenever stated in the composer.lock file, but composer update will update all the dependencies and create a new composer.lock file based on what is required in composer.json.
So many said only run composer update in development. But my question is doing composer update did replaced the old composer.lock file, if your app is going to break it will break, because there might be conflict with the new updated dependencies.
I came across with a situation where I must do composer update, the issue is related to pcntl extension. The only solution is to do composer update PHP pcntl module installation
I don't understand why people are afraid of running composer update on production.
TLDR;
Do not run composer update nor composer install in production. Execute it somewhere else and upload the result to the production server, but not to the same directory where the application is hosted. As a general rule, you shouldn't modify the application that's being served while it's being served. Create a different copy of the application and when it's ready replace it with the closest to instantaneous command you can (e.g. mv or ln -s).
But if you HAVE to run either: always run install and create a fresh installation; and never update. install is more predictable and reliable, with update you are at the mercy of any of the project's dependencies.
Composer works recursively. So even if you have very tight version constraints in your composer.json, by running composer update you would be updating not only your dependencies, but your dependencies' dependencies.
While most of the time this won't introduce breakage, sometimes it will. One dependency down the line may introduce a change of behaviour that may impact your code in a way you may have not tested against.
Also, it's basically using the wrong tool for the job. Composer is a dependency management tool, not a deployment tool. To deploy your code to production you should be using some sort of code deployment tool (even if that "tool" is as simple as an FTP upload and a couple of scripts).
The appropriate flow is:
Do all the require and update calls on your development machine, where you can test the project without risk. This generates a composer.lock, which is a known state for the whole project, with discrete installed versions.
Create a new installable version doing install --no-dev. On this step you also should dump an optimized autoloader, run after-install scripts, etc. I usually separate this in more than one step:
composer install --prefer-dist --no-scripts --no-progress --no-suggest --no-interaction --no-dev:
^^ This for a complete, silent installation of everything, excluding development dependencies.
composer dump-autoload --optimize --no-dev
^^ To dump an optimized autoloader script suitable for production.
composer run-script --no-dev post-install-cmd
^^ This is mostly for Symfony, but if you have any post-install scripts to run (e.g. to copy assets to your "public" directory, warm-up some type of cache, anything like that), this would be a good moment to do it.
The result of the above step should be tested (in what typically is a staging environment), and then pushed to production whole (your client code, the vendor folder, the configuration tailored for prod, etc); using whatever deployment method you prefer.
Note that if you use any "slow" push method (FTP, copy, rsync, etc), you shouldn't write directly to the application filesystem, but create a fresh copy of the application and once the file transfer is ready, use a quick method to replace "production" with the new release. A popular and effective way is use a symlink as "production" root, so you only need to update the symlink once all of the above is done and ready, without impacting a running application (that otherwise could be temporarily broken, by virtue of suddenly containing files that belong to different versions of the application).
My thoughts about this are,
The current working state of the system is very important as I would assume some tests have been run against it.
To do composer update would mean that, libraries that are part of the app would have their updates and which may lead to breakage in the system. Because they are libraries that depends on libraries that depends on libraries.
Finally, I would rather do this if composer-update is needed:
Checkout on a dev environment and composer update,
Ensure the app is thoroughly tested on a dev environment
then install on live/production with composer install
My thoughts here :
You should never use composer update without argument.
composer update reads every package listed on composer.json, and updates it to the latest available version compatible with the specified version constraints.
In a perfect world, all librairies would follow semver correctly, and it shouldn't have any side effects. But technically, that is never always true, and you could download a version incompatible with the previous one, or just a version with uncorrected bugs.
So, updating all your packages at once would probably lead to some issues, unless you have the time to check everything on your website to ensure nothing went wrong.
But of course, you'll have to update specific packages sometimes, so using composer update xxx/xxx is useful, assuming you'll check all your implementations of the package.
When the updated package is fully tested, you can commit your code to staging/production, and then run composer install to ensure you'll have the same exact version of package and dependencies on all your platforms.
Long story short, here's the process I use :
composer require xxx/xxx to install new packages
composer update xxx/xxx to update a specific package
composer install on all environments when the package.lock file has been updated.
Additional thoughts
I stumbled once upon an implementation which would give the exact version of the package in composer.json. The developer explained that this way you could use composer update without damage.
I disagree with this option, since even with the exact versions in composer.json, the dependencies are not fixed, and a composer update could lead to potential bugs in them.
I am reading/learning about Composer, the application-level package manager for PHP.
In this blog post written by lead dev Jordi Boggiano, he writes:
Composer on the other hand forces you to declare your project
dependencies in a one-stop location (composer.json at the root). You
just checkout the code, install dependencies, and they will sit in the
project directory, not disturbing anything else on the machine.
Another related feature is the composer.lock file that is generated
when you install or update dependencies. It stores the exact version
of every dependency that was used. If you commit it, anyone checking
out the project will be able to install exactly the same versions as
you did when you last updated that file, avoiding issues because of
minor incompatibilities or regressions in different versions of a
dependency.
If I understand Composer properly, when we're talking about packages downloaded/installed by Composer, we are talking about PHP code packages, ie, programming code written in PHP, and not system-level packages, eg, extensions to the PHP runtime installed on the server. So once these PHP code packages have been downloaded and added to a PHP project, I would have thought those packages become part of the PHP application source code, eg to be checked in to whichever version control system is being used for the project. If another developer comes along and checks out the code, why would they need to then "install the packages", as is stated in the blog post? Wouldn't they get a copy of all code packages when they check out the code from source control? This line in the blog post is confusing me, and making me think I don't understand Composer.
Any clarity on this would be greatly appreciated. Thanks.
The dependencies themselves should not be commited to source control. The composer.json and composer.lock files, on the other hand, should. There's various reasons for this, amongst them:
Every time you update the dependency you would have to commit the changes. That kind of tightly couples your code to the dependency, when it should be exactly the other way around.
The packages themselves are already in their own repository with their own history. Why repeat that in your project's history?
Those repositories can be huge, just muddling the waters around your project. Why carry around all that weight?
Instead, having each developer just run composer install (very important: not composer update) whenever they check out the project is much more efficient. Composer will install the dependencies from composer.lock, making sure everyone running the same commit is on the exact same page. The same goes for deploying.
You can read more about this here.
On the other hand, there might be situations where you have to commit your packages to get around a problem, like for example when you know you won't be able to run composer install on your production server (shared hosting)
Normally packages installed via composer don't get checked in to source control, only the code you write and the composer.json and composer.lock files.
This way the repository for your project does not get bloated with code you did not write and possibly don't really care that much about.
Yes its normal after cloning down your repository a developer will need to run the "composer install" command. The composer.lock file will ensure they get the same modules and versions of them you used when creating your project.
Not including the composer modules in your source control also allow you to easily update to the modules to get bug fixes and new features in new versions of them.
composer install will install whenever stated in the composer.lock file, but composer update will update all the dependencies and create a new composer.lock file based on what is required in composer.json.
So many said only run composer update in development. But my question is doing composer update did replaced the old composer.lock file, if your app is going to break it will break, because there might be conflict with the new updated dependencies.
I came across with a situation where I must do composer update, the issue is related to pcntl extension. The only solution is to do composer update PHP pcntl module installation
I don't understand why people are afraid of running composer update on production.
TLDR;
Do not run composer update nor composer install in production. Execute it somewhere else and upload the result to the production server, but not to the same directory where the application is hosted. As a general rule, you shouldn't modify the application that's being served while it's being served. Create a different copy of the application and when it's ready replace it with the closest to instantaneous command you can (e.g. mv or ln -s).
But if you HAVE to run either: always run install and create a fresh installation; and never update. install is more predictable and reliable, with update you are at the mercy of any of the project's dependencies.
Composer works recursively. So even if you have very tight version constraints in your composer.json, by running composer update you would be updating not only your dependencies, but your dependencies' dependencies.
While most of the time this won't introduce breakage, sometimes it will. One dependency down the line may introduce a change of behaviour that may impact your code in a way you may have not tested against.
Also, it's basically using the wrong tool for the job. Composer is a dependency management tool, not a deployment tool. To deploy your code to production you should be using some sort of code deployment tool (even if that "tool" is as simple as an FTP upload and a couple of scripts).
The appropriate flow is:
Do all the require and update calls on your development machine, where you can test the project without risk. This generates a composer.lock, which is a known state for the whole project, with discrete installed versions.
Create a new installable version doing install --no-dev. On this step you also should dump an optimized autoloader, run after-install scripts, etc. I usually separate this in more than one step:
composer install --prefer-dist --no-scripts --no-progress --no-suggest --no-interaction --no-dev:
^^ This for a complete, silent installation of everything, excluding development dependencies.
composer dump-autoload --optimize --no-dev
^^ To dump an optimized autoloader script suitable for production.
composer run-script --no-dev post-install-cmd
^^ This is mostly for Symfony, but if you have any post-install scripts to run (e.g. to copy assets to your "public" directory, warm-up some type of cache, anything like that), this would be a good moment to do it.
The result of the above step should be tested (in what typically is a staging environment), and then pushed to production whole (your client code, the vendor folder, the configuration tailored for prod, etc); using whatever deployment method you prefer.
Note that if you use any "slow" push method (FTP, copy, rsync, etc), you shouldn't write directly to the application filesystem, but create a fresh copy of the application and once the file transfer is ready, use a quick method to replace "production" with the new release. A popular and effective way is use a symlink as "production" root, so you only need to update the symlink once all of the above is done and ready, without impacting a running application (that otherwise could be temporarily broken, by virtue of suddenly containing files that belong to different versions of the application).
My thoughts about this are,
The current working state of the system is very important as I would assume some tests have been run against it.
To do composer update would mean that, libraries that are part of the app would have their updates and which may lead to breakage in the system. Because they are libraries that depends on libraries that depends on libraries.
Finally, I would rather do this if composer-update is needed:
Checkout on a dev environment and composer update,
Ensure the app is thoroughly tested on a dev environment
then install on live/production with composer install
My thoughts here :
You should never use composer update without argument.
composer update reads every package listed on composer.json, and updates it to the latest available version compatible with the specified version constraints.
In a perfect world, all librairies would follow semver correctly, and it shouldn't have any side effects. But technically, that is never always true, and you could download a version incompatible with the previous one, or just a version with uncorrected bugs.
So, updating all your packages at once would probably lead to some issues, unless you have the time to check everything on your website to ensure nothing went wrong.
But of course, you'll have to update specific packages sometimes, so using composer update xxx/xxx is useful, assuming you'll check all your implementations of the package.
When the updated package is fully tested, you can commit your code to staging/production, and then run composer install to ensure you'll have the same exact version of package and dependencies on all your platforms.
Long story short, here's the process I use :
composer require xxx/xxx to install new packages
composer update xxx/xxx to update a specific package
composer install on all environments when the package.lock file has been updated.
Additional thoughts
I stumbled once upon an implementation which would give the exact version of the package in composer.json. The developer explained that this way you could use composer update without damage.
I disagree with this option, since even with the exact versions in composer.json, the dependencies are not fixed, and a composer update could lead to potential bugs in them.
I'm already half way done with a project in Symfony2.
I need to install a couple of new vendor bundles using composer.
I already have everything (minus logs, cache and parameters.yml) in version control (including the vendor folder).
Problem is when using composer update, it deletes the .svn folders in the vendor folders that where updated. So it's basically impossible to commit now (gives me not a working copy error).
Additional information: I'm working locally and committing to a dev server and then once approved an application server. Therefore it has to be perfect (cannot just run php composer install or php composer update on the dev/application server after commit).
I also tried exporting everything and copying and pasting them back into the repo but that also didn't work (index page broke locally).
Regarding to vendor versioning the best way is not version vendors at all.
The only things you need to version are composer.json and composer.lock. This may cause a problem with vendors which doesn't have stable versions or with that for which you need not stable one (eg. master with particular commit).
As a solution you should create your own (private) vendor repository (let's say your own packagist). Composer has a tool for that, which is called Satis.
https://github.com/composer/satis
So my suggestion would be:
Create a private repository with Satis. You place every package you need in satis.json and whenever you need to update a version of vendor, or add new one, you only modify satis.json and rebuild repository.
In your project's composer.json you set your new private repository as the only repository and set option: packagist to false.
Now, every time you run composer install it will use only your private repository, so it's fast and you always sure that every environemnt has the same versions
-
I was in similar situation two years ago.
The hard lesson I learned was never to edit files within vendor. At first I totally rejected using composer and manually cloned everything I needed. Later on, I decided to fork projects I needed to edit and referenced my forks instead.
Composer supports private GitHub repos - you don't need to register it to Packagist in order to work.
You should not keep your vendor directory in your version control. This is how it is done in Symfony Standard Edition and you should follow this. Running composer install command should be a part of your deployment process
Including vendor packages in your codebase is not recommended, so if you need to maintain the same version of the packages you use on your local machine, the best way is to keep composer.lock in the VCS and running only composer install on other environments.
Additionally, if you want the prod deployment to be instant, without depending on the composer process, you could run composer install on the dev server, and once it's validated you can make your prod deployment script copy the vendor folder from the dev env.
I'm new to Composer (getcomposer.org) and wasn't sure how it works if I install a package locally using Composer and then push my codebase to my production server using Git. Do I have to run Composer again on the production server?
cheers,
J
When you setup your project, you add your dependencies into your composer.json file in your local project directory.
Once you have done this, you will need to run composer update. You can also run composer install, however, without a composer.lock file, composer install actually runs composer update.
Composer update goes out and resolves all the dependencies of all the libraries you are using, downloads them to the /vendor directory, creates an autoloader script and generates the composer.lock file.
For your project what you want to do is version your composer.json AND your composer.lock file.
On your production server, you will always run composer install, which insures that the libraries on your production server are the exact same ones you utilized in your development process.
composer install is also a lot faster as it does not have to do all the dependency management work, and can almost always just pull a specific commit#. It doesn't have to look at version strings. Thus is is usually very fast, once a server has already gone through it once.
In development the only time you should run composer update, is when you introduce a new library OR you have an issue where an underlying library has been changed and you know that you need to have composer go out and re-calculate the dependencies. composer update always recalculates and downloads the latest revisions of any library available even if the version level did not change. This means that there is a potential for something to have become broken, necessitating the potential for as full a set of regression tests as you might have available. In short, something having nothing to do with what you're actually changing could have broken, so you only want to introduce the potential for change when you are forced to.
Of course, if you did introduce a new library, you have no choice but to run composer update.
Once you run composer update, your composer.lock file will be updated (as expected) and the production server will pick this up when you run composer install on it.
As others stated, put the vendors in your gitignore. The point is that these are external libraries that you depend on, but that do not belong in your project, and should not be versioned. In the old days some people utilized git submodules, and it's a big PITA you really want to avoid, not to mention that submodules don't address dependencies of the libraries you included.
It depends how are you working. If you, like getcomposer.org says, are ignoring the "vendor" folder then you need to run it again. If you are versioning the "vendor" folder then you don't need to run it again.
Have in mind that composer will get in charge of managing your dependencies versions, so there is no need to put your dependencies files under versioning. If you put these files under git you will only make your repository bigger.
Read https://getcomposer.org/doc/01-basic-usage.md#installing-dependencies.
For clarification when you ignore the "vendor" folder Git don't track the files under the folder so if you clone the repo it will be like composer never was executed