I want to minimize traffic usage with composer packages load, so I need for production loads to only download the source/lib folders without examples/docs/readme.
I haven't found any information in the official composer documentation.
UPD
One of the best solution - add .gitattributes file into your project and add main rows from this gist
This is not really up to composer and instead a responsibility of package maintainers. You can use --prefer-dist to install a distribution instead of checking out the git repository as is usually done during development, but what this distribution contains is managed by the package maintainer.
If the packages you install allow for pull requests & issues, you could ask them to provide a .gitattributes file which is recognized by github when creating an archive or provide a pull request providing one. In this file you can specify which files and folders will be excluded from the archive using export-ignore. In any case since everything is bundled in a zip the bandwidth saved is probably negligible.
So, short answer is the only way to minimize bandwidth usage and allocated disk space for dependencies is using the --prefer-dist option - and omitting dev dependencies with --no-dev - when installing them for production.
Related
I am reading/learning about Composer, the application-level package manager for PHP.
In this blog post written by lead dev Jordi Boggiano, he writes:
Composer on the other hand forces you to declare your project
dependencies in a one-stop location (composer.json at the root). You
just checkout the code, install dependencies, and they will sit in the
project directory, not disturbing anything else on the machine.
Another related feature is the composer.lock file that is generated
when you install or update dependencies. It stores the exact version
of every dependency that was used. If you commit it, anyone checking
out the project will be able to install exactly the same versions as
you did when you last updated that file, avoiding issues because of
minor incompatibilities or regressions in different versions of a
dependency.
If I understand Composer properly, when we're talking about packages downloaded/installed by Composer, we are talking about PHP code packages, ie, programming code written in PHP, and not system-level packages, eg, extensions to the PHP runtime installed on the server. So once these PHP code packages have been downloaded and added to a PHP project, I would have thought those packages become part of the PHP application source code, eg to be checked in to whichever version control system is being used for the project. If another developer comes along and checks out the code, why would they need to then "install the packages", as is stated in the blog post? Wouldn't they get a copy of all code packages when they check out the code from source control? This line in the blog post is confusing me, and making me think I don't understand Composer.
Any clarity on this would be greatly appreciated. Thanks.
The dependencies themselves should not be commited to source control. The composer.json and composer.lock files, on the other hand, should. There's various reasons for this, amongst them:
Every time you update the dependency you would have to commit the changes. That kind of tightly couples your code to the dependency, when it should be exactly the other way around.
The packages themselves are already in their own repository with their own history. Why repeat that in your project's history?
Those repositories can be huge, just muddling the waters around your project. Why carry around all that weight?
Instead, having each developer just run composer install (very important: not composer update) whenever they check out the project is much more efficient. Composer will install the dependencies from composer.lock, making sure everyone running the same commit is on the exact same page. The same goes for deploying.
You can read more about this here.
On the other hand, there might be situations where you have to commit your packages to get around a problem, like for example when you know you won't be able to run composer install on your production server (shared hosting)
Normally packages installed via composer don't get checked in to source control, only the code you write and the composer.json and composer.lock files.
This way the repository for your project does not get bloated with code you did not write and possibly don't really care that much about.
Yes its normal after cloning down your repository a developer will need to run the "composer install" command. The composer.lock file will ensure they get the same modules and versions of them you used when creating your project.
Not including the composer modules in your source control also allow you to easily update to the modules to get bug fixes and new features in new versions of them.
I'm wondering how I would be able to load PHP packages without using composer to do so.
I'm unable to use composer as I'm using shared hosting, which imposes limitations like being unable to install composer due to a heavily jailed shell.
While there are certain packages which allow for the loading of a package with require_once('package/Autoloader.php');, i.e. Braintree & Twig.
However, not all packages can be loaded like this, example being,
Pug-PHP, which only includes installation with composer.
Can anyone help me load composer packages without composer?
This is not possible due to Packages being a dependency of the composer management system. Replicate the ENV locally and use #MichalGallovic 's suggestion.
There is also the option of git clone every package and dependency from their respective source repos; but at that point you will hate yourself.
Is the option of using a host that are not as strict an option?
Visit https://php-download.com/ They have a number of packages ready to go that will no longer need Composer. It finds and creates all required folders and creates a zips it for download. Saves me hours of drama.
When using composer to get dependencies a huge amount of unnecessary files are download such as documentations, test units, etc. These make the built file large. How to only include production files from composer folder?
Unfortunately, the most package developer don't exploit .gitattributes files and using or not using --prefer-dist is the same for them. I come up with the following gulp task: https://gist.github.com/salarmehr/b62703afb6617f4cacac
This isn't as easy as it looks at first glance.
Composer itself isn't the right place to fiddle with project's published code. So the next stop is .gitattributes.
Symfony tried to do this, and their experience made them revert this decision. Composer had information about using .gitattributes in the documentation, but removed it.
In essence, removing some part of a package from a distinct distribution path is likely to cause more problems than it solves. From my perspective, the CLI switch --prefer-dist and --prefer-source is a selector of either having to clone a huge repository that takes ages or download a ZIP with that exact version - but the results should be equal, i.e. I should not be forced to --prefer-source for ALL my dependencies just because one single package that decided to "optimize for deployment" decided to remove documentation and tests from their ZIP.
Yes, during development I usually look at their code and tests to help me understand what's going on - or what SHOULD go on, and isn't.
Conclusion: Composer is NOT a deployment tool. If you care about the size of your application, it is your task to remove everything you don't need or want, and probably optimize other things as well (minify JS and CSS, optimize images etc.). It should not be Composers' or any package maintainers task to do this optimization for you.
Here's our situation :
We have 3 different Laravel projects and all 3 projects rely on our Core project.
This Core project is a separate Laravel package hosted on our private repo and is used as a dependency for other projects.
Before, whenever something would change in the Core project we woud just run a composer update ourvendor/ourcorepackage on our servers for each project to pull in the core changes. However as of lately composer seems to suffer from serious memory issues when we try to run the update on our Digital Ocean staging environment with 512 MB Ram. See : https://github.com/composer/composer/issues/1898
The solution I always come across is people saying that you should always run composer install on your production servers. I can relate to that in terms of security because it can be dangerous if you update to a new version of some 3rd party package that can possibly break your code. But in our case we only update our own core package so we know what we're doing but this memory issue forces us to use the composer install method because it is less memory demanding.
So basically this is our current workflow :
When something changes in our core package we need to run a composer
update ourvendor/ourpackage on each project LOCALLY This generates a
composer.lock file
We commit the composer.lock file in our repo
On the servers for each project we run a git pull and run a composer
install. This will only update our core package and runs much faster
and has no memory issues vs composer update
However this solution raises 2 issues :
Since we're working with multiple devs on the same project we sometimes end up having merge conflicts for the composer.lock file when pulling in the changes locally.
Running a git pull on the server gives an error : Your local changes to the following files would be overwritten by merge: composer.lock
Please, commit your changes or stash them before you can merge.
So what am I supposed to do here? Before the pull on the server remove the composer.lock file?
How should we handle the merge conflicts for the composer.lock file?
It's a shame that composer update suffers from memory issues because that method seems much more logical. Just update the package you want and no hassle with the composer.lock file..
Please advice how a correct workflow with GIT and composer should be in our case and how to solve the conflicts above ?
Many thanks for your input
How can you test that a core update (or any other dependency that gets updated) doesn't break things in the projects using it if the developer don't do this step themselves?
That's why the usual workflow is expecting the composer update being run on a development machine having enough RAM (i.e. probably more than 1GB set as memory limit for PHP), and the update should be triggered manually by the developer (and if triggered automatically by a continuous integration build, the memory requirements apply to this machine as well).
There is no way around this memory requirement. A web server with only 512 MB RAM installed might be able to function as a staging server with barely any concurrent users present, but it shouldn't be used to update the Composer dependencies.
Personally I fix the merge conflicts in the composer.lock with a very easy system: Delete the lock file and run composer update. This will update all dependencies to the latest versions that satisfy the version requirements, and create a new working composer.lock file that get's committed during the merge.
I am not afraid to potentially update everything, because either it works as expected, or my tests will catch errors quickly.
I do select the 3rd party packages I use carefully:
they have to tag their versions, preferably using semantic versioning.
I do not use any branches for release versions (the rare occasions that someone used them during development were painful)
they should ship a new major version if they make backwards incompatible changes
the locally developed packages also follow these requirements
This works with around 270 packages served by our local Satis instance (probably also a factor to consider when trying to reduce memory footprint - only the packages known to Composer can end up in memory: Compare the ten thousand packages potentially available on packagist.org with 270 local packages). 60 packages of the 270 are locally developed by 20 developers, and randomly releasing new versions. The update failures in the last 2 years are very rare, and should be handled like other bugs: If a tagged version is detected to be incompatible, we release a bugfix release reverting the change, and tag the original change with a new major release, if the incompatible change is necessary.
So the workflow you ask for is probably like this:
Anytime, any developer should be able to run composer update on their local machine.
They should be able to detect if this breaks things on their local machine.
If nothing is broken, they commit the changes including the composer.lock file to Git
The staging server only runs composer install and will use exactly the versions that the developer used on his machine.
If nothing is broken on staging, that version is ready to be used on production.
Merging an already committed version on another developers machine will likely show merge conflicts with composer.lock.
Resolve conflicts on all other files.
The composer.lock file should be deleted.
From here, the workflow is like above, i.e.:
The developer should be able to run composer update on his local machine.
They should be able to detect if this breaks things on his local machine.
If nothing is broken... and so on.
Another approach (without doing composer update):
Copy your new (and deleted) lines from composer.json into a separate text file.
Use entire remote composer.json and composer.lock files.
During merge conflict mode do:
composer install
For every new package your wrote down in step 1 run composer require vendor/package:version
For every removed package your wrote down in step 1 run composer remove vendor/package
Testing!, Commiting, done!
This method will keep locks from remote branch (maybe master or develop branches), and only updates your new packages.
Sometime composer update can break things.
What I do is.
Discard all of my changes on composer.lock
Merge composer.json
Run composer install so that packages get installed according to composer.lock.
Take one of your package and run composer require vendor/package if you
added that package or composer remove vendor/package if you removed it.
if there are multiple packages it will be installed automatically from composer.json
A composer install normally takes a few minutes. And on a production environment it's feels too slow.
Is it possible to make a composer install to a temp directory and then switch it? If that is possible the downtime should be about zero.
Or are there any other way to do a composer install faster?
I created a composer plugin to download packages in parallel.
https://packagist.org/packages/hirak/prestissimo
$ composer global require hirak/prestissimo
Please try it. In my environment, composer install become 10 times faster.
You can sometimes speed up composer install significantly by using the --prefer-dist flag, which just happens to be recommended for production use:
--prefer-dist: Reverse of --prefer-source, composer will install from dist if possible. This can speed up installs substantially on build servers and other use cases where you typically do not run updates of the vendors.
composer install docs here: http://getcomposer.org/doc/03-cli.md#install
Edited To Clarify Sometimes
I say it sometimes speeds up composer install because there are plenty of factors that go into it feeling slow, not the least of which are network performance and the current Github status. A slow install can be really frustrating, but it's not always b/c of Composer.
You are asking two different and unrelated things.
Yes, it is a solution to build the next version of your site in a separate directory then put it in place after you moved the old version out of the way. It is, actually, the best solution.
This is how the deployment scripts I build work:
prepare the next version of the site in a separate directory (let's say /var/www/new); the following list of items and their order is not static, some projects need a different flow:
get the last version of the code from the repo;
remove the files that are not needed on the live site (.gitignore, IDE project files, placeholders etc);
run composer install;
copy/generate the configuration files containing the settings for the live servers (the ones stored in the repository contain dummy values);
change the user and permissions of all files; make some directories writable by the web server;
create symlinks, directories etc; for example, the directory that contains the user uploaded files is somewhere outside the server directory and a symlink to it is created during deploy to make its content available through the web server;
move the live code out of the way; I use mv to move the entire directory (/var/www/html to /var/www/old, for example);
move the prepared new version to the proper place (mv /var/www/new /var/www/html);
move the previous version into the archive (after I remove the content of vendor and other files that do not change or are external).
The advantages:
the downtime is zero (microseconds, probably, between steps #2 and #3);
a failed build doesn't affect the live site (if handled properly);
reverting to the previous version (if needed) can be done easily.
Regarding the other question, the only way I know to speed composer up is to avoid running it using a PHP that loads the xdebug extension. The xdebug extension shouldn't be loaded on the production server anyway.