I have a project with multiple modules. Each module uses Composer inside itself and is mostly independent of the other modules.
However some modules share dependencies that have different versions. These dependencies are backwards compatible for the most part and use semantic versioning.
I would like to ensure that the dependency with the highest semantic version take precedence. This would allow all modules to share the same dependency and backwards compatibility of these dependencies would ensure nothing breaks.
My plan was to do this by controlling the order in which I call require_once on the individual autoloaders. The code below is an example, which in practice is generated.
require_once(__DIR__ . '/moduleA/vendor/autoload.php');
require_once(__DIR__ . '/moduleB/vendor/autoload.php');
require_once(__DIR__ . '/moduleC/vendor/autoload.php');
The main assumption I was making was that if an autoloader is required before another, it would take precedence over later ones.
What I've found though is that the opposite is true. The autoloader that comes last seems to be taking precedence over the others.
Consider a class Foo\MyClass was a dependency shared between these modules. I'm expecting with the above load order, Foo\MyClass would be picked up from moduleA/vendor/....
Instead it is coming form moduleC/vendor/....
I could flip my generated order to workaround this, but I'd like to verify if there is a predictable order in which the PHP autoloaders.
Is there an order in which PHP executes Autoloaders? Do multiple Composer autoloaders affect this in any way?
Thanks.
Actually, you are in a mess, but you are half way out of this mess already without seeing.
What's bad about your situation is that your modules potentially CAN depend on incompatible third party libraries. You mention they use semantic versioning, but this only covers upwards compatibility, like "minor version increases if a new feature gets added in a compatible way to the older version". Which means that this newer version is not backwards compatible!
Assume module A is using version 1.0.7 of a library, and module B is using version 1.2.5. That library got a new method added to a class in version 1.2, and module B is using that method. Can module B run with the class version 1.0.7 of module A? Of course not. You want both modules to run with the highest compatible version for both modules, 1.2.5.
How to get this? Use only one Composer autoloader and only one central dependency definition.
If you could create a composer.json file that contains the dependencies for all the modules A, B and C, and each module states it's dependencies on other libraries, Composer would collect all these libraries, calculate the "best" usable version from it, and create an autoloader that would unambiguously load these libraries only.
Added benefit: Only one version per library without duplicates. Only one autoloader object with global knowledge about all available classes (which could optimize autoloading a bit).
And you are half way there. Each of your modules must already have a local composer.json which states the version requirements. Do add a definition for autoloading that module itself, and give it a name. You can then reference that name in the central composer.json (you'd probably need to add the repositories if they are private), and you are almost done. Maybe there is some fiddling with paths if you really need these modules in a defined path.
But that's about it.
And then you solve one other thing: What if module A needs some small part of module B? With Composer you can state that dependency together with all the libraries, and even if you forget to install module B, Composer would do it for you, or remind you about it.
Related
We have developed 2 WordPress plugins which are using same composer package.
Depending on plugin versions the package included in plugin may be changed, we constantly add new functionality to our package.
The problem is that for example plugin A have version 1.0.0 of package, and plugin B have version 1.0.1, WordPress loads only one package , from plugin which loaded first, so if plugin A loaded first then plugin B will use version 1.0.0 package.
We are including autoload.php on both plugins.
Is it possible to do some configuration in WordPress or from composer side to make every plugin load and work with package included on his vendor folder ?
Is it possible to do some configuration in WordPress or from composer side to make every plugin load and work with package included on his vendor folder ?
In general no. The option about namespace rewriting has already been given by Chris Haas, so if you need to rely on different code behind the same global static names, you have to provide a different name for each version.
Another option is you align both plugins to rely on the same dependency version-stability. The example you give with 1.0.0 and 1.0.1 versions, the API should be compatible and it should not be an issue (if the package follows semantic versioning).
From the context of your question, it seems the dependency has not yet matured enough that this is an option practically.
Instead it should be possible to have the plugin which is currently based on 1.0.0 to use 1.0.1 as well. Then the version conflict is solved, as now it is the same version and can use the same names. This may not be an ideal solution, but could get you back to a working version fast and gives you more room to consider a better way in the future (e.g. having a build for those plugins that use scoping).
Additionally you could/should wrap all access to third-party libraries once in your own code so that such problems aren't that deep reaching (dependency issues tend to be harder to resolve). This methodology is independent to Wordpress or Composer, just a recommendation on how to interface to third-party dependencies (or even Wordpress itself[1], which you may know better).
If however what you ask here is already such a wrapper you build your own (e.g. in form of a composer package), consider to adhere to semantic versioning and stabilize its API first.
PHP has no built-in utility to create archives apart from PHAR, e.g. that you can have imports via names but to different code. This is likely also the reason why Composer does not support it and instead provides PSR-0/PSR-4, classmap and file inclusion autoload configuration.
As you have the scenario to share the same namespaces across the plugins - there is no other option actually as it runs in the same PHP process - the first one wins. You may however make your own plugins interoperable so that they can establish a loading order in their own hierarchy, e.g. to prefix the one autoloader before the other conditionally if it exists.
I'd probably go with scoping in the first place and only share build utilities between the plugins nowadays and have a nice package for each plugin afterwards. Yes, scoping can be PITA, but if you have this early on, you don't have to solve this later which is much harder.
Compare for the level of WordPress plugins WordPress Plugin: How do I avoid "tight coupling"? and Multiple Custom Metabox Help - both well dated so it is easier to not take code verbatim but to develop own conclusions.
Is there a way, within PHP, to get a list of all libraries loaded by Composer's /vendor/autoload.php file?
Example:
require_once("./composer/vendor/autoload.php");
// In some way, get a list of libraries that were loaded:
$libraries = composer_get_libraries(); // you get the idea
I can't seem to find this in the Composer docs or google...
It's easy to list the installed libraries ('composer show') but this doesn't show which ones are autoloaded in PHP. The reason I'm asking is that I just installed a new library with composer require [library name], but in PHP the library's class is not found. I'm trying to debug this problem and it would be helpful to ensure the autoload.php is actually loading this library.
Even enabling a "debugger" option that would dump the autoloader activity to a log file would be helpful.
Update: looking into the When unit test code, I see there must be a
use When\When;
before instantiating a When object.
PHP itself has no notion of a library or a package, only the functions, classes, etc, it defines.
Composer, in turn, has no notion of how the code is being used; it may register one or more autoloaders using the spl_autoload_register function, based on the configuration of the installed packages, but it can also include certain files on every page load, which might register their own autoloader, or just define classes and functions directly.
There is therefore no concept anywhere of "the autoloader loading that library" - there might be a combined PSR-4 autoloader built by Composer, plus a combined classmap autoloader, plus a bunch of custom autoloaders, none of which have any need to "remember" which package they belong to.
When you ask for the class, PHP runs all the autoloaders that are registered, and each has a chance to define the class, usually by including a file.
All of this happens in PHP code, so a debugger such as XDebug can be used to see what autoloaders are actually run, and what each does, but beware that they may be hard-to-read machine-generated code.
A class not being autoloaded suggests one of three things:
Composer has not generated the correct autoloader code. You might try running composer dump-autoload
The package contains incorrect autoload information in its composer.json. This is unlikely unless nobody else has installed it using Composer.
You have made a mistake in the way you referenced the class, for instance missing the namespace, or mistyping it in a use statement.
Background
I am developing a framework in PHP. I started by making each component separately so that it can be used independently from the framework.
After creating four libraries A, B, C and D:
A has no dependencies
B and C require A
D requires A, B and C
Now I have some issues when releasing a new version of one library, I may have to change dependencies of others and have to release new versions for them. For example: new version of A means new version of B, C and D.
I looked how other frameworks like Symfony and Laravel solved this issue. I learned that they are using the subtree feature of Git and the replace feature of Composer. It works as follows:
Each component is in a read-only repository with his own composer.json
Each component can require other components but doesn't replace them.
The framework repository uses subtrees to include all components. So no need to require them using composer. But it should require all their dependencies (since this is no longer handled by Compser).
The framework replaces all its components.
I have also noticed that
A component repository contains only the source code (no unit tests !)
Laravel made the component Contracts just to store all the interafces of all component and each component is requiring that.
Questions
Is my explanation of how Laravel and Symfony solved the issue correct ?
Do I really have to remove tests from the components repositories and put them in the framework one ?
If yes, how can someone who wants just to use a single component be sure it passes the tests regardless of the whole framework being passing the wole tests ?
Do I have to make sure that all components dependencies are compatible and require them manually in the framework composer.json ?
What is the point of having a component for Interfaces ? This could not be used standalone anyway !
Is there a better way to solve this problem ?
P.S: here are links to A, B, C and D
Now I have some issues when releasing a new version of one library, I may have to change dependencies of others and have to release new versions for them. For example: new version of A means new version of B, C and D.
You have a multi-repo approach.
Edit of A => new version of A => version bump needed in B, C and D.
I think the most important thing is to get away from using dev-master and versionize your components properly, once they are stablized and ready to be out of dev-phase. Then you might use Composers range operators (caret ^ & tilde ~) to automatically update to the latest released version in a certain major.minor version range. This helps a great deal and takes the tedious manual version updating work out of your hands.
Is my explanation of how Laravel and Symfony solved the issue correct ?
It's not correct. The underlying development concept, publication and consumption of packages work differently, then to what you described.
They use a monolithic repo development style. It's a single repository, which contains code for group of packages. The opposite of the mono-repo is many-repo approach. The alternative are git submodules.
All modules/bundles of a framework and the framework core/kernel are in one repository! For Laravel it's https://github.com/laravel/framework/tree/5.4/src/Illuminate
Each module/bundle folder contains a composer.json and the framework itself contains a composer.json
This allows to "split out" the module folders into standalone read-only repositories. Using a custom git helper, e.g. git subsplit publish like Laravel uses https://github.com/laravel/framework/blob/636020a96a082b80fa87eed07d45c74fa7a4ba70/build/illuminate-split-full.sh or splitsh https://github.com/splitsh/lite, like Symfony uses
The development happens in the main repo.
Finally, from the user/consumer perspective (in the composer.json of your CMS/app whatever), you simply require a module/bundle from the "standalone read-only repository" source. This is many-repo, because your app depends on many repositories.
When you update a dependency using Composer, then Composer replaces your packages with a newer version.
Do I really have to remove tests from the components repositories and put them in the framework one ?
No. You could also leave the tests in the /moduleA/tests folder and adjust your unit test collector.
If yes, how can someone who wants just to use a single component be sure it passes the tests regardless of the whole framework being passing the wole tests ?
Two things. The subject under test is:
(a) the component, which is ideally independently testable and
(b) the framework, which consumes many components and tests functionalities, which rely on functions from multiple components (e.g. a core/kernel). You could also split a kernel out, once it stabilizes and is testable independently. (e.g. your component D)
Do I have to make sure that all components dependencies are compatible and require them manually in the framework composer.json ?
The monorepo developer perspective:
The developer/maintainer of a framework can only release a new version, when all unit-tests of all components and all unit-tests of the framework itself pass, right? Then he can start a subtree split and automatically versionize the new components.
The application developer perspective:
Yes. As the user of components of a monorep you are simply consuming standaloen dependencies (from the read-only repos). That means you have to maintain the versions of the components you require in your composer.json manually or automatically.
What is the point of having a component for Interfaces ? This could not be used standalone anyway !
Good question!
Maybe the developers want to do things differently and "keep things sorted"
Or, they have a bad optimization idea on their minds:
One could argue that interfaces are only development contracts.
When all components are written against interfaces you could simply pull the plug on them, after testing and before doing a production release.
In other words: you could leave the interfaces repository away
and run an interface removal, when you are releasing for production.
Leaving the interfaces repo away would lead to "interfaces X not found" fatal errors. Then you run an "optimizer pass" over the rest of the classes and remove all "implements interfaceX" strings. Less files to include. Less code to parse. Less IO. And now i will probably be killed in the comment section by suggesting this :) And no, Laravel or Symfony are not doing this.
Is there a better way to solve this problem?
I'd suggest to do the following: for a project with <5 components, use multi-repo. If >5 components, go monorepo.
In general there are not so many options to solve this:
git submodules
mono-repo
multi-repo
Each of the approaches has pro's and con's.:
Updating git submodules a.k.a. git version bumping and submodule updating leads to git madness, because it'll be constantly broken. and git madness leads to the dark side. :)
Mono-repo is easy to maintain and easy to publish. It gives you easy maintainace for the developer and multi-repo for the consumer. You can replace/rename/refactor across all modules/components at once.
Many-repo is hard to maintain, when you have a large number of components.
See also:
https://www.tomasvotruba.cz/blog/2017/01/31/how-monolithic-repository-in-open-source-saved-my-laziness/
DrupalCon New Orleans 2016: The Symfony Monolith Repository
Our current development setup uses a single Subversion repository containing multiple projects, each with branches, tags, and trunk. We then use a "sparse checkout" to select the projects, and branches of those projects, to work with.
The result is that the directory structure of a working copy matches that of the repository, including branch information, and we never use svn switch. (This style of working will probably be familiar to anyone who uses SVN, but may be surprising to those who don't.)
We are thinking of using Composer to manage both external and internal dependencies, but I'm not sure how this can work with the sparse checkout style of working.
I would like some way of using a directory within the existing checkout to satisfy a dependency, rather than each "root project" needing a separate copy.
For example:
sites/Foo/trunk
depends on lib Aaa, so references lib/Aaa/trunk
depends on lib Bbb 1.5.*, so references lib/Bbb/branches/release-1.5
sites/Bar/trunk
depends on lib Aaa 1.0.*, so references lib/Aaa/branches/release-1.0
depends on lib Bbb 1.5.*, so references lib/Bbb/branches/release-1.5
At present, if I edit the code in lib/Bbb/branches/release-1.5, I can test those changes on both sites, without needing to commit one and update the other.
Is there any way of using Composer to manage these dependencies?
(PS: Please don't answer with "give up on SVN, use Git, it is teh awesomez"; that is an answer to a different question.)
No - I do not believe that you can do this with Composer as standard: it expects to copy the files from whichever source (Packagist/VCS/Zips) to the local vendor folder, which is not what you want.
That said, I believe there are two potential ways you could get this working (at least in part):
Autoloader
You could try using the autoload field in the composer.json file to include the correct files into the project. You would still need to manage the checkouts of the relevant branches/versions manually (like I assume you do now), but you would be able to manage the inclusion of the internal libraries through Composer. This will probably require that your libraries are properly namespaced. The paths to the files for each namespace are relative to the root of the project, but can go below the root (via the /../ path) if required.
To be honest though, if you already have an autoloader for these files, there may not be much advantage to this solution. Relevant Docs
Composer Plugin
You could also write a composer plugin/"custom installer" that could probably manage this. This would have the advantage that you could have it manage checking out the correct parts of the sparse repository to have the correct version available, as well as doing correct wildstar version checking, but would be a much more difficult and riskier venture.
The basic process would be that you would define a new package type (e.g. 'internal-svn-package'). You would create the plugin as an external library that gets installed normally via Composer, which declares (via it's composer.json) that it handles this new type of package. Your custom logic would then be used for any packages that are listed with this custom type. I'm not sure how much of the internal composer logic for SVN checkouts you would be able to reuse however. Relevant Docs
I was wondering if it's a good idea to use PHP's namespace / autoloading to version my code. The file system would be something like /Class/<Package>/<version#>/<ClassName>, and a PSR4 autoloader maps new <OurVendorName>\<Package>\<version#>\<ClassName>() to the appropriate directory. We also have a /vendor/ directory for other vendors' packages we use.
My thoughts are to have a bunch of packages that run specific groups of logic, and a Slim router would use only the API package. The API package would essentially be a bunch of controllers pulling up the appropriate versions of the other packages and running them.
Doing so would allow the team to work independently on logic packages and only release new versions when backwards incompatible changes are being made. Other packages could migrate to new versions of their dependencies at their own pace without breaking anything. When we deprecate versions of packages, we can make the base object log the call stack upon instantiation to make refactoring easier to trace. Our API would also be versioned, so specific Slim routes could migrate to different versions of the API at their own pace.
I don't really see many flaws with this approach, but one possibility I considered was that it would be too much overhead. Between the filesystem holding multiple versions of every package and classes instantiating multiple versions of the same class would there be a potential performance hit? Also, would this level of granularity cause more headaches than it's worth? This is a project that is evolving very rapidly, so we need a system where we can quickly update our code without breaking anything.