I was wondering if it's a good idea to use PHP's namespace / autoloading to version my code. The file system would be something like /Class/<Package>/<version#>/<ClassName>, and a PSR4 autoloader maps new <OurVendorName>\<Package>\<version#>\<ClassName>() to the appropriate directory. We also have a /vendor/ directory for other vendors' packages we use.
My thoughts are to have a bunch of packages that run specific groups of logic, and a Slim router would use only the API package. The API package would essentially be a bunch of controllers pulling up the appropriate versions of the other packages and running them.
Doing so would allow the team to work independently on logic packages and only release new versions when backwards incompatible changes are being made. Other packages could migrate to new versions of their dependencies at their own pace without breaking anything. When we deprecate versions of packages, we can make the base object log the call stack upon instantiation to make refactoring easier to trace. Our API would also be versioned, so specific Slim routes could migrate to different versions of the API at their own pace.
I don't really see many flaws with this approach, but one possibility I considered was that it would be too much overhead. Between the filesystem holding multiple versions of every package and classes instantiating multiple versions of the same class would there be a potential performance hit? Also, would this level of granularity cause more headaches than it's worth? This is a project that is evolving very rapidly, so we need a system where we can quickly update our code without breaking anything.
Related
This question already has answers here:
Should composer.lock be committed to version control?
(10 answers)
Closed 1 year ago.
In an answer to How to get the exact version of included packages in my private repository, I made the statement that composer.lock should not be put under version control for a package. When installing a package, this file is not used after all.
I've peeked in a set of popular repositories, and most of them do not contain a lock file (like Symfony, Laravel, Guzzle, Monolog). On the other hand, the Doctrine repositories contain that file, and I'd like to know if there is any good reason to do so, or to omit the file.
Side note: this is about packages, libraries, however you want to call them. For applications, this is a different thing, as you want to stick to specific versions of each dependency when working together in a team or deploying to other systems. How to handle this different situation is covered in Should composer.lock be committed to version control?, but it does not contain too much arguments for my use case
Since the file is not used in any useful manner when installing the package, as a functionality of the library itself for the end user, it's at least not relevant to the user of the library.
Then the reasoning becomes whether it's useful to the developers of the library to have a locked set of dependencies that they need to perform development tasks, such as specific versions of testing frameworks etc. In those cases the argument can be that the composer.json file fulfils the same role as in a regular application - it locks down the dependencies to those that we know work.
However, there's a caveat here - when developing a library you really want the use case to be the same as what the user of the library experiences when he/she installs it. Considering this it usually makes more sense to lock down the explicit version in composer.json instead of relying on the lock file to provide the same functionality. That makes any CI solution install the correct set (the same as what a user would get) of dependencies when for running the tests. You can however make that process update the lock file locally before running the tests to have multiple test cases - one with the locked dependencies and one with the most recent versions (as the user would get).
Doctrine has made the decision that lock files should be committed for their own reasons, which are perfectly valid - in effect they come down to the tools used for their development workflows:
All Doctrine projects must commit the composer.lock file. Tools like phpstan and phpcs are quite fragile on patch releases and we don't want builds to start failing without us having made any changes to our own code. Whenever a dependency needs to be upgraded, the composer.lock file should be updated locally and the change submitted via pull request.
An argument can be made for both cases; it'll be up to the preferences of the project itself and its developers. I'm leaning towards it not being committed, since that more closely replicates what a user would experience when installing the library. However, there would still be local lock files present for each developer, meaning that what each developer has on their own computer when developing the library could differ. Committing the lock file would make that more similar across the board for all developers, but would require extra care to replicate the experience for users (and then, we're back to our original arguments again..).
My post was not about pure libraries, but a kind of module that has many dependencies on other libraries. The module is part of various applications. If, for example, I run a composer install without composer.lock when deploying my application, I might roll out stands that I have not tested. Therefore I fix the dependencies of my module release on a concrete status and of course commit the composer.lock. Therefore, the comparison to frameworks like Symfony is a bit lagging in my opinion, because nothing is deployed here.
Background
I am developing a framework in PHP. I started by making each component separately so that it can be used independently from the framework.
After creating four libraries A, B, C and D:
A has no dependencies
B and C require A
D requires A, B and C
Now I have some issues when releasing a new version of one library, I may have to change dependencies of others and have to release new versions for them. For example: new version of A means new version of B, C and D.
I looked how other frameworks like Symfony and Laravel solved this issue. I learned that they are using the subtree feature of Git and the replace feature of Composer. It works as follows:
Each component is in a read-only repository with his own composer.json
Each component can require other components but doesn't replace them.
The framework repository uses subtrees to include all components. So no need to require them using composer. But it should require all their dependencies (since this is no longer handled by Compser).
The framework replaces all its components.
I have also noticed that
A component repository contains only the source code (no unit tests !)
Laravel made the component Contracts just to store all the interafces of all component and each component is requiring that.
Questions
Is my explanation of how Laravel and Symfony solved the issue correct ?
Do I really have to remove tests from the components repositories and put them in the framework one ?
If yes, how can someone who wants just to use a single component be sure it passes the tests regardless of the whole framework being passing the wole tests ?
Do I have to make sure that all components dependencies are compatible and require them manually in the framework composer.json ?
What is the point of having a component for Interfaces ? This could not be used standalone anyway !
Is there a better way to solve this problem ?
P.S: here are links to A, B, C and D
Now I have some issues when releasing a new version of one library, I may have to change dependencies of others and have to release new versions for them. For example: new version of A means new version of B, C and D.
You have a multi-repo approach.
Edit of A => new version of A => version bump needed in B, C and D.
I think the most important thing is to get away from using dev-master and versionize your components properly, once they are stablized and ready to be out of dev-phase. Then you might use Composers range operators (caret ^ & tilde ~) to automatically update to the latest released version in a certain major.minor version range. This helps a great deal and takes the tedious manual version updating work out of your hands.
Is my explanation of how Laravel and Symfony solved the issue correct ?
It's not correct. The underlying development concept, publication and consumption of packages work differently, then to what you described.
They use a monolithic repo development style. It's a single repository, which contains code for group of packages. The opposite of the mono-repo is many-repo approach. The alternative are git submodules.
All modules/bundles of a framework and the framework core/kernel are in one repository! For Laravel it's https://github.com/laravel/framework/tree/5.4/src/Illuminate
Each module/bundle folder contains a composer.json and the framework itself contains a composer.json
This allows to "split out" the module folders into standalone read-only repositories. Using a custom git helper, e.g. git subsplit publish like Laravel uses https://github.com/laravel/framework/blob/636020a96a082b80fa87eed07d45c74fa7a4ba70/build/illuminate-split-full.sh or splitsh https://github.com/splitsh/lite, like Symfony uses
The development happens in the main repo.
Finally, from the user/consumer perspective (in the composer.json of your CMS/app whatever), you simply require a module/bundle from the "standalone read-only repository" source. This is many-repo, because your app depends on many repositories.
When you update a dependency using Composer, then Composer replaces your packages with a newer version.
Do I really have to remove tests from the components repositories and put them in the framework one ?
No. You could also leave the tests in the /moduleA/tests folder and adjust your unit test collector.
If yes, how can someone who wants just to use a single component be sure it passes the tests regardless of the whole framework being passing the wole tests ?
Two things. The subject under test is:
(a) the component, which is ideally independently testable and
(b) the framework, which consumes many components and tests functionalities, which rely on functions from multiple components (e.g. a core/kernel). You could also split a kernel out, once it stabilizes and is testable independently. (e.g. your component D)
Do I have to make sure that all components dependencies are compatible and require them manually in the framework composer.json ?
The monorepo developer perspective:
The developer/maintainer of a framework can only release a new version, when all unit-tests of all components and all unit-tests of the framework itself pass, right? Then he can start a subtree split and automatically versionize the new components.
The application developer perspective:
Yes. As the user of components of a monorep you are simply consuming standaloen dependencies (from the read-only repos). That means you have to maintain the versions of the components you require in your composer.json manually or automatically.
What is the point of having a component for Interfaces ? This could not be used standalone anyway !
Good question!
Maybe the developers want to do things differently and "keep things sorted"
Or, they have a bad optimization idea on their minds:
One could argue that interfaces are only development contracts.
When all components are written against interfaces you could simply pull the plug on them, after testing and before doing a production release.
In other words: you could leave the interfaces repository away
and run an interface removal, when you are releasing for production.
Leaving the interfaces repo away would lead to "interfaces X not found" fatal errors. Then you run an "optimizer pass" over the rest of the classes and remove all "implements interfaceX" strings. Less files to include. Less code to parse. Less IO. And now i will probably be killed in the comment section by suggesting this :) And no, Laravel or Symfony are not doing this.
Is there a better way to solve this problem?
I'd suggest to do the following: for a project with <5 components, use multi-repo. If >5 components, go monorepo.
In general there are not so many options to solve this:
git submodules
mono-repo
multi-repo
Each of the approaches has pro's and con's.:
Updating git submodules a.k.a. git version bumping and submodule updating leads to git madness, because it'll be constantly broken. and git madness leads to the dark side. :)
Mono-repo is easy to maintain and easy to publish. It gives you easy maintainace for the developer and multi-repo for the consumer. You can replace/rename/refactor across all modules/components at once.
Many-repo is hard to maintain, when you have a large number of components.
See also:
https://www.tomasvotruba.cz/blog/2017/01/31/how-monolithic-repository-in-open-source-saved-my-laziness/
DrupalCon New Orleans 2016: The Symfony Monolith Repository
I'm using Laravel to create a versioned JSON API that I'll then access with AngularJS to populate my pages. Currently I'm returning the JSON using versioned controllers (eventually I plan to implement dependency injection & versioned repositories instead).
I've attempting to namespace my controllers in the format App\Controllers\API\v1.0 but when I visit localhost:800/api/1.0/companies in my browser, where the JSON is returned, I am notified that Class App\Controllers\API\v1.0\CompaniesController does not exist
. I'm guessing this is because the . is being interpreted like one of the backslashes? I can make the application function by changing the namespace to App\Controllers\API\v1 or App\Controllers\API\v1\v0 , but the former doesn't allow for proper versioning and the latter seems a bit ... inelegant.
What conventions should I use for my namespaces to adequately version my API? Any advice would be appreciated! :)
APIs should only use major versions externally. Following best practice semantic versioning, major versions change when you introduce backwards-incompatible changes to a project.
If you're just adding features, or modifying existing ones in backwards-compatible ways, then you just do it and your existing consumers are not affected (though, they can then make use of the new changes if they want). Your API can change versions from 1.0.0 to 1.1.0 internally, but the version as exposed to your consumers is still just "v1".
If you're just patching bugs, the same applies. Change from 1.0.1 to 1.0.2 internally, but the API should stay at "v1".
Now, if you rename/remove resources, or do some other drastic changes that will break existing clients, your internal version might go from 1.2.0 to 2.0.0, because breaking backwards compatibility requires a major version change. Because of this, the new major version has to be exposed to your API's consumers as "v2".
So, in keeping with this, both your namespaces and your URLs should reflect only the major version (e.g. "v1"), and you should make sure that you never break backwards-compatibility within a major version.
Some background first
Our company, a small startup with only four developers, is starting the refactoring of our products into reusable modules to simplify the development process, increase productivity and, along the way, we would like to introduce unit tests where fits.
As usual on a small startup, we can't afford wasting too much development time but, as we see, this is extremely important for the success of our business on a medium and long term.
Currently, we have two end-user products. Both are Laravel (PHP) applications built on top of our own internal business layer, mainly composed of webservices, restful apis and a huge database.
This business layer provides most of the data for these products, but each of them makes completely different use of it. We plan to build other products on the near future besides maintaining and improving those two that are almost finished.
For that to happen, we intend to abstract the common logic of those (and the future) products into reusable and decoupled modules. The obvious choice seems to be Composer, even with our little knowledge about it.
Now to the real question
I would like to ask other opinions on how to develop internal packages on a test driven fashion. Should each module be a composer package with it's own unit tests and requiring it's dependencies, or should we build a single package with each module namespaced?
To clarify a bit, we would like to have, for instance, a CurlWrapper module and that would be required on our InternalWebserviceAPI module (and a few others).
I personally like the idea of having completely separate packages for each module and declaring dependencies on composer.json, which would mentally enforce decoupling and would allow us to publish some of those packages as opensource someday. It also may simplify breaking changes on those modules because we could freeze it's version on the dependents that will need to be updated.
Although, I also think this separation may add a lot of complexity and may be harder to maintain and test, since each module would need to be a project on it's own and we don't have all that man power to keep track of so many small projects.
Is really Composer the ideal solution for our problem? If so, which would recommend: single package or multiple packages?
Edit 1:
I would like to point out that most of these modules are going to be:
Libraries (ie obtaining an ID from an youtube URL or converting dates to "x seconds ago")
Wrappers (like a chainable CURL wrapper)
Facades (of our multiple webservices, those require the other two kinds)
Yes, composer is the way to go and I recommend you to use single packages.
You don't know when you need these modules. It is better to create many single packages and be able to include them all (or a single one), than creating big packages and need to put more time in breaking a package in multiple ones when you need some classes from it.
For instance, see the Symfony2 project. That is a lot of components which are all required for the full-stack Symfony2 framework, but you can also use some components in your own project (like Drupal8 is doing). Moreover, Symfony2 gets more and more packages, it seems so usefull to have small packages that people put time in breaking some big packages in pieces.
An alternative to using single packages: use separate composer.json files for each subproject.
This has the benefit of letting you keep all of your libraries in the same repository. As you refactor the code, you can also partition autoload and dependencies by sub-library.
If you get to the point that you want to spin the library off into its own versioned package, you could go the final step and check it into its own repository.
In the Symfony 2 docs it's said:
A bundle should not embed third-party libraries written in JavaScript, CSS, or any other language.
Then how should I do that? I want to install Twitter Bootstrap, DataTables, and many other things as dependencies using Composer. But the only way I can think of is creating a bundle and embedding them.
What is the correct way to do this?
You should use Bower by Twitter. It is a package manager for HTML, CSS and Javascript. It was created to address this very issue you are having.
EDIT:
As of now, there are very good package managers for JS Libraries such as Bower, Jam or Component.
Versioning system
Semantic Versioning - Composer advises to use the semantic versioning system. It uses a X.Y.Z setting, in which X is the major version, Y is the minor version and Z is the patch version. Y and Z should always be backwards compatible while X reflects changes in code which MIGHT break backwards compatibility.
Embeding
Embeding should be read as copy and pasting the code (and binary) as part of your library, rather then requiring it as a third party (vendor) package/bundle. Its like including query.js in a resource folder or copying and pasting propel code to a folder inside your bundle.
Why not embed 3rd party libs
A bundle should not embed third-party libraries written in JavaScript, CSS, or any other language.
This statement comes from a best practice point of view. Embeding (as in copy/paste) third party libraries of any kind (PHP libs especially) is usually not a good idea. For instance, imagine that BUNDLE A uses LIBRARY FOO v1.4.1, and BUNDLE B also uses LIBRARY FOO but with a different version v1.5.2. If any of the BUNDLES (A or B) embeds FOO lib, they might (most probably will) become incompatible. For instance, php classes and functions cannot be redeclared. Any of the bundles can, of course, use workarounds to mitigate this problem, such as namespacing their version of FOO or autoloading rules, but this can rise other problems as well besides surely increasing memory usage as there are 2 versions of the same thing parsed by PHP.
If a PHP package does not follow this best practice, the errors that arise are usually easy to spot (with error: cannot redeclcare function blablabla). With Javascript Libraries, however, that is not true. You can redeclare functions (as they are object properties). So if now FOO is a JS Lib instead, and BUNDLE A and B embed them into their libraries, when they are included, strange problems can arise. For instance, a function can be redeclared that lacks a crucial functionality for one of the bundles and break it.
Symfony is a PHP framework.
It deals with PHP libraries/bundles. Symfony advises to require a library as dependency instead of embedding it since it uses Composer as a Package manager, which takes care of downloading and loading the require packages. As far as I remember, when 2 bundles/packages use the same library, if they have different version requirements, the most actual is used, unless its backwards incompatible. Composer then reports a conflict that you have to manually resolve.
However... There is no way to handle javascript libraries properly. That's because Composer is a package for PHP libraries. You can workaround this in two ways I can think of: (there are probably more and best ways to handle this, I just thought of these two, read them as suggestions)
Create a PHP wrapper around the javascript library and including it (although this potentially creates the same problem if another bundle decides to do the same thing but giving the package a different name)
Creating a bundle which requires the javascript library as a third party dependency through composer. Since the javascript library won't probably have a composer.json file in it's repository (sometimes they live as a standalone minified file), this can be accomplished by creating a custom composer installer, forking the javascript repository (in gitHub for instance) adding a composer.json to it, etc... However, you will need to constantly maintain and upgrade the said library, which can be troublesome.
You will have to keep in mind that:
JS and CSS libs have to be exposed publicly, so that the client can access it (security considerations)
Symfony is a PHP framework and deals with server-side packages. JS/CSS are client side. This as to be taken in consideration so it works properly.
One of the main ideas behind symfony (as with other PHP Frameworks) is code reusability within and between projects. Pure Javascript Library are reusable in themselves. They are usually self contained. Besides, there is no real gain in "bundling" a JS library from the server side. You don't need any kind of bundle to achieve reusability.
My Approach
Since the composer system is so appealing, specially when deploying bundles/packages/libraries to other people, my approach to using third party javascript/css libraries was to create a dependency manager specific to JS/CSS that other packages/bundles could rely on to take care of their JS/CSS dependencies without worrying about this.
My sugestion
If you are planing to release your project to the public, namely as a symfony bundle, you should plan carefully how to approach this.
If your project is self contained (personal use or to a client, not widespread use) then this has much less relevancy since you (the programmer) have total control in what third party tools you use and include in your project. These are just best practice "suggestions" to avoid
future problems.