There is a main application, let's call it APP.
APP has several dependencies (including open source projects and proprietary libraries).
There are multiple clients that use their own instance of APP (on different domains that I manage). Some of these clients use a slightly adjusted version of APP. I implemented this by creating a specific module (let's call it SM) for each client that I just add to their instance of APP (so that I don't change any of the code from APP).
Currently, I've implemented this as follows:
Develop APP locally, use Composer to update its dependencies (composer update), push APP on central repo
For each regular client, pull APP from central repo and install the Composer dependencies (composer install)
For clients with specific implementation, create a new SM (specific module), that has the following composer.json file:
...
"require": {
"APP": "X.X.X"
}
...
Then apply the same steps as before for this SM (composer update locally, PUSH to central repo, PULL from central repo, composer install).
Everything is fine, except for two issues that I'd like to overcome:
composer.lock from APP will be ignored by SM (since APP is loaded as a library in the vendor/ folder, and composer ignores the composer.lock files of libraries); this is not good at all, as I will not be confident that the specific clients will use the exact same libraries as APP.
Each time I fix a bug or implement a new feature in APP (and this happens frequently - a few times a day), apart from the steps that I perform for the regular clients, I also need to rebuild the SMs (since one of their libraries - APP - was updated to a new version that I need to use). This is an overhead since most of the changes that I perform are inside APP (and not SM). So, if it was the other way (APP having SM as a dependency), it would have been working faster (since I wouldn't need to composer update on each SM).
Are there any known workflows or best practices that cover this scenario in order to mitigate the two issues above or at least to decrease the complexity of the upgrade/deployment process?
Please note that most of the steps above are already automated, so my question is not about the automation part, but the complexity of this architecture
I implemented this by creating a specific module (let's call it SM) for each client that I just add to their instance of APP
For clients with specific implementation, create a new SM (specific module), that has the following composer.json file:
It's an application with a client specific module (next to other dependencies).
The application has the module as dependency (APP having SM as a dependency).
And not: the module pulls the application as it's vendor dependency in.
This will only cause extra steps to take during the development phase (your issue 2).
I would suggest to refactor the application and it's modules until you get the following folder structure:
|-application #< the application has dependencies
|-src
|-tests
|-vendor
|-framework #< maybe your application is framework based
|-libs #< more dependencies
|-... #< other modules
|-sm #< the client specific module
This allows to pull in dependencies, which extend "the application" for client-specific needs.
This overcomes your issue 1, because APP is the main repository and contains the lock file. It's essential to lock the versions, so that all developers are bound to the same versions and also for packaging exactly the same set of versions.
So, if it was the other way (APP having SM as a dependency), it would have been working faster (since I wouldn't need to composer update on each SM).
Yes! The need to rebuild the module, each time you change APP would vanish, if you start to "develop inside APP" with module dependencies.
And for multiple clients, simply use multiple application repos, which have a custom set of requirements. 10 clients, 10 application repos, 10 composer.json files. Run composer install no-dev then pre-package each repo and place zip into downloads. Done.
You can use a "container" or "packaging" project here, where the composer.json of each project would include the app and the specific modules. You might utilize the caret or tilde operator to specify a version range for the app ("vendor/app": "^1.2.3") and then simply update and repackage, after a new version of the application is released. This approach should work with the composer autoloading, because the application will remain inside the vendor folder, too. Only a little wrapper is needed, to set the composer autoloader up and switch over to your application.
Or, if the application is really modular. Just package the main application and provide the client-specific modules as extra downloads. With this approach upgrades will have multiple download steps: upgrade app, upgrade modules. Think of it as "wordpress-style" updates/upgrades.
You might reduce the complexity of the upgrade/deployment process further by dropping the composer install --no-dev part on the client machine
by building "client-specific application archives" on the developer machine.
These are basically the "--no-dev" package of the application with all it's dependencies, including the client-specific module(s) = pre-packaged.
Like, Application-v1.2.3-WithModuleAForClientA-v3.2.1.zip.
On the dev machine: composer install --no-dev --optimize-autoloader + zip.
To install or upgrade simply download to the client, extract, execute the upgrade script. Done.
Related
I am working on Laravel webapp right now and kept vendor directory out of git (version control) so far and every time for fresh install I used to have composer install command added to automated script and everything was fine.
Now just 2 days back I added added laravelcollective (https://laravelcollective.com/) to my project for helping me with forms and html in blade templates. Now somehow one of the dependency requires me to generate GIT private token to install it and that is pain as it would hurt my automation. I can still hack it by calling the url and scrapping html to read token and stuff like that but I don't like it. And then I thought is it good idea to keep vendor directory out of SVN/GIT? Isn't source code for a product contain all dependencies within itself? I am not talking about stuffing JRE in the installer but when it comes to libraries of a product in native language.
I would like to hear more about it on industry standards or best practices on this.
P.S:
This question is much generic and not just limited to laravel or even php for the matter.
Now somehow one of the dependency requires me to generate GIT private token to install it and that is pain as it would hurt my automation.
You're just running into Github's rate limits for package downloads for anonymous users. No reason you can't automate this. Generate a Github token (you only need to do it once - they get very high rate limits for authenticated requests), then have your automation use that token like so:
composer config -g github-oauth.github.com <oauthtoken>
https://getcomposer.org/doc/articles/troubleshooting.md#api-rate-limit-and-oauth-tokens
Well, for production environment you usually run a build process first in your CI software. If 'composer install' fails during the build – application won't be deployed to production environment, so you are safe.
Yes, most (99%+) people keep 'vendor' folder out of the repo because it's a third-party code, it's not yours. You may not even have rights to host it in your repo.
If you want to be sure that your production version will have all the dependencies in order, the way you had them during CI, and will always release – you could build Docker images and ship them to production. Then, everything comes prepackaged.
I've created post-receive hook for git repository which checkouts commit to web folder /var/www/myproject. Since I'm not committing dependencies (framework files), when checked out I have to install dependencies and init the framework. I see two approaches:
1) When checked out run the following commands from the project directory:
composer install
php init --env=Development --overwrite=n
#other commands to setup db connection, credentials etc.
2) Install dependencies one level up of checked out project /var/www and have links one level up. But it seems that with this approach I'll still need to run php init.
What's the common approach for such deployment?
An important feature of the advanced app is having separate backend and frontend apps. So if you check out your repo to /var/www/myproject, two important directories will be created: /var/www/myproject/frontend/web and /var/www/myproject/backend/web. You usually configure your web server to have these two directories as web roots for two different domains (like example.com and admin.example.com).
Yii2 advanced app introduces the concept of "environments". An environment is basically a set of config files, that, among other things, include db credentials. So if you don't mind having credentials in your repo, push-deployment is possible.
So if you plan to have multiple servers with multiple configs, you just create an environment for each server and use it to deploy.
So here are the steps you need to take after checking out the repo.
Install composer dependencies: composer install
Pretty straightforward. Keep in mind that fxp/composer-asset-plugin needs to be installed globally for composer to fetch bower and npm dependencies.
Check out a certain environment for current setup: init --env=your_server_environment --overwrite=All
You should overwrite everything. If you have updated some parameter in your environment, it will be applied to the current setup.
Migrate the database: yii migrate --interactive=0
That's all there is to it.
I am going to rewrite this question to be more clear. I have the following application structure:
applications/
api/
public/
composer.json
frontend/
public/
composer.json
backend/
public/
composer.json
common/
vendor/
... composer libraries here
How can I make that every single application's composer install gets installed into common/vendor, so that way I can have the most up to date version of the library in wherever is used with just one composer update; while at the same time only load the libraries that are in the composer.json file of each application. So, when I include vendor/autoload.php, only the needed libraries are loaded.
EDIT: Edited the whole question. Please reread
You have to create one bigger meta project that requires the API package, the frontend and the backend. You can define which directory should be used for placing dependencies for this meta project, and should be able to define for the special packages API, frontend and backend that they should go into their respective directories and not the common folder.
Updating that meta package will have to check more dependencies, but it is guaranteed that you either get the newest possible versions that conform to your version constraints (which may NOT install the newest version available if one of your packages requires a lower version). That way you would avoid installing dependencies that are not allowed for one of the projects, and you would be immediately notified if you attempt to install conflicting versions.
Note that I wouldn't recommend this at all. I would write a script, placing it at applications/updatecomposer.sh and add all the commands necessary to update each project individually. You gain all the flexibility that Composer is about, because essentially you want to have the central library installation of PEAR back. This central installation and the resulting inability to update any of the PEAR packages without risking to break something is one of the reasons that PEAR is considered dead.
Or think about any pre-Composer originating framework like Zend Framework 1. Having this installed in a central point that every application is using will effectively prevent you from ever updating it, if you are not prepared to also deal with incompatibilities in ALL your applications at the same time. Just an example: Updating from any ZF 1.11 to ZF 1.12 (the currently maintained up to date version) is a potentially backwards-incompatible change, because at least one abstract class (dealing with REST interfaces) got new abstract methods that have to be implemented.
When putting together a PHP project with composer, on installation / deployment, composer would fetch the dependencies usually from their original sources.
This could lead to problems when deploying, when a source (maybe only temporarily) becomes unavailable.
Is there any included mechanism to keep at least the current, stable versions of the dependencies some where to be always able to deploy the current version to other instances?
Right now there is no one click solution for this, but I plan to work on something soon that will give you more reliability.
Broker looks like a tool which could serve as a proxy to keep files, and is now integrated into Satis (see https://github.com/researchgate/broker)
broker is a full repository proxy for composer. It takes a composer file, downloads all
requirements and all dependencies, and then publishes a new repository with all these
packages. Instead of packagist or satis, all packages, including dist and source files will > be served directly by broker.
Note: this project is not actively maintained anymore. Since satis supports a similar
functionality now, you should use satis instead.
We created a ZF2 project with skeleton app and it works fine for a simple test application. Now we are working on a real project. My question is what we should store in the repository (SVN), the whole project structure or just the new source code? ZF2 comes with a vendor directory which is almost 31MB in size (which has the ZF libraries). Should we store the whole vendor folder in SVN?
This is the first time we are using PHP and ZF so are not clear in how we will deliver the complete project to production from SVN. Also what is the build process if at all exists. Any clues/links to "ZF2 project packaging" is appreciated.
No, don't include dependencies in your repository! Putting your dependencies under version control doesn't do any good, it just blows up your repo for no reason.
You want to add the skeleton to your repository and your own library but definitely not the framework or any other dependencies.
The way to go is to use composer for dependency installation and some kind of build tool like Phing to automate installation of your project.
See the relevant chapter on phptherightway for more information on how to build your application.
The most simple build process doesn't even need a build tool
checkout your project from SVN/git
run php composer.phar install to install the needed dependencies (defined in your composer.json)
But most probably you want to do some more stuff like setup up the environment, deleting some files, etc.
A word about ZF packages. They're not available from packagist but you can install them with composer anyways. You just have to add the dedicated repository to your composer.json as described here: http://framework.zend.com/downloads/composer