Configure cache on GitLab runner - install dependencies by composer only once - php

I have problem with gitlab cache config.
I use GitLab Community Edition 11.7.7, in docs we have:
https://docs.gitlab.com/ee/ci/caching/#caching-php-dependencies
So, my config:
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- vendor/
stages:
- code-style
- analysis
before_script:
- composer global require hirak/prestissimo
- composer install --ignore-platform-reqs --no-scripts
php-cs-fixer:
stage: code-style
script:
- vendor/bin/php-cs-fixer fix --dry-run --diff --verbose --config=.php_cs.dist src tests
php-cs-fixer2:
stage: analysis
script:
- vendor/bin/php-cs-fixer fix --dry-run --diff --verbose --config=.php_cs.dist src tests
Unfortunately, composer re-installs all dependencies in every job.
I would like to install all dependencies through the composer only once, how can I achieve it?

Your .gitlab-ci.yml configuration is similar to my own that run on self-hosting Gitlab at my company, I think it should works as well, So I have created an sample repo that runs pipelines based on your configuration in order to reproduce your problem.
As you can see here, I was trying to make a working pipeline by complete the configuration, For example, wrong image, missing composer executable...something like that, The final configuration is not much diffrent to yours.
image: php:7.2-alpine
cache:
key: $CI_COMMIT_REF_SLUG
paths:
- vendor/
stages:
- code-style
- analysis
before_script:
- php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
- php -r "if (hash_file('sha384', 'composer-setup.php') === 'a5c698ffe4b8e849a443b120cd5ba38043260d5c4023dbf93e1558871f1f07f58274fc6f4c93bcfd858c6bd0775cd8d1') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;"
- php composer-setup.php --install-dir=/usr/local/bin --filename=composer
- export PATH="/usr/local/bin:$PATH"
- php -r "unlink('composer-setup.php');"
- composer global require hirak/prestissimo
- composer install --ignore-platform-reqs --no-scripts
php-cs-fixer:
stage: code-style
script:
- vendor/bin/php-cs-fixer fix --dry-run --diff --verbose --config=.php_cs.dist src tests || true
allow_failure: true
php-cs-fixer2:
stage: analysis
script:
- vendor/bin/php-cs-fixer fix --dry-run --diff --verbose --config=.php_cs.dist src tests
allow_failure: true
The last job #97744833 have a clue to answer your problem.
Compare the last part of outputs from previous failed job https://gitlab.com/Gasol/php-composer-cache-on-gitlab-ci/-/jobs/358118759#L191
187 Checked all files in 0.006 seconds, 10.000 MB memory used
191 ERROR: Job failed: exit code 8
to https://gitlab.com/Gasol/php-composer-cache-on-gitlab-ci/-/jobs/358121571#L190
The cache is creating when build success.
190 Creating cache master...
191 vendor/: found 4860 matching files
192 Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/15472690/master
193 Created cache
196 Job succeeded
So, after cache created, The job on the next stage should be ok for downloading cache archive.
https://gitlab.com/Gasol/php-composer-cache-on-gitlab-ci/-/jobs/358121572#L20
18 Checking cache for master...
19 Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/15472690/master
20 Successfully extracted cache
https://gitlab.com/Gasol/php-composer-cache-on-gitlab-ci/-/jobs/358121572#L46
And there are nothing to install when cache extracted. That's the result you want.
43 $ composer install --ignore-platform-reqs --no-scripts
44 Loading composer repositories with package information
45 Installing dependencies (including require-dev) from lock file
46 Nothing to install or update
47 Generating autoload files

Related

Travis composer install issue

I want to use travis with my simple php projects. My project structure is next:
My .travis.yml
language: php
sudo: required
before_install:
- cd http
before_script:
- cd http
install:
- composer self-update
- composer-install --no-interaction
script:
- phpunit --configuration phpunit.xml
and I want to run trevis into http folder, my composer.json and phpunit are there. However, as a result of my build, I received:
How can I solve this issue and run travis correctly? Thanks

How to use the image between steps in bitbucket-pipelines

I would be grateful if someone could help me with setting up bitbucket-pipeline for my php env. What I'm trying to succeed is:
Build image in the first step
In the second step run some QA stuff like unit test, code sniffer etc.
Deploy to preprod envirement
Currently I'm stuck with re-using image from the first step. This is how my bitbucket-pipelines.yml looks like:
pipelines:
branches:
develop:
- step:
name: Build docker image
caches:
- composer
script:
- apt-get update && apt-get install -y unzip
- docker-php-ext-install mysqli pdo_mysql json sockets
- curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
- composer install
- parallel:
- step:
caches:
- composer
name: Unit tests
script:
- vendor/bin/phpunit
- step:
caches:
- composer
name: Code sniffer
script:
- composer phpcs:all
- step:
name: Deploy to preprod
script:
- echo "Deployment"
What I get here is:
bash: vendor/bin/phpunit: No such file or directory - from "Unit tests" step
and bash: composer: command not found - from "Code sniffer" step
I already tried to set docker/composer to cache, save image in the first step and import it in the second, but still not working.
One way to share stuff between different steps is to make use of Artifacts.
Basically, bitbucket spins up each step in a separate docker container. So a way to re-use material between steps is create an artifact.
The link should give you enough information.
For example:
- step: &build
caches:
- node
name: Build
script:
- npm install
- npm run build
artifacts: # defining the artifacts to be passed to each future step.
- dist/**
- step: &read-from-artifact
name: Read from the artifact saved in the previous step
script:
- echo "Read artifact (dist folder) saved in previous step"
- ls -la dist/

Cache git repositories installed with Composer from source on Travis CI (and other CI tools)?

I work on several Symfony bundles hosted on GitHub and tested automatically with Travis CI.
The longest part of the tests is the installation of the requirements by Composer.
I configured Travis CI to install packages with composer update --prefer-dist and cache the $HOME/.composer/cache directory. Tests took a total time of 18 minutes, thanks to caching:
Installing doctrine/lexer (v1.0.1)
Loading from cache
But a week ago I saw a message from Composer:
Installing doctrine/lexer (v1.0.1)
Downloading: Connecting... Failed to download doctrine/lexer from dist: Could not authenticate against github.com
I changed the configuration to composer update --prefer-source because of this. This seems to be a common practice across Symfony bundles. The tests suite took 28 minutes.
I know that I can register GitHub keys in Travis CI in order to avoid the API limit while using the Composer --prefer-dist option.
Are they some other ways to cache the dependencies? E.g by cloning the dependencies repositories in a cache?
GitHub has removed the API rate limits, composer can now be used with --prefer-dist and the zip files can be cached. Here is an example of configuration in .travis.yml:
cache:
directories:
- $HOME/.composer/cache
# …
install:
- composer update --prefer-dist
Here is the announce:
Hi Niels and Jordi,
We've spent some time digging in, considering all the options to resolve your issues, weighing your needs against infrastructure strain and availability.
I'm happy to report that our Infrastructure team believes that due to their work on our Git backend since these APIs were introduced, we're now able to drop these rate limits on the API side. We deployed those changes a couple of hours ago. Getting an archive link 1 via the API will no longer count against your hourly rate limit (authenticated or not). This should make Composer installs happy.
Let us know if you see any funny business.
Cheers,
Wynn Netherland
Platform Engineering Manager, GitHub
Source.
I tested caching of the vendor/ directory and it worked. I used tar in order to create an uncompressed archive $HOME/vendor-cache/ and configured Travis CI for this directory.
Commands have two goals:
Extract the vendor/ from cache if it's available
Put the vendor/ in the cache after the tests
Here is an example .travis.yml file:
sudo: false
language: php
cache:
directories:
- $HOME/.composer/cache
# This is where vendor/ backups will be stored
- $HOME/vendor-cache/
php:
- […]
env:
- SYMFONY_VERSION="2.7.*"
- SYMFONY_VERSION="2.8.*"
- SYMFONY_VERSION="3.0.*"
before_install:
# Create an hash corresponding to the PHP version and the dependencies
- tohash="${SYMFONY_VERSION}"
- cachefile="`echo -n "$tohash" | sha1sum | cut -d " " -f 1`.tar"
# Extract cache archive if the file exists
- if [[ -f $HOME/vendor-cache/$cachefile ]]; then tar -xf $HOME/vendor-cache/$cachefile ; fi
install:
- composer self-update
- composer update --profile --no-progress
script: php ./vendor/bin/phpunit
# Create cache archive from vendor/ directory
before_cache:
- if [[ -f $HOME/vendor-cache/$cachefile ]]; rm -fv $HOME/vendor-cache/$cachefile ; fi
- tar -cf $HOME/vendor-cache/$cachefile vendor/
Here is a fully annotated .travis.yml file with more verbose output:
sudo: false
language: php
cache:
directories:
- $HOME/.composer/cache
# This is where vendor/ backups will be stored
- $HOME/vendor-cache/
git:
depth: 5
php:
- […]
env:
- SYMFONY_VERSION="2.7.*"
- SYMFONY_VERSION="2.8.*"
- SYMFONY_VERSION="3.0.*"
before_install:
# Create an hash corresponding to the PHP version and the dependencies
- echo "Vendor cache content:" ; ls -lh $HOME/vendor-cache/
- echo "Values used for hash:"
- tohash="${SYMFONY_VERSION}"
- echo "$tohash"
- cachefile="`echo -n "$tohash" | sha1sum | cut -d " " -f 1`.tar"
- echo "cachefile = ${cachefile}"
# Extract cache archive if the file exists
- if [[ -f $HOME/vendor-cache/$cachefile ]]; then echo "Extract cache archive"; tar -xf $HOME/vendor-cache/$cachefile ; echo "Done" ; else echo "Cache archive does not exist" ; fi
- if [[ -d vendor/ ]]; then echo "Size of vendor directory extracted from cache:" ; du -hs vendor/; else echo "vendor/ directory does not exist"; fi
install:
- composer self-update
- composer update --profile --no-progress
script: php ./vendor/bin/phpunit
# Create cache archive from vendor/ directory
before_cache:
- if [[ -f $HOME/vendor-cache/$cachefile ]]; then echo "Delete previous cache archive"; rm -fv $HOME/vendor-cache/$cachefile ; echo "Done" ; else echo "No cache archive to delete" ; fi
- echo "Create cache archive" ; tar -cf $HOME/vendor-cache/$cachefile vendor/ ; echo "Done"
- echo "Size of cache archive:" ; ls -lh $HOME/vendor-cache/$cachefile
By using this method, composer update took 30 seconds, instead of about 2 minutes without cache (note that the comparison is not perfect, some other changes are applied but that's still a good estimate).
It's preferable to limit the number of parallel builds the first time you launch builds, so that caches won't suffer from a race condition.

Travis configuration for composer packages

While working on a laravel 5.1+ package I have this need to run automated tests through travis-ci.org. The difference with regular automated tests is the requirement to include this package into a framework and set specific configuration options to run the tests.
So the requirement would be:
install laravel
add my package as dependency
set some travis specific configurations like the travis database access
run migrations of laravel
run migrations specific for package or run an artisan command
run package specific unit tests
I searched everywhere; asked on laravel forums, asked in a travis community chat and saw this topic being closed as too localized (although an answer would have certainly been helpful now). I'm hoping my question is fit to remain open.
At this time I have the following configuration:
language: php
php:
- 5.5
- 5.6
- hhvm
addons:
hosts:
- system.hyn.me
- tenant.hyn.me
before_install:
- sudo composer self-update
install:
- composer create-project laravel/laravel
- cd ./laravel
- composer require hyn-me/multi-tenant ~0.1.0
- composer update
before_script:
- cp .env.travis .env
- export APP_ENV="testing"
- php artisan migrate -q -n --path ./vendor/hyn-me/multi-tenant/src/migrations
- cd ./vendor/hyn-me/multi-tenant
script: phpunit
Yet my knowledge of travis (thus far) is limited and before I send in an unneeded number of commits to fix my problems I'd rather have your opinion on what would be a good method to test integration into a framework.
Ps. this concerns the package hyn/multi-tenant.
Advise on how to keep this question as generic as possible would be helpful. I hope without explicitly mentioning best practice and requesting integration into framework examples helps in defining the scope of the answers.
So after weeks of pushing commits into travis, I finally made this work.
The .travis.yml:
language: php
sudo: true
php:
- 5.5
- 5.6
- 7.0
- hhvm
addons:
hosts:
- system.hyn.me
- tenant.hyn.me
install:
# fix ipv6 issue that prevented composer requests and resulted in failing builds
- sudo sh -c "echo 'precedence ::ffff:0:0/96 100' >> /etc/gai.conf"
# updates composer on travis
- travis_retry composer self-update
# clear composer cache, might speed up finding new tags
- travis_retry composer clear-cache
# set the global github token, so connections won't be cancelled
- composer config -g github-oauth.github.com $GITHUB_TOKEN
# create a new database for the hyn connection
- mysql -e 'create database hyn;' -uroot
- mysql -e "grant all privileges on *.* to 'travis'#'localhost' with grant option;" -uroot
# create a new laravel project in the subfolder laravel (default composer behaviour)
- composer create-project laravel/laravel
# set global variables
- export DB_USERNAME=travis DB_DATABASE=hyn DB_PASSWORD= QUEUE_DRIVER=sync
script:
# run the script calling unit tests and so on
- ./scripts/travis.sh
after_script:
- if [[ $TRAVIS_PHP_VERSION != '7.0' ]]; then php vendor/bin/ocular code-coverage:upload --format=php-clover ${TRAVIS_BUILD_DIR}/coverage.clover; fi
And the scripts/travis.sh
#!/bin/bash
# e causes to exit when one commands returns non-zero
# v prints every line before executing
set -ev
cd ${TRAVIS_BUILD_DIR}/laravel
BRANCH_REGEX="^(([[:digit:]]+\.)+[[:digit:]]+)$"
if [[ ${TRAVIS_BRANCH} =~ $BRANCH_REGEX ]]; then
echo "composer require ${TRAVIS_REPO_SLUG}:${TRAVIS_BRANCH}"
composer require ${TRAVIS_REPO_SLUG}:${TRAVIS_BRANCH}
else
echo "composer require ${TRAVIS_REPO_SLUG}:dev-${TRAVIS_BRANCH}"
# development package of framework could be required for the package
composer require hyn-me/framework "dev-master as 0.1.99"
composer require "${TRAVIS_REPO_SLUG}:dev-${TRAVIS_BRANCH}#${TRAVIS_COMMIT}"
fi
# moves the unit test to the root laravel directory
cp ./vendor/${TRAVIS_REPO_SLUG}/phpunit.travis.xml ./phpunit.xml
phpunit
# phpunit --coverage-text --coverage-clover=${TRAVIS_BUILD_DIR}/coverage.clover
This code might change due to new Laravel versions or changes in travis. If this is the case, you will find the latest release here.

Travis-CI: how to cache composer executable between builds?

On every travis build, composer self-update is run. And it gets updated on every build. Is it possible to cache composer executables like we do it with vendor dir via
cache:
directories:
- vendor
- $HOME/.composer/cache
I thought about caching the whole /home/travis/.phpenv/versions/5.5/bin/composer but I feel this is not right because the contents of this folder may change without notifying caching system about a change (when travis updates php version for example).
Any suggestions (except for custom composer, of course)?
I'd recommend not updating composer itself, but let travis handle it. (its automatically updated every 30/60 days)
Also i can recommend using the new containerized infrastructure to speed up the runs and allow caching...
language: php
sudo: false
cache:
directories:
- $HOME/.composer/cache
php:
- 5.5
- 5.6
- 7
- hhvm
install:
- composer install
script: vendor/bin/phpunit
The sudo: false statement indicates the use of containers. The cache: statement makes sure composer caches correctly.
If you really want to cache the composer binary:
language: php
php:
- 5.5
- 5.6
- 7
- hhvm
cache:
directories:
- $HOME/.composer/cache
install:
- travis_retry composer self-update && composer --version
- export PATH="$HOME/.composer/vendor/bin:$PATH"
- travis_retry composer install --prefer-dist --no-interaction
script: vendor/bin/phpunit
Also just as a heads up, if testing for HHVM and you need to set the datetime, have a look at https://github.com/travis-ci/travis-ci/issues/2523. My way of solving this is adding a .ini-file in my test-directory with the datetime and setting this in the correct folder for all test-runners. This is prepended in the install:-step:
- mkdir -p /home/travis/.phpenv/versions/$(phpenv version-name)/etc/conf.d
- phpenv config-add test/phpconfig.ini
Anyway, a bit more information than you requested, but i hope this helps someone looking for composer/travis/stuff :)

Categories