I would be grateful if someone could help me with setting up bitbucket-pipeline for my php env. What I'm trying to succeed is:
Build image in the first step
In the second step run some QA stuff like unit test, code sniffer etc.
Deploy to preprod envirement
Currently I'm stuck with re-using image from the first step. This is how my bitbucket-pipelines.yml looks like:
pipelines:
branches:
develop:
- step:
name: Build docker image
caches:
- composer
script:
- apt-get update && apt-get install -y unzip
- docker-php-ext-install mysqli pdo_mysql json sockets
- curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
- composer install
- parallel:
- step:
caches:
- composer
name: Unit tests
script:
- vendor/bin/phpunit
- step:
caches:
- composer
name: Code sniffer
script:
- composer phpcs:all
- step:
name: Deploy to preprod
script:
- echo "Deployment"
What I get here is:
bash: vendor/bin/phpunit: No such file or directory - from "Unit tests" step
and bash: composer: command not found - from "Code sniffer" step
I already tried to set docker/composer to cache, save image in the first step and import it in the second, but still not working.
One way to share stuff between different steps is to make use of Artifacts.
Basically, bitbucket spins up each step in a separate docker container. So a way to re-use material between steps is create an artifact.
The link should give you enough information.
For example:
- step: &build
caches:
- node
name: Build
script:
- npm install
- npm run build
artifacts: # defining the artifacts to be passed to each future step.
- dist/**
- step: &read-from-artifact
name: Read from the artifact saved in the previous step
script:
- echo "Read artifact (dist folder) saved in previous step"
- ls -la dist/
Related
Github Action.
I have a PHP Project (using Yii2 Framework) that I need to build into a docker image, which is those project also need MongoDB extension.
My plan is in Github Action - CI phase is like this:
Commit to main branch
Enable php extension
Run composer install --no-dev
Build docker image using dockerfile, which is in those dockerfile, I copy vendor`s folder into an image container,
..so on
So, for those, I start with a configuration: 01 - Init Composer.yml like this:
name: Init composer
on:
push:
branches: ["main"]
jobs:
run-composer:
name: composer initialization
runs-on: ubuntu-latest
container:
image: composer:latest
volumes:
- ${{ github.workspace }}:/app
steps:
- name: Check out the repo
uses: actions/checkout#v3
- name: PHP setup with Extension
id: setup-php
uses: shivammathur/setup-php#v2
with:
php-version: '8.1'
extensions: mongodb-mongodb/mongo-php-driver#v1.9
- name: Cache Composer packages
id: composer-cache
uses: actions/cache#v3
with:
path: vendor
key: ${{ runner.os }}-php-${{ hashFiles('**/composer.lock') }}
restore-keys: |
${{ runner.os }}-php-
- name: Running composer install
run: composer install --no-dev
I got error like this:
Run shivammathur/setup-php#v2
/usr/bin/docker exec fccfaf138a6147da33e0f9ca9add2947ae76ff4b439c3fe9c35f53042e6419fa sh -c "cat /etc/*release | grep ^ID"
/bin/bash /__w/_actions/shivammathur/setup-php/v2/src/scripts/run.sh
==> Setup PHP
/__w/_actions/shivammathur/setup-php/v2/src/scripts/linux.sh: line 216: sudo: command not found
✗ PHP Could not setup PHP 8.1
Error: The process '/bin/bash' failed with exit code 1
Any advise is so appreciated.
Running inside the composer image, it doesn't have the sudo command to perform actions to compile the extension.
In action shivammathur/setup-php it is possible enable composer, see tools support
Try running it directly in ubuntu, setting the php and composer extension first.
I had a similar problem here when running it in self hosted server.
Looks like shivammathur already fixed in the "develop" branch (see here)
It will be fixed in the next version, you can follow the issue about it here
I have a very simple pipelines, I am just trying to get the app to build before I look to adding anything else to it.
But, even in its simple state, I can't get composer to successfully patch a file.
The entire composer install works flawlessly until this point:
- Applying patches for drupal/core
https://www.drupal.org/files/issues/2021-03-05/3007386-language-negotiator-weight-12.patch (Language negotiation weight)
Could not apply patch! Skipping. The error was: Cannot apply patch https://www.drupal.org/files/issues/2021-03-05/3007386-language-negotiator-weight-12.patch
[Exception]
Cannot apply patch Language negotiation weight (https://www.drupal.org/files/issues/2021-03-05/3007386-language-negotiator-weight-12.patch)!
After it started failing, I added some debug to the config including trying to cat the file that was to be patched, but it is not there.
+ cat docroot/core/modules/language/src/LanguageNegotiator.php
cat: can't open 'docroot/core/modules/language/src/LanguageNegotiator.php': No such file or directory
This is the composer patch chunk:
"extra": {
"patches": {
"drupal/core": {
"Language negotiation weight": "https://www.drupal.org/files/issues/2018-10-17/3007386-language-negotiator-weight-2.patch"
}
},
This is the pipeline yml:
# Template PHP Build
# This template allows you to validate your PHP application.
# The workflow allows running tests and code linting on the default branch.
image: php:7.3-alpine
pipelines:
default:
- parallel:
- step:
name: Prepare environment
script:
- /sbin/apk update
- /sbin/apk add git
- curl -sS https://getcomposer.org/installer | php -- --install-dir=/usr/local/bin --filename=composer
- docker-php-ext-install mysqli pdo pdo_mysql
- /sbin/apk add libpng-dev
- docker-php-ext-install mbstring
- docker-php-ext-install gd
- pwd
- ls -la
- cat docroot/core/modules/language/src/LanguageNegotiator.php
- /usr/local/bin/composer install
caches:
- composer
Do I need to change directories or do something similar?
I have problem with gitlab cache config.
I use GitLab Community Edition 11.7.7, in docs we have:
https://docs.gitlab.com/ee/ci/caching/#caching-php-dependencies
So, my config:
cache:
key: ${CI_COMMIT_REF_SLUG}
paths:
- vendor/
stages:
- code-style
- analysis
before_script:
- composer global require hirak/prestissimo
- composer install --ignore-platform-reqs --no-scripts
php-cs-fixer:
stage: code-style
script:
- vendor/bin/php-cs-fixer fix --dry-run --diff --verbose --config=.php_cs.dist src tests
php-cs-fixer2:
stage: analysis
script:
- vendor/bin/php-cs-fixer fix --dry-run --diff --verbose --config=.php_cs.dist src tests
Unfortunately, composer re-installs all dependencies in every job.
I would like to install all dependencies through the composer only once, how can I achieve it?
Your .gitlab-ci.yml configuration is similar to my own that run on self-hosting Gitlab at my company, I think it should works as well, So I have created an sample repo that runs pipelines based on your configuration in order to reproduce your problem.
As you can see here, I was trying to make a working pipeline by complete the configuration, For example, wrong image, missing composer executable...something like that, The final configuration is not much diffrent to yours.
image: php:7.2-alpine
cache:
key: $CI_COMMIT_REF_SLUG
paths:
- vendor/
stages:
- code-style
- analysis
before_script:
- php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"
- php -r "if (hash_file('sha384', 'composer-setup.php') === 'a5c698ffe4b8e849a443b120cd5ba38043260d5c4023dbf93e1558871f1f07f58274fc6f4c93bcfd858c6bd0775cd8d1') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;"
- php composer-setup.php --install-dir=/usr/local/bin --filename=composer
- export PATH="/usr/local/bin:$PATH"
- php -r "unlink('composer-setup.php');"
- composer global require hirak/prestissimo
- composer install --ignore-platform-reqs --no-scripts
php-cs-fixer:
stage: code-style
script:
- vendor/bin/php-cs-fixer fix --dry-run --diff --verbose --config=.php_cs.dist src tests || true
allow_failure: true
php-cs-fixer2:
stage: analysis
script:
- vendor/bin/php-cs-fixer fix --dry-run --diff --verbose --config=.php_cs.dist src tests
allow_failure: true
The last job #97744833 have a clue to answer your problem.
Compare the last part of outputs from previous failed job https://gitlab.com/Gasol/php-composer-cache-on-gitlab-ci/-/jobs/358118759#L191
187 Checked all files in 0.006 seconds, 10.000 MB memory used
191 ERROR: Job failed: exit code 8
to https://gitlab.com/Gasol/php-composer-cache-on-gitlab-ci/-/jobs/358121571#L190
The cache is creating when build success.
190 Creating cache master...
191 vendor/: found 4860 matching files
192 Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/15472690/master
193 Created cache
196 Job succeeded
So, after cache created, The job on the next stage should be ok for downloading cache archive.
https://gitlab.com/Gasol/php-composer-cache-on-gitlab-ci/-/jobs/358121572#L20
18 Checking cache for master...
19 Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/15472690/master
20 Successfully extracted cache
https://gitlab.com/Gasol/php-composer-cache-on-gitlab-ci/-/jobs/358121572#L46
And there are nothing to install when cache extracted. That's the result you want.
43 $ composer install --ignore-platform-reqs --no-scripts
44 Loading composer repositories with package information
45 Installing dependencies (including require-dev) from lock file
46 Nothing to install or update
47 Generating autoload files
A little context: I am new to docker and dont know of any best-practices yet. My task is to create a webservice based on php and i decided to use docker-compose for provisioning.
Because we are all fancy devs, we know - there is no way to build a php application without using composer these days.
My question:
Should i install dependencies in the build? So by adding this to my Dockerfile:
RUN cd /app && composer install --no-interaction --prefer-source --optimize-autoloader
Or should i install dependencies during development and build the container image with included dependencies?
I only know of one way NOT to do it: install dependencies locally on the dev machine and build the container afterwards. But how would the "best practice" look like?
And - because i am a newbee in this field - how would i run a "composer require some/package" for my app service container?
By the way
I also noticed a message "Do not run Composer as root/super user!" when building the container. I added COMPOSER_ALLOW_SUPERUSER=1 to my Env file (as seen here), but this message still appears.
Is it possible to NOT execute composer install as root in a docker container? Can i ignore that message?
Thanks in advance,
Philipp
For the moment i am working towards the following solutuion:
Development environment:
Add the whole src as volume:
volumes:
- .:/app
initial composer install command
docker-compose exec app composer install
install new composer package:
docker-compose exec app composer require some/package
The package will be installed via the container in your composer package directory (/vendor by default of course), the source code can be inspected in your ide, etc.
Production environment:
Only app state defined as volume, for example:
volumes:
- public/uploads:/app/public/uploads
add /vendor to .dockerignore
and run composer install during build
RUN cd /app && composer install --no-dev --no-interaction --optimize-autoloader
So you will have a pre-built image to deploy, but still be able to develop in a more agile way.
There are still two downsides in this solution, maybe someone has input for me:
1) build becomes kind of slow - it seems composer caching does not work properly
A solution (i dont know if this is possible in composer) could be a volume for the composer cache directory.
2) all packages created by docker are owned by root, so you can only work on those files/folders with sudo/root session.
I have no idea how to fix this properly. Maybe i could create a user with the name of the dev user for the development container and have this user run commands and php-fpm, etc... For production root would be okay i guess.
I am still very new to docker and open for better solutions, so i wont accept this answer.
UPDATED
To add new user in docker user in Dockerfile. as follows
RUN useradd -ms /bin/bash newuser
USER newuser
if you want to create home user too
WORKDIR /home/newuser
or
mkdir /home/newuser
if you want to back to root user, you can do like
USER root
two-sides-of-a-coin -
It is a good practice to do on building images period which I also do the same as you.
we will get latest update from public repositories but another side of coin is that we cannot ensure they are the same version as we develop.
you need to handle this by specifying exact version in composer.json (do not use ^, *).
While working on a laravel 5.1+ package I have this need to run automated tests through travis-ci.org. The difference with regular automated tests is the requirement to include this package into a framework and set specific configuration options to run the tests.
So the requirement would be:
install laravel
add my package as dependency
set some travis specific configurations like the travis database access
run migrations of laravel
run migrations specific for package or run an artisan command
run package specific unit tests
I searched everywhere; asked on laravel forums, asked in a travis community chat and saw this topic being closed as too localized (although an answer would have certainly been helpful now). I'm hoping my question is fit to remain open.
At this time I have the following configuration:
language: php
php:
- 5.5
- 5.6
- hhvm
addons:
hosts:
- system.hyn.me
- tenant.hyn.me
before_install:
- sudo composer self-update
install:
- composer create-project laravel/laravel
- cd ./laravel
- composer require hyn-me/multi-tenant ~0.1.0
- composer update
before_script:
- cp .env.travis .env
- export APP_ENV="testing"
- php artisan migrate -q -n --path ./vendor/hyn-me/multi-tenant/src/migrations
- cd ./vendor/hyn-me/multi-tenant
script: phpunit
Yet my knowledge of travis (thus far) is limited and before I send in an unneeded number of commits to fix my problems I'd rather have your opinion on what would be a good method to test integration into a framework.
Ps. this concerns the package hyn/multi-tenant.
Advise on how to keep this question as generic as possible would be helpful. I hope without explicitly mentioning best practice and requesting integration into framework examples helps in defining the scope of the answers.
So after weeks of pushing commits into travis, I finally made this work.
The .travis.yml:
language: php
sudo: true
php:
- 5.5
- 5.6
- 7.0
- hhvm
addons:
hosts:
- system.hyn.me
- tenant.hyn.me
install:
# fix ipv6 issue that prevented composer requests and resulted in failing builds
- sudo sh -c "echo 'precedence ::ffff:0:0/96 100' >> /etc/gai.conf"
# updates composer on travis
- travis_retry composer self-update
# clear composer cache, might speed up finding new tags
- travis_retry composer clear-cache
# set the global github token, so connections won't be cancelled
- composer config -g github-oauth.github.com $GITHUB_TOKEN
# create a new database for the hyn connection
- mysql -e 'create database hyn;' -uroot
- mysql -e "grant all privileges on *.* to 'travis'#'localhost' with grant option;" -uroot
# create a new laravel project in the subfolder laravel (default composer behaviour)
- composer create-project laravel/laravel
# set global variables
- export DB_USERNAME=travis DB_DATABASE=hyn DB_PASSWORD= QUEUE_DRIVER=sync
script:
# run the script calling unit tests and so on
- ./scripts/travis.sh
after_script:
- if [[ $TRAVIS_PHP_VERSION != '7.0' ]]; then php vendor/bin/ocular code-coverage:upload --format=php-clover ${TRAVIS_BUILD_DIR}/coverage.clover; fi
And the scripts/travis.sh
#!/bin/bash
# e causes to exit when one commands returns non-zero
# v prints every line before executing
set -ev
cd ${TRAVIS_BUILD_DIR}/laravel
BRANCH_REGEX="^(([[:digit:]]+\.)+[[:digit:]]+)$"
if [[ ${TRAVIS_BRANCH} =~ $BRANCH_REGEX ]]; then
echo "composer require ${TRAVIS_REPO_SLUG}:${TRAVIS_BRANCH}"
composer require ${TRAVIS_REPO_SLUG}:${TRAVIS_BRANCH}
else
echo "composer require ${TRAVIS_REPO_SLUG}:dev-${TRAVIS_BRANCH}"
# development package of framework could be required for the package
composer require hyn-me/framework "dev-master as 0.1.99"
composer require "${TRAVIS_REPO_SLUG}:dev-${TRAVIS_BRANCH}#${TRAVIS_COMMIT}"
fi
# moves the unit test to the root laravel directory
cp ./vendor/${TRAVIS_REPO_SLUG}/phpunit.travis.xml ./phpunit.xml
phpunit
# phpunit --coverage-text --coverage-clover=${TRAVIS_BUILD_DIR}/coverage.clover
This code might change due to new Laravel versions or changes in travis. If this is the case, you will find the latest release here.