composer cache not working on bitbucket pipeline build - php

Question

I got this in my bitbucket pipeline:
pipelines:
branches:
develop:
- step:
caches:
- composer
name: unit tests - Delivery
image: totersapp/laravel-docker:phpredis
script:
- echo "memory_limit=512M" > /usr/local/etc/php/conf.d/memory-limit-php.ini
- ln -f -s .env.pipelines .env
- composer install
services:
- postgres
- redis
every time it run it is downloading the same files (even on the second run).. any ideas why?
This is the log for composer install:
+ composer install
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Package operations: 199 installs, 0 updates, 0 removals
- Installing kylekatarnls/update-helper (1.2.0): Downloading (connecting...)Downloading (0%) Downloading (100%)
- Installing ocramius/package-versions (1.4.2): Downloading (connecting...)Downloading (0%) Downloading (30%)Downloading (35%)Downloading (65%)Downloading (95%)Downloading (100%)
- Installing symfony/polyfill-ctype (v1.13.1): Downloading (connecting...)Downloading (0%) Downloading (100%)
update: couldn't override default
Based on this answer I tried this:
- step:
caches:
- composer
image: totersapp/laravel-docker:phpredis
script:
- composer install
..
definitions:
caches:
composer: /composer/cache
My problem is that I'm trying to call composer with a custom cache directory (in this case it's /composer/cache) but then again I don't have a way to call the composer install command while supplying the custom cache directory as an option (ie which is what the bitbucket tutorial is doing with bundle in this example:
- bundle install --path vendor/bundle
said another way, I cannot run something like this:
- composer install --cache-directory /composer/cache
Update 2: discrepancy between composer.json and composer.lock
First as a sanity check, I simply cloned the repo of Nico
Haase, and ran the pipelines, passed ✅
I copied and pasted the contents of his bitbucket-pipelines.yml into mine, and I copied and pasted his composer.json, I ran the pipeline. Failed ❌
But I got this in the composer install log:
+ composer install --ignore-platform-reqs --no-scripts
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Warning: The lock file is not up to date with the latest changes in composer.json. You may be getting outdated dependencies. Run update to update them.
Package operations: 199 installs, 0 updates, 0 removals
- Installing kylekatarnls/update-helper (1.2.0): Downloading (connecting...)Downloading (0%) Downloading (100%)
- Installing ocramius/package-versions (1.4.2): Downloading (connecting...)Downloading (0%) Downloading (30%)Downloading (35%)Downloading (65%)Downloading (95%)Downloading (100%)
- Installing symfony/polyfill-ctype (v1.13.1): Downloading (connecting...)Downloading (0%) Downloading (100%)
So i figured it's b/c the composer.lock file is out of sync with composer.json, so I added composer update to my pipeline.. this loaded from cache! ✅but failed here ❌:
composer update
Loading composer repositories with package information
Updating dependencies (including require-dev)
Package operations: 108 installs, 0 updates, 0 removals
- Installing ocramius/package-versions (1.4.2): Downloading (connecting...)Downloading (0%) Downloading (30%)Downloading (35%)Downloading (65%)Downloading (80%)Downloading (100%)
- Installing symfony/flex (v1.6.0): Downloading (connecting...)Downloading (100%)
Symfony operations: 1 recipe (4c6f09f5995a77b64f54dd80675d3bfe)
- Configuring symfony/flex (>=1.0): From github.com/symfony/recipes:master
Loading composer repositories with package information
Updating dependencies (including require-dev)
Restricting packages listed in "symfony/symfony" to "5.0.*"
Prefetching 106 packages
- Downloading (0%) (5%) (10%) (16%) (21%) (31%) (36%) (41%) (47%) (52%) (57%) (65%) (70%) (75%) (80%) (85%) (90%) (95%) (100%)
Package operations: 106 installs, 0 updates, 0 removals
- Installing psr/container (1.0.0): Loading from cache
- Installing symfony/service-contracts (v2.0.1): Loading from cache
..
..
Executing script cache:clear [KO]
[KO]
Script cache:clear returned with error code 1
!! Fatal Error: composer.lock was created for PHP version 7.2.9 or higher but the current PHP version is 7.1.32.
note: in my original composer.json file, i found this line
"config": {
..
"cache-dir": "~/.cache/composer",
},
so I removed that line, but that didn't help either.
Still digging (for the record this is what my composer.json looks like, i kept the cache-dir part in it for reference)

Answer 1

According to a question on the Bitbucket forums, this can happen if the docker image you are using is putting the downloaded files in a unusual position. According to the documentation about caches, the usual directory that is cached is ~/.composer/cache, while the docker file of the image uses /composer.
Please try to use the following configuration. It contains a custom cache (which needs a custom name!) that puts the customized composer folder in the cache:
pipelines:
branches:
develop:
- step:
caches:
- composer-custom
definitions:
caches:
composer-custom: /composer/cache
As you can see on https://bitbucket.org/haasenico/composer-cache/addon/pipelines/home#!/results/4, the cache is used and seems to work for my short example

Answer 2

To make this work I basically followed Nico Haase's answer, but I also had to do the following:
remove "cache-dir": "~/.cache/composer" from my composer.json
ensure that the cache file saved is in sync with whatever is going on in composer.json.. you cannot ask composer.json to load the dependencies from a cache file that has irrelevant content. To do this, I simply deleted all caches file and started again, it worked this time!