Github Action CI should fail when code coverage is not 100% - php

I want to check the code coverage percentage in the Github Action CI on push and pull requests. I'm using Symfony. I've found actions for Javascript but not for PHP or Symfony-based.
I've already created a GitHub action workflow which is as below:
name: Running Code Coverage
on: [push, pull_request]
jobs:
build:
name: Code Coverage
runs-on: ubuntu-latest
strategy:
matrix:
php-version: [8.1]
steps:
- name: Checkout repository
uses: actions/checkout#v2
- name: Set up PHP ${{ matrix.php-version }}
uses: shivammathur/setup-php#v2
with:
php-version: ${{ matrix.php-version }}
- name: Upload coverage to Codecov
uses: codecov/codecov-action#v1
with:
token: ${{ secrets.CODECOV_SECRET_TOKEN }}
fail_ci_if_error: true
files: ./coverage_report.xml
But I cannot achieve my end goal using the above as it uploads the code coverage to codecov and there I can see the %age.
My goal is that the CI should fail when the code coverage is below a threshold like 100%.

Referencing from the official docs on threshold:
threshold (number): Allow the coverage to drop by X%, and posting a success status.
Here is an example codecov.yml reference from the same docs:
coverage:
status:
project:
default:
# basic
target: auto
threshold: 0%
base: auto
flags:
- unit
paths:
- "src"
# advanced settings
branches:
- master
if_ci_failed: error #success, failure, error, ignore
informational: false
only_pulls: false

Related

Sonar-scanner all files

I am using sonarqube to analyze the code of my project in PHP, everything is set up and partially working, the problem is as follows, I do a check with the Sonar scanner on my pull-requests and merge with the branch master, the analysis it is being carried out, but only in the modified files. I would need to analyze all the code at least on the merge with the master branch. When I go to Project -> code, I only have a few files in the master branch.
I would like to know if there is any parameter that can be passed in the scanner so that it always analyzes all files as it is done with the scanner run locally.
Code scanner
name: Analyze pull request
on:
pull_request:
types: [opened, edited, reopened, synchronize]
branches:
- master
jobs:
SonarQube-Scanner-pull_request:
runs-on: ubuntu-latest
steps:
- name: Setup sonarqube
uses: warchant/setup-sonar-scanner#v1
- name: 'Checkout repository on branch: ${{ github.REF }}'
uses: actions/checkout#v2
with:
ref: ${{ github.HEAD_REF }}
- name: Retrieve entire repository history
run: |
git fetch --prune --unshallow
- name: Run an analysis of the PR
env:
# to get access to secrets.SONAR_TOKEN, provide GITHUB_TOKEN
GITHUB_TOKEN:
run: sonar-scanner
-Dsonar.host.url=
-Dsonar.login=
-Dsonar.projectKey=Project
-Dsonar.qualitygate.wait=true
-Dsonar.pullrequest.key=${{ github.event.number }}
-Dsonar.pullrequest.branch=${{ github.HEAD_REF }}
-Dsonar.pullrequest.base=${{ github.BASE_REF }}
-Dsonar.pullrequest.github.repository=${{ github.repository }}
-Dsonar.scm.provider=git
-Dsonar.java.binaries=/tmp
enter image description here
Thank you for your help
Can you try by giving sonar.projectBaseDir and sonar.sources in sonar analysis properties.
Find more details here Alternate Analysis Directory

Run suites of included files in codeception

I have an application with different modules. For each module, there are separate tests written which are divided in different suites.
For example in tests/Custom/codeception.yml
namespace: Test\Custom
paths:
tests: .
data: _data
support: _support
log: _output
coverage:
enabled: true
remote: false
whitelist: { include: ['../../../../src/*'] }
suites:
Business:
path: Business
...
View:
path: View
....
So every module has its own codeception.yml.
To have one "master config" there is a codeception.yml in the first level of the test folder which is including all the subfolder module yml with
namespace: Test
actor: Tester
include:
- tests/*
...
When running a build caused by a branch update in the repository we usually don't want to run other tests than business because they are slow and fragile.
For this i try to run codecept run Business and expect, that just the Business suites from the included files will be executed.
But this will not work, instead i get the error message
Suite 'Business' could not be found
But when i run one module explicit with Business suite
codecept run -c tests/Custom Business
it works.
My first assumption was, that the files get not proper included to the master yml so i tried the general call
codecept run
and it works! But this unfortunately runs all suites.
So, why the suites cant be seen by the master codeception.yml? How can i run all tests Business suites without pointing explicit to the module path?
Thanks

Codeception acceptance tests report that coverage is 0%

I'm trying to test my Yii2 application by codeception.
I want to see coverage of my code.
I'm running selenium server:
java -jar -Dwebdriver.chrome.driver=chromedriver /path/to/selenium-server-standalone-3.141.59.jar
And i'm running my test in the following way:
./vendor/bin/codecept run `tests/acceptance/callIn/FinishCallCest.php:submitBlank --coverage`
When I'm runinng codeception acceptance test without --coverage everything works fine. And if I'm running codeception unit tests with --coverage I get correctly coverage report.
But, if I run codeception acceptance test with --coverage I seeing following:
localhost:8080 link opens in my browser. And i see "NOT FOUND 404" page from Yii2.
Then redirect to http://localhost:8080/index-test.php?r=call-in%2Ffinish-call&callId=1
happens, and I see the page and test is successfully done.
Why does the localhost:8080 page load first? And how can I fix the error with 0% coverage?
Here's my codeception.yml
actor: Tester
paths:
tests: tests
log: tests/_output
data: tests/_data
helpers: tests/_support
settings:
bootstrap: _bootstrap.php
memory_limit: 1024M
colors: true
modules:
config:
Yii2:
configFile: 'config/test.php'
cleanup: false
coverage:
c3_url: http://127.0.0.1:8080/index-test.php/
enabled: true
whitelist:
include:
- models/*
- controllers/*
config:
test_entry_url: http://localhost:8080/idnex-test.php
Here's my acceptance.suite.yml
class_name: AcceptanceTester
modules:
enabled:
- Yii2:
part: [orm] # allow to use AR methods
cleanup: false
- WebDriver:
url: 'http://test:test#localhost:8080'
auth: ['test', 'test']
browser: chrome
restart: true
window_size: 1024x768
And here's a codeception test log:
Acceptance Tests (1) ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Modules: Yii2, WebDriver
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
FinishCallCest: Submit blank finish form
Signature: calIn\FinishCallCest:submitBlankFinishForm
Test: tests/acceptance/callIn/FinishCallCest.php:submitBlankFinishForm
Scenario --
Destroying application
Starting application
[ConnectionWatcher] watching new connections
[Fixtures] Loading fixtures
[Fixtures] Done
[GET] http://test:test#localhost:8080/
[Cookies] [{"domain":"localhost","httpOnly":true,"name":"_csrf","path":"/","secure":false,"value":"c99eee70d806de0660302db950c436a04f5a6316b5847caed6de7d1d2e2d896aa%3A2%3A%7Bi%3A0%3Bs%3A5%3A%22_csrf%22%3Bi%3A1%3Bs%3A32%3A%22PXMEKYNF-k-StsR38slXolelVVLuFKTK%22%3B%7D"},{"domain":"localhost","expiry":1543848123.606772,"httpOnly":true,"name":"_identity","path":"/","secure":false,"value":"dacb46519512cd901d6c8ceafb494a3d2893b6cf41a196e111c9c522aa4605c9a%3A2%3A%7Bi%3A0%3Bs%3A9%3A%22_identity%22%3Bi%3A1%3Bs%3A77%3A%22%5B%22corebofs000080000ktnhd8g9vnbio68%22%2C%22corebofs000080000ktnhd8g9vnbio68%22%2C18000%5D%22%3B%7D"},{"domain":"localhost","httpOnly":true,"name":"PHPSESSID","path":"/","secure":false,"value":"06okm3re6q6ncjj6q878sdq88j"}]
I am on page "/index-test.php?r=call-in%2Ffinish-call&callId=1"
[GET] http://test:test#localhost:8080/index-test.php?r=call-in%2Ffinish-call&callId=1
I wait 5
I click "//*[#id="w3"]/div[1]/div[1]/h4/a"
I wait 5
I see "Завершение звонка","//*[#id="w1"]/div[4]/button"
I click "//*[#id="w1"]/div[4]/button"
PASSED
Destroying application
[ConnectionWatcher] no longer watching new connections
[ConnectionWatcher] closing all (0) connections
Suite done, restoring $_SERVER to original
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Time: 24.18 seconds, Memory: 14.00MB
OK (1 test, 1 assertion)
Code Coverage Report:
2018-12-03 09:42:01
Summary:
Classes: 0.00% (0/84)
Methods: 0.00% (0/389)
Lines: 0.00% (0/2706)
Remote CodeCoverage reports are not printed to console
HTML report generated in coverage

How to fail a build on Scrutinizer based on phpcs output?

I've tried configuring Scrutinizer to fail whenever the CodeSniffer analysis returns some errors with no luck.
Here's what I tried so far:
V1:
build:
tests:
override:
command: './vendor/bin/phpcs ./src --report=checkstyle --report-file=cs-data'
analysis:
file: 'cs-data'
format: 'php-cs-checkstyle'
V2:
build:
tests:
override:
- 'phpcs-run'
My tools and build_failure_conditions:
tools:
php_code_sniffer:
config:
standard: "PSR2"
checks:
php:
code_rating: true
duplication: true
build_failure_conditions:
- 'issues.label("coding-style").new.exists'
What's missing?
TLDR; It looks like you're not specifying which folders to scan for errors.
Try using the following:
build:
tests:
override:
- 'phpcs-run --standard=custom/standard/ruleset.xml --ignore=app/*/*/Test/ app/dir1/ app/dir2/ app/other/folder'
Some things that required clarification for me:
scrutinizer currently does not support failing the phpcs command (similar to phpunit) as that would prevent the issues from being displayed in the UI.
the workaround for that is to set-up a failure condition (which you already did) like for example whether there are coding-style issues, then that would fail the build.
I'm currently using:
build_failure_conditions:
- 'issues.label("coding-style").new.exists'
Which makes the inspection look like this
More info about how to set-up failure conditions:
https://scrutinizer-ci.com/docs/configuration/build_status

How can i debug a failed ci build?

i created some tests with PHPUnit and laravel. The tests are running locally successful, but as a gitlab ci job there is an error.
Here is the ci log:
There was 1 failure:
1) Tests\Feature\AuthTest::testAuthLoggedInIsAdmin
Expected status code 200 but received 500.
Failed asserting that false is true.
/builds/XXX/webproject/vendor/laravel/framework/src/Illuminate/Foundation/Testing/TestResponse.php:55
/builds/XX/webproject/tests/Feature/AuthTest.php:53
For the better debug and finding a solution i need the error trace from the project or the error.log from the webserver.
What is the best practise to debug errors in the ci?
ok, the solution is very simple... i don't delete the question. Here is the solution:
You have to add an artifact to the job. You can set the artifact, that only on failures the artifact is created. If there is an failure on the build, the hole project is dumped to a seperated location. Now you can browse each files on the dump.
.gitlab-ci.yml
stages:
- test
- deploy
php-7.0:
stage: test
type: test
services:
- mysql:latest
image: tetraweb/php:7.0
script:
- bash .gitlab-ci.sh
- php vendor/bin/phpunit --coverage-text --colors=never
artifacts:
when: on_failure
name: "${CI_BUILD_STAGE}_${CI_BUILD_REF_NAME}_FAILED"
paths:
- "."
untracked: false
expire_in: 1 day
deploy:
stage: deploy
# etc...
Here are more informations about artifacts.
You can download the artifacts in the pipelines section in gitlab:

Categories