I am trying to integrate sonarcloud with Bitbucket pipeline. I have a simple test project with single file named index.php consisting few lines.
I am exactly not sure what method should I follow.
I have created sonar-project.properties files that consists following lines.
sonar.sourceEncoding=UTF-8
sonar.host.url=https://sonarcloud.io/dashboard?id=first_ftp
sonar.projectKey=first_ftp
Here is my yml file
image: aariacarterweir/lamp-gitftp:latest
clone:
depth: full # SonarCloud scanner needs the full history to assign issues properly
definitions:
caches:
sonar: ~/.sonar/cache # Caching SonarCloud artifacts will speed up your build
steps:
- step: &build-test-sonarcloud
name: Build, test and analyze on SonarCloud
caches:
- node
- sonar
script:
- npm install --quiet
- npm run test -- --code-coverage --no-watch --no-progress --browsers=ChromeHeadlessNoSandbox
- pipe: sonarsource/sonarcloud-scan:0.1.5
variables:
SONAR_TOKEN: ${SONAR_TOKEN}
EXTRA_ARGS: '-Dsonar.sources=src -Dsonar.tests=src -Dsonar.test.inclusions="**/testing/**,**/*.spec.ts" -Dsonar.typescript.lcov.reportPaths=coverage/lcov.info'
When I check into sonarcloud website, It tells
The main branch has no lines of code
Can anybody help me how to configure properly sonarcloud with bitbucket for php project.
I've had the same issue - in my case it was because of the name of the project - it was called ---------Test, and apparently, by default, Sonar will treat that as a test assembly and will exclude it from analysis.
Before running code analysis in bitbucket pipelines, you would have to first manually run analysis(only once). By following below steps.
Related
PHP isn't a natively supported language in AWS Lambda, but I thought I'd try my hand at getting one working, using a custom Docker image. I am using this official AWS example to structure the image.
I don't quite understand the pieces yet. I will add what files I have to this post.
Firstly, my Dockerfile:
# Demo of a PHP-based lambda
#
# See example:
# https://github.com/aws-samples/php-examples-for-aws-lambda/blob/master/0.7-PHP-Lambda-functions-with-Docker-container-images/Dockerfile
FROM php:8.0-cli-alpine
WORKDIR /root
# Install Composer
COPY bin bin
RUN sh /root/bin/install-composer.sh
RUN php /root/composer.phar --version
# Install Composer deps
COPY composer.json composer.lock /root/
RUN php composer.phar install
# Install runtimes
COPY runtime /var/runtime
COPY src /var/task/
# Entrypoint
CMD ["index"]
Based on the example I also have:
A PHP listener program at /var/runtime/bootstrap (nearly verbatim copy of the example repo)
Composer dependencies at /root/vendor/... that are loaded by the bootstrap
A trivial index file at /var/task/index.php (verbatim copy of the example repo)
One change I have made is to base the image on an Alpine image from PHP, rather than to use an Amazon Linux image. I wonder, could there be something in the Amazon image that is needed?
The test I am using is the "hello world" in the AWS Lambda UI:
{
"key1": "value1",
"key2": "value2",
"key3": "value3"
}
Anyway, I have used docker login and docker push to get the image to ECR. When I run the hello world test inside the AWS dashboard, I am getting this set of error logs in CloudWatch:
2021-11-13T19:12:12.449+00:00 IMAGE Launch error: fork/exec /usr/local/bin/docker-php-entrypoint: permission denied Entrypoint: [docker-php-entrypoint] Cmd: [index] WorkingDir: [/root]
2021-11-13T19:12:12.493+00:00 START RequestId: 38da1e10-4c93-4109-be10-32c58f83a2fb Version: $LATEST
2021-11-13T19:12:12.502+00:00 IMAGE Launch error: fork/exec /usr/local/bin/docker-php-entrypoint: permission denied Entrypoint: [docker-php-entrypoint] Cmd: [index] WorkingDir: [/root]
2021-11-13T19:12:12.504+00:00 END RequestId: 38da1e10-4c93-4109-be10-32c58f83a2fb
2021-11-13T19:12:12.504+00:00 REPORT RequestId: 38da1e10-4c93-4109-be10-32c58f83a2fb Duration: 9.20 ms Billed Duration: 10 ms Memory Size: 128 MB Max Memory Used: 3 MB
2021-11-13T19:12:12.504+00:00 RequestId: 38da1e10-4c93-4109-be10-32c58f83a2fb Error: fork/exec /usr/local/bin/docker-php-entrypoint: permission denied Runtime.InvalidEntrypoint
That makes a lot of sense, as I don't understand the entry point of "index" either, but it's there as a CMD in the example Dockerfile. Is this an alias for something? I would be inclined to use something like php /var/runtime/bootstrap, but I'd rather understand things, rather than guessing.
I believe I might be able to use Lambda RIE to test this locally, but I wonder if that would tell me what I already know - I need to fix the CMD.
For what it's worth, I can't see how the index.php file is triggered in the lambda either. How does that get invoked?
Update
I am wondering if the parent image in the example (public.ecr.aws/lambda/provided) has an ENTRYPOINT that would shed some light on the apparently confusing CMD. I wonder if I will investigate that next.
Update 2
The ponderance that I might have to use the Amazon Linux image parent was a false steer - this official resource shows the use of standard Python and Node images.
I decided to try repairing the main Docker command:
CMD ["php", "/var/runtime/bootstrap"]
However it doesn't like that:
START RequestId: d95a29d3-6764-4bae-9976-09830c1c17af Version: $LATEST
IMAGE Launch error: fork/exec /usr/local/bin/docker-php-entrypoint: permission denied Entrypoint: [docker-php-entrypoint] Cmd: [php,/var/runtime/bootstrap] WorkingDir: [/root]
IMAGE Launch error: fork/exec /usr/local/bin/docker-php-entrypoint: permission denied Entrypoint: [docker-php-entrypoint] Cmd: [php,/var/runtime/bootstrap] WorkingDir: [/root]
END RequestId: d95a29d3-6764-4bae-9976-09830c1c17af
REPORT RequestId: d95a29d3-6764-4bae-9976-09830c1c17af Duration: 19.88 ms Billed Duration: 20 ms Memory Size: 128 MB Max Memory Used: 3 MB
RequestId: d95a29d3-6764-4bae-9976-09830c1c17af Error: fork/exec /usr/local/bin/docker-php-entrypoint: permission denied
Runtime.InvalidEntrypoint
Update 3
No matter what I do, I seem to run into problems with entrypoints. I've even tried a runtime script to chmod +x on the various binaries it doesn't like, but of course if I try to kick that off in the ENTRYPOINT, the container believes that /bin/sh cannot be executed. This is getting rather silly - containers are just not behaving correctly in the Lambda environment.
Update 4
I have now tried to move away from Alpine, in case a more standard OS will work correctly. No joy - I get the same. I'm now at the random-trying things point, and this is rather slow going, given that the build-push-install loop is cumbersome.
This question looks promising, but putting the bootstrap file in /var/task seems to go directly against the example I am working from.
This problem was tricksy because there were two major interlocking problems - a seemingly excessive permissions requirement, and what struck me as a non-standard use of the ENTRYPOINT/CMD systems.
Working solution
The Dockerfile that works is as follows:
# Demo of a PHP-based lambda
#
# See example:
# https://github.com/aws-samples/php-examples-for-aws-lambda/blob/master/0.7-PHP-Lambda-functions-with-Docker-container-images/Dockerfile
FROM php:8.0-cli-alpine
WORKDIR /root
# Install Composer
COPY bin bin
RUN sh /root/bin/install-composer.sh
RUN php /root/composer.phar --version
# Install Composer deps
COPY composer.json composer.lock /root/
# Move deps to /opt, /root has significant permission issues
RUN php /root/composer.phar install && \
mv /root/vendor /opt/vendor
# Install runtimes
COPY runtime/bootstrap /var/runtime/
COPY src/index.php /var/task/
RUN chmod 777 /usr/local/bin/php /var/task/* /var/runtime/*
# The entrypoint seems to be the main handler
# and the CMD specifies what kind of event to process
WORKDIR /var/task
ENTRYPOINT ["/var/runtime/bootstrap"]
CMD ["index"]
So, that resolves one of my nagging questions about Amazon Linux - it is not required. Note that although I installed Composer dependencies in /root, they could not stay there - even 777 perms on them seemed to be insufficient.
As you can see I used 777 permissions on things in /var. 755 might work, maybe even 750 would work - but the key here is that Amazon appears to be a user that is not the build (root) user. That tripped me up a lot.
Now the ENTRYPOINT is used to run the bootstrap script, which appears to be doing general mediation between events on the AWS side and "use cases" in /var/tasks. The normal purpose of a Docker entrypoint is as a command wrapper to CMD, so to use CMD as a "default lambda type" seems to significantly violate the principle of least surprise. I would have thought the lambda type would be defined by the incoming event, not by any lambda-side setting.
Testing
To test this lambda I use this event in the Lambda UI:
{
"queryStringParameters": { "name": "halfer" }
}
And the demo code will respond with:
{
"statusCode": 200,
"headers": {
"Content-Type": "application/json",
"Access-Control-Allow-Origin": "*",
"Access-Control-Allow-Headers": "Content-Type",
"Access-Control-Allow-Methods": "OPTIONS,POST"
},
"body": "Hello, halfer"
}
Suffice it to say this feels rather brittle. Admittedly the demo code is not production quality, but even so, I suspect this would need a pipeline to do a real AWS Lambda test prior to merging down or deployment.
Performance
Here is why lambdas are tempting, especially for infrequent calls such as crons - they are instantiated quickly and die quickly, leaving no running infra. In one of my demo calls I have:
Init duration 188.75 ms
Duration 39.45 ms
Billed duration 229 ms
Deeper understanding
Having worked with the pieces I think I can now explain them rather better, and what I thought of as unusual architectural choices may actually have some purpose. I fear however this design ideology is not sufficiently documented, so engineers working with Docker-based AWS Lambdas have to spend additional time figuring the pieces out.
Consider the processing loop in the demo runtime:
// This is the request processing loop. Barring unrecoverable failure, this loop runs until the environment shuts down.
do {
// Ask the runtime API for a request to handle.
$request = getNextRequest();
// Obtain the function name from the _HANDLER environment variable and ensure the function's code is available.
$handlerFunction = $_ENV['_HANDLER'];
require_once $_ENV['LAMBDA_TASK_ROOT'] . '/' . $handlerFunction . '.php';
// Execute the desired function and obtain the response.
$response = $handlerFunction($request['payload']);
// Submit the response back to the runtime API.
sendResponse($request['invocationId'], $response);
} while (true);
This picks up $_ENV['_HANDLER'] from the Lambda environment, and AWS derives that from the CMD of the image. Now, in PHP the env vars in $_ENV are static for the duration of the process, so it is perhaps a slightly odd choice to read this in a loop and include the file in a loop - it would have been better to do this in an initialisation phase, returning a clean error if the include isn't found.
However, here's the likely purpose of this system: AWS Lambdas let users customise the CMD in the web dashboard. So in an example enterprise let's say that there's three lambdas - one for responding to a web event, one for a scheduler, and one for responding to SNS topics. The handlers for each of these could be added to the same image, allowing the three lambdas to share an image - all they need to do is to supply a CMD override, and each one will load and use the right handler.
I have my own php image, which I would like to use for my project to run tests on.
container: rela589n/doctrine-event-sourcing-php:latest
services:
test_db:
image: postgres:13-alpine
env:
POSTGRES_DB: des
POSTGRES_USER: des_user
POSTGRES_PASSWORD: p#$$w0rd
steps:
- uses: actions/checkout#v2
- whatever_needed_to_run_tests_inside_container
This fails on checkout action with such error:
EACCES: permission denied, open '/__w/doctrine-event-sourcing/doctrine-event-sourcing/6977c4d4-3881-44e9-804e-ae086752556e.tar.gz'
And this is logical as in fresh docker container there's no such folder structure. What i thought to do is run checkout action inside virtual machine provided runs-on: ubuntu-20.04 and configure volume for docker so that it will have access to code. However I have no idea neither is it a good practice to do this way nor how to implement this. I guess even if it is possible to do this way it won't work for other actions.
Had the same issue when trying to use my own Docker image. In my case, installing everything I need on the fly was not an option, so I had to fix this issue.
It appears that GitHub runs the Docker image with user 1001 named runner and group 121 named docker. After adding the group, adding the user and adding the user to sudoers the problem was solved.
Notice that the checkout path starts with /_w which is strange. If I perform actions/checkout#v2 without my container, the path is /home/runner. Not sure how to solve that yet.
Thanks, this really helped me to find the issue when trying to deploy a CDK project from within a Docker container on Github Actions.
I was getting a permission denied error after checking out the code, and trying to deploy it.
Error: EACCES: permission denied, mkdir '/__w/arm-test/arm-test/cdk.out/asset.7d21b14f781f8b0e4ebb3b30c66614a80f71a2c1637298e5557a97662fce0abe'
This issue had the workaround of running the container with the same user and group as the Github Actions runner, so that it matched with the permissions of the source code directory: https://github.com/actions/runner/issues/691
jobs:
configure:
runs-on: ubuntu-latest
outputs:
uid_gid: ${{ steps.get-user.outputs.uid_gid }}
steps:
- id: get-user
run: echo "::set-output name=uid_gid::$(id -u):$(id -g)"
clone-and-install:
needs: configure
runs-on: ubuntu-latest
container:
image: mcr.microsoft.com/vscode/devcontainers/base:ubuntu
options: --user ${{ needs.configure.outputs.uid_gid }}
steps:
- uses: actions/checkout#v2
Since there is no official troubleshooting information, it is an uphill struggle for anyone without a more complete knowledge of working with similar systems to install.
I want to address issues specifically for OSX Catalina and the Selenium server/Chromedriver configuration and put them all in one place. I'll also include my problem at the end.
It seems to me that most of the problems revolve around:
An incorrect installation of either Selenium, Codeception or Chromedriver
Conflicting versions of the above (or php?)
Problems with the port the Selenium server runs on
Coding mistakes: typos, old code etc
Lack of knowledge of terminal/java commands
Path problems
Anything I don't know about/my problem
I'll start with the things I have learnt along the way, please help to fill in the gaps or correct anything I state below if you are able.
Incorrect installation
Installing via homebrew seems the most foolproof way.
Selenium
brew install selenium-server-standalone
Chromedriver
brew tap homebrew/cask
brew cask install chromedriver
Then you can run brew doctor to check installation is okay and follow instructions or google issues. More brew commands.
Check conflicting versions
Installed Google Chrome and Chromedriver should be the same version.
Check Chrome by going to "Chrome" at the top left of the app and the "about Google Chrome".
Check Chromedriver by running brew info chromedriver
Port problems
You might have an old server running on a port you're trying to use. Shut it down going to this page in google chrome.
http://localhost:<port number>/selenium-server/driver?cmd=shutDownSeleniumServer
You can specify the port you run Selenium in terminal using the -port flag. Like this
java -jar <location of your server> -port 8090
Then match this in your acceptance.suite.yml config file
url: 'http://localhost:<port number>/wd/hub'
Coding mistakes in config file
The most common ones I've seen is having a slash at the end of the line of code above or not including '/wd/hub'
Formatting is like so:
actor: AcceptanceTester
modules:
enabled:
- WebDriver:
url: 'http://192.168.88.100:<port number>/wd/hub'
browser: chrome
- \Helper\Acceptance
And to format extensions (automatically run) is like this:
extensions:
enabled:
- Codeception\Extension\RunProcess:
- chromedriver
Lack of knowledge of terminal/java commands
Run a codeception test in terminal using
php vendor/bin/codecept run acceptance --steps
You must have first used
cd <location of your code>
Paths
I don't know how to go about checking paths are configured correctly. (are they inside config files that point to the other programs?) but I keep seeing hints of them scattered around.
My Issue
From below log, I think that chromedriver is loading and there is something happening in the browser, but it does not load my website. Instead it loads the page below. Help greatly appreciated!
Log:
Codeception PHP Testing Framework v4.1.8
Powered by PHPUnit 9.4.2 by Sebastian Bergmann and contributors.
Running with seed:
Acceptance Tests (1) --------------------------------------------------------------------------------------------------
[RunProcess] Starting chromedriver
QuizCest: New user_complete quiz
Signature: QuizCest:newUser_completeQuiz
Test: tests/acceptance/QuizCest.php:newUser_completeQuiz
Scenario --
I am on page "/"
I see "Fragrant"
FAIL
-----------------------------------------------------------------------------------------------------------------------
[RunProcess] Stopping chromedriver
Time: 00:02.653, Memory: 10.00 MB
There was 1 failure:
---------
1) QuizCest: New user_complete quiz
Test tests/acceptance/QuizCest.php:newUser_completeQuiz
Step See "Fragrant"
Fail Failed asserting that on page /wd/hub/static/resource/hub.html
--> Sessions
Create Session | Refresh Sessions
Chrome
e999201a1381fb966c9d02785c094ae6 | Capabilities | Take Screenshot | Delete Session | Load Script
Mac OS X 10.15.7 | v3.141.59 | re82be7d358
--> contains "Fragrant".
Scenario Steps:
2. $I->see("Fragrant") at tests/acceptance/QuizCest.php:14
1. $I->amOnPage("/") at tests/acceptance/QuizCest.php:13
Artifacts:
Png: /Applications/MAMP/htdocs/fragrantmandarin.com/tests/_output/QuizCest.newUser.completeQuiz.fail.png
Html: /Applications/MAMP/htdocs/fragrantmandarin.com/tests/_output/QuizCest.newUser.completeQuiz.fail.html
FAILURES!
Tests: 1, Assertions: 1, Failures: 1.
My Config file:
actor: AcceptanceTester
modules:
enabled:
- WebDriver:
url: 'http://localhost:4444/wd/hub'
browser: chrome
- \Helper\Acceptance
extensions:
enabled:
- Codeception\Extension\RunProcess:
- chromedriver
step_decorators: ~
This excellent explanatory guide on how Selenium works led to a breakthrough.
I mistakenly thought the url for the WebDriver was the Selenium server url, but it was actually my MAMP url.
So I changed the port in my code to my MAMP url:
- WebDriver:
url: 'http://localhost:8888/wd/hub'
browser: chrome
- \Helper\Acceptance
And voila, a day of work finally complete. Hallelujah! Hope this helps someone else out.
I'm running Symfony Insight with a gitlab CI in order to improve my code quality.
I recently added google recaptcha, which needs a public (used in twig) and a private token (used in controller).
To do a clean job, I added them both as environment variables, added the pub one as twig global variable, and i get the private one with getenv.
I a am now getting the "A Symfony application should be bootable" error on symfony insight, with the following stacktrace :
Symfony\Component\DependencyInjection\Exception\EnvNotFoundException: Environment variable not found: "RECAPTCHA_PUBLIC_KEY". in /home/foobar/code/vendor/symfony/dependency-injection/EnvVarProcessor.php:97
Note that everything is working properly on both local and php unit tests (also ran by CI)
I'm guessing that the insight isn't finding the env var when trying to initiate the global twig variable.
Here is my .env.dist file : (the keys are google testing key, not my real ones, don't worry)
###> google-recaptcha ###
RECAPTCHA_PRIVATE_KEY=6LeIxAcTAAAAAGG-vFI1TnRWxMZNFuojJ4WifJWe
RECAPTCHA_PUBLIC_KEY=6LeIxAcTAAAAAJcZVRqyHh71UMIEGNQ_MXjiZKhI
###< google-recaptcha ###
Here is my twig variable declaration :
globals:
recaptcha_public_key: '%env(RECAPTCHA_PUBLIC_KEY)%'
And my gitlab-ci insight config:
insight:
stage: insight
image: php:7.1
tags:
- symfony4
script:
- curl -o insight.phar -s https://get.insight.symfony.com/insight.phar
- php insight.phar analyze --no-interaction --no-ansi cc8f0c98-ce1b-4b1e-acc4-9dfafd4bafc4 -v --reference=$CI_COMMIT_SHA --user-uuid=23888e4d-ec4f-479b-90c6-ee454c7bfc88 --api-token=b1dcbef05392e237a5ee5d29ea348b9ab7179245b0f086c3490478b0ae643272 --fail-condition="counts.critical > 0 or counts.major > 0"
only:
- develop
- master
variables:
MYSQL_DATABASE: higalsymf
DATABASE_HOST: mysql
MYSQL_ROOT_PASSWORD: root
The first idea: you can insert tokens in variables section in gitlab-ci.yml.
Second idea is add command cp .env.dist .env in section before_script in gitlab-ci.yml.
i created some tests with PHPUnit and laravel. The tests are running locally successful, but as a gitlab ci job there is an error.
Here is the ci log:
There was 1 failure:
1) Tests\Feature\AuthTest::testAuthLoggedInIsAdmin
Expected status code 200 but received 500.
Failed asserting that false is true.
/builds/XXX/webproject/vendor/laravel/framework/src/Illuminate/Foundation/Testing/TestResponse.php:55
/builds/XX/webproject/tests/Feature/AuthTest.php:53
For the better debug and finding a solution i need the error trace from the project or the error.log from the webserver.
What is the best practise to debug errors in the ci?
ok, the solution is very simple... i don't delete the question. Here is the solution:
You have to add an artifact to the job. You can set the artifact, that only on failures the artifact is created. If there is an failure on the build, the hole project is dumped to a seperated location. Now you can browse each files on the dump.
.gitlab-ci.yml
stages:
- test
- deploy
php-7.0:
stage: test
type: test
services:
- mysql:latest
image: tetraweb/php:7.0
script:
- bash .gitlab-ci.sh
- php vendor/bin/phpunit --coverage-text --colors=never
artifacts:
when: on_failure
name: "${CI_BUILD_STAGE}_${CI_BUILD_REF_NAME}_FAILED"
paths:
- "."
untracked: false
expire_in: 1 day
deploy:
stage: deploy
# etc...
Here are more informations about artifacts.
You can download the artifacts in the pipelines section in gitlab: