I am running docker-container on Amazon EC2. Currently I have added AWS Credentials to Dockerfile. Could you please let me know the best way to do this?
A lot has changed in Docker since this question was asked, so here's an attempt at an updated answer.
First, specifically with AWS credentials on containers already running inside of the cloud, using IAM roles as Vor suggests is a really good option. If you can do that, then add one more plus one to his answer and skip the rest of this.
Once you start running things outside of the cloud, or have a different type of secret, there are two key places that I recommend against storing secrets:
Environment variables: when these are defined on a container, every process inside the container has access to them, they are visible via /proc, apps may dump their environment to stdout where it gets stored in the logs, and most importantly, they appear in clear text when you inspect the container.
In the image itself: images often get pushed to registries where many users have pull access, sometimes without any credentials required to pull the image. Even if you delete the secret from one layer, the image can be disassembled with common Linux utilities like tar and the secret can be found from the step where it was first added to the image.
So what other options are there for secrets in Docker containers?
Option A: If you need this secret only during the build of your image, cannot use the secret before the build starts, and do not have access to BuildKit yet, then a multi-stage build is a best of the bad options. You would add the secret to the initial stages of the build, use it there, and then copy the output of that stage without the secret to your release stage, and only push that release stage to the registry servers. This secret is still in the image cache on the build server, so I tend to use this only as a last resort.
Option B: Also during build time, if you can use BuildKit which was released in 18.09, there are currently experimental features to allow the injection of secrets as a volume mount for a single RUN line. That mount does not get written to the image layers, so you can access the secret during build without worrying it will be pushed to a public registry server. The resulting Dockerfile looks like:
# syntax = docker/dockerfile:experimental
FROM python:3
RUN pip install awscli
RUN --mount=type=secret,id=aws,target=/root/.aws/credentials aws s3 cp s3://... ...
And you build it with a command in 18.09 or newer like:
DOCKER_BUILDKIT=1 docker build -t your_image --secret id=aws,src=$HOME/.aws/credentials .
Option C: At runtime on a single node, without Swarm Mode or other orchestration, you can mount the credentials as a read only volume. Access to this credential requires the same access that you would have outside of docker to the same credentials file, so it's no better or worse than the scenario without docker. Most importantly, the contents of this file should not be visible when you inspect the container, view the logs, or push the image to a registry server, since the volume is outside of that in every scenario. This does require that you copy your credentials on the docker host, separate from the deploy of the container. (Note, anyone with the ability to run containers on that host can view your credential since access to the docker API is root on the host and root can view the files of any user. If you don't trust users with root on the host, then don't give them docker API access.)
For a docker run, this looks like:
docker run -v $HOME/.aws/credentials:/home/app/.aws/credentials:ro your_image
Or for a compose file, you'd have:
version: '3'
services:
app:
image: your_image
volumes:
- $HOME/.aws/credentials:/home/app/.aws/credentials:ro
Option D: With orchestration tools like Swarm Mode and Kubernetes, we now have secrets support that's better than a volume. With Swarm Mode, the file is encrypted on the manager filesystem (though the decryption key is often there too, allowing the manager to be restarted without an admin entering a decrypt key). More importantly, the secret is only sent to the workers that need the secret (running a container with that secret), it is only stored in memory on the worker, never disk, and it is injected as a file into the container with a tmpfs mount. Users on the host outside of swarm cannot mount that secret directly into their own container, however, with open access to the docker API, they could extract the secret from a running container on the node, so again, limit who has this access to the API. From compose, this secret injection looks like:
version: '3.7'
secrets:
aws_creds:
external: true
services:
app:
image: your_image
secrets:
- source: aws_creds
target: /home/user/.aws/credentials
uid: '1000'
gid: '1000'
mode: 0700
You turn on swarm mode with docker swarm init for a single node, then follow the directions for adding additional nodes. You can create the secret externally with docker secret create aws_creds $HOME/.aws/credentials. And you deploy the compose file with docker stack deploy -c docker-compose.yml stack_name.
I often version my secrets using a script from: https://github.com/sudo-bmitch/docker-config-update
Option E: Other tools exist to manage secrets, and my favorite is Vault because it gives the ability to create time limited secrets that automatically expire. Every application then gets its own set of tokens to request secrets, and those tokens give them the ability to request those time limited secrets for as long as they can reach the vault server. That reduces the risk if a secret is ever taken out of your network since it will either not work or be quick to expire. The functionality specific to AWS for Vault is documented at https://www.vaultproject.io/docs/secrets/aws/index.html
The best way is to use IAM Role and do not deal with credentials at all. (see http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html )
Credentials could be retrieved from http://169.254.169.254..... Since this is a private ip address, it could be accessible only from EC2 instances.
All modern AWS client libraries "know" how to fetch, refresh and use credentials from there. So in most cases you don't even need to know about it. Just run ec2 with correct IAM role and you good to go.
As an option you can pass them at the runtime as environment variables ( i.e docker run -e AWS_ACCESS_KEY_ID=xyz -e AWS_SECRET_ACCESS_KEY=aaa myimage)
You can access these environment variables by running printenv at the terminal.
Yet another approach is to create temporary read-only volume in docker-compose.yaml. AWS CLI and SDK (like boto3 or AWS SDK for Java etc.) are looking for default profile in ~/.aws/credentials file.
If you want to use other profiles, you just need also to export AWS_PROFILE variable before running docker-compose command.
export AWS_PROFILE=some_other_profile_name
version: '3'
services:
service-name:
image: docker-image-name:latest
environment:
- AWS_PROFILE=${AWS_PROFILE}
volumes:
- ~/.aws/:/root/.aws:ro
In this example, I used root user on docker. If you are using other user, just change /root/.aws to user home directory.
:ro - stands for read-only docker volume
It is very helpful when you have multiple profiles in ~/.aws/credentials file and you are also using MFA. Also helpful when you want to locally test docker-container before deploying it on ECS on which you have IAM Roles, but locally you don't.
Another approach is to pass the keys from the host machine to the docker container. You may add the following lines to the docker-compose file.
services:
web:
build: .
environment:
- AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
- AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
- AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}
The following one-liner works for me even when my credentials are set up by aws-okta or saml2aws:
$ docker run -v$HOME/.aws:/root/.aws:ro \
-e AWS_ACCESS_KEY_ID \
-e AWS_CA_BUNDLE \
-e AWS_CLI_FILE_ENCODING \
-e AWS_CONFIG_FILE \
-e AWS_DEFAULT_OUTPUT \
-e AWS_DEFAULT_REGION \
-e AWS_PAGER \
-e AWS_PROFILE \
-e AWS_ROLE_SESSION_NAME \
-e AWS_SECRET_ACCESS_KEY \
-e AWS_SESSION_TOKEN \
-e AWS_SHARED_CREDENTIALS_FILE \
-e AWS_STS_REGIONAL_ENDPOINTS \
amazon/aws-cli s3 ls
Please note that for advanced use cases you might need to allow rw (read-write) permissions, so omit the ro (read-only) limitation when mounting the .aws volume in -v$HOME/.aws:/root/.aws:ro
Volume mounting is noted in this thread but as of docker-compose v3.2 + you can Bind Mount.
For example, if you have a file named .aws_creds in the root of your project:
In your service for the compose file do this for volumes:
volumes:
# normal volume mount, already shown in thread
- ./.aws_creds:/root/.aws/credentials
# way 2, note this requires docker-compose v 3.2+
- type: bind
source: .aws_creds # from local
target: /root/.aws/credentials # to the container location
Using this idea, you can publicly store your docker images on docker-hub because your aws credentials will not physically be in the image...to have them associated, you must have the correct directory structure locally where the container is started (i.e. pulling from Git)
You could create ~/aws_env_creds containing:
touch ~/aws_env_creds
chmod 777 ~/aws_env_creds
vi ~/aws_env_creds
Add these value (replace the key of yours):
AWS_ACCESS_KEY_ID=AK_FAKE_KEY_88RD3PNY
AWS_SECRET_ACCESS_KEY=BividQsWW_FAKE_KEY_MuB5VAAsQNJtSxQQyDY2C
Press "esc" to save the file.
Run and test the container:
my_service:
build: .
image: my_image
env_file:
- ~/aws_env_creds
If someone still face the same issue after following the instructions mentioned in accepted answer then make sure that you are not passing environment variables from two different sources. In my case I was passing environment variables to docker run via a file and as parameters which was causing the variables passed as parameters show no effect.
So the following command did not work for me:
docker run --env-file ./env.list -e AWS_ACCESS_KEY_ID=ABCD -e AWS_SECRET_ACCESS_KEY=PQRST IMAGE_NAME:v1.0.1
Moving the aws credentials into the mentioned env.list file helped.
for php apache docker the following command works
docker run --rm -d -p 80:80 --name my-apache-php-app -v "$PWD":/var/www/html -v ~/.aws:/.aws --env AWS_PROFILE=mfa php:7.2-apache
Based on some of previous answers, I built my own as follows.
My project structure:
├── Dockerfile
├── code
│ └── main.py
├── credentials
├── docker-compose.yml
└── requirements.txt
My docker-compose.yml file:
version: "3"
services:
app:
build:
context: .
volumes:
- ./credentials:/root/.aws/credentials
- ./code:/home/app
My Docker file:
FROM python:3.8-alpine
RUN pip3 --no-cache-dir install --upgrade awscli
RUN mkdir /app
WORKDIR /home/app
CMD python main.py
I've created a Docker container with an Ubuntu base image. Setting the environment variables through a .env file. When running the container, I can see the variables being passed through using the shell terminal.
I want to able to get the env varibles in my wp-config. I am using getenv but it is not working..
Any suggestions..
Thanks
You can set the environment variable for your docker container in 2 ways
In docker run command use docker run -e VARIABLE=VALUE ...
In docker-compose file you can set in like:
environment:
- DEBUG=1
https://docs.docker.com/compose/environment-variables/#set-environment-variables-in-containers
You use .env file, so you certainly use docker-compose. If not use docker-compose, .env will not make effect. And the .env file must be placed in the directory where docker-compose is run from.
Whole solution could be something like:
.env
MY_VARIABLE=abc
docker-compose.yml
version: '3'
services:
my_service:
environment:
- MY_VARIABLE="${MY_VARIABLE}"
wp-config.php
echo getenv('MY_VARIABLE');
I guess you did not get env because you did not do - MY_VARIABLE="${MY_VARIABLE}" in docker-compose.yml, the value in .env will not be automatically act as an environment variable to container, you need to handle it in compose file. FYI.
Detail refers to offical guide
Both previous answers, I could already pass the env variables to my apache environment served by docker. I just needed to add Pass env_name to the .htaccess file for each env variable.
I could then get the values via the $SERVER['env_name'] within my php application..
I'm using the php:7-fpm Docker image but I cannot put my application in /var/www/html. Instead, I want to put it in /opt/foo. /opt/foo is a volume. How can I do this without replacing the whole PHP-FPM configuration?
PHP-FPM defaults to the working directory, but because the image sets the working directory before it sets the command, you can't customize it with WORKDIR. So the only way to do it neatly seems to be appending to the PHP-FPM configuration file:
RUN echo 'chdir = /opt/foo' >> /usr/local/etc/php-fpm.d/www.conf
Background
I'm trying to use Wercker to run my test for a PHP application. Wercker uses Docker containers to setup a test environment to execute tests in. It uses Environment Variables to expose the connection params for any connected services like MySQL and Elasticsearch. Example MYSQL_PORT_3306_TCP_ADDR = 127.0.1.1
My core Docker containers is running Ubuntu 14.04 with PHP and Apache already installed on the container.
Problem
I can't seem to access the Environment Variables via php $_SERVER or $_ENV when running via Apache. It works fine if I run the script via CLI php ./db_connect.php or if I run PHP using its build in server php -S localhost:8000. However If I try and access a page via the Apache virtual host, the Environment Variables are not available.
Progress
I have setup Apache with the mod used to allow environmental variables "I think"
sudo a2enmod env
sudo service apache2 restart
I'm trying to access the Environment Variables in my script.
$database_host = $_SERVER["MYSQL_PORT_3306_TCP_ADDR"];
$database_username = $_SERVER["MYSQL_ENV_MYSQL_USER"];
$database_password = $_SERVER["MYSQL_ENV_MYSQL_PASSWORD"];
$database_name = $_SERVER["MYSQL_ENV_MYSQL_DATABASE"];
$elasticsearch_host = $_SERVER["ELASTICSEARCH_PORT_9300_TCP_ADDR"];
I can add new variables in my .htaccess, I just don't get all the system environmental variables.
SetEnv TEST_VAR test
I have read this question How to get system environment variables into PHP while running CLI & Apache2Handler? but i'm not sure what its suggesting to do.
Question
How do I expose System Environment Variables to Apache and PHP?
If you are using php official Docker image you have to explicitly pass environment variables from Apache to PHP.
You can do something like this:
In your Dockerfile:
FROM php:7.2-apache
RUN echo 'SetEnv MYSQL_USER ${MYSQL_USER}' > /etc/apache2/conf-enabled/environment.conf
environment.conf is an arbitrary name, but it should be placed in /etc/apache2/conf-enabled/.
In docker-compose.yml:
version: '2'
services:
yourservice:
build: ./yourimage
image: yourimage
ports:
- 8080:80
volumes:
- ../html:/var/www/html
environment:
MYSQL_USER: foo
In your PHP script:
echo getenv('MYSQL_USER');
Here is the solution:
Docker will pass these to apache but you have to configure apache to make them available to PHP.
Setup the values in your local .env file
MYSQL_PORT_3306_TCP_ADDR=1234
MYSQL_ENV_MYSQL_USER=development
MYSQL_ENV_MYSQL_PASSWORD=password
Then add these as environment params in the docker-compose.yml file
version: 2
services:
web:
build: php:5.6-apache
environment:
MYSQL_PORT_3306_TCP_ADDR:${MYSQL_PORT_3306_TCP_ADDR}
MYSQL_ENV_MYSQL_USER: ${MYSQL_ENV_MYSQL_USER}
MYSQL_ENV_MYSQL_PASSWORD: ${MYSQL_ENV_MYSQL_PASSWORD}
Then to pass these to PHP set these as environment params in your Virtual Host config
<VirtualHost *:80>
ServerName some-project
ServerAdmin webmaster#localhost
DocumentRoot /var/www/some-project
# Set apache environment variables
SetEnv MYSQL_PORT_3306_TCP_ADDR ${MYSQL_PORT_3306_TCP_ADDR}
SetEnv MYSQL_ENV_MYSQL_USER ${MYSQL_ENV_MYSQL_USER}
SetEnv MYSQL_ENV_MYSQL_PASSWORD ${MYSQL_ENV_MYSQL_PASSWORD}
</VirtualHost>
These will now be available to access in PHP via the $_SERVER super global array.
<?php
echo $_SERVER['MYSQL_ENV_MYSQL_USER'];
if you're using php-fpm, you can pass env vars from OS to PHP-FPM through clear_env ini setting in a file like /path/to/php/fpm/pool.d/www.conf:
clear_env = no
it works with environment vars set via docker-compose.yml as below:
version: "..."
services:
php-fpm:
image: php:7-fpm
environment:
MY_VAR: my-value
IMPORTANT: Of course, the risk is that your PHP-FPM will get access to all OS env vars. If passing all vars is a problem, you can also pass only specific vars via www.conf or another php ini config file:
; Pass environment variables like LD_LIBRARY_PATH. All $VARIABLEs are taken from
; the current environment.
env[MY_VAR] = $MY_VAR
With docker-compose you can retrieve the operating system's environment variables set with the environment option of the docker-compose.yml file through php's $_ENV variable.
version: 2
services:
web:
build: php:5.6-apache
environment:
MYSQL_USER: "user"
MYSQL_PASSWORD: "passwd"
should give you
$_ENV['MYSQL_USER'] = user
$_ENV['MYSQL_PASSWORD'] = passwd
I'm not sure how Wercker maps environment variables to the containers, but there's this open issue that I think might help
https://github.com/wercker/wercker/issues/63
you can check get env with php command like below
php -r "echo getenv('MYSQL_USER');"
I have an index.php file that is running some script making mysql requests. For obvious security reason, i'd like to remove the mysql credentials (host, password, user, database) from this script and replace them with some apache environment variables.
I tried to create such variables in /etc/apache2/envvars using following line of code : export MYSQL_USER='my_user' and then I intend to get it back using getEnv php function like this : getenv('MYSQL_USER') but this returns nothing.
Any idea ? Thanks !
Four steps :
in /etc/apache2/envvars : export MYVAR='value'
in /etc/apache2/apache2.conf : PassEnv MYVAR
Restart apache
in the php file located wherever apache2 is running: echo $_SERVER['MYVAR'];
In the virtual host for Apache you could use
SetEnv VARIABLE_NAME value
and the get the value of this variable from your PHP code using
$variable = getenv('VARIABLE_NAME');
You can read more on SetEnv here http://httpd.apache.org/docs/2.2/mod/mod_env.html#setenv
If you would like to store passwords in an environment variable, but you don't want other users to be able to read the password or key, you could follow the following steps (tested on Ubuntu 18.04):
Create a file /etc/apache2/conf-available/aws_key.conf
Change the ownership to root sudo chown root:root /etc/apache2/conf-available/aws_key.conf
Change the file permission, so it is only readable for root sudo chmod 600 /etc/apache2/conf-available/aws_key.conf
Put the password or key in the file, e.g. SetEnv AWS_SECRET_ACCESS_KEY abcdefg12345+-987654321
Enable the configuration file sudo a2enconf aws_key
Restart Apache sudo service apache2 restart
This key is accessible in PHP via $_SERVER['AWS_SECRET_ACCESS_KEY'].