Compose: Interpolate Variables set by environment or env_file

[darkn3rd]

Summary

Enhancement request that environment variables set from within the docker-compose.yml file could be used elsewhere.

This would make docker-compose more dynamic instead of current static nature. Right now have to compose some wrapper script to compose docker-compose to compose containers.

Docker-Compose file examples

version: '2'
services:
  database:
    environment:
      - POSTGRES_VERSION="9.4"
    image: "postgres:${POSTGRES_VERSION}"

version: '2'
services:
  web:
    environment:
      - APP_ROOT="/app"
    build: .
    command: ${APP_ROOT}/wrapper.sh
    volumes:
      - ./logs/:${APP_ROOT}/logs
    ports:
      - "8080:8080"

[blankenhaus]

Would https://docs.docker.com/compose/env-file/ not do it ?

[darkn3rd]

For this enhancement request, I would also like the env_file as described the docs (https://docs.docker.com/compose/compose-file/#env-file) to be supported as well, so that they are interpolated in the docker-compose.yml.

The default command-line supported for a static .env doesn't support having multiple types of env files, such as test.env and dev.env. In the interim solution would to have, ENV_FILE_PATH=${RAILS_ENV}.env docker-compose up, such as https://github.com/docker/compose/pull/3399.

Purposefully keeping examples simple (would use something more complex, e.g. local dev mount points, test assets, persistent store links, etc):

db.env

POSTGRES_VERSION="9.4"

docker-compose.yml

version: '2'
services:
  database:
    env_file: 
      - ./db.env
    image: "postgres:${POSTGRES_VERSION}"

Or another example

app.env

APP_ROOT="/app"

docker-compose.yml

version: '2'
services:
  web:
    env_file: 
      - ./app.env
    build: .
    command: ${APP_ROOT}/wrapper.sh
    volumes:
      - ./logs/:${APP_ROOT}/logs
    ports:
      - "8080:8080"

Currently, this outputs that following in 1.7.1:

WARNING: The POSTGRES_VERSION variable is not set. Defaulting to a blank string.

[blankenhaus]

Ok, now I understand what you want. That makes sense to me.
Another way to solve this is to have a default developer .env file inside each of your images. Then in production you map in a different file via the --volume option.

[darkn3rd]

Not sure I understand, as I thought .env was pulled from $PWD and used to launch containers, not within the container image itself. My use case is to support dynamic environments without specifying numerous environment variables prefixed to docker-compose, or using an external tool like ansible or vagrant.

[blankenhaus]

Right, so you can get there by mapping those specific .env files into the container via the -v flag. Then inside your container your startup script always sources in .env.

[cecchisandrone]

+1 for this feature

[darkn3rd]

I have been using .env heavily now. Given this understanding, I would recommend for compose variables that there's a section outside of services: where we can have an area to define variables. This would be the similar functionality of .env introduced in 1.7.0, but just within the docker-compose.yml. These variables would be inherited by any extends. External environment variables would override this behavior.

Thus given above, you'd have:

version: '2'
variables:
  - APP_ROOT="/app"
services:
  web:
    env_file: 
      - ./app.env
    build: .
    command: ${APP_ROOT}/wrapper.sh
    volumes:
      - ./logs/:${APP_ROOT}/logs
    ports:
      - "8080:8080"

I used variables: for this example, but could be whatever is appropriate, such as globals: or compose_env:, whatever makes sense.

[jpooler]

I support fixing the hardcoding of the env_file per: #3399

I did some digging and don't think it would take a ton of effort, but am not familiar enough(or maybe a lack of more complex coding skills) with the codebase to implement.

I was thinking of something along the lines of the logic of the process_service method in compose/config/config.py being called/passing the service_dict to compose/config/environment.py:from_env_file to replace:

env_file = os.path.join(base_dir, 'env')

with

env_file = os.path.join(base_dir, service_dict['env_file'])

Obviously there's a few steps between that,happy to offer help/collaborate with anyone else on this too.

I am also happy to open a new issue, but thought this issue made the most sense to add this to.

[aanand]

While I understand the use cases for such a feature, I don't think it's a good idea to overload env_file with extra meaning. It's supposed to be simply analogous to the docker run --env-file parameter.

[jpooler]

Totally agree, but the behavior I'm currently seeing isn't inline with the docs (I am happy to open a separate issue if you would like). Basically I'm trying to specify a custom ENV file that's not "${PWD}/.env"

https://docs.docker.com/compose/compose-file/#/env-file
Doc example:

env_file:
  - ./common.env
  - ./apps/web.env
  - /opt/secrets.env

Leads me to believe the below _should work_

My compose file:

version: '2'
services:
  APP:
    env_file: './jarrod.env'
    build:
      args:
        NODE_ENV: ${NODE_ENV}
      context: ../../APP
    image: APP:${git_branch}
    ports:
     - "XX:XX"
    environment:
      - NODE_ENV=${NODE_ENV}

my jarrod.env file:

cat jarrod.env
#Note if you have this value specified in your terminal, it will take precedence over this file!!!
NODE_ENV=FAIL
git_branch=FAIL

When I try to build the image:

docker-compose build APP
WARNING: The NODE_ENV variable is not set. Defaulting to a blank string.
WARNING: The git_branch variable is not set. Defaulting to a blank string.
ERROR: Couldn't find env file: /Users/jpooler/Tools/docker_compose/.env

jpooler [~/Tools/docker_compose] (master)$ docker-compose version
docker-compose version 1.8.0-rc2, build c72c966
docker-py version: 1.9.0-rc2
CPython version: 2.7.9
OpenSSL version: OpenSSL 1.0.2h  3 May 2016

 jpooler :[~/Tools/docker_compose] (master)$ docker version
Client:
 Version:      1.12.0-rc2
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   906eacd
 Built:        Fri Jun 17 20:35:33 2016
 OS/Arch:      darwin/amd64
 Experimental: true

Server:
 Version:      1.12.0-rc3
 API version:  1.24
 Go version:   go1.6.2
 Git commit:   91e29e8
 Built:        Sat Jul  2 00:17:11 2016
 OS/Arch:      linux/amd64

[aanand]

@jpooler This is working normally. env_file is analogous to docker run --env-file: all variables defined in any of the specified files are passed through to the container. They are not made available for environment variable interpolation.

We should perhaps make the docs clearer, but this isn't a bug.

[lucaspottersky]

They are not made available for environment variable interpolation.

@aanand you mean you can't use ${XXX} on the docker-compose.yml for variables defined in "env_files"? If that's the case, indeed the docs need to be clearer about that.

[guycalledseven]

I hate wrapper scripts as well, but I don't think that compose yml file is good place to define vars. IMHO that would lead to completely unreadable mix of vars (who is setting or reading from where?).

I have been using .env file heavily and if compose would interpolate env_file.yml with .env/shell vars the same way it does in compose.yml I would be so happy. :) (https://github.com/docker/compose/issues/3934).

Now if ADD/COPY would be able to pipe through something like setenv (https://github.com/subfuzion/envtpl) I wouldn't need any post-scripts at all.

[darkn3rd]

To clear the scope, this is not about docker variables for the Dockerfile, which are already handled. This is for having the .env inside the docker-compose.yml.

In any automation system, you want to have DRY (don't repeat yourself) and also a way to have defaults that can be overridden. Docker Compose added this in 1.7.0 with the .env file for docker-compose (not to be confused with .env files for Docker used by docker-compose). This is just to have only ONE file, so that the variables are at the top.

My initial confusion when writing this up was around the time the feature was released, and the documentation (still likely) was less than adequate. It was easy to confuse the .env file for docker-compose with .env files for Docker engine that Docker-Compose uses. My proposal would be to have a section above services called, compose_env: which can list vars or a file of your choosing.

So TL;DR, don't have two files, a .env and a docker-compose.yml, just have one file docker-compose.yml with docker compose's env incorporated in the same yaml file.

[jpswade]

Workaround solution: Put this in a docker-up.sh file:

#!/usr/bin/env bash
set -a
. db.env
set +a
docker-compose up -d --build

The set -a automatically exports all variables. The . db.env will load the variables. The set +a sets exports back to normal. Then simply docker-compose up as normal.

[egel]

Yeah, the workaround solution from @jpswade works perfectly but is there any progress with providing native reading variables from multiple files by env_files: (as @jpooler commented few post before)?

[jacobrask]

The workaround from @jpswade means you cannot override environment variables though. FOO=bar docker-up.sh will still use FOO from db.env.

[Wirone]

I personally think that docker-compose.yml should NOT define any new env variables, but only read exported ones (system wide) and support env files. Then, with interpolation, it should use them and pass to services. With YAML anchors it's already possible to reuse whole groups of variables.

It's better to separate Docker infrastucture (which compose represents) from config values. It's, for example, much safer because you can copy/paste docker-compose.yml without thinking about clearing sensitive data because everything is outside of this file.

Interpolation itself should work better, but this is totally different story...

[dovry]

A year of no activity on this issue :(

Would absolutely love for this to be a feature. (bump)

[col-panic]

this feature would be really helpful! (bump)

[dlupu]

+1

[ndeloof]

As noted by @aanand https://github.com/docker/compose/issues/3435#issuecomment-232174433, there seem to be some confusion here.
env_file is used to declare the environment file used when creating container from, as a 1:1 mapping of the docker run CLI --env-file option. It isn't involved when parsing the compose file to do any variable substition, so can't be used to customize container definition.

variable substitution as requested here is well supported using an .env file.

[clvLabs]

Just got here _following the rabbit hole_ :sweat_smile:

I would also love to be able to have variables interpolated from an env_file, and I also hate wrapper scripts.

Seeing this issue is closed I'm following the path of @jpswade

Workaround solution: Put this in a docker-up.sh file:

#!/usr/bin/env bash
set -a
. db.env
set +a
docker-compose up -d --build

But, reading @jacobrask 's comment...

The workaround from @jpswade means you cannot override environment variables though. FOO=bar docker-up.sh will still use FOO from db.env.

Made an updated version of the docker-up.sh script and wanted to share it... maybe someone else will follow the _rabbit hole_ and find it :smile:

#!/bin/bash

# Need to manually export env variables
# See: https://github.com/docker/compose/issues/3435

set -a

for assignment in $(cut -s -d= -f1,2 docker.env)
do
  varname=$(echo "$assignment" | cut -s -d= -f1 -)
  dockerenvvalue=$(echo "$assignment" | cut -s -d= -f2 -)
  bashenvvalue=${!varname}
  eval "$varname=\${$varname-$dockerenvvalue}"
done

set +a

docker-compose up --detach

Differences from the original version:

• My env file is named docker.env instead of db.env
• Lines in docker.env not containing an = will be ignored (allow comments)
• This script is a lot more verbose in writing (shorten it if you will, I'll keep it _long_ for my script)
• Default values from the spawning shell are preserved
• Service is not rebuilt on startup (in my case, I'm using a private registry)