Compose: env variables from .env file are not set in docker compose file

[spatchamatla]

ENVIRONMENT VARIABLES from .env are setting to empty string in docker compose file

Context information (for bug reports)

Output of docker-compose version

docker-compose version 1.23.2, build 1110ad01
docker-py version: 3.6.0
CPython version: 3.6.6
OpenSSL version: OpenSSL 1.1.0h  27 Mar 2018

Output of docker version

Client: Docker Engine - Community
 Version:           18.09.2
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        6247962
 Built:             Sun Feb 10 04:12:39 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.2
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       6247962
  Built:            Sun Feb 10 04:13:06 2019
  OS/Arch:          linux/amd64
  Experimental:     true

Output of docker-compose config
(Make sure to add the relevant -f and other flags)

WARNING: The POSTGRES_VERSION variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_USER variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_PASSWORD variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_DB variable is not set. Defaulting to a blank string.
services:
  testpg:
    environment:
      POSTGRES_DB: ''
      POSTGRES_PASSWORD: ''
      POSTGRES_USER: ''
      POSTGRES_VERSION: '9.6'
    expose:
    - 5432
    image: 'testpgimage:'
    ports:
    - published: 5432
      target: 5432
version: '3.7'

Steps to reproduce the issue

postgreServer-test.yml

# To execute this docker-compose yml file use docker-compose -f <file_name> up
# Add the "-d" flag at the end for detached execution
version: "3.7"
services:
    testpg:
        env_file:
          - POSTGRES_SERVER.env
        image: testpgimage:${POSTGRES_VERSION}
        ports:
            - 5432:5432
        environment:
            - POSTGRES_USER=${POSTGRES_USER}
            - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
            - POSTGRES_DB=${POSTGRES_DB}
        expose:
            - 5432

POSTGRES_SERVER.env

POSTGRES_VERSION=9.6
POSTGRES_USER=testuser
POSTGRES_PASSWORD=testPWD
POSTGRES_DB=testdb

Expected result

services:
  testpg:
    environment:
      POSTGRES_VERSION='9.6'
      POSTGRES_USER='testuser'
      POSTGRES_PASSWORD='testPWD'
      POSTGRES_DB='testdb'
    expose:
    - 5432
    image: 'testpgimage:9.6'
    ports:
    - published: 5432
      target: 5432
version: '3.7'

Additional information

OS version / distribution, docker-compose install method, etc.
Mac OS 10.14.3, docker installed as desktop

[ps23]

The following pulls the environment variables into the container that is created. It does NOT substitute the placeholders in the yml file.

env_file:
          - POSTGRES_SERVER.env

Steps to correct your approach

postgreServer-test.yml

version: "3.7"

services:
    testpg:
        image: testpgimage:${POSTGRES_VERSION}
        ports:
            - 5432:5432
        environment:
            - POSTGRES_USER=${POSTGRES_USER}
            - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
            - POSTGRES_DB=${POSTGRES_DB}
        expose:
            - 5432

.env

POSTGRES_VERSION=9.6
POSTGRES_USER=testuser
POSTGRES_PASSWORD=testPWD
POSTGRES_DB=testdb

docker-compose -f postgreServer-test.yml config

services:
  testpg:
    environment:
      POSTGRES_DB: testdb
      POSTGRES_PASSWORD: testPWD
      POSTGRES_USER: testuser
    expose:
    - 5432
    image: testpgimage:9.6
    ports:
    - published: 5432
      target: 5432
version: '3.7'

[spatchamatla]

Thanks for replying @ps23. Can I also know which OS you ran on... I'm running in Mac OS 10.14.3

tried above with .env file which resides in the same folder as .yml file, but still not populating

docker-compose -f postgreServer-test.yml config

WARNING: The POSTGRES_VERSION variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_USER variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_PASSWORD variable is not set. Defaulting to a blank string.
WARNING: The POSTGRES_DB variable is not set. Defaulting to a blank string.
services:
  testpg:
    environment:
      POSTGRES_DB: ''
      POSTGRES_PASSWORD: ''
      POSTGRES_USER: ''
    expose:
    - 5432
    image: 'testpgimage:'
    ports:
    - published: 5432
      target: 5432
version: '3.7'

[stbnrivas]

+1

LSB Version: :core-4.1-amd64:core-4.1-noarch
Distributor ID: Fedora
Description: Fedora release 29 (Twenty Nine)
Release: 29
Codename: TwentyNine

Docker version 18.09.5, build e8ff056

docker-compose version 1.22.0, build f46880f

# .env-bug
VALUE=not-null

#docker-compose.yml
version: "3.7"
services:
  cli:
    image: debian:stretch-slim
    env_file: .env-bug
    environment:
      - VALUE=$VALUE
    command: echo -e "VALUE=$VALUE VALUE=${VALUE}"

it is run with

docker-compose --file docker-compose.yml up

however

docker-compose --file docker-compose.yml config

WARNING: The VALUE variable is not set. Defaulting to a blank string.
services:
cli:
command: echo -e "VALUE= VALUE="
environment:
VALUE: ''
image: debian:stretch-slim
version: '3.7

[s9gf4ult]

I have similar issue. The difference for me is that docker-compose --file docker-compose.yml up does not work either. But the

export VALUE=non-null
docker-compose -f docker-compose.yml up                                                                                                
Recreating yoba_cli_1_9b9c55ff203c ... done
Attaching to yoba_cli_1_9b9c55ff203c
cli_1_9b9c55ff203c | VALUE=non-null VALUE=non-null
yoba_cli_1_9b9c55ff203c exited with code 0

works.

The package versions are:

% docker --version
Docker version 18.09.7, build 2d0083d657f82c47044c8d3948ba434b622fe2fd
% docker-compose --version
docker-compose version 1.23.1, build b02f130

OS:

% nixos-version 
19.03.git.35841f8 (Koi)
# nixpkgs version is 35841f87afa7faaeb3fd0f558f9226ebdae0b954

[funkyfuture]

the question that i'd have to y'all is from which working directory are invoking Compose. because i encounter the same when i have one of the subfolders beside of a Compose project as a working directory.

[ghost]

I encountered the same problem but we figured out that the problem was the makefile that was triggering docker-compose and that was not passing the environment variables.

[ndeloof]

There's an obvious confusion here between:

• .env file which is used for variable substitution in the docker-compose.yaml file, and should resolve when you run docker-compose config
• env_file element in a service definition, which is just setting the environment file to send to docker engine when starting container, as a definition of _runtime_ environment. Compose will not parse this file which is opaque to him.

So there's nothing wrong here.

[Darksoulsong]

So env_file option is not for interpolating variables, too bad. I'm having undefined variables in a situation where the .env file is inside a directory and I call docker-compose -f from the parent.

It seems compose is not aware from the .env files in cases like this, and I thought env_file would help. So, how do I handle this?

[ps23]

source <PATH>/.env
docker-compose up

or

env $(cat <PATH>/.env | xargs) docker-compose up

[Darksoulsong]

@ps23 Thanks for your reply!

However, it seems compose can't read the env variables set. I created a run.sh file at the root of my project and the result is this:

run.sh
#!/bin/bash

source ./db/.env
# env $(cat ./db/.env | xargs)

echo ${CONTAINER_DB_NAME}
echo ${DB_PATH}


docker-compose config

exit 0

and this is the output:

db
./../data/db
WARNING: The CONTAINER_DB_NAME variable is not set. Defaulting to a blank string.
WARNING: The DB_PATH variable is not set. Defaulting to a blank string.
services:
  db:
    container_name: ''
    environment:
      MYSQL_DATABASE: ''
      MYSQL_PASSWORD: ''
      MYSQL_ROOT_PASSWORD: ''
      MYSQL_USER: ''
    image: mariadb:latest
    restart: unless-stopped
    volumes:
    - .:/var/lib/mysql:rw
version: '3.0'

The variable content is printed, but it's not passed down to compose. I'm running this in a mac os, BTW.

[Darksoulsong]

Nevermind, this one did the trick:
eval $(egrep -v '^#' ./db/.env | xargs) docker-compose config

Thanks!

[m-jarmula]

Hmm I had similar issue, as it turned out I called docker-compose up from the service directory 😅 so I guess you should call it from the docker-compose.yml directory level

[mcnesium]

So why is docker-compose not sourcing the .env files relatively to the passed docker-compose.yml? The command env $(cat <PATH>/.env | xargs) does its job but it makes the script less slim and has an imho unnecessary repetition in it, e.g.:

for DIR in "${DIRS[@]}"; do
    env $(cat "$DIR/.env" | xargs) docker-compose -f "$DIR/docker-compose.yml" up -d
done

compared to

for DIR in "${DIRS[@]}"; do
    docker-compose -f "$DIR/docker-compose.yml" up -d
done

I know I can also do

HOMEDIR=$(pwd)
for DIR in "${DIRS[@]}"; do
    cd $DIR
    docker-compose up -d
    cd $HOMEDIR
done

but that feels even worse :disappointed:

[javierguzman]

I have a similar issue; If I have my .env in the parent directory the env_file compose option works; If I have it where my docker-compose.yml lives, it does not work. Of course, I change the path in each scenario but still...

[ps23]

@mcnesium Because you can pass in multiple yml files from different locations on a single compose command

[cmosguy]

Save your self the hassle and use the technique posted here: https://github.com/docker/compose/issues/6581#issuecomment-547080034

[saggzz]

Maybe this sounds stupid and its obvious. But i had this similar issue until i sorted the variables in the .env file and docker-compose.yml in the same order.

[MCardus]

Maybe this sounds stupid and its obvious. But i had this similar issue until i sorted the variables in the .env file and docker-compose.yml in the same order.

To sort the .env variables also fixed the issue for me