Compose: `.env` with `export` Lines No Longer Works

[berwyn]

Description of the issue

I recently upgraded Compose to 1.24.0-rc1 (build 0f3d4dda) and now Compose's automatic parsing of my .env file fails. I keep export statements in my .env file so I can easily source it in addition to using it as a standard .env. In previous versions of Compose, this worked fine and didn't give me any issues, however with this new update I instead get an error about spaces inside a value.

Context information (for bug reports)

Output of docker-compose version

docker-compose version 1.24.0-rc1, build 0f3d4dda

Output of docker version

Docker version 18.09.1, build 4c52b90

Output of docker-compose config
(Make sure to add the relevant -f and other flags)

services:
  db:
    image: mdillon/postgis:10-alpine
    ports:
    - 5432:5432/tcp
    volumes:
    - pgdata:/var/lib/postgresql/data:rw
  elasticsearch:
    environment:
      discovery.type: single-node
    image: docker.elastic.co/elasticsearch/elasticsearch:6.1.3
  memcached:
    image: memcached:1.5.10-alpine
  redis:
    image: redis:4.0.6-alpine
  web:
    build:
      context: /Users/berwyn/dev/<repo>
    depends_on:
    - db
    - elasticsearch
    - memcached
    - redis
    environment:
      DB_HOST: db
      DB_USERNAME: postgres
      FOUNDELASTICSEARCH_URL: elasticsearch
      REDIS_CACHE_HOST: redis
      # Several more removed variables here
    ports:
    - 3000:3000/tcp
    stdin_open: true
    tty: true
    volumes:
    - /Users/berwyn/dev/<repo>:/app:rw
version: '3.0'
volumes:
  pgdata: {}

Steps to reproduce the issue

1. Create a .env file with something like export FOO=1
2. Attempt to run a one-off container docker-compose run --rm service bash
3. Notice the error

Observed result

ERROR: In file ./.env: environment variable name `export FOO` may not contains whitespace.

Expected result

The container runs and FOO is correctly set in it.

Stacktrace / full error message

ERROR: In file ./.env: environment variable name `export FOO` may not contains whitespace.

Additional information

macOS 10.14.3
Docker & Friends installed using Cask (cask install docker-edge)

[esvm]

I'm getting the same error in a similar environment (MAC OS). Some news about this without rollback version?

[rene-s]

My guess is that this is a backwards-incompatible ~bug~fix that has not been announced in advance. Not a new bug.

See https://github.com/docker/compose/blob/master/CHANGELOG.md#bugfixes

The best approach would be to avoid exporting variables in .env files.

edit:typo

[thaJeztah]

Looks related to https://github.com/docker/compose/pull/6403 @shin- @hirochachacha

[shin-]

I don't think that we ever supported this, but moreover, it was certainly never the intent: see our docs which clearly state

Compose expects each line in an env file to be in VAR=VAL format.

[hirochachacha]

Presumably, docker-compose's .env format has not been supporting export syntax since day one.

https://docs.docker.com/compose/env-file/

I also checked the following test case with old docker-compose version:

$ cat .env
VAR2=2
export VAR3=3
$ cat docker-compose.yaml
version: "3"
services:
  env_test:
    image: alpine
    environment:
      - VAR1=1
      - VAR2=${VAR2}
      - VAR3=${VAR3}
    command: env
$ docker-compose up && docker-compose down
WARNING: The VAR3 variable is not set. Defaulting to a blank string.
Creating network "d_default" with the default driver
Creating d_env_test_1 ... done
Attaching to d_env_test_1
env_test_1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
env_test_1  | HOSTNAME=f137dd513f8f
env_test_1  | VAR1=1
env_test_1  | VAR2=2
env_test_1  | VAR3
env_test_1  | HOME=/root
d_env_test_1 exited with code 0
WARNING: The VAR3 variable is not set. Defaulting to a blank string.
Removing d_env_test_1 ... done
Removing network d_default

As you can see, export VAR3=3 line in .env wasn't recognized by docker-compose.

I also tested source .env scenario, but I cannot see nice output:

$ cat .env
VAR2=2
export VAR3=3
$ cat docker-compose.yaml
version: "3"
services:
  env_test:
    image: alpine
    environment:
      - source .env
    command: env
$ docker-compose up && docker-compose down
Creating network "d_default" with the default driver
Creating d_env_test_1 ... done
Attaching to d_env_test_1
env_test_1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
env_test_1  | HOSTNAME=468523d0ef4b
env_test_1  | HOME=/root
d_env_test_1 exited with code 0
Removing d_env_test_1 ... done
Removing network d_default

So, if I understand correctly, this is not a bug.

[morganchristiansson]

Previously this was a warning and these lines were ignored. Now docker-compose fails with an error.

My setup was relying on this behaviour. For now I've downgraded to 1.23.2 which has old behaviour.

[hirochachacha]

I really cannot understand the use-case of the old behavior...
If someone find the misconfiguration, they should fix the configuration.
If you don't want to evaluate some lines, you could comment out these lines.
That makes your intentions clearer.

[morganchristiansson]

My usecase is currently to use default .env file to set COMPOSE_PROJECT_NAME and IMAGE_TAG

The same file is also used by a python application that has a very long JSON array that is split across many lines using single quotes. .env is read from inside the container by python-dotenv which permits much wider syntax.

I don't expect docker-compose to parse these values. The previous behaviour of ignoring them worked fine for me. With new behaviour I'll have to tell docker-compose not to use default .env file.

[hirochachacha]

Thank you for the response.

But I don't think copying .env into the docker image is a best practice.
In that approach, you need to recreate images per environments.
If you cannot apply .env settings on the fly, that's not a kind of envvars.
Perhaps, you can save it as settings.py or something.

You maybe know, a work-around for multi-line support is using base64 encoding.

[berwyn]

I have a much different use-case for this, actually. As per the original comment, I like to source my .env for use with orchestration scripts. Using dotenv libraries from Ruby, Node, .NET, etc are perfectly happy to ignore the export lines and parse the rest, and previously docker-compose was also happy to do this.

Very handy to support both docker-compose up and source .env && scripts/do-some-ci-thing.sh

[morganchristiansson]

@hirochachacha we use .env for application default values. Then actual environment variables take precedence.

Can we get old behaviour of warning of ignored lines back?

[hirochachacha]

I see. In the context of the original change, I'm fine with warning instead of error.
While I still feel those tricks are sort of implementation specific hack, but I have no strong opinion about that.

[jonmchan]

Please note that ruby's dotenv recommends or allows you to store the .env file with the caveat that you do not put any sensitive information in there. Since ruby's dotenv supports combining multiple .env files such as .env.local (which would be .gitignore'd and never committed), using .env file for general default environment variables is totally legitimate for that library.

Reference: https://github.com/bkeepers/dotenv#what-other-env-files-can-i-use

[morganchristiansson]

Also worth noting that while ignoring these lines is acceptable. Preferred behaviour is to support leading export keyword.

Most dotenv libraries allow optional leading export keyword:

dotenv library |
------- | -------
ruby-dotenv | https://github.com/bkeepers/dotenv#usage
python-dotenv | https://github.com/theskumar/python-dotenv#usages
godotenv | https://github.com/joho/godotenv#usage

EDIT: nodejs dotenv libraryies don't seem to allow export keyword?

[teury]

Same mistake when using this new version

ERROR: In file ./.env: environment variable name 'RECAPTCHA_PUBLIC_KEY ' may not contains whitespace.

[based64god]

I was sharing a .env between compose and other, related, scripts in my repo. I needed these values to be exported for some of these scripts to work. Compose was parsing them correctly before, but now I need to keep duplicates of my .env. Feels bad.

[teury]

pip install docker-compose==1.23.2

[hirochachacha]

I'm in favor of https://github.com/docker/compose/issues/6741 - add command line option that can disable loading .env feature.
Then, advanced people can do anything they want while docker-compose and docker can keep their simplicity and compatibility.

[tilo]

Not a fan of breaking people's setup by introducing a not backwards-compatible change, and then declaring it a "feature".
If you don't care about the export, how hard can it be to parse out the key=value pairs when reading the .env file?

[Quintasan]

@hirochachacha I'm in favour of you not breaking anyone's setup and forcing people to change everything about their deployments just because you thought it would be a good idea to error out on lines that you can't/won't parse.

HTML doesn't stop rendering because it encounters an invalid tag and this kind of logic would best apply here.

[esvm]

Is there a way to ignore the env file? Because I have an env file not to docker-compose, but to my app. And when I run docker-compose ... it fails because is trying to use the env file

[rosskarchner]

Just speaking for my team-- we had conventions around using a (bash-formated) .env file before we adopted Compose. The previous behavior, where compose tried to read the file, ignoring lines it didn't understand, was annoying (but acceptable). We haven't upgraded to 1.24.x yet, but this is going to be pretty disruptive.

[Quintasan]

@esvm there should be a flag --skip-env-file once #6850 gets merged.

[morganchristiansson]

Hallelujah! Switching to python-dotenv in #7150 solves my compatibillity issues as that's what my target app is using to parse .env

python-dotenv has multi-line variables which is it's in my requirements https://github.com/bkeepers/dotenv#multi-line-values

edit: It's not in any released version yet.

[philwhln]

Just downloaded Gitter.im https://gitlab.com/gitlab-org/gitter/webapp to test out and totally fails on this issue. Not sure why docker-compose couldn't continue to just ignore the export line 🤷‍♂

[morganchristiansson]

This is fixed in 1.26.0-rc2 and later. 1.26.0 is not released yet.