Compose: '--env-file' option is ignored

[KlaasH]

Description of the issue

The --env-file option added in PR #6535 doesn't appear to work. The commands accept the argument, but the variables from the specified env file don't make it into the environment.

I tried the option with run and up, both before and after the subcommand. For up it just gets ignored in either position. When I put it after run I get the usage message, but if I add the option into the options list in the run docstring, it accepts the option but ignores it, just like when it's in the other position.

Context information (for bug reports)

Output of docker-compose version

docker-compose version 1.25.0-rc1, build 8552e8e

Output of docker version

Docker version 18.09.5, build e8ff056

Steps to reproduce the issue

This makes a super-simple compose environment using the busybox image, creates default and override .env files, and runs the container to show the bug.

cat <<\EOF > docker-compose.yml
version: '3.7'
services:
  test:
    image: busybox
    env_file: .env.conf
EOF

echo 'WHEREAMI=config' > .env.conf
echo 'WHEREAMI=override' > .env.override

docker-compose --env-file .env.override run --rm test env | grep WHEREAMI

Observed result

WHEREAMI=config

Expected result

WHEREAMI=override

Additional information

If you remove the env_file line from the docker-compose.yml the variable disappears, so it's not a matter of preferring the config, it's just not managing to get the file from --env-file loaded into the environment.

Installation details: Ubuntu 18.04, using system Docker and installing docker-compose with pipenv, both as docker-compose==1.25.0-rc1 and by installing the cloned repo at master (same result).

[chris-crone]

Hi @KlaasH,

Thanks for the report and the reproduction steps. I've been able to reproduce this locally so it's definitely a bug.

[chris-crone]

Debugging a bit, I believe there are two issues:

• We should use toplevel_options here
• In the same function, --env-file flag is never passed to the container

[KlaasH]

After looking at this more, I think I misunderstood the feature. I was focused on the env_file container option, but that's not what --env-file replaces. It wouldn't make sense for a project-level argument to override a service-level setting. This overrides the project-level environment file, i.e. the one that gets loaded into the environment in which the services are initialized.

So my original example wasn't doing a sensible test--.env.conf was forcing the value, regardless of what was in the environment. I should have been looking at something like:

cat <<\EOF > docker-compose.yml
version: '3.7'
services:
  test:
    image: busybox
    env_file: .env.conf
EOF

echo 'WHEREAMI=default' > .env
echo 'WHEREAMI' > .env.conf
echo 'DEFAULT_CONF_LOADED=true' >> .env.conf
echo 'WHEREAMI=override' > .env.override

docker-compose --env-file .env.override run --rm test env

That does load the override--WHEREAMI comes back as override with the option and default without (and DEFAULT_CONF_LOADED gets set either way).

I think there might still be an issue here. At the least it might be worth tweaking the description of the option in the usage text to highlight the distinction. But actually, I'm not currently seeing how this makes sense as an argument to any subcommands. Maybe I'm still misunderstanding things, but it seems like this should be a top-level option only and handled within TopLevelCommand such that subcommands don't have to deal with it.

If it makes sense to close this issue, since it's off base as reported, I could write a new one to capture that idea (assuming it's not also off base...)

[MasterCATZ]

having same issue with docker-compose.yml
it is completely ignoring enviromental changes

change a setting like these , and it just keeps using the first settings you tried WTF

environment:
- MARIADB_HOST=sql.mywesite.com
      - MARIADB_PORT_NUMBER=3306
      - OPENCART_DATABASE_USER=bitnami_opencart
      - OPENCART_DATABASE_NAME=bitnami_opencart_mywebsite
      - OPENCART_DATABASE_PASSWORD=mysupersecret
      - OPENCART_HOST=mywebsite.com
      - ALLOW_EMPTY_PASSWORD=no

docker-compose stop
docker-compose rm
docker system prune

docker images
docker image remove da414fd3b337

if I reboot it will then follow the changes .. what am I missing .. this is getting ridiculous must be a hidden temp file somewhere ...

[ulyssessouza]

@MasterCATZ I didn't understand your issue.
Could you please describe minimum steps to reproduce your issue?

[digitalist]

related bug here:

15:42:10-user~/b/slicer (master)$ ls .env
.env
15:42:22-user~/b/slicer (master)$ ls .env # this is a project file
.env
15:42:32-user~/b/slicer (master)$ ls .env # this is a project file, I can't rename it, it doesn't relate to docker-compose
.env
15:43:00-user~/b/slicer (master)$ cat docker-compose.env # this is an empty file I want to use, I don't need any env vars but those defined in .yml

15:43:45-user~/b/slicer (master)$ docker-compose --env-file=docker-compose.env up # this fails because compose tries to read project-related '.env' file
ERROR: In file ./.env: environment variable name 'YII_DEBUG   ' may not contains whitespace.

[ulyssessouza]

@digitalist Unfortunately as you said, it's not a docker-compose's problem. You have to fix the name of the .env file

[thaJeztah]

@ulyssessouza in the last case (custom --env-file specified), shouldn't the .env file be ignored altogether?

[thaJeztah]

I do notice that --env-file looks to be defined twice; once as top-level option:

https://github.com/docker/compose/blob/1.25.0-rc1/compose/cli/main.py#L279-L280

And onces for docker-compose build / docker-compose down / docker-compose up;

https://github.com/docker/compose/blob/1.25.0-rc1/compose/cli/main.py#L432-L435

https://github.com/docker/compose/blob/1.25.0-rc1/compose/cli/main.py#L492-L495

https://github.com/docker/compose/blob/1.25.0-rc1/compose/cli/main.py#L1067-L1068

Could it be that the flag on build/down/up is overriding the top-level --env-file flag?

[digitalist]

@ulyssessouza in the last case (custom --env-file specified), shouldn't the .env file be ignored altogether?

absolutely impossible. I had to patch pip-version of docker-compose to be able to run this project, and, by the way, .env files often used as... literally dotenv files on *nix system for user's own needs.

[digitalist]

    # compose/cli/main.py:125
    # --env-file gets lost here, handler for 'up' tests for --env-file
    # quick fix with
    # command_options['--env-file'] = options.get('--env-file', None)
    with errors.handle_connection_errors(project.client):
        handler(command, command_options)

[digitalist]

@KlaasH here's why you're doing the right thing with this patch - it fixes two bugs
https://twitter.com/strangeqargo/status/1149427260488933377