Compose: Inconsistent env var parsing from docker-compose.yml

[endophage]

If I declare my env vars in bash, I get the following:

10:07 $ export TESTVAR=test
10:08 $ echo $TESTVAR
test
10:08 $ export TESTVAR="test"
10:08 $ echo $TESTVAR
test

However, if I create the following files:
Dockerfile:

FROM ubuntu:latest
COPY ./test.sh /test/test.sh
WORKDIR /test
ENTRYPOINT [ "/test/test.sh" ]

test.sh:

#!/bin/bash
echo "--------------TEST RUN---------------"
echo $TESTVAR

case $TESTVAR in
    test)
        echo "no quotes"
    ;;
esac
echo "------------END TEST RUN-------------"

docker-compose.yml

testthing:
    dockerfile: Dockerfile
    build: .
    environment:
        - TESTVAR="test"

I get the following output:

10:07 $ docker-compose up
Recreating compose_testthing_1
Attaching to compose_testthing_1
testthing_1 | --------------TEST RUN---------------
testthing_1 | "test"
testthing_1 | ------------END TEST RUN-------------
compose_testthing_1 exited with code 0

Changing the environment variable to TESTVAR=test produces:

10:07 $ docker-compose up
Creating compose_testthing_1
Attaching to compose_testthing_1
testthing_1 | --------------TEST RUN---------------
testthing_1 | test
testthing_1 | no quotes
testthing_1 | ------------END TEST RUN-------------
compose_testthing_1 exited with code 0

For the record, docker itself appears to handle the quotes correctly (test:latest is the same Dockerfile above being run directly):

10:11 $ docker run -it --rm -e TESTVAR="test" test:latest 
--------------TEST RUN---------------
test
no quotes
------------END TEST RUN-------------

Also, compose behaves correctly when using the map syntax, i.e. TESTVAR: "test" and TESTVAR: test behave the same way, and consistently with bash (no quotes is printed).

[dnephin]

I think this is working as expected.

In the docker example above, bash is interpreting the quotes for you, so the application never gets "test", it gets tests.

In the compose example using map, yaml is interpreting the quotes for you, so Compose never sees the quotes.

In the list of strings example: TESTVAR="test" is interpreted by yaml as the literal string 'TESTVAR="test"'.

I don't think compose should alter the expected behaviour of yaml, and remove quotes from the string.

[endophage]

@dnephin I agree with your break down, the problem is that the resulting behaviour is as if compose is setting export TESTVAR="\"test\"". It's confusing and unexpected, which is compounded by the fact things just silently don't work as expected (because values don't match up).

If this really is the intended behaviour, it should be documented.

[endophage]

Also, should note that because the list syntax TESTVAR="TEST" is being parsed as a single string, my expectation was even more strongly that the equivalent of export TESTVAR="TEST" or docker run -e TESTVAR="TEST" would occur.

It seems like compose must be doing some internal parsing, splitting the var name and value, then recomposing them in a different format.

[dnephin]

True, TESTVAR=test is split into key/value so that we can handle overriding the value from other sources, and injecting values from the host. However, I don't think that's what is causing the problem. Docker never sees the quotes when you use them from bash.

There are plenty of other cases where this is also the case:

Try sending it to the API directly:

>>> client.create_container('alpine:3.3', command='sh', environment=['TESTVAR="TEST"'])
>>> c['Config']['Env']
[u'TESTVAR="TEST"']

Or from an env_file with docker-cli:

$ echo 'TESTVAR="test"' > env_file
$ docker run -ti --env-file=env_file alpine:3.3 env | grep TESTVAR
TESTVAR="test"

In both cases the double quotes are still there.

Edit: My point is that the quotes are not actually part of the API definition. They're read and handled by bash, so I don't think you can use them outside of bash and expect them to be correct.

It just so happens that yaml strings support the same character, so it will work if you define a string value, but you can't add them in other places and expect them to get stripped.

[endophage]

hmmm, it would at least be good to document. This problem came to light because I was getting complaints about true being ambiguous because YAML treats it as a boolean so I _had_ to wrap it in quotes. For consistency, I just tried to make all my env vars formatted identically, which caused the problems and confusion above.

[cr7pt0gr4ph7]

At first, my opinion was to allow TESTVAR="test", but after viewing the section on "Plain Scalars" in the YAML 1.2 Spec, I'm very much against it, because it interacts badly with YAML's parsing of unquoted strings (which is what TESTVAR="test" is interpreted as).

To illustrate:

# Resulting value is given in comments
# Using dicts
environment:
  PLAIN_STRING: test          # test
  QUOTED_STRING: "test"       # test
  ESCAPED_STRING: "tes\u0074" # test
  WITH_COLON: "test: abc"     # test: abc

# Using lists
environment:
  - PLAIN_STRING=test          # test
  - QUOTED_STRING="test"       # "test"
  - ESCAPED_STRING="tes\u0074" # "tes\u0074" # Note the uninterpreted escapes
  - WITH_COLON="test: abc"     # CRASHES (Yaml Spec says: Plain scalars must never contain the “: ” and “ #” character combinations)

My suggestion would therefore be to output a warning, like it is done for true.

[kojiromike]

This is also unintuitive behavior in env_file. Suppose you want to set PIP_TRUSTED_HOSTS with more than one host. In docker-compose's env_file syntax, you would have to write

PIP_TRUSTED_HOSTS=pip1 pip2

In shell, this would be taken as _export PIP_TRUSTED_HOSTS=pip1 for the command pip2_. Since the shell is already so finicky with its quotes-handling, I beg you not to diverge too much from it in env_file.

[AshleyAitken]

This has caused me grief as well...

[junneyang]

so, what's the solution?

[ernsheong]

My use case was to pass custom parameters to a command, e.g. OPTIONS="--foo --bar", so this is not going to work in an .env file

A stopgap solution might be something like https://stackoverflow.com/questions/9733338/shell-script-remove-first-and-last-quote-from-a-variable where you have to explicitly do some processing when using some variables...

[reverland]

So, whats the proper way if I wanna env file which I can set -a;source env;set +a work in bash still work in deploy? any idea?

just mount the file inside the container and run source directly?

[kamermans]

I'm probably beating a dead horse here, but this has really caused some major confusion in my scenario as well. I've created an example to demonstrate all the possible ways you can screw up environment variables in docker-compose. I'm using it as a reference to get the correct behavior as it is unpredictable in my opinion.

Here's my docker-compose.yml (the command just prints out the values of the env vars):

env-array:
    image: python:3
    volumes:
        - ./test.py:/test.py
    command: /test.py
    environment:
        - "TEST1=this is a normal string"
        - "TEST2=this is a $UNQUOTED string"
        - "TEST3=this is a $QUOTED string"
        - TEST4="this is a $UNQUOTED string"
        - TEST5="this is a $QUOTED string"
        - TEST6=this is a $UNQUOTED string
        - TEST7=this is a $QUOTED string

env-map:
    image: python:3
    volumes:
        - ./test.py:/test.py
    command: /test.py
    environment:
        TEST8: this is a normal string
        TEST9: this is a $UNQUOTED string
        TEST10: this is a $QUOTED string
        TEST11: "this is a $UNQUOTED string"
        TEST12: "this is a $QUOTED string"

and my .env file:

UNQUOTED=unquoted
QUOTED="quoted"

Let's start the betting to see which tests come out with no quotes at all in the values themselves 🎲 🎲

Results:

$ docker-compose run --rm env-array
TEST1: [this is a normal string]
TEST2: [this is a unquoted string]
TEST3: [this is a "quoted" string]
TEST4: ["this is a unquoted string"]
TEST5: ["this is a "quoted" string"]
TEST6: [this is a unquoted string]
TEST7: [this is a "quoted" string]

$ docker-compose run --rm env-map
TEST8: [this is a normal string]
TEST9: [this is a unquoted string]
TEST10: [this is a "quoted" string]
TEST11: [this is a unquoted string]
TEST12: [this is a "quoted" string]

Test 1 and 8 are controls, so they don't count. That leaves only test 2 and 6 from the array syntax, and 9 and 11 from the map syntax:

# Acceptable array syntax:
- "TEST2=this is a $UNQUOTED string"
- TEST6=this is a $UNQUOTED string

# Acceptable map syntax:
TEST9: this is a $UNQUOTED string
TEST11: "this is a $UNQUOTED string"

Particularly troubling was test 5, which ended up with a quoted word inside a quoted sentence, effectively breaking out of the quotes, ala SQL injection.

My biggest complaint is that docker-compose does not allow you to quote values in .env files. This flies in the face of all the dotEnv implementations that I've seen, across multiple languages, and breaks compatibility with my applications, since they can no longer share information with Docker via .env files reliably. For now I'm just going to keep trimming quotes from env vars, but I really think this should be addressed.

[kamermans]

If you would support changing the behavior to clean quotes from values in .env, I'm happy to submit a PR.

I'm not a native Python speaker, but this should do the trick:

(k, v) = env.split('=', 1)
if len(v) >= 2:
    if v[0] == v[-1] and v[0] in ['"', "'"]:
        v = v[1:-1]
return (k, v)

right here:
https://github.com/docker/compose/blob/7ae632a9ee7530fcf81e212baa3e588f477ea862/compose/config/environment.py#L21

[hlarsen]

it appears i'm having problems due to docker-compose keeping quotes as part of the value when using the env_file. considering (as noted previously) most/all other env file implementation do allow quoting of the values, will this be fixed/changed at any point?

per the comments in the thread i can understand (even if i don't agree with) the rationale for not "altering the expected behavior of yaml" , however since we have both environment and env_file, maybe have env_file allow quoting of the values? this would both keep the spirit of the yaml handling and give others an avenue for more proper dotenv handling, though i can see some additional documentation being needed for the difference between them.

this issue has been hanging open for a few years, so if this isn't going to happen, maybe it should get an official response and be closed.

[hardcodet]

.env files simply may _need_ quotes due to special characters in some of the settings, and they work just fine in just about any environment. Given that docker-compose supports external .env files, they should be parsed accordingly - as external files, with their own schema. So docker-componse should be interested in the key/value pairs on those files, and be aware the values may or may not be in quotes, rather than blindly reading them, assuming the files adhere to it's own preferred schema.

Obviously, that's a biased view, but it feels cleaner then asking for docker-compose specific .env files, which just would end up in a mess.

[jensenak]

dotenv is a specific format used by more than just docker-compose. The dotenv format allows quoted values. The expected behavior of dotenv files is bash-like.

So in my opinion, docker-compose should make one of two possible changes:
A) Properly handle dotenv files with quoted values just as source .env works in bash
B) Use a different format that is intentionally distinct from .env, e.g. yaml, toml, ini, or similar.

The current behavior seems totally incorrect and surprising, considering what people expect. I don't think the expectations are unreasonable either, since the env_file is used to set environment variables in a running container.

[kolaente]

Any updates on this?

[kamermans]

I'll put up a pull request that fixes the problem. Does anyone feel strongly about supporting multi-line, quoted values like this?:

FOO="my name is John Smith,
it's nice to meet you"
BAR=something else

I suspect it will be more difficult to convince the maintainers to merge if I include support for that.

[kamermans]

PR added - please show it some love if you want this fixed!

[archfz]

A possible temporary solution is to run this which removes the quotes
sed -r 's/^([^=]+=)\"(.*)\"$/\1\2/g' .env > .env.docker
and just load .env.docker in env file for docker compose till this lands.

[cerw]

Still no solutions?

[nesl247]

@cerw When 1.26 releases, it will have support. The feature was already merged in.

[brandonhill]

Too bad the fix didn't make it into the cli. Same issue using --env-file with docker run.

[archfz]

I am still experiencing this same issue in 1.27. Had to put back the hack of compiling separate docker env file.