Compose: Interpolate environment variables in docker-compose.yml

[aanand]

(I'm creating a fresh issue for this as the old one has accumulated rather a lot of baggage.)

It should be possible to pass in environment variables to the value of any* configuration entry in docker-compose.yml. A lot of people want to do it, it's good for portability and I'm satisfied it's not going to cause chaos.

I have some reckons.

Required variables and optional defaults

It's useful to be able to specify that a variable that _must_ be present in the environment, i.e. that Compose will refuse to run if it isn't. However, this will be a pain when you've got lots of them, so it should either be something you explicitly enable, or it should be possible to specify a default value.

The MVP implementation does not need to have either feature, but there should be a clear path to implementing both in a backwards-compatible way.

Syntax

There's a strong case for implementing an established standard, as long as it's not heavyweight - our requirements for functionality are minimal.

• POSIX parameter expansion is OK. It has a few too many features, but we could implement a subset of them:
  
  
  
  • ${VARIABLE} - outputs empty string if VARIABLE is unset
  
  
  • ${VARIABLE-default} - outputs default if VARIABLE is unset
  
  
  • ${VARIABLE?} - errors out if VARIABLE is unset
  
  

https://github.com/docker/compose/pull/845 implemented a Bash-style ${VARIABLE:default} syntax, which is similar to POSIX parameter expansion but slightly different.

• Python's format syntax would be trivial to implement, but is tied to the Python language, and has rather too many features.

Implementation

Python's os.path.expandvars function implements the most basic case of POSIX parameter expansion:

>>> from os.path import expandvars
>>> expandvars('${HOME}')
'/Users/aanand'

However, it's got at least 2 problems:

1. An unset variable doesn't expand to an empty string - instead, it results in no expansion:

```

expandvars('${UNSET}')
'${UNSET}'
```

1. Malformed syntax doesn't error out - instead, it also results in no expansion:

```

expandvars('${HOME')
'${HOME'
```

So far, https://github.com/docker/compose/pull/845 is the closest we've got, but I'm fundamentally wary of an implementation that relies on regular expressions. Templating is a non-trivial job, and people are going to put all kinds of broken stuff in, so we need something that's robust, strict and errors out with helpful messages. Two important requirements:

• If someone puts in something malformed, Compose will not run.
• It is possible to escape any of the special characters used in the template syntax.

There may well be good Python implementations of Bash-like variable interpolation out there already - if not, creating something standalone would be far preferable to bloating the Compose codebase.

*Actually, are there any configuration keys for which we _shouldn't_ allow interpolation?

[kojiromike]

How far do you want to go with these established UNIX standards? (FWIW, it's not a defacto standard, it's an actual standard.)

As someone who occasionally accidentally tries to use POSIX parameter expansions in Dockerfiles, if they were at all supported in docker-compose.yml it would make me a happy camper.

[aanand]

@kojiromike Hmm, so POSIX parameter expansion is actually what I was going for, but reading over the docs it looks like I misremembered the syntax/semantics.

Edit: I've updated my thoughts on syntax in the description.

[Vad1mo]

I have been following the old thread and we urgently wanted to have this feature. Finally the pain was too big and we created a yaml preprocessor bahs script to substitute variables in POSIX style. it worked fine but eventually we stopped using it, because it had one issue. You have to run the preprocessor first and set all the parameters before you get the final solution. Now we are using the docker yaml extends feature. Because it allows us to checkin the actual configuration and just execute it on the target. We know better what is going to happen.

Even though I was a supporter of docker-compose passing variables, I am now not so sure.

As an ideal solution I would rather see docker extends done right. In a sense that would be a solution that fits both. So what is broken in docker extends? It is basically the fact that you have to write all entries in the inherited file. It is not a merge where you enter only what you want to override.

Look at out actual example and how verbose it is. There are only two lines that matter.

#Common 
elasticsearch:
  image: zinvoice/elasticsearch
  hostname: elasticsearch
  restart: always
  dns: 172.17.42.1
  ports:
    - "9200:9200"
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro
    - /data/elasticsearch:/opt/elasticsearch/data/elasticsearch

logstash:
  image: zinvoice/logstash
  hostname: logstash
  dns: 172.17.42.1
  restart: always
  ports:
    - "5000:5000"
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro

kibana:
  image: zinvoice/kibana
  hostname: kibana
  dns: 172.17.42.1
  restart: always
  ports:
    - "5601:5601"
  volumes:
    - /etc/localtime:/etc/localtime:ro
    - /etc/timezone:/etc/timezone:ro

logspout:
  image: zinvoice/logspout
  hostname: logspout
  command: logstash://logstash.docker:5000
  restart: always
  dns: 172.17.42.1
  ports:
    - "8003:8000"
  volumes:
    - /var/run/docker.sock:/tmp/docker.sock

doorman:
  image: zinvoice/doorman
  hostname: doorman
  restart:  always
  dns: 172.17.42.1
  ports:
    - "8085:8085"

# inherited
elasticsearch:
  extends:
    file: ../common.yml
    service: elasticsearch

logstash:
  extends:
    file: ../common.yml
    service: logstash

kibana:
  extends:
    file: ../common.yml
    service: kibana

logspout:
  extends:
    file: ../common.yml
    service: logspout

doorman:
  environment:
    - DOORMAN_GITHUB_APPID=xxxxxxxx
    - DOORMAN_GITHUB_APPSECRET=xxxxxx
  links:
    - nginxtrusted
  extends:
    file: ../common.yml
    service: doorman

So my recommendation fix docker extends an make it less verbose. You don't even have to write that much code as YAML provides all the functionality you need. If you stick with standard YAML the file could be analyzed or created by other tools and UIs.

Take a look at YAML "node anchors" and YAML "file merge" it might be the perfect solution.

FYI: this discussion continuous now on #1380

[aanand]

@Vad1mo I agree that extends falls short in your case. There are a lot of things we can do to improve that experience - could you open a separate issue for it, so we don't get sidetracked here?

[Vad1mo]

Of course! I just wanted to highlight that this could be an easy and elegant alternative.
If compose extends gets you half the way too variable passing, then an improved compose-extends will make variable passing obsolete. Having less concepts to understand make easier for the user.

[mattes]

My use case is to allow $PWD in volumes, so every developer in the team can clone a repo to wherever and paths still get mounted correctly.

elasticsearch:
  image: zinvoice/elasticsearch
  volumes:
    - $PWD:/app

[dnephin]

@mattes I believe that is already supported, I think .:/app is supported as well

[kojiromike]

@aanand As a PoC I did a dirty hackup of POSIX PE in Python. For the Saturdays.

[aanand]

@kojiromike Looks great. Let me know if you plan to continue on it.

[kojiromike]

@aanand I intend to, but it definitely has a few bugs right now (and I think it may have been a bad idea to use shlex). Bug reports and PRs are welcome, of course.

[nafg]

@dnephin how about $HOME / ~?

[dnephin]

@nafg Both of those are supported for the host path of a volume

[nafg]

@dnephin interesting, b/c somehow I ended up with a directory named '$HOME'...

[andrerom]

@aanand Like the "${VARIABLE:default}" proposal, with global_extends (or "import") this would become rather powerful.

Q: Would this allow to specify port number that is exposed to host? like - "${WEB_PORT:80}:80"?
Use case is to be able to easily spin up several instances of an app on same machine/cluster, typically listening to different ports or assigned to different local domain names.

[aanand]

Yes, you'd be able to do that.

[alexanderilyin]

I'd like to use vars in volumes together with docker-compose scale my_app=3. I have this docker-compose.yml

server:
  image: alexanderilyin/docker-teamcity-server
  ports:
   - "8111:8111"
  volumes:
    - .TeamCity:/root/.BuildServer
  links:
   - mysql
mysql:
  image: alexanderilyin/docker-mysql
  volumes:
    - .MySQL:/var/lib/mysql
  environment:
    MYSQL_DATABASE: teamcity
    MYSQL_USER: teamcity
    MYSQL_PASSWORD: teamcity
    MYSQL_ALLOW_EMPTY_PASSWORD: yes
agent:
  image: alexanderilyin/docker-teamcity-agent
  links:
   - server

And I want to be able use scale for agents and use dynamical volumes for them to keep data between launches, for example:

agent:
  image: alexanderilyin/docker-teamcity-agent
  volumes:
    - .agent_{$AGENT_INSTANCE_ID}:/opt/buildAgent
  links:
   - server

[Maxim-Filimonov]

I hope it would be possible to interpolate variables as part of image name too
We are using https://github.com/openshift/source-to-image to build local container on CI for every branch and then run tests on it using docker-compose.
Running tests with dynamic image is quite complicated with docker-compose and requires manual template rendering.. :-1:

[andrerom]

But you can set COMPOSE_PROJECT_NAME to control prefix per run to be able to do that already right? If so, no need to have complex logic and unreadable yml files around names.

[Maxim-Filimonov]

@andrerom don't follow. According to docs that controls the following Sets the project name, which is prepended to the name of every container started by Compose while we are trying to set an image property instead:

web:
  image: <I_AM_DYNAMIC>

[andrerom]

ah, my mistake.

Thought you meant

<I_AM_DYNAMIC>:
  image: nginx

Dynamic image (and build) reference would indeed make a lot of sense. For instance switching between debug and non debug base containers for your programming language for instance would be a good use case for this.

[andrerom]

Additional use case _(which might be what @Maxim-Filimonov has in mind)_: Being able to override which tag to use of an image, so you can use :latest by default, but change to easily test something else without changing yml file _(needed for CI use cases basically)_.

[Maxim-Filimonov]

@andrerom that is exactly our use case :+1:

[ghost]

Will this also work for things like??

web:
  environment:
    - FOO=${whoami}

[thaJeztah]

@k0377 I don't think they will, because that's really something that's handled by the shell, but you could add the result in an environment variable and use that.

In this particular case, the $USER environment variable will probably give you the same.

[askoretskiy]

@aanand Why not use any of existing template engines that are already present? Jinja2 is there and works fine.

As mentioned before -- implementing our own templating is non-trivial task (and regexps are not that cool) so that we should use already existing one, that proven to be solid.

[askoretskiy]

Alternatively we might use YAML ancors and references https://gist.github.com/bowsersenior/979804

But then we are limited on variables usage (inject variable name into middle of content).

[DoubleMalt]

+1 for Jinja2: it would certainly fit the mold and ansible uses it for
exactly that use case (templating in yml files)

On Tue, May 26, 2015 at 1:25 PM, tonnzor [email protected] wrote:

@aanand https://github.com/aanand Why not use any of existing template
engines that are already present? Jinja2 is there and works fine.

As mentioned before -- implementing our own templating is non-trivial task
(and regexps are not that cool) so that we should use already existing one,
that proven to be solid.

—
Reply to this email directly or view it on GitHub
https://github.com/docker/compose/issues/1377#issuecomment-105493447.

[aanand]

Jinja2 does a _lot_ more than we need:

• conditionals
• looping
• extension/inheritance
• comments
• filters

We're not adding any of that stuff to Compose. If Jinja2 can be configured to just interpolate variables, then it might be a candidate.

[DoubleMalt]

Actually looping might be interesting.

Assume you have a list of customers that you want to start containers for
where you put some customer specific variables into the environment.

Extension/Inheritance might be interesting to enhance the current
rudimentary extension mechanism.

Filters can be great to do something with existing variables.

On Tue, May 26, 2015 at 1:56 PM, Aanand Prasad [email protected]
wrote:

Jinja2 does a _lot_ more than we need:

• conditionals
• looping
• extension/inheritance
• comments
• filters

We're not adding any of that stuff to Compose. If Jinja2 can be configured
to just interpolate variables, then it might be a candidate.

—
Reply to this email directly or view it on GitHub
https://github.com/docker/compose/issues/1377#issuecomment-105498909.

[aanand]

They _might_ be interesting features, but they come with far more complexity than I'm comfortable introducing to both Compose and the file format, and we'd be tying both to a specific templating language with (as far as I know) a single implementation and no spec. It's simply not feasible.

[askoretskiy]

@aanand Some notes here:

1. Jinja2 is solid and it takes minutes to do pre-processing of YAML with it:

from jinja2 import Template
template = Template('Hello {{ name }}!')
template.render(name="Aanand")
Hello Aanand!

If you want more security -- you could use immutable sandbox:

from jinja2.sandbox import ImmutableSandboxedEnvironment
env = ImmutableSandboxedEnvironment()
template = env.from_string('Hello {{ name }}!')
template.render(name="Aanand")
Hello Aanand!

In our case it would be:

import os
from jinja2.sandbox import ImmutableSandboxedEnvironment
env = ImmutableSandboxedEnvironment()
template = env.from_string('Hello {{ name }}!')
template.render(**os.environ)

1. Don't we want filters? With filter you could define default value easily ( {{ value|default("default") }} )
2. Do we really need to care about users that use extended Jinja features to screw YAML file? In the same manner user could produce invalid YAML file manually. I think we should keep it simple -- try to process given Jinja template and return error if there was an error or produced YAML is invalid (the same as you do now).
3. If you don't see Jinja2 as solution -- it would be great at least to use {{ variable }} as syntax.
4. Django uses regexp to parse and generate template. It is production-grade for a long time and lives fine with it.

import os
import re
template = "Hello {{ name }}!"
re.sub("{{\s_([a-zA-Z0-9_]+?)\s_}}", lambda m: os.environ.get(m.group(1), ''), template)

In any case -- we need to have this feature rolling, whatever solution we take.

[thaJeztah]

I'm +1 on using a generic templating solution if templates are considered. E.g. http://mustache.github.io, which is available in many languages. This is just an example, other templating engines can be considered equally

[DoubleMalt]

@aanand I totally understand your point. I also like the simplicity and
succinctness of the compose dsl.

Maybe this should be done as an external project, say meta-composer. It
takes a compose.tpl.yml and a variables.yml, creates a docker-compose.yml
and off we go.
As @tonnzor showed t could be done with a tiny piece of python code.

This would deliver powerful templating to those who need it without
introducing complexity for simple tasks.

On Tue, May 26, 2015 at 4:52 PM, Sebastiaan van Stijn <
[email protected]> wrote:

I'm +1 on using a _generic_ templating solution if templates are
considered. E.g. http://mustache.github.io, which is available in many
languages. This is just an example, other templating engines can be
considered equally

—
Reply to this email directly or view it on GitHub
https://github.com/docker/compose/issues/1377#issuecomment-105551631.

[kojiromike]

Hmm… So is the proposal now to use a templating language inside compose.yml (which is a descriptive language for composing Docker containers), for things like command and entrypoint, which already accept both exec and sh -c style values? This could be confusing, since after template rendering the resultant shell command would still presumably be interpreted, so if a variable happened to expand to * it would further be glob expanded. Escaping sequences in one language or another becomes tricky when you have this many layers of fall-through interpretation.

[thaJeztah]

@kojiromike I'm not sure a templating engine is wanted, but if it's to be used! better use something well known. The basic question is; should docker-compose write the substitution from-scratch, or use something existing.

[nafg]

On Tue, May 26, 2015, 11:02 AM Christoph Witzany [email protected]
wrote:

@aanand I totally understand your point. I also like the simplicity and
succinctness of the compose dsl.

Maybe this should be done as an external project, say meta-composer. It
takes a compose.tpl.yml and a variables.yml, creates a docker-compose.yml
and off we go.

You can do that today without any new project. I'm sure jinja can be
invoked from the command line somehow. Personally I just use the envsubst
command.

What would be really helpful is if compose could read the file from stdin.
That would eliminate the need for an intermediate file.

As @tonnzor showed t could be done with a tiny piece of python code.

This would deliver powerful templating to those who need it without
introducing complexity for simple tasks.

On Tue, May 26, 2015 at 4:52 PM, Sebastiaan van Stijn <
[email protected]> wrote:

I'm +1 on using a _generic_ templating solution if templates are
considered. E.g. http://mustache.github.io, which is available in many
languages. This is just an example, other templating engines can be
considered equally

Reply to this email directly or view it on GitHub
https://github.com/docker/compose/issues/1377#issuecomment-105551631.

Reply to this email directly or view it on GitHub
https://github.com/docker/compose/issues/1377#issuecomment-105554730. src="
https://ci6.googleusercontent.com/proxy/iSBXyl7D8PwFM4p1mGPHCR7bQctunieGbhyGkvo0QIMIjmAYE3I0Mt96yl1fGrqcuOzxV4APP8ZRIw-5_qd6nzps9Mpr6jTAydCC4xs8JDgqm93aIbWvN1eMlxykrz7iwYooyAQdqL4RFJokeEbnBkZm5mhgKg=s0-d-e1-ft#https://github.com/notifications/beacon/AAGAUO8xqz29B2SUoG7QFPUy848_JJW9ks5oNIJlgaJpZM4EMysO.gif
">

[kevana]

+1 for reading the file from stdin. I have no problem with using an external template solution but not having intermediate files around would be nice.

[dnephin]

That sounds like a great first step, and a common feature of many cli tools. Let's do that

[Maxim-Filimonov]

:+1:

[kojiromike]

So for example

envsubst compose.tmpl.yml | docker-compose -f - up -d

wfm. :+1:

[thaJeztah]

Just noticed that docker/distribution handles overriding values in the yml file via environment variables, but using a different approach https://github.com/docker/distribution/blob/master/docs/configuration.md#override-configuration-options

^^ @aanand

[Maxim-Filimonov]

@thaJeztah that would work for us too. We can use environment variables to override commands then

DOCKER_COMPOSE_IMAGE_NAME='my_image:is_dynamic'

[aanand]

Interesting approach, but I'm not a fan - verbose environment variable names, lots of duplication if you want to use one value in multiple places, everything's implicit, no interpolation within strings.

[thaJeztah]

@aanand not really sold on that approach either, but wanted to point to it because its another project within the "Docker" organization.

[olalonde]

Just stumbled on https://github.com/kelseyhightower/confd which might be of interest. It uses http://golang.org/pkg/text/template/#pkg-overview

[thaJeztah]

@olalonde unfortunately, docker-compose is written in Python, so Go-templates won't work.

[andrerom]

@aanand I'm +20 on your original proposal, with the tweak that even images and especially tags should be possible to inject. Just go for it, would save all of us a lot of wrappers and unneeded config handling ;)

[DoubleMalt]

I wrote a little python package that helps me with that. The plan is to tunnel all commands through to docker compose so you can use it equivalently.
Check it out at https://github.com/webcrofting/meta-compose/

[hadim]

meta-compose looks really nice. It should be integrated to docker-compose !

[smari]

Big +1 here -- I'm not excited about template preprocessing, but pulling environment variables one way or another would be great. POSIX expansion is probably cleaner than Jinja2, but either way is fine.

[keshy]

Big +1 from here too. My use case is more for adding dynamic advertising ID support for kafka container which is vital for producers of data (which could be other containers).

[jake-low]

I'm also excited about this feature.

POSIX expansion is probably cleaner than Jinja2, but either way is fine.

I think another argument in favor of POSIX expansion is that it's logic-less. Jinja2 supports some degree of conditional / loop logic (as most templating engines do, even the ones that claim to be "logic-less"). Mixing templating logic and YAML is pretty weird in my experience. Can someone think of a use case for such logic? If not, it might be best to specifically avoid support for now.

[hadim]

It would be nice to have a clear answer from devs about this feature. Reading various issues and PR, it's not clear if you really want to implement it or not.

If you do, with what kind of mechanism ? If you don't, people could start building some third party tools to manage the feature.

Thank you !

[hadim]

Ok I just saw https://github.com/docker/compose/pull/76. I guess the answer is there...

[peter-leonov]

Walked a few cycles over the linked issues/PRs.

AFAIK, nginx community refused to adopt any template engine for config files, even for a simple variable substitution. Why? Maybe, they are still choosing an ideal template engine :smile:. Result? Pain (relatively)!

[aanand]

@hadim

Reading various issues and PR, it's not clear if you really want to implement it or not.

This issue was supposed to provide the definitive answer to that question, so I'm sorry I failed to be clear: yes, we want to implement it, and with a POSIX-style syntax.

[hadim]

thank you @aanand !

[peter-leonov]

Thanks, @aanand.

[gtmtech]

+1 for me. I need to pass --dns=(address of the docker0 bridge), and I need it to work if that ever changes in future versions of docker, so an environment variable and/or shell is perfect. meta-compose doesnt work for me, as it should support remote DOCKER_HOST and e.g. via docker-swarm, not just locally.

[lxe]

:+1: This would be very nice. Currently I either end up generating the .yml file via another script, or just not using docker-compose altogether, and manually --link-ing dockers.

[Jasperswaagman]

:thumbsup:

[iadknet]

I think basic environment variable interpolation would be very useful for simple things.

A couple of simple use-cases:

• Being able to dynamically specify the tag of an image to pull from a remote repository.
• Being able to set the port mapping for a container.

Tools like Ansible already do templating very well, so I'm not sure that a full template engine is needed. But not being able to have any dynamic content in the comose.yml file is very limiting.

[kadishmal]

Regarding merged PR #1488, I am particularly interested in piping a config file to docker-compose. I can't understand why docker-compose cannot pick up from a node process.

var spawn = require('child_process').spawn;

var compose = spawn('docker-compose', ['--file' + '-' + 'up']);

compose.stdin.setEncoding = 'utf-8';

compose.stdout.on('data', function (data) {
    console.log('"docker-compose --file - up" stdout: "%s".', data);
});

compose.stderr.on('data', function (data) {
    console.log('"docker-compose --file - up" returned an error: "%s".', data);
});

compose.on('close', function (code) {
    if (code !== 0) {
        console.log('"docker-compose --file - up" existed with an erroneous code: "%s".', code);
    } else {
        console.log('"docker-compose --file - up" existed with code: "%s". SUCCESS!', code);
    }
});

compose.stdin.write("redis: {\"image\": \"redis\"}\n");
compose.stdin.end();

Any examples on how to pipe a data from Node.js?

[kadishmal]

Another thing I've found is that docker-compose 1.4.0-RC1 is sending some seemingly normal messages like Starting... or Attaching... to the stderr instead of stdout.

[aanand]

@kadishmal Could you open separate issues for these please?

[aanand]

Another candidate syntax/implementation: Python's string templating, as specified in PEP 0292 and implemented in string.Template.

It's very similar to POSIX parameter expansion:

• $foo expands to the value of foo
• ${foo} expands to the value of foo
• $, ${, $}, ${}, ${foo, $ {foo}, ${ foo}, ${foo } are errors

Drawbacks:

• No default value or "required value" syntax. Still, we could decide later if we want those enough to merit writing our own templating code.
• The error messages don't expose any machine-readable information about where the syntax error is (without performing regex matching on the error string, that is).
• The escape syntax differs from POSIX: $$ instead of \$.

This might actually be a blessing in disguise: YAML doesn't like \$, and requires you to double-escape. I don't think telling people to type \\$ just to get a dollar sign is going to fly.

I've spiked out an implementation in #1765.

[adamkdean]

+1

[fazy]

I'm not sure if this is the right place for this, or if I should make a new issue.

I think the env precedence should be the other way around, i.e. a variable from the shell that invokes docker-compose should override any variable within docker-compose.yml, which in turn should override any variable defined by the container.

Here's what currently happens when I try it:

docker-compose.yml:

test:
    image: ubuntu
    environment:
        - FOO="from compose"

and then run it with the env command:

docker-compose run test env | grep FOO

gives FOO="from compose", as expected. But then:

FOO="from shell" docker-compose run test env | grep FOO

also gives FOO="from compose", but here I was expecting FOO="from shell".

Some people might still also need variable interpolation for other use cases, but changing this would satisfy the "default" case - effectively the environment: definition/value in docker-compose.yml is the default, and it can be overridden at runtime if needed, without the need for any extra YAML syntax.

[smileart]

@fazy you didn't take into account that the env command was executed in the isolated test container in which FOO's value is from compose (just as it should be and as it was configured in a docker-compose file). But outside of that container if docker-compose process had some kind of a print function for the environment variable you setup before the command it would have printed 'from shell' cause it is the value for the host (as well as for the docker-compose process) you are running it on. Maybe you were expecting the FOO value to be from shell in this case but personally I would be pretty much surprised if it was. (I hope despite my English you will understand my point).

[fazy]

@smileart thanks, I think I see what you're saying.

However, the test container is not completely isolated, it gets its environment from docker-compose (or at least, docker-compose is able to set environment variables into the launched container), and docker-compose itself can see the "outer" environment variable.

You can see with this docker-compose.yml:

test:
    image: ubuntu
    environment:
        - FOO

Then the command:

FOO="from shell" docker-compose run test env | grep FOO

does indeed give the "from shell" result.

So my question is about precedence. By only specifying the variable name here, - FOO, I can inject the variable from outside. But if I specify - FOO=something _and_ inject a variable from outside, which should take precedence? IMHO the variable specified on the command line should take precedence over the config file.

[smileart]

@fazy Oh, sorry, I haven't tried FOO="from shell" docker-compose run test env | grep FOO without specifying its value in the docker-compose.yml and I didn't know it gives us host's FOO value. So it not just would be it's already queer for me :smiley: I thought setting up an environment variable before docker-compose will influence on docker-compose and docker-compose ONLY without throwing it into the container. Now I see what did you mean.

[ct-clearhaus]

I just stumbled upon the drawback of escaping $ mentioned in https://github.com/docker/compose/issues/1377#issuecomment-124571722. First I did just FOO=ba$e then FOO='ba$e' (forgetting that it's taken "bare"), then FOO=ba\$e, then FOO=ba\\$e, then I gave up and went to the docs, just to be surprised to find that "$ is the escape char for $". To me, this wasn't particularly "least surprise-ish".

However, I don't know what the good solution would be.

[albers]

@ct-clearhaus Compose is not the only program that uses $ for escaping $. You will also find this in makefiles. So for some people this idiom is quite familiar.

[nottrobin]

I love the existing variable substitution implementation. However I really could use the ability to set a default, as per @aanand's original proposal. I think POSIX syntax is perfect:

${ENV-default}

My specific usage is that I want to be able to specify the host port that the service runs on:

PORT=8123 docker-compose up

By adding this to my docker-compose.yml:

web:
  ports:
    - "${PORT-8000}:5000"

Is this feature still the plan and in the pipeline?

I tried solving my problem with extends, but it got pretty messy. Not only did I have to duplicate almost all of my docker-compose.yml just to change one setting, there's also no way to _change_ an exposed port setting, you can only add to the list of ports exposed, which isn't ideal for me.

[dkirrane]

Why does docker-compose not fail when environment variables are not set? It just logs a warning and continues. Wouldn't just returning and error be a better approach...
WARNING: The FOO variable is not set. Defaulting to a blank string.

[hugochinchilla]

+1 for POSIX syntax for declaring default values

[mgor]

Maybe I'm missing something obvious, but I'd like to be able to use environment variables from an env_file to set the value of an environment variable, e.g.:

docker-compose.env:

DB_PASSWORD=test

docker-compose.yaml:

...
service:
    database:
        env_file:
            - ./docker-compose.env
        environment:
            - MYSQL_PASSWORD=${DB_PASSWORD}
    webserver:
        env_file:
            - ./docker-compose.env
        environment:
            - WORDPRESS_DB_PASSWORD=${DB_PASSWORD}

Could it be accomplished in some other way? I don't want to have a yaml template file that needs to to be piped through envsubst.

[dnephin]

Why not just put this value directly in the env_file the way you want it?

2636 will support an env file for default values

[mgor]

That would mean having a variable that must be the same value in two places, makes it easier if you only need to change one. #2636 looks promising.

[darkn3rd]

Desperately need mechanism to support default variables now, as existing limitations force us to use wrapper scripts to help docker-compose. I need stuff like NODE_ENV=${NODE_ENV:-dev} to work, and for convenience would be nice to have SOME_NUMBER=$((96*60)) to work. Was this slated for an upcoming version?

[koliyo]

+1 for default values

[composer22]

+1 for default values. this is becoming critical for us.

[metal3d]

I agree @darkn3rd - I need to get user id and user group id to setup them in container. The only way I found is to force my team to export 2 vars... or to use a makefile I made to export set them.

If only I can do:

    user: $((id -u)):$((id -g))

that will resolve my entire problems

[OJFord]

@mgor Sounds like you could just pass it through envsubst?

env $(cat docker-compose.env | xargs) envsubst < docker-compose.tmpl > docker-compose.yml

should do it (without polluting the persisting environment), I think.

[goliatone]

@OJFord @mgor No intention to highjack the thread but I built a couple CLI tools to have a cleaner workflow; envset and slv.

envset development -- slv docker-compose.tpl > docker-compose.yml

envset will load variables from an env file into the current shell session, slv does template replace using environmental variables.

[metal3d]

I agree @OJFord but that's not what I need...
Let me precise: we are a team of 40 developpers that uses different docker-compose stack. We are using git to get code.

If I ask them to modify docker-compose.yml that is delivered by our git, so I'm sure that someone will push a modified docker-compose.yml file... Trust me, that will be the case.

I can "generate a base composer file" that is ignored by git and extended by docker-compose.yml, but to generate it I will need to give 'em a Makefile or a bashscript... The day will come when a "base docker file" change will be needed, and the team will not be aware that they will need to re-run the generation.

Same for a "env" file, that is very nice, but it does'nt work with "build" and I need to ask my team to generate this file.

Really, if docker-compose can get values from bash (or any other solution that returns something else that an ENV var) in the yaml file will resolve a lot of needs.

My example in my previous command is a perfect one: I need to get user id and gid and thoses values are not set by ENV vars. So I need to ask my team to write their IDs in a .env file... Simple for me and you, not for all.

[metal3d]

To precise: I need to give a docker-compose file that should not be changed by the team, because it's on a git repository.

[metal3d]

This pull-request is a simple example that works for me. Maybe you can help me to do better.

I tried with environment and user directives from docker-compose.yml file. Works well for now.

[marcellodesales]

The default value should be there... Very useful... Developers and OPS are using either the docker logs or syslog... So, They usually have to create the default LOG_FORMAT shown below... We could just have the default value and only use it when switching to syslog...

default:
  extends:
    file: base.yml
    service: base-${LOG_FORMAT:docker}
  labels:
    - "net.company.npmjs.datacenter=${DATA_CENTER}"
    - "net.company.npmjs.env=${ENV}"
    - "net.company.npmjs.hostname=${HOSTNAME}"
    - "net.company.npmjs.role=${NPMO_ROLE}"
    - "net.company.npmjs.log=${LOG_FORMAT}"

base-syslog:
  log_driver: syslog
  log_opt:
    tag: "{{.ImageName}}/{{.Name}}/{{.ID}}"

base-docker:
  log_driver: json-file
  log_opt:
    max-size: "128m"
    max-file: "4"

When will this be available? I'm on Compose 1.7.0 and this is still not there :(

[nottrobin]

Please please please give us default values!

[pataquets]

@marcellodesales: Perhaps you can take advantage of using a docker-compose.override.yml file in some way. Check that feature.

Also +1 on env vars. It is our major pain point with docker-compose nowadays.

[metal3d]

I would insist on my PR #3367 to be able to get certain values from host. :)

[marcellodesales]

@pataquets I don' think I want to create yet other override file... our base.yml file, as shown above, shows all the supported stuff in terms of logger driver, etc... I just want to switch and have default values. I would think that we would need to maintain yet more yml files. But I will keep that in mind just in case.

[zerthimon]

+1

[trippel]

+1

[gbmeuk]

+1

[panga]

+1

[alburena]

Any notice about use environment variables in docker-compose?

+1

[ChrisCooney]

+1

[darkn3rd]

FYI: Environment vars for docker-compose works as of 1.7.0. You can also set docker-compose default variables in .env in the same directory as your root docker-compose.yml file. This is not to be confused with docker engine envfiles, as that's a different thing.

[LouWii]

Is there a way to set the service name as a variable ?

Instead of writing this

services:
   site_db:
     image: mysql:5.7

We could write

services:
   ${CONTAINER_NAME}:
     image: mysql:5.7

My goal is to keep the same docker-compose.yml across multiple sites, and only change the .env file. Right now, I still need to modify the container name because I'm running several apps on the same host. And I'd like to have each service to have its own name for clarity.

[thaJeztah]

@LouWii you can use

services:
    site_db:
      container_name: "${CONTAINER_NAME}"
      image: mysql:5.7

or (compose-file format 2.1 and up)

services:
    site_db:
      container_name: "${CONTAINER_NAME:-defaultname}"
      image: mysql:5.7

But why not set the project-name? The project name is _intended_ for that, as it prefixes/namespaces the container-names that are created to prevent conflicting with other projects on the same host. See https://docs.docker.com/compose/reference/envvars/#/composeprojectname

[LouWii]

@thaJeztah Thanks ! I'm still learning how Docker and docker compose work. Setting the project name seems to be a better option, it makes total sense in my use.

[samhh]

Is there any way to script within the interpolated braces? For example ${HOST_PORT + 1}.

[nafg]

You could pipe the file through Jinja or something...

On Tue, Jan 24, 2017, 5:36 AM Sam A. Horvath-Hunt [e-mail]
wrote:

Is there any way to script within the interpolated braces? For example ${HOST_PORT

• 1}.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/docker/compose/issues/1377#issuecomment-274767368,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAGAUN5ZrU39dnVVVASwIHr5mGqJFxh3ks5rVdRIgaJpZM4EMysO
.

[logicminds]

Am I able to escape $?

environment:
   PATH: "$PATH:/home/appuser/.bundler/bin"

Currently this results in the host's PATH variable being interpolated and not the container

[alaagalal]

is a docker-compose.yml file only one found ?
how can i add or modify it?
Thanks in advance

[samrocketman]

@logicminds though I couldn't find it documented anywhere, I found $$ interpolates to an escaped $.

environment:
   PATH: "$$PATH:/home/appuser/.bundler/bin"

[binarymist]

@elquimista has a solution that I've found useful https://github.com/mhart/alpine-node/issues/48#issuecomment-430902787