Compose: Pass variables inside fig.yml on runtime

[dkapanidis]

Is it possible to configure a fig.yml in such way that you can pass on execution time variables instead of hard-wiring them inside the fig.yml?

I think a generic variable injection during execution would be quite useful for various use cases.

e.g:

jenkins:
  image: aespinosa/jenkins:latest
  ports:
    - "8080"
  hostname: ${HOSTNAME}

HOSTNAME=ci fig up

That could inject the variable HOSTNAME inside the fig.yml during execution and execute a docker run with hostname ci.

ps. This is different than passing environment variables inside docker which is already supported (http://www.fig.sh/yml.html#environment)

[mieciu]

Good idea - i've also found it really handy when passing env. variable as a parameter in automated test run :smile:
But does your example imply passing all the variables from current users shell session? I'd rather see this like in docker

fig up -d -e HOSTNAME:ci

[mieciu]

Another thing is: which container should set this inline passed variable? First one, the second one, both?

[dkapanidis]

the format fig up -d -e HOSTNAME:ci sounds good by me.

as for which container, I would say make it global for the entire file. If the author of fig.yml doesn't want to exist conflicts between containers he will use different variables to different containers, or he can share the same variable to both if needed.

[mieciu]

That's great :+1:

There are two possible options right now:

• After fig up -d -e HOSTNAME:ci we pass variable to all the containers, and let's say that we can use variable names to detect, which container should use the variable, for instance:
  Having web and db container we can simply run:

fig up -d -e WEBS_ENDPOINT=192.168.0.40 -e DBS_ENDPOINT=192.168.0.50

So that besides the fact, that each container has two defined endpoints it still can use this one defined for him.

• We can also use container names like in scale=? approach, but with syntax:
  fig up -d -e <container_name>:<VARIABLE_NAME>:<VARIABLE>, but does it look good:
  fig up -d -e web:HOSTNAME:ci ??
  Maybe something like:
  fig up -d -c web -e HOSTNAME:web -e ROLE:ci -c db -e PASS:pass
  where web and db are containers names defined in fig.yml

[mieciu]

But the first idea looks definiately best to me

[dkapanidis]

can you show me a mockup of a fig.yml file and the respective CLI command
so I can understand better the first option?

are the variable names arbitrary or they have to match the container name?

2014-09-30 11:47 GMT+02:00 mieciu [email protected]:

But the first idea looks definiately best to me

—
Reply to this email directly or view it on GitHub
https://github.com/docker/fig/issues/495#issuecomment-57290241.

[mieciu]

Well, I thought about just arbitrary variable names, but matching them to container name will be okay as well (as FIG use it 'natively' http://www.fig.sh/env.html). So maybe this is the proper way :+1:

I also don't consider modyfing fig.yml as necessary, passing them just inline, like in

fig up -d -e HOSTNAME:ci

is 100% okay to me.

[mieciu]

So let's say that :

fig up -d -e WEB_HOSTNAME:ci

will pass variable to all web containers, and if no prefix is detected (like HOSTNAME:ci ), variables will be passed to all the containers

[dkapanidis]

ok, then we're probably talking about different things.

I understand what you propose as passing variables _inside_ the container,
but during the creation of docker command do nothing with them. My initial
proposal is to pass variables inside _fig.yml_ before parsing it.

In my use case I need to be able to set hostname not as variable inside the
container, but as part of the docker command:

e.g. docker -h '$HOSTNAME' jenkins

which in fig.yml format translates:

jenkins:
  image: aespinosa/jenkins:latest
  hostname: ci

but that ci I'd like to define it in runtime. If I pass my vriable as you
say WEB_HOSTNAME:ci then the container _needs_ to know what to do with it.

what you propose is already done with
http://www.fig.sh/yml.html#environment but
only the white-list of things that are defined in the fig.yml as possible
variable entries, which seems sufficient (the author of fig.yml should
explain what entries are parsable from the container)

Environment variables with only a key are resolved to their values on the machine Fig is running on, which can be helpful for secret or host-specific values
(taken from fig site)

[matleh]

I too would love to have some kind of variable expansion in the fig.yml. As of now, I have to use a template fig.yml and create the real fig.yml with some wrapper script.

My current use-case: Have functional test run agains a docker container, but be able to specify the tag of the image under test.

[sidabs]

+1 on this request!

[mrfelton]

Also looking for this support so that I can let a container know the hosts IP address so that it can connect to it.

[manolitto]

+1

[codeitagile]

+1

[dzmitry-paulenka]

This is really handy, if you store fig.yml in VCS. For example, our team uses the volumes: option, but everybody maps containers' folder to different path on the host machine. So we need a way to modify some parameters before 'fig up' execution, without changing the fig.yml itself.

These are file fragments to clarify:

# run.sh
export DIST_FOLDER=~/work/project/dist
fig up

# fig.yml
tomcat:
  image: internal/tomcat
  volumes:
    - "$DIST_FOLDER:/dist:ro"

[dkapanidis]

This is the exact same use case I have and why I proposed it: have fig.yml
in VCS and avoid changes on file to customize executions.

On Wed, Nov 26, 2014, 2:34 PM Dzmitry Paulenka [email protected]
wrote:

This is really handy, if you store fig.yml in VCS. For example, our team
uses the _volumes:_ option, but everybody maps containers' folder to
different path on the host machine. So we need a way to modify some
parameters before 'fig up' execution, without changing the fig.yml itself.

These are file fragments to clarify:

run.shexport DIST_FOLDER=~/work/project/dist

fig up

fig.ymltomcat:

image: internal/tomcat
volumes:
- "$DIST_FOLDER:/dist:ro"

—
Reply to this email directly or view it on GitHub
https://github.com/docker/fig/issues/495#issuecomment-64604460.

[matleh]

My use-case is similar: I have a fig.yml defining the setup for a suite of functional tests. Now I want to be able to run these tests against different versions.

# fig.yml
app:
  image: mickey/theapp:${APP_VERSION:latest}
test:
  image: mickey/tests
  links:
   - app

# run
APP_VERSION=1.3 fig up

[bushong1]

+1

[baptistedonaux]

+1

[fmahnke]

+1

[byrnedo]

+1

[balokir]

+1

[decoomanj]

+1

Are there any plans to implement this feature?

[dnephin]

I think this feature is already partially implemented. Many fields already support environment variables.

The one example I see here that isn't implemented yet is environment variable support for image tags. That should be a small change, PRs welcomed :)

I believe all it needs is an os.environ() in fig/service.py:_get_image_name()

[decoomanj]

Yep, sorry, I was unclear. I was referring to the environment-resolution inside Fig itself.

[mingfang]

+1

[relwell]

+:100:

[jmreicha]

+1

[jlrigau]

+1

[nathanpalmer]

+1 - This could be useful for our postgres image. Otherwise we aren't able to share the same configuration.

db:
  image: postgres:9.3
  volumes:
    - /$APP_NAME/postgresql/:/var/lib/postgresql/data/
  environment:
    - POSTGRES_USER=$CURRENT_USER
    - POSTGRES_PASSWORD=
  ports:
    - "5432:5432"

[JonLundy]

+1

[danggrianto]

+1

[NoumanSaleem]

:+1:

[relwell]

I have a few suggestions for how to implement this change:

• Fig should support two separate methods for accessing this data:
  
  
  
  • Environment variables, namespaced with FIG_VARIABLE_X
  
  
  • Example: If you're looking for HOSTNAME in fig.yml, the environment variable should be stored as FIG_VARIABLE_HOSTNAME
  
  
  • A separate YAML file
  
  
  • By default, we would search the root folder's variables.yml file
  
  
  • The user can specify the location of this file as an environment variable at FIG_VARIABLES_YML
  
  
  • The user can also pass this file via the command line for any command as --fig-variables=/foo/bar/variables.yml
  
  
• Create a class called FigConfig which wraps the all access of config data from fig.yml
  
  
  
  • This config class should have all the APIs available from the return value of yaml.safe_load to minimize changes to interactions throughout the library
  
  
  • This config class should include hooks for interpolating values first from templates.yaml, overwriteable by values set as environment variables
  
  
• Replace all existing calls to yaml.safe_load with instantiating a new instance of FigConfig with the provided filename.

If everyone feels good about this proposal, I would be happy to get to work on it and issue a PR.

[mingfang]

+1
Can't wait!

[dnephin]

We already support environment variables in some fields without this name mangling. ${HOSTNAME} would be used directly. I think this is a lot more intuitive than having to namespace them.

How would the values in this yaml file be represented? Would it make more sense as a file with just one key=value per line? Since this is more of a "nice to have" , I would probably wait on this.

FigConfig

I like the idea of a class instead of a dict. It makes validation easier. You can centralize all config validation in that class (or before creating it), instead of scattering validation all over. I would not try to emulate the dict interface. I think you lose a lot of the benefit of using a class if you do that.

Another related request is to be able to provide default values for things. Something like: ${HOSTNAME:localhost} is used in other files (tox.ini at least, possibly others), to provide defaults.

A change to support environment variables in most fields might be a good time to introduce this feature.

As a first past, I would suggest just adding extra support for environment variables in many fields using os.environ() is probably the easiest way to support this with minimal code changes.

[dkapanidis]

+1 on avoiding namespacing, since it is already provided as such in lot of
fields.
+1 also on providing default values, this is actually really cool,
otherwise by putting variables without defaults you miss the fanciness of
fig up

On Wed Jan 07 2015 at 9:20:47 PM Daniel Nephin [email protected]
wrote:

We already support environment variables in some fields without this name
mangling. ${HOSTNAME} would be used directly. I think this is a lot more
intuitive than having to namespace them.

How would the values in this yaml file be represented? Would it make more
sense as a file with just one key=value per line? Since this is more of a
"nice to have" , I would probably wait on this.

FigConfig

I like the idea of a class instead of a dict. It makes validation easier.
You can centralize all config validation in that class (or before creating
it), instead of scattering validation all over. I would not try to emulate
the dict interface. I think you lose a lot of the benefit of using a
class if you do that.

Another related request is to be able to provide default values for
things. Something like: ${HOSTNAME:localhost} is used in other files
(tox.ini at least, possibly others), to provide defaults.

A change to support environment variables in most fields might be a good
time to introduce this feature.

As a first past, I would suggest just adding extra support for environment
variables in many fields using os.environ() is probably the easiest way
to support this with minimal code changes.

—
Reply to this email directly or view it on GitHub
https://github.com/docker/fig/issues/495#issuecomment-69085278.

[relwell]

On the namespacing stuff, I think that's just a case of wanting to avoid interactions or dependencies between other systems on someone's machine. I'm happy to make namespacing environment variables a fully separate discussion, especially since it sounds like there's already support for some bare environment variables.

I think you're right that a bare INI-style format makes more sense here. I initially wanted to avoid introducing "yet another" form of config management into this proposal.

So maybe the proposal should be something more like:

• Add access to os.environ() for all fig.yml fields
• Introduce a FigConfig class responsible for wrapping data access from fig.yml
• The FigConfig class should have hooks that can access data from the environment (provided the ${} format). The default value should be provided with a semicolon for the provided string.

We can de-scope the template file for now, since users could create a separate file with their environment variables to source from for the time being.

I'm happy to try to take this chunk of work, either in one PR or two. I'm leaning towards doing it in a single PR since we'd actually want to take os.environ() out of all currently exposed fields and have that access happen directly in the Config class.

[zakkie]

+1

[acofer]

+1

[marcellmars]

+1

[rejinka]

+1, especially @relwell's point of having a seperate YAML file

[ohord]

+1

[loa]

+1

Especially like this one:

# fig.yml
app:
  image: django:${DJANGO_VERSION:latest}
  environment:
    - SECRET_KEY=${SECRET_KEY:ONLYFORTESTING}

[hadim]

+1

[NoumanSaleem]

fyi environment variables are capable of being referenced when declared without a value as the docs state

Environment variables with only a key are resolved to their values on the machine Fig is running on, which can be helpful for secret or host-specific values.

For the rest, try envsubst which can replace variable references using either $var or ${var} syntax

envsubst < "fig.tpl" > "fig.yml"
fig up

[thomasdavis]

+1 and I don't believe env vars should be namespaced. I want to able to write my code as normal and expect that env vars will be available as they normally would when on a given host.

[hadim]

I agreee with @thomasdavis .

[ilkka]

@NoumanSaleem gives good advice but since envsubst isn't available on OS X without installing gettext from e.g. homebrew which apparently can cause some problems, I slapped together a Go equivalent: https://github.com/ilkka/substenv

[pmoust]

Agreed with @thomasdavis

+1 for ENV passed on runtime

[Vad1mo]

I hate +1 but this is the most requested feature and discussed feature here.

Can we sum up the discussion here?

1. What variables are already supported in RC2 and where is it documented?
2. What is the current status?
3. Is this feature request accepted and when can one expect a solution?

[wololock]

Agreed with @Vad1mo, +1

[Vad1mo]

This is a simple script that can be used before docker-compose

https://gist.github.com/Vad1mo/9ab63f28239515d4dafd

here is the one liner to install:
sudo curl https://gist.githubusercontent.com/Vad1mo/9ab63f28239515d4dafd/raw/aa54b91f4c3671097789a54a9f42ba679b89dbaf/replace-var -o /usr/local/bin/replace-var && sudo chmod 755 /usr/local/bin/replace-var

[fullergalway]

+1

[ghost]

I hate to +1 this -- but this really is the most requested feature in github right now...

[fr3nd]

definitely +1
What's the current status? Is this feature being worked on?

[wildsurfer]

+1

[grjones]

+1

[tobijb]

+1

[coding2012]

+1, my use-case for this is to have 3 environments (a bunch of machines) that are all basically duplicates, but I want different port for each one. Rather than having a different fig file to maintain, I'd rather have a .sh file I can add a parameter like 2000, and that would spin up an environment that uses ports like 2001, 2002, 2003. I can do the addition logic in bash script, but I'll need something to pass to fig.

[noisy]

+1

[rhardih]

+1

[joshes]

+1

[montells]

+1

[pvlg]

+1

[davibe]

+1

[bpletzer]

+1

[mduheaume]

+1

[jaydp17]

+1

[johanhaleby]

+1

[thijsterlouw]

+1

[g3rd]

+1

[hd-deman]

+1

[cpopov]

+1

[radutoev]

+1

[ggregoire]

+1

[BMouser]

+1

[christianwoehrle]

+1

[abraverm]

+1

[Vad1mo]

let me sum up the progress so far:

[funkyfuture]

adding '+1'-comments and naughty pictures certainly don't help in a debate.

i wish i had the rights to delete them. :-P

[abraverm]

In my opinion, maintainer has limited time and energy and he needs to choose where to invest them. '+1' is the simplest way for a user to show which feature\bug is more important for them. There are different kind of levels of user participation in OSS projects. '+1' on the bug\feature says it in single flag what they think. And I wish you just had a button '+1' with a counter.

[javiervivanco]

+1

[docteurklein]

isn't it already possible by using the env_file ?
for example:

echo 'VAR=test' > env && docker-compose up -d

Might not, because "Environment variables specified in environment override these values.", which I find a little strange, cause env_file is meant to allow custom (per-install) params IMHO

[dkapanidis]

Compose.yml file does not define just env variables. So that would not work
for setting dynamically the hostname or the exposed port number etc.

On Wed, Apr 15, 2015, 9:39 AM Florian Klein [email protected]
wrote:

isn't it already possible by using the env_file
https://docs.docker.com/compose/yml/#env_file ?
for example:

echo 'VAR=test' > env && docker-compose up -d

Might not, because "Environment variables specified in environment
override these values."

—
Reply to this email directly or view it on GitHub
https://github.com/docker/compose/issues/495#issuecomment-93242626.

[mgasner]

+1

[justinwiley]

What are the security implications of this? I.e. can an unprivileged user can set HOSTNAME or other environment variables referenced in the YAML to something nefarious, that would lead to unintended code execution...either through failed YAML parsing or via its use after parsing?

[dkapanidis]

Well, good question, I suppose if someone has privileges to execute a user-defined compose.yml file, you should assume he 'owns' the machine, since he can activate --privileged mode on a container of his choice.

If the compose.yml file is not user-defined, then the hostname can only be set if the author of the compose.yml file is letting it be defined by variable. In that case, hostname substitution shouldn't execute any arbitrary shell commands to be resolved, otherwise command injection could be possible by unprivileged users.

[jsegura]

+1

[olalonde]

+1

[dlouwers]

:+1:

[ebuildy]

Come on, even Symfony 2 has variables support in YML!

Or we need something in top of docker-compose.

[goloroden]

+1

[Ovski4]

+1

[cyppan]

+1

[bobrik]

Every time you write your +1, 73 people get useless email. That doesn't make any sense. I bet maintainers got your point already.

[cyppan]

Sorry for this additional spam, but until Github supports votes in another way you're gonna see a lot of ugly, space consuming and useless +1... You have a nice car btw :p

[funkyfuture]

fyi, yesterday i requested automated censorship for '+1'- spamming content on github.

if you want something implemented, make a sponsoring offer!

[aanand]

OK! Sincere thanks for the +1s folks, it's genuinely helpful to know how many people want this feature. I've created a new issue at https://github.com/docker/compose/issues/1377 to track it. Head over there with your thoughts.

[olalonde]

In case anyone doesn't know, you can unsubscribe from notifications via the Unsubscribe button on the right. /meta

[phazei]

I wanted to share a rather simple temporary solution I had for variable expansion:

#!/bin/bash
apply_shell_expansion() {
    declare file="$1"
    declare data=$(< "$file")
    declare delimiter="__apply_shell_expansion_delimiter__"
    declare command="cat <<$delimiter"$'\n'"$data"$'\n'"$delimiter"
    eval "$command"
}
#in case you want to store the variables in a local file
source ./local.env
if ! [ -n "$DOCKER_DATA_PATH" ] ; then
    echo "DOCKER_DATA_PATH not set, using default value"
fi
DOCKER_DATA_PATH=${DOCKER_DATA_PATH:=//c/projects/dockerdata}

apply_shell_expansion docker-compose.tpl.yml > docker-compose.yml

Then your docker-compose.tpl.yml can be commited, each person can have local.env that's sourced into the script, and it just uses bash's own variable expansion, quick and easy.

Can either put default values for all the vars in the script, or just use the default variable syntax directly inside the tpl.yml

Edit: actually, that script seems to be bash only, since boot2docker only has sh, sh would be:

#!/bin/sh
apply_shell_expansion() {
    local file="$1"
    local data=$(cat "$file")
    local delimiter="__apply_shell_expansion_delimiter__"
    local command="cat <<$delimiter"$'\n'"$data"$'\n'"$delimiter"$'\n'""
    eval "$command"
}
#in case you want to store the variables in a local file
source ./local.env
if ! [ -n "$DOCKER_DATA_PATH" ] ; then
    echo "DOCKER_DATA_PATH not set, using default value"
fi
DOCKER_DATA_PATH=${DOCKER_DATA_PATH:=//c/projects/dockerdata}

apply_shell_expansion docker-compose.tpl.yml > docker-compose.yml

[garystafford]

For a small number of variables ('tokens'), I use a simple shell script along with a templated version of my YAML file. Here's an actual example:

files:

docker-compose-template.yml
docker-compose.yml
compose_replace.sh

run:

sh compose_replace.sh

script:

#!/bin/sh

# variables
base_url_token="{{ base_url }}" # find all these...
base_url="api.foo.com" # replace with url of public rest api
host_ip_token="{{ host_ip }}" # find all these...
host_ip=$(docker-machine ip $(docker-machine active)) # replace with ip of host running NGINX

# output
echo ${base_url_token} = ${base_url}
echo ${host_ip_token} = ${host_ip}

# find and replace
sed -e "s/${base_url_token}/${base_url}/g" \
    -e "s/${host_ip_token}/${host_ip}/g" \
    < docker-compose-template.yml \
    > docker-compose.yml

this in docker-compose-template.yml:

  extra_hosts:
   - "{{ base_url }}:{{ host_ip }}"

becomes this in docker-compose.yml:

  extra_hosts:
   - "api.acme.com:192.168.99.100"

[apobbati]

Github needs to implement an upvote/importance feature for issues.

[aanand]

@apobbati +1

(Environment variable interpolation has been implemented and merged in https://github.com/docker/compose/pull/1765, if that's what you're getting at.)

[funkyfuture]

i admit there's room for misinterpretations, so refresh your knowledge about bolshevik strategies!

[levivm]

+1