Compose: Using shell command in docker-compose.yml

[zkanda]

Hello, is there a way to use shell commands in docker-compose.yml file?
Here is my use case:

 version: '2'
 services:
   ci:
     image: jenkins
     volumes:
       - ./data:/var/jenkins_home
       - /var/run/docker.sock:/var/run/docker.sock
       - $(command -v docker):/usr/bin/docker
     groupadd:
       - $(stat -c %g /var/run/docker.sock)
     ports:
       - "8080:8080"
       - "50000:50000"

Currently it's giving me this error:

ERROR: Invalid interpolation format for "volumes" option in service "ci": "${command -v docker}:/usr/bin/docker"

[shin-]

Hi @zkanda ,

Sorry, this is not something we support. Usually, this is done by setting environment variables and using variable substitution inside the Compose file instead.

[zkanda]

@shin- thanks, I can workaround with environment variables. Also .env seems very useful as well.
Relevant docs: https://docs.docker.com/compose/environment-variables/#/the-env-file

[dtothefp]

@zkanda did you ever get this to work, I tried in my .env file

DOCKER_BIN=`which docker`

and then in the docker-compose.yml

jenkinsmaster:
  build: jenkins-master
  volumes:
    - /var/run/docker.sock:/var/run/docker.sock
    - ${DOCKER_BIN}:/usr/bin/docker
  ports:
    - "50000:50000"

But I keep getting

Cannot create container for service jenkinsmaster: create `which docker`: "`which docker`" includes invalid characters for a local volume name, only "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed.

So it looks like commands are interpreted as strings in the .env file?

[shin-]

Commands are not expanded by Compose, in .env or elsewhere.

[marzlarz]

This would be very useful to have ....

[CpuID]

+1

[jeffijoe]

Here's a use case. When setting up Kafka, it needs the IP of the host machine. I have the following script to get that for me:

DOCKER_HOST_IP=$(ifconfig | grep 'inet .*br' | sed -E 's/.*inet (.*) netmask.*/\1/')

I reference that IP in the env. I can't just run docker-compose up anymore because I need to run that beforehand. And I need to inform the other devs on the team of this.

[bfwg]

+1

[davi5e]

Would be useful to get the current user, as in:

services:
  foo:
    image: bar:latest
    user: ${CURRENT_USER-$(id -u):$(id -g)}

There must be other use cases too...

By the way, I don't get the fact that ${BAZ-default} works in Compose if we can't execute the remainder of the Bash functionalities... (as far as I know, the - is one of Bash's variable manipulation functionalities, as substitution with ${VAR/search/replace}, for example)

[binarymist]

Yes @davi5e ... What I was trying for:

version: "3.7"

services:
  orchestrator:
    build: .
    ports:
      - "2000:2000"
    # Use type host until we can test link
    network_mode: "host"
    environment:
      - NODE_ENV=development
      - LOCAL_USER_ID=${id -u}

[Baathus]

@davi5e another use case is when you want to create a docker volume and link it to your git repository directory. This absolute bath will wary for each developer on the team.

[sido420]

+1
Plz reopen the issue

[sarkortrantor]

+1
my use case it to publish a range of ports that is determined as a function of a "base" port

[thejoshsmith]

This would be very useful to have for setting UID/GID variables as this is pretty much a requirement for using Docker in a development environment (without needed to use scripts to set this into an .env file).

[con-f-use]

+1

Could use this to label/tag the images generated by docker compose with the git tag and commit hash:

In .env:

GIT_VERSION=$(git describe --always --dirty --abbrev)
OUTER_PORT=6970

In docer-complse.yml:

version: '3'
services:
  nginx:
    restart: always
    build:
        context: ./nginx
        labels:
          org.label-schema.schema-version: "1.0"
          org.label-schema.version: "${GIT_VERSION}"
          org.lavel.schema.url: "https://mydocu-server.company.com/vcs/${GIT_VERSION}"
    ports:
      - ${OUTER_PORT}:8080

[ackerleytng]

For those doing this in development, if your shell sets the UID variable, you can pass that in when building the local development image. Since the local development image is not shared, you don't have to worry about uid conflicts with your co-workers.

[laflaneuse]

+1

[Hultner]

@con-f-use Doing that will just send the actual command and not the result from my testing, the only hack I've been successful so far is writing a script wrapping docker compose which sets up the environment variable.

[juliojmphjv]

+1

[wadkar]

TL;DR if you want to export those variables from .env file:

# set the path of .env file here
ENV_FILE="${ENV_FILE:-local.env}"
while IFS= read -r line; do
  export "$line"
done < <( grep --color=never -E -v -e '^#' -e '^[[:space:]]*$' "${ENV_FILE}" )

How to use:

• Type it on your command line and press enter to modify your current shell environment

OR

• Save it as bash script and source it

Caveats:

• Saving this as a script and running it won't magically export those variables in your current shell environment. You'll have to source it or do some fancy eval.
• (Maybe a good thing?) It will overwrite existing variables with same name in $ENV_FILE

Explanation: here

[mbrevda]

Have a look at direnv to automate all that https://direnv.net/

On Fri, Jun 28, 2019, 12:57 PM Sudarshan Wadkar [e-mail]
wrote:

TL;DR if you want to export those variables from .env file:

set the path of .env file here

ENV_FILE="${ENV_FILE:-local.env}"while IFS= read -r line; do
export "$line"done < <( grep --color=never -E -v -e '^#' -e '^[[:space:]]*$' "${ENV_FILE}" )

How to use:

• Type it on your command line
• Save it as bash script and source it

Caveats:

• Saving this as a script and running it won't magically export those
  variables in your current shell environment. You'll have to source it or do
  some fancy eval.
• (Maybe a good thing?) It will overwrite existing variables with same
  name in $ENV_FILE

Explanation: here
https://explainshell.com/explain?cmd=while+IFS%3D+read+-r+line%3B+do+export+%22%24line%22%3B+done+%3C+%3C%28grep+--color%3Dno+-v+-e+%27%5E%23%27+-e+%27%5E%5B%5B%3Aspace%3A%5D%5D*%24%27+%24%7BENV_FILE%7D%29

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/docker/compose/issues/4081?email_source=notifications&email_token=AACHRDBQXQZTC5FXTOFL6UTP4XOBJA5CNFSM4CUISSSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYZUTRQ#issuecomment-506677702,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACHRDGJ6FAQBEDLBSRVS6TP4XOBJANCNFSM4CUISSSA
.

[wadkar]

Yeah, direnv is nice but I am not sure if it will be useful with Docker's .env file which has simple lines with plain VAR=VAL syntax (as per this link.) The idea being .env file is committed to the repository and the build script can run it on my local or docker-compose can run it on the staging/deploy.

[mbrevda]

There not really environment variables
https://en.wikipedia.org/wiki/Environment_variable if they are part of
the code base and not part of the environment...

On Fri, Jun 28, 2019 at 2:04 PM Sudarshan Wadkar [e-mail]
wrote:

Yeah, direnv is nice but I am not sure if it will be useful with Docker's
.env file which has simple lines with plain VAR=VAL syntax (as per this
https://docs.docker.com/compose/env-file/#syntax-rules link.) The idea
being .env file is committed to the repository and the build script can
run it on my local or docker-compose can run it on the staging/deploy.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/docker/compose/issues/4081?email_source=notifications&email_token=AACHRDCEKOMK2VI3A43QBC3P4XV5BA5CNFSM4CUISSSKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYZY53A#issuecomment-506695404,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AACHRDF3Q3YOZPHUIUZYEPTP4XV5BANCNFSM4CUISSSA
.

[wadkar]

Oh, okay.

I like to think that the .env file turns into environment variable when you execute docker-compose up.

My use case was to use the same .env file as the source of truth to run a local dev instance (possibly without help of docker-compose). So yeah, maybe those lines in the .env file aren't really environment variables, but they surely turn into one when you deploy.

I still don't see how direnv is going to help in that case.

[sido420]

Another use case is to put a command that generates a password assigning it to env var in the container.
This way the pasword would actually not be written in .env or docker-compose.yml files.

I think this is going to be a huge benefit for many security related use cases.

[jack3311]

+1

[stat1x]

+1

[omoustaouda]

+1, would be hugely useful for setting uid and gid without over-complicating the solution

```
/_/\
( o.o )

^ <```

[prathik]

+1 I have a use case as mentioned above to add git branch and commit in the compose file which will eventually go into the dockerfile and then be added as a version in the manifest of the jar built in a stage of a multistage docker build process.

[logrusorgru]

  gateway:
    image: name
    build:
      context: ./project
      dockerfile: build/Dockerfile
      args:
        - GIT_COMMIT=${$(git rev-list -1 HEAD):-unspecified}
        - GIT_DATE=${$(git log -1 --date=short --pretty=format:%ct):-unspecified}

So, currently, for 7 projects in my compose file I have to set 7x2 environments. Sad.

[four43]

@shin- Could you re-open this as a feature request? It seems like it hasn't been solved by nice work-arounds and there is some solid need for it.

[dukekautington3rd]

+1 Probably could have achieved this a different way, but I needed to engage the docker engine from within a container to reboot a neighboring container. In the case where my Letsencrypt cert is expired, but I need to restart the NGinx container after the renewal. Problem is, how do I tell docker-compose inside the container what the project name is for this group of containers? PROJECT=(basedir ~+). It's manually coded for now :(

[rafaelbattesti]

+1 to that feature request
Another use case is to grab secrets from a cli/http (e.g. vault, 1Password) and plug into the environment.
i.e.

...
    environment:
      - SERVICE_USERNAME=$(vault kv get -field=username kv/service/credentials)
      - SERVICE_PASSWORD=$(vault kv get -field=password kv/service/credentials)
...

[TonyBrobston]

I have the same need as @rafaelbattesti, except I'm using lastpass.

...
     environment:
         - TRPASSWD=$(lpass show --password Transmission)
....

[POD666]

+1

[dburnsii]

+1

[esale]

@shin- Could you re-open this as a feature request? It seems like it hasn't been solved by nice work-arounds and there is some solid need for it.

@four43, maybe @shin- or others don't read comments on closed issues? Shall we open another one to get attention?

[sjlnk]

The variable expansion in docker-compose file would be a very useful feature to have ...

[PierrickVoulet]

+1

[four43]

@esale - Yeah I guess it kind of depends. I could see it being out of scope for this case. Don't go overboard on some DSL too much. At that point template the docker-compose file with some other tool? Are we crossing concerns too much?

[ntsd]

If you're using .env file you can replace shell output by a command like this

eval "echo \"$(cat .env.example)\"" > .env

Example .env.example file

DOCKER_BIN=$(which docker)
DOCKER_COMPOSE_BIN=$(which docker-compose)

Create the .env file

DOCKER_BIN=/snap/bin/docker
DOCKER_COMPOSE_BIN=/snap/bin/docker-compose