Compose: Variable substitution

[gponsu]

This new feature is amazing, but I had a problem. I explain:

In our dev team, to enter in docker with a user with the same UID and GID that of the host machine, we made the following:

# docker-compose.yml
app:
  image: rails:4.2.4
  working_dir: /app
  ...
  extends:
    file: docker-custom.yml
    service: app

The docker-custom.yml file we have untracked of git, so everyone can define their own settings.

# docker-custom.yml
app:
  # Set your user and group id.
  # View your user id with command 'id -u' and your group id with 'id -g'.
  user: '1000:1000'
  environment:
    BUNDLE_JOBS: 4
    BUNDLE_PATH: /app/.bundle
    BUNDLE_APP_CONFIG: /app/.bundle
  # You can use env_file option instead of environment if you are using one file
  # to save environment variables
  # env_file: .env

With the new functionality of Variable substitution, I was excited, because we could spare us all this, and minimize it to this line in docker-compose.yml:

user: "${UID}:${GID}"

Unfortunately this does not work, because UID and GID are shell variables (not env variables), and os.environ do not get them.

Would it be possible get the shell envs too? Something like (I know nothing about python :disappointed:):

os.system("set")

He had also thought of another possibility, if the option user is user: host or similar, internally get the uid and gid with:

os.getgid()
os.getuid()

And setup docker with this config for user.

Thank you very much for everything, and apologies for my horrible English.

[oinak]

:+1:

I have the exact same problem, as soon as I saw variable substitutio I tried for $UID to no avail

I could make the file ownership issues of containers mounting code folders go away.

Please please please enable shellvars along with envvars to be sunstituted on docker-compose.yml files.

Thank you for the excellent work.

[dnephin]

The only different is that UID isn't exported. I don't think it's correct to go reading un-expected shell variables.

I would suggest export UID.

[deeky666]

:+1: I think this is a common use-case when mouting directories from the host and accessing them inside the container. Having to export extra environment variables on the host side is unnecessary extra configuration on the host side which more or less defeats this whole purpose.
Until using docker composer, I was evaluating the user and group id via id -u and id -g in the run command, which is not (yet) possible in docker composer configuration.
Some sort of solution to this would be very welcome.

[cookandy]

+1

Another use case would be trying to set the hostname of a container to the hostname of your host:

myapp:
  hostname: ${HOSTNAME}

My workaround right now is to set it before calling docker-compose

HOSTNAME=$HOSTNAME docker-compose up

[dvddarias]

I have exactly the same problem. In our team all you need to do to setup a new developer is
to run:

git clone ....
cd ....   
export UID   
docker-compose up   

We need the same user on the container and the host because of the big amount of Ruby on Rails commands that generate new files on the project volume. The sad part is that you have to export the UID variable on every new shell session for docker-compose up to work or add the variable to your shell startup script, which is sub-optimal on such a common use case.

This problem is specific to programmers using docker-compose as a tool:

Using docker CLI directly you don't have this problem because it is assumed that you run them within a shell and there you can use the variable $UID without exporting it.

Using docker through the API is also possible getting the current user id with os.geteuid() or something similar.

When using docker-compose i haven't find the way of doing it without running commands, so the "same user as the host" service requirement can't be made effective on any of the repository files.

A solution may be to setup some specific environment variable (let's say COMPOSE_UID) to the same UID as the user running the docker-compose command with os.geteuid() and then use it within the file as ${COMPOSE_UID}.

[mknapik]

👍
Same use case, handling shared directories at CI server.

[edannenberg]

Can't spam enough :+1: for this. UID/GID of the current host process should be provided by docker-compose in some way.

[ScreamingDev]

So as my #4725 got closed due to duplicates I need to ask the remaining question here:

How to solve/bypass this?

I guess it can't be left that way it is as long as there is no alternative.
It really is something that makes work a lot easier for developers.
Doesn't matter on production server but for dev and test server it helps.

Does the world build a wrapper around docker-compose having export UID in it?
Is it feasible by docker compose to solve this?

[durkode]

I'm also going to add my voice to this. Developing with Django from within a container and would love for this to work out of the box without having to export UID and GID.

I personally am not concerned how it is done, but a way for files written to the mounted volume in the container to match the host user without having to export UID and GID is my end aim. Currently running the export isn't a big deal, it would just be nicer if this wasn't the case

[sokratisg]

To be honest we sort of worked around this issue by doing something similar to:

1. Make sure UID is exported in each user's bash shell. Just create the following file and restart:
   /etc/profile.d/docker.sh

export UID

1. Add the necessary arguments in the docker-compose.yml file which is used for local development purposes:

...
  app:
    build:
      context: .
      args:
        username: $USER
        userid: $UID
...

1. Use these arguments in the respective Dockerfile which is also used solely for local development:

...
ARG username
ARG userid
RUN useradd --no-create-home --user-group --shell /bin/bash --home-dir $APP_HOME --uid $userid $username
RUN chown -R $username: $APP_HOME
...

Only drawback: UID conflicts with images that by default use the same one with any local user. (i.e. official node images have node user with UID=1000 and that conflicts with the initial UID a typical Ubuntu desktop assigns to its first user, tha one you're going to use post-installation)

[edannenberg]

Only drawback: UID conflicts with images that by default use the same one with any local user. (i.e. official node images have node user with UID=1000 and that conflicts with the initial UID a typical Ubuntu desktop assigns to its first user, tha one you're going to use post-installation)

Just use a static user like node in the image and change it's UID on container start as needed. The username is irrelevant when it comes to permissions, all that matters is a matching UID.

[sokratisg]

That is true if you use an image from a Dockerfile. Not true if you just want to start a node app by pulling the default image without any fiddling around or any custom shell script as your command.

[edannenberg]

@sokratisg You seem to use a local Dockerfile already, replacing useradd with usermod --uid $userid nodeshould do the trick.

[sokratisg]

@edannenberg excuse me but I think I may have been misunderstood, maybe my comment wasn't very clear. I'll try to give you an example.

Let's say your desktop user has UID=1000 (default for first user, at least on Ubuntu) and that you've written a node app that outputs or generates some local files. You want them to be written with the correct ownership (UID=1000) since you're also working locally and want to have read/write access on them without using sudo for your favorite editor/IDE. If you go with the solution I described then upon startup of the node container you will end up with a UiD conflict since official node images come pre-baked with a node user that has the same UID (1000).

This is what I was trying to describe; that this workaround is not universal and comes with some possibilities of a conflict so this issue is still relevant.

[edannenberg]

@sokratisg I got that, the problem is that your solution tries to add another user with an already existing UID. My point was that you can avoid that by just changing the UID of an existing static user in the image, i.e. node. In theory this will still leave the possibility of a conflict, but in practice the image will most likely only have one user with a UID that could conflict with the typical user UID space (>= 1000, everything below 1000 is usually reserved for system users).

A typical nodejs compose file in our projects looks like this:

version: '2.2'
services:
  app:
    build:
      dockerfile: ${PWD}/docker/Dockerfile
      context: ./docker
      args:
        - NODEJS_UID=${UID}
        - NODEJS_GID=${GID}
    user: nodejs
    working_dir: ${PWD}
    volumes:
      - ~/.npm:/home/nodejs/.npm
      - ${PWD}:${PWD}

The nodejs user already exists in the pulled image, the local Dockerfile just changes the UID/GID of said user.

[vipseixas]

My workaround for the usecase of ownership of mounted volumes is to use a impersonate.sh script as the entrypoint of the image, like this:

#!/bin/bash

USER_NAME="deployer"

files=(/work/source/*) && DIR=${files[-1]}
USER_ID=$(stat -c "%u" ${DIR})

echo "Using ${USER_ID} from owner of dir ${DIR}"

id "${USER_NAME?}" >/dev/null 2>&1

if [[ $? -ne 0 ]]; then
    echo Creating user ${USER_NAME} with UID ${USER_ID}
    useradd --home-dir /work --uid ${USER_ID} --shell /bin/bash --no-create-home ${USER_NAME}
fi

su --login --preserve-environment --command "/scripts/deploy.sh $1 $2 $3" ${USER_NAME}

/work/source is where I mount my external source code inside the container.

[38leinaD]

+1

[ChristianKreuzberger]

Django Developer here:
Using $UID and $GID within docker-compose.yml would be so great! Please please please!

[CodeCorrupt]

One better and more versatile would be the ability to include shell script in the docker-compose. Something like this would be helpful

services:
  service_name:
    environment:
      - UID=$(id -u)
      - GID=$(id -g)

I'm currently accomplishing this by wrapping the docker-compose up in a shell script that sets those variables, however that's a little clunky. This could also be useful for dynamically setting other fields outside of just environment variables.

Edit: A lot of the comments are saying to just use $UID and $GID. I don't know about other distros but my Ubuntu Server 16.4 doesn't have that by default so adding the following to your .bashrc can add that and has been working well for me.

UID=$(id -u)
GID=$(id -g)

[reduardo7]

I use something like this:

version: "2.1"

services:
  app:
    volumes:
      - /etc/passwd:/etc/passwd:ro
      - /etc/group:/etc/group:ro
    user: $UID:$GID

[awilkins]

Using a .env file ; sadly, you can't use substitutions in a .env file, it's all literals.

[rcdailey]

Maybe I've got the wrong idea here, but why do folks use the user and group ID of the current user when they run their services? So far, I've personally been creating system users in Ubuntu for this, one per service. So for my plex container, I have a user called plex and a shared group called docker_services. My next service, sonarr, will have a user called sonarr and also use the group docker_services. This allows me to assign very specific permissions to bind mount directories, and prevent other services from touching files they aren't supposed to.

I have a shell script I run right now to launch my services with users in this way, but unfortunately since I can't run subshells inside the yml file, I haven't been able to use docker compose:

docker run -d \
    --name plex \
    --restart unless-stopped \
    --network host \
    -e PLEX_UID="$(id -u plex)" \
    -e PLEX_GID="$(id -g plex)" \
   #...etc....

Is this not the recommended way to run permissions for containers? I'd rather not start them under my personal account I use to SSH into the machine itself, since this user account was not meant to be used by services running 24/7.

Assuming what I'm doing isn't completely bonkers, then I think there's still some ways to go before I use docker compose. Right now I use a series of bash scripts to start my services, semantically similar to docker-compose up. This is the only way I've been able to have the flexibility to do some system introspection while running docker containers.

[javabrett]

Maybe I've got the wrong idea here, but why do folks use the user and group ID of the current user when they run their services?

Largely in a development context, in cases where it is desirable to use bind-mounts, and you can't, or it's not easy or practical to prevent the container from creating files. If such files are created with users/groups that don't match the host-container, they become painful to clean-up in development-cycles.

[Wirone]

@gponsu We're using environment variable which is documented in our README.md:

Add export DOCKER_COMPOSE_RUN_AS_USER="$(id -u):$(id -g)" to your ~/.bashrc or other place that will define it every time when starting terminal.

and then in docker-compose.yml:

user: "${DOCKER_COMPOSE_RUN_AS_USER:?You must define user:group for permissions handling, look at README}"

The :? part enforces that variable, so if it's not defined, build will fail.

But yeah, I came here because I was looking for executing Bash script in docker-compose.yml because I wanted to pass git describe result as ENV variable.

[lygstate]

looking for better solution

[ivanbogomoloff]

Working for me.

environment:
    APP_USER_UID: $APP_USER_UID

and run docker-compose with export like this

export APP_USER_UID=$UID && docker-compose up -d

[MtDalPizzol]

Just ran into a scenario where I needed to pass the UID an GID to compose. I'll just live what I did here so maybe it helps someone out there. I'm running Mint (Ubuntu based distro) and here is the step by step that got this working for me:

1. Edit ~/.bashrc file adding the following to the end of it:

export COMPOSE_UID=$(id -u)
export COMPOSE_GID=$(id -g)

1. Add the environment vars to docker-compose.yml

version: "3"
services:
  storage:
    image: richarvey/nginx-php-fpm:latest
    volumes:
      - .:/var/www/html
    environment:
      WEBROOT: /var/www/html/www
      PUID: ${COMPOSE_UID}
      PGID: ${COMPOSE_GID}
    ports:
      - 8081:80

1. Restart the terminal or start a fresh one.

2. Now you can run compose without any additional magic:

$ docker-compose up

[egdoc]

When using sudo to launch docker-compose, one can use the SUDO_UID and SUDO_GID environment variables which contain respectively the UID and GID of the user who launched the command:

version: '3'
services:
    php-apache:
        build:
            context: ./php-apache
            args:
                APACHE_UID: "${SUDO_UID}"
        ports:
            -  8080:80
        volumes:
            -   ./DocumentRoot:/var/www/html:z

[slykar]

Use an .env file

I often have a sample .env file for my projects.
You can set required variables there.

# .env
UID=$UID
GID=$GID

docker-compose command picks this file automatically and makes the declared env vars available in the YAML file.

Ask users to create an override file and set user: manually.

# docker-compose.override.yaml
...
user: 1000:1000
...

[rvdbogerd]

After hours of searching, @slykar came with the missing piece. Finally I've got it working.

Small addition:

Why not add the uid and gid to the docker-compose.yml directly? Then you are sure it will work for everyone. Like:

# docker-compose.yml
user: "$UID:$GID"

[awilkins]

The difficulty is that UID and GID are not really environment variables.

❯ env | grep UID

~ 
❯ echo $UID
1000

They're inserted magically by your shell. I've used the .env hack mentioned by @slykar, usually with an .env.template committed to source ; this is basically the solution in this comment but solidified into a config file.

version: '2'
services:
  hello_world:
    image: ubuntu
    command: [/bin/echo, 'Hello, user : ${UID} in group ${GID}']

❯ docker-compose up
WARNING: The UID variable is not set. Defaulting to a blank string.
WARNING: The GID variable is not set. Defaulting to a blank string.
Creating network "env_default" with the default driver
Creating env_hello_world_1 ... done
Attaching to env_hello_world_1
hello_world_1  | Hello, user :  in group 
env_hello_world_1 exited with code 0

If you add the .env file suggested above by @slykar verbatim you just get exactly what's in it ... you have to fill it with your own values.

❯ docker-compose up
Recreating env_hello_world_1 ... done
Attaching to env_hello_world_1
hello_world_1  | Hello, user : $UID in group $GID
env_hello_world_1 exited with code 0

I usually do this with a shell script. (e.g. sed -i "s/\$UID/$UID/" .env)

What would be nice is if docker-compose called getuid() and getgid() on Linux and populated it's internal environment table with these vars, the same way that shells do.

[Jim-Bar]

As @Wirone signaled, it's possible to work around this issue pretty well by using:

user: "${UID:?Please export UID}:${GID:?Please export GID}"

docker-compose will immediately fail with the specified error message if the UID or GID is not exported.

But indeed it would be practical to have Compose either let us do a $(id -u):$(id -g) or provide us with the uid/gid of the caller. Maybe not using environment variables like UID and GID because that can be confusing if they are indeed not exported, but by any other means.

[Snowshield]

For people who are looking for a solution in the current version of docker-compose.

Default linux user have 1000 user id and 1000 group id. Using this values is most common case among developers.

So i found very practical to substitute these values as default in docker-compose to simplify usage of docker-compose CLI.

user: "${UID:-1000}:${GID:-1000}"

If developer needs some custom values of user id and group id or he needs full control over execution in automating tools so he can pass manually any values of UID and GID via environment variables, i.e.

1. Custom values

   UID=1001 GID=1001 docker-compose up
   

2. Full control

   UID=$(id -u) GID=$(id -g) docker-compose up
   

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it had not recent activity during the stale period.