Moby: Allow COPY command's --chown to be dynamically populated via ENV or ARG

[mavericksid]

How to use COPY --chown without losing portability (recently merged from this issue)? I'll try to explain with an example wherein I find --chown causing portability issues.

Let's say I got a base image (mybase) which I use for running other containers as non-root user (let's say with UID 1000) using gosu in ENTRYPOINT.
exec /usr/local/bin/gosu ${USER_NAME} "$@"

I use this base image (mybase) for building other container (let's say mycontainer) Dockerfile where I COPY some files from host to container using --chown

FROM mybase
RUN mkdir -p /tmp/build
COPY --chown=1000 . /tmp/build

Now I want the files generated via my build tool (from mycontainer) to be shared on a Docker Volume mapped with my host. Generated files would be available at specified host location but the owner of those files would be user with UID 1000. Doesn't this mean I'll need to have a user with UID 1000 on host machine for generated files to be accessible? Isn't there a way to provide value for --chown dynamically? I tired using ENV and ARG but they are not accessible in --chown.

ARG owner
COPY --chown=$owner . /tmp/build

it gives me an error

unable to convert uid/gid chown string to host mapping: can't find uid for user $owner: no such user: $owner

Steps to reproduce the issue:

1. Add this to Dockerfile

FROM openjdk:8

ARG owner
ADD --chown=$owner . /tmp/build/platform3

CMD echo 'test'

1. Build image
   sudo docker build -t test -f Dockerfile .

Describe the results you received:
Result of running docker build

unable to convert uid/gid chown string to host mapping: can't find uid for user $owner: no such user: $owner

Describe the results you expected:
Expected result should be to get owner information from build arguments

Output of docker version:

Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:38 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:41:20 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 15
Server Version: 17.09.0-ce
Storage Driver: aufs
 Root Dir: /docker/pd0/aufs
 Backing Filesystem: extfs
 Dirs: 18
 Dirperm1 Supported: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
init version: 949e6fa
Security Options:
 apparmor
Kernel Version: 3.13.0-129-generic
Operating System: Ubuntu 14.04.5 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 14.69GiB
Name: ip-172-31-35-149
ID: KR63:RIDQ:SQNY:3ESC:5OA6:4DFT:QXSB:YE6M:727G:H7O2:REIS:RSFA
Docker Root Dir: /docker/pd0
Debug Mode (client): false
Debug Mode (server): false
Username: mavericksid
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

[thaJeztah]

Now I want the files generated via my build tool (from mycontainer) to be shared on a Docker Volume mapped with my host. Generated files would be available at specified host location but the owner of those files would be user with UID 1000. Doesn't this mean I'll need to have a user with UID 1000 on host machine for generated files to be accessible?

If I understand your use-case correctly;

• the Docker container produces some artefacts; those are created by the container's process, thus are owned by the process running inside the container (e.g. 1000:50).
• your Dockerfile copies files from the host back to the image, but then has to match the user/group of the container's process?

I think the _correct_ approach would be;

• Files inside the container (and image) have the ownership/permissions that are required inside the container. If you copy files _into_ the container image, that user is known, so you can change the permissions to match that user
• If the container's process produces artefacts that should be accessible by other users, the permissions should be changed by that process (assuming the artefacts are created at _runtime_, not during docker build, this could be done by an entrypoint script that does a chmod to set the correct permissions).

Isn't there a way to provide value for --chown dynamically? I tired using ENV and ARG but they are not accessible in --chown.

I'm not sure this is the correct approach for your use-case (see my bullets above), but you can create a user that uses the desired uid/gid and --chown to that user/group:

FROM openjdk:8
ARG uid=1001
ARG gid=51

RUN addgroup --gid $gid mygroup \
 && adduser --disabled-password --gecos "" --no-create-home --uid $uid --gid $gid myuser
COPY --chown=myuser:mygroup . /tmp/build/platform3

Also see https://github.com/moby/moby/issues/34482, and https://github.com/moby/moby/issues/34819 which may be relevant

[lmakarov]

I have the same issue - COPY --chown does not understand variables set via ENV or ARG earlier in the Dockerfile:

...
ARG AGENT_USER=agent
ARG AGENT_HOME=/home/agent
...
COPY --chown=$AGENT_USER:$AGENT_USER config/.ssh/config $AGENT_HOME/.ssh/config
...

This results in:

...
Step 15/16 : COPY --chown=$AGENT_USER:$AGENT_USER config/.ssh/config ${AGENT_HOME}/.ssh/config
unable to convert uid/gid chown string to host mapping: can't find uid for user $AGENT_USER: no such user: $AGENT_USER

If I hardcode the values, then everything works:

...
ARG AGENT_USER=agent
ARG AGENT_HOME=/home/agent
...
COPY --chown=agent:agent config/.ssh/config $AGENT_HOME/.ssh/config
...

[msmshariq]

I am also experiencing issue reported by @lmakarov where the the --chown doesn't honour ARG defined in the Dockerfile.

Guess this should be fixed?

[thaJeztah]

Guess this should be fixed?

It's not a bug; environment-var expansion in command options is not implemented yet (this issue is a feature request)

[darkmattercoder]

Would be great to have that expanded by var-values some time in the future

[thasmo]

This would really make my day!

[thasmo]

What's the status on this issue; will it be relabeled to be included in 18.x?

[thaJeztah]

No work has been done on this yet; if someone wants to work on this, I think this would be something that can be accepted as an enhancement.

When implementing, I think variable expansion should be implemented for the Dockerfile --options in general (so not just for --chown)

[phlax]

adding a +1 to be notified on progress

[dimaspivak]

This would be super useful for a number of OSS projects I work on. As it stands, doing COPY --from=${SOME_IMAGE} only works with some sed hackery on the Dockerfile.

[jwarnier]

adding a +1 to be notified on progress

[ericberberich]

As @dimaspivak pointed out: this case is specific - as the other example in #36986

In general: Allow to interpret $ARG in COPY

[acwest]

It seems rather inconsistent that $ARG processing is supported in the arguments for COPY and ADD, except for the -- arguments

[batmat]

adding a +1 to be notified on progress

FWIW @jwarnier for that purpose, you have the subscribe button on the right.

[erolg]

+1

[himslm01]

I've just fallen into this trap - trying to use ARG variable in COPY --from.
Is there any news from the devs whether is intentional or a mistake which will be fixed in due course?

[arthurdm]

+1! This would be useful for the WebSphere Application Server Docker image as well. Any updates?

[glensc]

@arthurdm:

The "--chown" COPY argument requires Docker 17.09 and up:

• https://github.com/moby/moby/pull/27303#issuecomment-338921602

EDIT: disregard my comment, different topic :(

[stephan-henningsen]

Hello, World.

Just wanted to say that it's 2019 and variable expansions in command options is still a high-valued feature request, at least for COPY --chown =)

[binarymist]

What's the work around to lack of string expansion other than hard coding?

[m-hu]

Run chown ${envvar}

For instance

Le dim. 31 mars 2019 à 06:04, Kim Carter [e-mail] a
écrit :

What's the work around to lack of string expansion other than hard coding?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/moby/moby/issues/35018#issuecomment-478310131, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACcfr0IGust5-OyR_NHdLTXH7Yqf4Wg3ks5vcDPHgaJpZM4PnKcX
.

[binarymist]

Thanks @pptime

Run chown ${envvar}

That's where I came from, it's too slow and adds an unnecessary additional image layer.

[m-hu]

Yeah, I agree. It is anyway a trade-off to make. More layers sometimes
yield better cache usage.

Le dim. 31 mars 2019 à 11:30, Kim Carter [e-mail] a
écrit :

Thanks @pptime https://github.com/pptime

Run chown ${envvar}

That's where I came from, it's too slow and adds an unnecessary additional
image layer.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/moby/moby/issues/35018#issuecomment-478326004, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACcfr-zlxpREIOx3ScGMPU7sfWJVUkBZks5vcIBQgaJpZM4PnKcX
.

[gertvdijk]

@pptime @binarymist

Run chown ${envvar}

For instance

it's too slow and adds an unnecessary additional image layer.

More layers sometimes yield better cache usage.

It's not just a small layer that's added. Try to add a 100 MB file with COPY, then RUN chown [...] in the next layer, build the image, and inspect with dive.

Result: double data in your image. Efficiency down to 50% of the layer you add. Really not okay.

[m-hu]

All right , that's horrible. Both for dev experience cause and storage
efficiency cause.
Could that be possible to remove or compress the intermediate layer like
the idea of git squash or fix-up?

Le dim. 31 mars 2019 à 12:05, Gert van Dijk [e-mail] a
écrit :

@pptime https://github.com/pptime @binarymist
https://github.com/binarymist

Run chown ${envvar}

For instance

it's too slow and adds an unnecessary additional image layer.

More layers sometimes yield better cache usage.

It's not just a small layer that's added. Try to add a 100 MB file with
COPY, then RUN chown [...] in the next layer, build the image, and
inspect with dive https://github.com/wagoodman/dive.

Result: double data in your image. Efficiency down to 50% of the layer you
add. Really not okay.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/moby/moby/issues/35018#issuecomment-478328116, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACcfr27v-s1yMwFnasIGRN3Pi0s_8CRhks5vcIiCgaJpZM4PnKcX
.

[binarymist]

Could that be possible to remove or compress the intermediate layer like the idea of git squash or fix-up?

That's what multistage builds try and solve, but this issue is about something more fundamental I believe... something that seems intuitive, but doesn't yet exist sadly.

[m-hu]

I guess the process is to first integrate moby/buildkit#926, then run vndr to update #39008.

[m-hu]

If extendedly we need to have it compatible with the envvar set by a run command, then we need to think about an alternative implementation.

[thaJeztah]

This is fixed on master now through https://github.com/moby/buildkit/pull/926, and https://github.com/moby/moby/pull/39132;

mkdir -p test && cd test
echo somefile > somefile.txt

cat > Dockerfile -<<'EOF'
FROM alpine

RUN addgroup --gid 456 onegrp \
 && adduser --disabled-password --no-create-home --uid 456 -G onegrp oneusr


RUN addgroup --gid 567 twogrp \
 && adduser --disabled-password --no-create-home --uid 567 -G twogrp twousr


ARG USR=oneusr
ARG GRP=onegrp

CMD ls -la /somefile.txt

COPY --chown=${USR}:${GRP} somefile.txt somefile.txt
EOF

Build the dockerfile, once without a build-arg (to use the default onegrp / oneusr), and once with:

docker build -t one .
docker build -t two --build-arg USR=twousr --build-arg GRP=twogrp .

Run the images to see that the user/group was set accordingly:

docker run --rm one
-rw-r--r--    1 oneusr   onegrp           9 Apr 25 17:37 /somefile.txt

docker run --rm two
-rw-r--r--    1 twousr   twogrp           9 Apr 25 17:37 /somefile.txt

[katsar0v]

In which version of docker can we expect this feature?

[liejuntao001]

In which version of docker can we expect this feature?

Per my experiment this starts to work from docker 19.03.0

[iktuz]

Just faced this situation. Upgrading docker it is not an alternative. Even so, it is not a straight solution, but works:

FROM ${IMAGE:-my-image}
ARG NAME

# I have already created a 'simple-user' and already inside its user's directory
RUN mkdir dependencies

# Here is the simple trick, switch to root
USER root

# Do the copies (no --chown flag required)
COPY ./dependencies/a ./dependencies/a
COPY ./dependencies/b ./dependencies/b
COPY ./dependencies/c ./dependencies/c
COPY ./dependencies/d ./dependencies/d

# It is time for admin performing all necessary modifications
RUN chmod -R +x ./dependencies/*.sh
# Variable substitution will work gracefully here, so root will delivery the directory ownership to the original 'simple-user'
RUN chown -R ${NAME}:${NAME} ./dependencies/*

# Switch back to 'simple-user' and continue doing the rest of the job
USER ${NAME}

[HacKanCuBa]

Just faced this situation. Upgrading docker it is not an alternative. Even so, it is not a straight solution, but works:

FROM ${IMAGE:-my-image}
ARG NAME

# I have already created a 'simple-user' and already inside its user's directory
RUN mkdir dependencies

# Here is the simple trick, switch to root
USER root

# Do the copies (no --chown flag required)
COPY ./dependencies/a ./dependencies/a
COPY ./dependencies/b ./dependencies/b
COPY ./dependencies/c ./dependencies/c
COPY ./dependencies/d ./dependencies/d

# It is time for admin performing all necessary modifications
RUN chmod -R +x ./dependencies/*.sh
# Variable substitution will work gracefully here, so root will delivery the directory ownership to the original 'simple-user'
RUN chown -R ${NAME}:${NAME} ./dependencies/*

# Switch back to 'simple-user' and continue doing the rest of the job
USER ${NAME}

What do you mean by "switch to root"? you are already root by default. And COPY didn't honor USER command (not sure if it does now). Anyway, COPY --chown ... is several times faster than RUN chown ... (as in, more than 10 time faster in my trials but don't trust me, try it yourself)

[iktuz]

Just faced this situation. Upgrading docker it is not an alternative. Even so, it is not a straight solution, but works:

FROM ${IMAGE:-my-image}
ARG NAME

# I have already created a 'simple-user' and already inside its user's directory
RUN mkdir dependencies

# Here is the simple trick, switch to root
USER root

# Do the copies (no --chown flag required)
COPY ./dependencies/a ./dependencies/a
COPY ./dependencies/b ./dependencies/b
COPY ./dependencies/c ./dependencies/c
COPY ./dependencies/d ./dependencies/d

# It is time for admin performing all necessary modifications
RUN chmod -R +x ./dependencies/*.sh
# Variable substitution will work gracefully here, so root will delivery the directory ownership to the original 'simple-user'
RUN chown -R ${NAME}:${NAME} ./dependencies/*

# Switch back to 'simple-user' and continue doing the rest of the job
USER ${NAME}

What do you mean by "switch to root"? you are already root by default. And COPY didn't honor USER command (not sure if it does now). Anyway, COPY --chown ... is several times faster than RUN chown ... (as in, more than 10 time faster in my trials but don't trust me, try it yourself)

In my case, with my Dockerfile flow, the command executions are performed by a not root user. I do this because I am preparing an image with Elasticsearch and root can not execute it by default for security reasons. In this way, I saw myself obligated creating a simple user (firstly as root), and then continuing all the flow as a simple user. The issue occurs when I try to copy something from host to the image as simple user. In the end, I need to switch back in the flow as root, perform the copies, give the due permissions to the simple user over the files and come back with him. As I said before, this is not a straight solution, just a way. Best regards.

[m-hu]

Do you happen to have any restriction in terms of docker version in your
working environment?

Le lun. 30 sept. 2019 à 13:09, iktuz [e-mail] a écrit :

Just faced this situation. Upgrading docker it is not an alternative. Even
so, it is not a straight solution, but works:

FROM ${IMAGE:-my-image}
ARG NAME

I have already created a 'simple-user' and already inside its user's directory

RUN mkdir dependencies

Here is the simple trick, switch to root

USER root

Do the copies (no --chown flag required)

COPY ./dependencies/a ./dependencies/a
COPY ./dependencies/b ./dependencies/b
COPY ./dependencies/c ./dependencies/c
COPY ./dependencies/d ./dependencies/d

It is time for admin performing all necessary modifications

RUN chmod -R +x ./dependencies/*.sh

Variable substitution will work gracefully here, so root will delivery the directory ownership to the original 'simple-user'

RUN chown -R ${NAME}:${NAME} ./dependencies/*

Switch back to 'simple-user' and continue doing the rest of the job

USER ${NAME}

What do you mean by "switch to root"? you are already root by default. And
COPY didn't honor USER command (not sure if it does now). Anyway, COPY
--chown ... is several times faster than RUN chown ... (as in, more than
10 time faster in my trials but don't trust me, try it yourself)

In my case, with my Dockerfile flow, the command executions are performed
by a not root user. I do this because I am preparing an image with
Elasticsearch and root can not execute it by default for security reasons.
In this way, I saw myself obligated creating a simple user (firstly as
root), and then continuing all the flow as a simple user. The issue occurs
when I try to copy something from host to the image as simple user. In the
end, I need to switch back in the flow as root, perform the copies, give
the due permissions to the simple user over the files and come back with
him. As I said before, this is not a straight solution, just a way. Best
regards.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/moby/moby/issues/35018?email_source=notifications&email_token=AATR7L2R4A62B2ZFPD22WITQMHM7JA5CNFSM4D44U4L2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD75I4KI#issuecomment-536514089,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AATR7L5WARB32MDC7BXBDL3QMHM7JANCNFSM4D44U4LQ
.

[iktuz]

Just for now. Version upgrade is strongly considered for sure.

[ahallur]

I'm on 19.03.3 and I still face this issue.

[m-hu]

@ahallur could you please a test case?

[ahallur]

I've the exact same issue as the OP has. .
I've the following in my Dockerfile

ARG USER=user
ARG GROUP=group
COPY --chown=${USER}:${GROUP} /path/to/my/main.py .

When I try to build it, it fails with the error message. .

unable to convert uid/gid chown string to host mapping: can't find uid for user ${USER}: no such user: ${USER}

Also I'm running an up to date docker version

$ docker version
Client: Docker Engine - Community
 Version:           19.03.3
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        a872fc2
 Built:             Tue Oct  8 00:55:12 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.3
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       a872fc2
  Built:            Tue Oct  8 01:01:15 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

It builds fine if I hardcode the user and group.

[m-hu]

Yeah, you need to firstly have the user and group

I've the exact same issue as the OP has. .
I've the following in my Dockerfile

ARG USER=user
ARG GROUP=group
COPY --chown=${USER}:${GROUP} /path/to/my/main.py .

When I try to build it, it fails with the error message. .

unable to convert uid/gid chown string to host mapping: can't find uid for user ${USER}: no such user: ${USER}

Also I'm running an up to date docker version

$ docker version
Client: Docker Engine - Community
 Version:           19.03.3
 API version:       1.40
 Go version:        go1.12.10
 Git commit:        a872fc2
 Built:             Tue Oct  8 00:55:12 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.3
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.10
  Git commit:       a872fc2
  Built:            Tue Oct  8 01:01:15 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

It builds fine if I hardcode the user and group.

Could you please provide a complete Dockerfile?
Tried to run it, it turns out to be working fine

[hostalp]

I tried the same for ADD and it doesn't work either:

ARG AS_OS_USER=user
ARG AS_OS_USER_GROUP=group
ADD --chown=${AS_OS_USER}:${AS_OS_USER_GROUP} 03-as-preconfig.sh ./

unable to convert uid/gid chown string to host mapping: can't find uid for user ${AS_OS_USER}: no such user: ${AS_OS_USER}

But using the same user and group directly works:

ADD --chown=user:group 03-as-preconfig.sh ./

Note that I use the same ARGs elsewhere in the Dockerfile (RUN, USER) and they work there alright.

docker version

Client: Docker Engine - Community
 Version:           19.03.5
 API version:       1.40
 Go version:        go1.12.12
 Git commit:        633a0ea
 Built:             Wed Nov 13 07:25:41 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.5
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.12
  Git commit:       633a0ea
  Built:            Wed Nov 13 07:24:18 2019
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.2.10
  GitCommit:        b34a5c8af56e510852c35414db4c1f4fa6172339
 runc:
  Version:          1.0.0-rc8+dev
  GitCommit:        3e425f80a8c931f88e6d94a8c831b9d5aa481657
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

[m-hu]

Copy works, add does not. It could be interesting to have it also in ADD, as it provides some capabilities that copy does not provide

[hostalp]

^ Confirmed COPY working, but ADD not. I switched to COPY since I use a local content at that place.

[kikihakiem]

I think variable expansion should be implemented for the Dockerfile --options in general (so not just for --chown)

anyone already working on this? If not, I'm interested to implement it

[m-hu]

It seems that the variable is not always expanded in the same way, that is probably why this requirement is associated to a specific keyword. To see if it could be achievable with a non invasive modification.