Kaniko: COPY changes ownership of target directory

[everflux]

Actual behavior
Copying everything to the workdir using
COPY . .
changes the file ownernship of the target directory to root.
For images that use a non-root user and a workdir that is owned by the this user, but not root, this leads to permission problems.

Expected behavior
Target directory is kept as is, but files are added.

To Reproduce
Steps to reproduce the behavior:

1. Use a container image with USER 1000 as last instruction in base image
2. Have in base image RUN mkdir /app && chown 1000:1000 /app
3. In Dockerfile used by kaniko COPY . .

Additional Information
Dockerfile

FROM trion/ng-cli:latest AS ngcli

RUN ls -la; touch foo; ls -la
COPY . .
RUN ls -la; rm foo; ls -la

Output

INFO[0083] args: [-c pwd; ls -la; touch foo; ls -la]    
/app
total 8
drwxr-xr-x 2 node root 4096 Feb  2 20:56 .
drwxr-xr-x 1 root root 4096 Feb  2 21:42 ..
total 8
drwxr-xr-x 2 node root 4096 Feb  2 21:42 .
drwxr-xr-x 1 root root 4096 Feb  2 21:42 ..
-rw-r--r-- 1 node root    0 Feb  2 21:42 foo
INFO[0083] Taking snapshot of full filesystem...        

INFO[0107] COPY . .
INFO[0108] Taking snapshot of files...           

INFO[0108] RUN ls -la; rm foo; ls -la                   
INFO[0108] cmd: /bin/sh                                 
INFO[0108] args: [-c ls -la; rm foo; ls -la]            
total 524
drwxr-xr-x 8 root root   4096 Feb  2 21:43 .
drwxr-xr-x 1 root root   4096 Feb  2 21:42 ..

Workaround
This is how I managed to get the build working, though it feels like a crutch

COPY . /tmp/src
RUN cp -r /tmp/src/* .; cp /tmp/src/.* .

[hobti01]

This behaviour is different than building with docker.

Dockerfile

FROM alpine
RUN adduser -u 1000 -G root -D pipeline
USER pipeline
COPY . /home/pipeline/

docker build copies the files as root but does not change the target directory ownership

...
drwxr-sr-x    1 pipeline root        4.0K Mar  6 14:23 .
drwxr-xr-x    1 root     root        4.0K Mar  6 14:03 ..
-rwxr-xr-x    1 root     root         835 Mar  6 13:52 run.sh

kaniko build changes target directory ownership (and also writable bit on files?)

...
drwxr-sr-x    1 root     root        4.0K Mar  6 14:17 .
drwxr-xr-x    1 root     root        4.0K Mar  6 14:18 ..
-rwxrwxrwx    1 root     root         835 Mar  6 14:17 run.sh

[hobti01]

COPY --chown has no effect

Dockerfile

FROM alpine
RUN adduser -u 1000 -G root -D pipeline
USER pipeline
COPY --chown=pipeline:root . /home/pipeline/

docker

drwxr-sr-x    1 pipeline root          4096 Mar  6 14:34 .
drwxr-xr-x    1 root     root          4096 Mar  6 14:03 ..
-rwxr-xr-x    1 pipeline root           835 Mar  6 13:52 run.sh

kaniko

drwxr-sr-x    1 root     root        4.0K Mar  6 14:34 .
drwxr-xr-x    1 root     root        4.0K Mar  6 14:35 ..
-rwxrwxrwx    1 root     root         835 Mar  6 14:34 run.sh

[Kars-de-Jong]

I have a similar issue.
I have the following simple Dockerfile:

FROM jetty:alpine
COPY jetty/webapps/*.xml /var/lib/jetty/webapps/
COPY jetty/target/webapps/*.war /opt/webapps/

When building this image with kaniko, running this image fails with the following error:

/docker-entrypoint.sh: line 91: can't create /var/lib/jetty/jetty.start: Permission denied

It works fine when using docker to build it.
The difference is the ownership of /var/lib/jetty (and /var/lib/jetty/webapps).
When built with docker:

drwxr-xr-x 100/101           0 2019-04-02 10:49 var/lib/jetty/
drwxr-xr-x 100/101           0 2019-04-02 10:48 var/lib/jetty/webapps/

When built with kaniko:

drwxr-xr-x 0/0               0 2019-04-01 10:46 var/lib/jetty/
drwxr-xr-x 0/0               0 2019-04-01 10:46 var/lib/jetty/webapps/

When any command is used before the COPY statements that makes kaniko unpack the base image, it works fine:

FROM jetty:alpine
RUN true
COPY jetty/webapps/*.xml /var/lib/jetty/webapps/
COPY jetty/target/webapps/*.war /opt/webapps/

So basically any steps that just add files to the image do not take the original permissions into account.

[gugahoi]

Also just got caught by this, my dockerfile is as follows:

[...]
WORKDIR /usr/src/app
RUN useradd -ms /bin/bash personifi \
    && chown personifi:personifi /usr/src/app
USER personifi
[...]
COPY --chown=personifi:personifi config/ ./config
[...]
RUN cp config/database.yml.sample config/database.yml \
    && chown personifi:personifi config/database.yml \
    && bundle exec rake assets:precompile
# FAIL ^^^^
[...]

Google cloud build outputs the following error:

Step #0: cp: cannot create regular file 'config/database.yml': Permission denied
Step #0: error building image: error building stage: waiting for process to exit: exit status 1

This leads me to think the directory is not appropriately configured with correct ownership with kaniko however works perfectly fine with docker.

[swistaczek]

This issue is a duplicate of issue #9.

[tejal29]

We will resurrect #250 and try to get this in.

[tejal29]

FROM alpine
RUN adduser -u 1000 -G root -D pipeline
USER pipeline
COPY --chown=pipeline:root . /home/pipeline/

Hey @hobti01 I verified COPY --chown on PR #962 and it now copies the files with correct permissions.
` / # ls -al /home/pipeline/ total 347440 drwxr-xr-x 8 pipeline root 4096 Feb 3 23:00 . drwxr-xr-x 4 root root 4096 Feb 3 22:58 .. -rw-r--r-- 1 pipeline root 920 Feb 3 23:00 cleanup.go -rw-r--r-- 1 pipeline root 1408 Feb 3 23:00 cmd.go

[tejal29]

I have a similar issue.
I have the following simple Dockerfile:

FROM jetty:alpine
COPY jetty/webapps/*.xml /var/lib/jetty/webapps/
COPY jetty/target/webapps/*.war /opt/webapps/

When building this image with kaniko, running this image fails with the following error:

/docker-entrypoint.sh: line 91: can't create /var/lib/jetty/jetty.start: Permission denied

It works fine when using docker to build it.
The difference is the ownership of /var/lib/jetty (and /var/lib/jetty/webapps).
When built with docker:

drwxr-xr-x 100/101           0 2019-04-02 10:49 var/lib/jetty/
drwxr-xr-x 100/101           0 2019-04-02 10:48 var/lib/jetty/webapps/

When built with kaniko:

drwxr-xr-x 0/0               0 2019-04-01 10:46 var/lib/jetty/
drwxr-xr-x 0/0               0 2019-04-01 10:46 var/lib/jetty/webapps/

When any command is used before the COPY statements that makes kaniko unpack the base image, it works fine:

FROM jetty:alpine
RUN true
COPY jetty/webapps/*.xml /var/lib/jetty/webapps/
COPY jetty/target/webapps/*.war /opt/webapps/

So basically any steps that just add files to the image do not take the original permissions into account.

@Kars-de-Jong I verified your docker file on PR #962 and looks like the permissions are now correct

/ # ls -al var/lib/jetty/
total 24
drwxr-xr-x    6 jetty    jetty         4096 Feb  3 23:04 .
drwxr-xr-x   11 root     root          4096 Feb  3 23:04 ..
drwxr-xr-x    3 jetty    jetty         4096 Feb  3 23:04 lib
drwxr-xr-x    2 jetty    jetty         4096 Feb  3 23:04 resources
drwxr-xr-x    2 jetty    jetty         4096 Feb  3 23:04 start.d
drwxr-xr-x    2 jetty    jetty         4096 Feb  3 23:04 webapps

[tejal29]

@everflux I ran your dockerfile on #962 and looks like it works.

/ #  /kaniko/executor -f dockerfiles/Dockerfile_test1 --context=dir:///workspace --destination=gcr.io/tejal-test/test --tarPath=image.tar
INFO[0000] Resolved base name trion/ng-cli:latest to trion/ng-cli:latest 
INFO[0000] Using dockerignore file: /workspace/.dockerignore 
INFO[0000] Resolved base name trion/ng-cli:latest to trion/ng-cli:latest 
INFO[0000] Retrieving image manifest trion/ng-cli:latest 
INFO[0001] Retrieving image manifest trion/ng-cli:latest 
INFO[0002] Built cross stage deps: map[]                
INFO[0002] Retrieving image manifest trion/ng-cli:latest 
INFO[0002] Retrieving image manifest trion/ng-cli:latest 
INFO[0003] Unpacking rootfs as cmd RUN ls -la; touch foo; ls -la requires it. 
INFO[0012] Taking snapshot of full filesystem...        
INFO[0015] RUN ls -la; touch foo; ls -la                
INFO[0015] cmd: /bin/sh                                 
INFO[0015] args: [-c ls -la; touch foo; ls -la]         
total 8
drwxr-xr-x 2 node root 4096 Feb  3 23:14 .
drwxr-xr-x 1 root root 4096 Feb  3 23:16 ..
total 8
drwxr-xr-x 2 node root 4096 Feb  3 23:17 .
drwxr-xr-x 1 root root 4096 Feb  3 23:16 ..
-rw-r--r-- 1 node root    0 Feb  3 23:17 foo
INFO[0015] Taking snapshot of full filesystem...        
INFO[0017] COPY . .                                     
INFO[0017] Taking snapshot of files...                  
INFO[0028] RUN ls -la; rm foo; ls -la                   
INFO[0028] cmd: /bin/sh                                 
INFO[0028] args: [-c ls -la; rm foo; ls -la]            
total 347436
drwxr-xr-x 8 407936 89939      4096 Feb  3 23:17 .
drwxr-xr-x 1 root   root       4096 Feb  3 23:16 ..
-rw-r--r-- 1 node   root          0 Feb  3 23:17 foo
rm: cannot remove 'foo': Permission denied
total 347436
drwxr-xr-x 8 407936 89939      4096 Feb  3 23:17 .
drwxr-xr-x 1 root   root       4096 Feb  3 23:16 ..
-rw-r--r-- 1 node   root          0 Feb  3 23:17 foo

[tejal29]

@everflux @hobti01 and @Kars-de-Jong @gugahoi @swistaczek
Can you please verify your dockerfile on this image

gcr.io/kaniko-project/executor:550
gcr.io/kaniko-project/executor:debug550

Thanks a lot in advance!
Tejal

[Kars-de-Jong]

@tejal29 I can confirm it works now!

$ docker run test:latest
2020-02-04 12:16:39.932:INFO::main: Logging initialized @516ms to org.eclipse.jetty.util.log.StdErrLog
2020-02-04 12:16:40.524:INFO:oejs.Server:main: jetty-9.4.18.v20190429; built: 2019-04-29T20:42:08.989Z; git: e1bc35120a6617ee3df052294e433f3a25ce7097; jvm 1.8.0_212-b04
...

Output of ls -l in /var/lib of the container:

/var/lib $ ls -l
total 16
drwxr-xr-x    2 root     root          4096 May  9  2019 apk
drwxr-xr-x    1 jetty    jetty         4096 Feb  4 12:16 jetty
drwxr-xr-x    2 root     root          4096 May  9  2019 misc
drwxr-xr-x    2 root     root          4096 May  9  2019 udhcpd
/var/lib $ 

Thanks!

(I only tested with gcr.io/kaniko-project/executor:550 by the way).

[tejal29]

will be available in next release on Friday.