Kaniko: Symlinks to absolute paths don't work properly

[cpointner]

Actual behavior
Ubuntu images have a symlink /var/run that points to the absolute path /run. Any file that gets created inside /var/run does not end up in the image. If i change the link to be relative everything works as expected.

Expected behavior
Symlinks to absolute paths must be based on the image root.

To Reproduce

Create a temporary directory and copy the following Dockerfile to it:

FROM ubuntu:bionic
RUN mkdir -p /var/run/foo /run/bar
CMD [ "find", "/run" ]

Inside the directory run the following commands:

docker run --rm -it -v $(pwd):/workspace gcr.io/kaniko-project/executor:latest --dockerfile=/workspace/Dockerfile --context=/workspace --destination=symlinks:kaniko --tarPath /workspace/symlinks-kaniko.tar --no-push
docker load < symlinks-kaniko.tar
docker run --rm -it symlinks:kaniko

This yields the following output:

/run
/run/bar
/run/systemd
/run/systemd/container
/run/lock
/run/utmp
/run/mount
/run/mount/utab

Whereas the image built using:

docker build -t symlinks:docker .
docker run --rm -it symlinks:docker

yields the following output:

/run
/run/bar
/run/foo
/run/systemd
/run/systemd/container
/run/lock
/run/utmp
/run/mount
/run/mount/utab

Mind the missing foo in the first output.

At the time i tested this the kaniko image latest tag pointed to sha256:9c40a04cf1bc9d886f7f000e0b7fa5300c31c89e2ad001e97eeeecdce9f07a29

Triage Notes for the Maintainers

| Description | Yes/No |
|----------------|---------------|
| Please check if this a new feature you are proposing |

• - [ ]

|
| Please check if the build works in docker but not in kaniko |

• - [x]

|
| Please check if this error is seen when you use --cache flag |

• - [x]

|
| Please check if your dockerfile is a multistage dockerfile |

• - [ ]

|

[cvgw]

Hi @cpointner is it possible this is a duplicate of https://github.com/GoogleContainerTools/kaniko/issues/506 and not specific to symlinks?

[MarkusTeufelberger]

Why should "/var/run" be a special case that's treated differently by kaniko? Maybe #506 is the special case and the symlink issue the general one (with #506 being one instance of the underlying issue that absolute symlinks don't work properly in the "chroot" environment of kaniko's build).

[cvgw]

@MarkusTeufelberger /var/run is part of the explicit list of paths ignored by kaniko https://github.com/GoogleContainerTools/kaniko/blob/b0577768490fc619fdb93e7221096e303caa0b65/pkg/util/fs_util.go#L52

It may be that there is a general issue with absolute symlinks, but I think it would be helpful to have a repro case that doesn't involve /var/run.

As to why /var/run is ignored, I'm not sure. I believe it was ignored for performance reasons. I think @priyawadhwa would be able to give more context.

[cpointner]

You are right the following Dockerfile works as expected:

FROM ubuntu:bionic
RUN mkdir -p /dest /src && ln -s /dest /src/link
RUN mkdir -p /dest/foo /src/link/bar
CMD [ "find", "/dest" ]

Still i don't get the rationale behind this. Why would somebody want to mount /var/run/docker.sock during image build inside the image that is about to be built?

[tejal29]

@cpointner its is very common to mount docker.sock during building an image.
Bind mounting the Docker daemon socket into a container is used to communicate with and control the docker daemon inside the container.

That being said, we want to add a feature to ignore the explicit list of paths if it is present in the dockerfile.

[cpointner]

@tejal29 sorry but i still don't get why someone needs this during image build time. I know there are some use-cases for mounting the docker socket to the container once it is started but i have never seen somebody use this during image build time...

However thanks for your response. What do you mean by

if it is present in the dockerfile

How would this work?

[tejal29]

@cpointner i meant, if /var/run is present as destination in Dockerfile then we should be not whitelisting it.
@cvgw just added a PR to to remove /var/run from default whitelist and add another option to command line additional-whitelist.

[dinvlad]

We've just run into this issue in Cloud Build:

error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/docker.sock: device or resource busy

The error seems repeatable, and has never manifested before.. We're using :debug image, whose latest version was published an hour ago.

[dinvlad]

I think this is due to https://github.com/GoogleContainerTools/kaniko/commit/72bfed18506977054bb05589e87cb7b920b11823#diff-d36eb675aa49a7b471e3a2be77005b18 (I'll create a new issue, if needed).

EDIT: I can see that --additional-whitelist=/var/run needs to be used, from now on. It is a nuisance, however..

[tejal29]

@cpointner I verified your issue on the latest kaniko image
This is now fixed on the master

/kaniko/executor -f Dockerfile --context=dir://workspace --destination=gcr.io/tejal-test/test1 --whitelist-var-run=false

tejaldesai@@example$ docker run --rm -it gcr.io/tejal-test/test1
Unable to find image 'gcr.io/tejal-test/test1:latest' locally
latest: Pulling from tejal-test/test1
23884877105a: Already exists 
bc38caa0f5b9: Already exists 
2910811b6c42: Already exists 
36505266dcc6: Already exists 
2a57e2e14962: Pull complete 
Digest: sha256:ef8954e8564cb486fc22abf4271859a258b2b6ecb203b6ab7bece1bd8fa2123e
Status: Downloaded newer image for gcr.io/tejal-test/test1:latest
/run
/run/utmp
/run/mount
/run/mount/utab
/run/lock
/run/bar
/run/foo
/run/systemd
/run/systemd/container
tejaldesai@@example$ 

[tejal29]

I am closing this now, please re-open if you see this again!