Kaniko: COPY file Unpacking rootfs fail

[noony]

Actual behavior
Image is not built. End with error : error building image: error building stage: Unable to add parent dir /var to layered map: lstat /var: no such file or directory

Expected behavior
Image is built like docker build

To Reproduce
In a directory create Dockerfile and script.sh with this content :
Dockerfile

FROM rabbitmq:3.6
COPY script.sh /usr/local/bin/
CMD ["script.sh"]
````
script.sh

echo "hey"

Launch build and result will be :

docker run -v $(pwd):/workspace
gcr.io/kaniko-project/executor:debug
--dockerfile=/workspace/Dockerfile --context=/workspace
--destination=test.test.com/otto:latest --no-push
INFO[0000] Downloading base image rabbitmq:3.6
2019/01/24 16:52:08 No matching credentials were found, falling back on anonymous
INFO[0001] Executing 0 build triggers
INFO[0001] Taking snapshot of full filesystem...
INFO[0001] Skipping paths under /kaniko, as it is a whitelisted directory
INFO[0001] Skipping paths under /workspace, as it is a whitelisted directory
INFO[0001] Skipping paths under /sys, as it is a whitelisted directory
INFO[0001] Skipping paths under /dev, as it is a whitelisted directory
INFO[0001] Skipping paths under /proc, as it is a whitelisted directory
INFO[0001] Skipping paths under /busybox, as it is a whitelisted directory
INFO[0001] Using files from context: [/workspace/script.sh]
INFO[0001] COPY script.sh /usr/local/bin/
INFO[0001] Taking snapshot of files...
error building image: error building stage: Unable to add parent dir /var to layered map: lstat /var: no such file or directory

If you add a ls /usr it works :
```Dockerfile

FROM rabbitmq:3.6
RUN ls /usr

COPY script.sh /usr/local/bin/
CMD ["script.sh"]

log :

INFO[0000] Downloading base image rabbitmq:3.6
2019/01/24 16:53:56 No matching credentials were found, falling back on anonymous
INFO[0001] Executing 0 build triggers
INFO[0001] Unpacking rootfs as cmd RUN ls /usr requires it.
INFO[0010] Taking snapshot of full filesystem...
INFO[0011] Skipping paths under /kaniko, as it is a whitelisted directory
INFO[0011] Skipping paths under /workspace, as it is a whitelisted directory
INFO[0011] Skipping paths under /sys, as it is a whitelisted directory
INFO[0011] Skipping paths under /dev, as it is a whitelisted directory
INFO[0011] Skipping paths under /proc, as it is a whitelisted directory
INFO[0011] Skipping paths under /busybox, as it is a whitelisted directory
INFO[0012] RUN ls /usr
INFO[0012] cmd: /bin/sh
INFO[0012] args: [-c ls /usr]
bin
games
include
lib
local
sbin
share
src
INFO[0012] Taking snapshot of full filesystem...
INFO[0013] Skipping paths under /kaniko, as it is a whitelisted directory
INFO[0013] Skipping paths under /workspace, as it is a whitelisted directory
INFO[0013] Skipping paths under /sys, as it is a whitelisted directory
INFO[0013] Skipping paths under /dev, as it is a whitelisted directory
INFO[0013] Skipping paths under /proc, as it is a whitelisted directory
INFO[0013] Skipping paths under /busybox, as it is a whitelisted directory
INFO[0014] No files were changed, appending empty layer to config. No layer added to image.
INFO[0014] Using files from context: [/workspace/script.sh]
INFO[0014] COPY script.sh /usr/local/bin/
INFO[0014] Taking snapshot of files...
INFO[0014] CMD ["script.sh"]
INFO[0014] Skipping push to container registry due to --no-push flag

gcr.io/kaniko-project/executor debug 0c4048081679 21 hours ago 47MB

It seems that adding --single-snapshot command argument fix the problem temporary too.

[dlorenc]

Thanks! I've been able to repro this.

[dlorenc]

This was caused by an unfortunate interaction between the strange way volumes are handled in Dockerfiles and several optimizations we made around unpacking base images and snapshotting.

[noony]

Hi, thanks for your quick response. I'll try your pull request :)

[noony]

Hi, It works on version v0.9.0 thanks for your fix !

[f0o]

Hi it seems like this issue just appeared again.

I got build jobs failing that were working just 3h ago.

Error:

INFO[0006] Unpacking rootfs as cmd COPY . /src requires it. 
error building image: error building stage: failed to get filesystem from image: error removing var/run to make way for new symlink: unlinkat /var/run/secrets/kubernetes.io/serviceaccount/..2020_01_27_14_28_26.514721149: read-only file system

Running as tagged image debug_793 solves it - So somewhere regression was introduced.

[littlexsparkee]

I had the same issue, seemed to be fixed by specifying v0.16.0 in the cloud build yaml.

[evheny0]

I've had similar issue:

INFO[0012] Unpacking rootfs as cmd RUN apk add --no-cache build-base ca-certificates curl git nginx tzdata sudo postgresql-client postgresql-dev nodejs yarn openssh requires it. 
error building image: error building stage: failed to get filesystem from image: chtimes /bin/busybox: invalid argument

and specifying v0.16.0 worked, thanks @littlexsparkee