Actual behavior
Since v0.18.0, COPY tries to resolve some symlinks it shouldn't try to resolve, which is a regression.
Expected behavior
I would expect kaniko to let those symlinks remain symlinks when copied in the image.
To Reproduce
For instance, we can create a symlink that points to nothing to make kaniko crash:
mkdir example
cd example/
echo "FROM alpine\nCOPY . /example" > Dockerfile
ln -s ./missing symlink
docker run --rm -it -v $(pwd):/workspace:ro gcr.io/kaniko-project/executor:debug --no-push
:point_down: :point_down: :point_down:
[โฆ]
error building image: error building stage: failed to execute command: lstat /workspace/missing: no such file or directory
If we create the symlink target, it builds successfully but the symlink is replaced with a plain file within the image:
touch missing
docker run --rm -it -v $(pwd):/workspace:ro --entrypoint='' gcr.io/kaniko-project/executor:debug /busybox/sh -c '/kaniko/executor --no-push && ls -l /example/'
:point_down: :point_down: :point_down:
[โฆ]
total 4
-rw-r--r-- 1 1000 1000 29 Mar 6 14:28 Dockerfile
-rw-r--r-- 1 1000 1000 0 Mar 6 14:28 missing
-rw-r--r-- 1 1000 1000 0 Mar 6 14:28 symlink
Whereas in v0.17.1 (or using docker build) the symlink is not resolved:
$ docker run --rm -it -v $(pwd):/workspace:ro --entrypoint='' gcr.io/kaniko-project/executor:debug-v0.17.1 /busybox/sh -c '/kaniko/executor --no-push && ls -l /example/'
[โฆ]
total 8
-rw-r--r-- 1 1000 1000 29 Mar 6 14:29 Dockerfile
-rw-r--r-- 1 1000 1000 0 Mar 6 14:29 missing
lrwxrwxrwx 1 root root 9 Mar 6 14:29 symlink -> ./missing
Additional Information
This is a regression from v0.18.0. It worked properly in v0.17.1.
Note that Kaniko v0.18.0 changed the way symlinks work to better match actual Docker's behavior. That probably introduced this bug as a side-effect.
Triage Notes for the Maintainers
| Description | Yes/No |
|----------------|---------------|
| Please check if this a new feature you are proposing |
--cache flag | FWIW, this was probably introduced by #943 .
@gilbsgilbs Thanks for the issue.
We changed copy to be as close to docker build.
I verified your case on docker version 1.9.03
Dockerfile
tejaldesai@@tmep$ cat Dockerfile
FROM scratch
COPY deadlink workdir/
My workdir
tejaldesai@@tmep$ ls -al
total 24
drwxr-xr-x 5 tejaldesai primarygroup 4096 Mar 6 15:49 .
drwxr-x--- 11 tejaldesai primarygroup 4096 Mar 6 11:57 ..
lrwxrwxrwx 1 tejaldesai primarygroup 4 Mar 6 15:48 deadlink -> test
Dockerbuild log
tejaldesai@@tmep$ docker build .
Sending build context to Docker daemon 7.68kB
Step 1/2 : FROM scratch
--->
Step 2/2 : COPY deadlink workdir/
COPY failed: stat /usr/local/google/docker/tmp/docker-builder885367293/test: no such file or directory
We also made changes to make COPY behavior is mimic to docker build
Dockerfile
FROM scratch
COPY sym.link workdir/
My dir
tejaldesai@@tmep$ ls -al
drwxr-xr-x 5 tejaldesai primarygroup 4096 Mar 6 15:56 .
drwxr-x--- 11 tejaldesai primarygroup 4096 Mar 6 11:57 ..
-rw-r--r-- 1 tejaldesai primarygroup 37 Mar 6 15:56 Dockerfile
-rw-r--r-- 1 tejaldesai primarygroup 6 Mar 6 15:56 hello
lrwxrwxrwx 1 tejaldesai primarygroup 5 Mar 6 15:56 sym.link -> hello
Run docker build
context: path ",." not found
tejaldesai@@tmep$ docker build .
Sending build context to Docker daemon 8.704kB
Step 1/2 : FROM scratch
--->
Step 2/2 : COPY sym.link workdir/
---> 3725f395af68
Successfully built 3725f395af68
Then, i exploded the built image.
tejaldesai@@tar$ tar -xvf 48233d291de8fc27f595bf188b1ed62de4ca3d3a562add526f302e8220e8385c/layer.tar
workdir/
workdir/sym.link
tejaldesai@@tar$ ls -al workdir/
total 12
drwxr-xr-x 2 tejaldesai primarygroup 4096 Mar 6 15:57 .
drwxr-xr-x 4 tejaldesai primarygroup 4096 Mar 6 15:59 ..
-rw-r--r-- 1 tejaldesai primarygroup 6 Mar 6 15:56 sym.link
tejaldesai@@tar$
If you see the sym.link is a regular file.
However, if you copy a directory, then symlinks are preserved.
Dockerfile
FROM scratch
COPY src/ workdir/
My dir
tejaldesai@@tmep$ ls -al src/
total 12
drwxr-xr-x 2 tejaldesai primarygroup 4096 Mar 6 15:19 .
drwxr-xr-x 4 tejaldesai primarygroup 4096 Mar 6 16:01 ..
lrwxrwxrwx 1 tejaldesai primarygroup 5 Mar 6 15:02 sym1.link -> test1
-rw-r--r-- 1 tejaldesai primarygroup 10 Mar 6 15:02 test1
lrwxrwxrwx 1 tejaldesai primarygroup 19 Mar 6 15:19 zSym.link -> ../anotherSrc/hello
tejaldesai@@tmep$ ls -al anotherSrc/
total 12
drwxr-xr-x 2 tejaldesai primarygroup 4096 Mar 6 15:18 .
drwxr-xr-x 4 tejaldesai primarygroup 4096 Mar 6 16:01 ..
-rw-r--r-- 1 tejaldesai primarygroup 14 Mar 6 15:18 hello
tejaldesai@@tmep$
Build Docker image
Sending build context to Docker daemon 7.68kB
Step 1/2 : FROM scratch
--->
Step 2/2 : COPY src/ workdir/
---> Using cache
---> 0ab1ea7e4e3f
Successfully built 0ab1ea7e4e3f
Exploded image tar
tejaldesai@@tar$ ls -al workdir/
total 12
drwxr-xr-x 2 tejaldesai primarygroup 4096 Mar 6 15:19 .
drwxr-xr-x 4 tejaldesai primarygroup 4096 Mar 6 16:03 ..
lrwxrwxrwx 1 tejaldesai primarygroup 5 Mar 6 15:02 sym1.link -> test1
-rw-r--r-- 1 tejaldesai primarygroup 10 Mar 6 15:02 test1
lrwxrwxrwx 1 tejaldesai primarygroup 19 Mar 6 15:19 zSym.link -> ../anotherSrc/hello
tejaldesai@@tar$
In this case, zSymlink is a deadlink since anotherSrc wasn't copied over.
FROM scratch
COPY src/ workdir/
My workdir
tejaldesai@@src$ ls -al
total 12
drwxr-xr-x 2 tejaldesai primarygroup 4096 Mar 6 16:21 .
drwxr-xr-x 5 tejaldesai primarygroup 4096 Mar 6 16:20 ..
lrwxrwxrwx 1 tejaldesai primarygroup 1 Mar 6 16:21 deadlink -> t
lrwxrwxrwx 1 tejaldesai primarygroup 5 Mar 6 15:02 sym1.link -> test1
-rw-r--r-- 1 tejaldesai primarygroup 10 Mar 6 15:02 test1
lrwxrwxrwx 1 tejaldesai primarygroup 19 Mar 6 15:19 zSym.link -> ../anotherSrc/hello
tejaldesai@@src$
Docker build
Step 1/2 : FROM scratch
--->
Step 2/2 : COPY src/ workdir/
---> 81a9999565db
Successfully built 81a9999565db
Exploded tar
tejaldesai@@tard$ ls -al workdir/
total 12
drwxr-xr-x 2 tejaldesai primarygroup 4096 Mar 6 16:23 .
drwxr-xr-x 4 tejaldesai primarygroup 4096 Mar 6 16:24 ..
lrwxrwxrwx 1 tejaldesai primarygroup 1 Mar 6 16:21 deadline -> t
lrwxrwxrwx 1 tejaldesai primarygroup 5 Mar 6 15:02 sym1.link -> test1
-rw-r--r-- 1 tejaldesai primarygroup 10 Mar 6 15:02 test1
lrwxrwxrwx 1 tejaldesai primarygroup 19 Mar 6 15:19 zSym.link -> ../anotherSrc/hello
I'm also seeing build failures that are likely caused by this when using npm link during image build. All fine in v0.17.1.
@WestonThayer can you please provide your dockerfile?
I am not very familiar with npm link
@tejal29 I agree that Kaniko is closer to Docker's behavior in the cases you described and that the title of this issue may be slightly incorrect (I'll fix it). But I insist, the case I gave in the original post still is a regression regarding docker 19.03 and kaniko 0.17.1 behavior. Kaniko v0.18.0 tries to resolve some symlinks it shouldn't resolve. The regression label should be kept. Are you unable to reproduce the two repro I gave in the original post?
[nils@buks example]$ lsb_release -a
LSB Version: 1.4
Distributor ID: Arch
Description: Arch Linux
Release: rolling
Codename: n/a
[nils@buks example]$ uname -a
Linux buks 5.5.6-arch1-1 #1 SMP PREEMPT Mon, 24 Feb 2020 12:20:16 +0000 x86_64 GNU/Linux
[nils@buks example]$ docker --version
Docker version 19.03.6-ce, build 369ce74a3c
[nils@buks example]$ cat Dockerfile
FROM alpine
COPY . /example
[nils@buks example]$ ls -la
total 4
drwxr-xr-x 2 nils nils 80 Mar 7 06:13 .
drwxrwxrwt 16 root root 360 Mar 7 06:23 ..
-rw-r--r-- 1 nils nils 28 Mar 7 06:13 Dockerfile
lrwxrwxrwx 1 nils nils 9 Mar 7 06:13 symlink -> ./missing
works :point_down:
[nils@buks example]$ sudo docker build --no-cache -t testimage .
Sending build context to Docker daemon 2.56kB
Step 1/2 : FROM alpine
---> e7d92cdc71fe
Step 2/2 : COPY . /example
---> 881fa1ebd1ec
Successfully built 881fa1ebd1ec
Successfully tagged testimage:latest
[nils@buks example]$ sudo docker run --rm -it testimage ls -la /example
total 8
drwxr-xr-x 1 root root 34 Mar 7 05:23 .
drwxr-xr-x 1 root root 148 Mar 7 05:23 ..
-rw-r--r-- 1 root root 28 Mar 7 05:13 Dockerfile
lrwxrwxrwx 1 root root 9 Mar 7 05:13 symlink -> ./missing
Crashes :point_down:
[nils@buks example]$ sudo docker run --rm -it -v $(pwd):/workspace:ro gcr.io/kaniko-project/executor:debug-v0.18.0 --no-push
INFO[0000] Resolved base name alpine to alpine
INFO[0000] Resolved base name alpine to alpine
INFO[0000] Retrieving image manifest alpine
INFO[0001] Retrieving image manifest alpine
INFO[0002] Built cross stage deps: map[]
INFO[0002] Retrieving image manifest alpine
INFO[0002] Retrieving image manifest alpine
INFO[0003] Unpacking rootfs as cmd COPY . /example requires it.
INFO[0004] Taking snapshot of full filesystem...
INFO[0004] Resolving paths
INFO[0004] COPY . /example
error building image: error building stage: failed to execute command: lstat /workspace/missing: no such file or directory
works :point_down:
[nils@buks example]$ sudo docker run --rm -it -v $(pwd):/workspace:ro gcr.io/kaniko-project/executor:debug-v0.17.1 --no-push
INFO[0000] Resolved base name alpine to alpine
INFO[0000] Resolved base name alpine to alpine
INFO[0000] Retrieving image manifest alpine
INFO[0001] Retrieving image manifest alpine
INFO[0002] Built cross stage deps: map[]
INFO[0002] Retrieving image manifest alpine
INFO[0002] Retrieving image manifest alpine
INFO[0003] Unpacking rootfs as cmd COPY . /example requires it.
INFO[0003] Taking snapshot of full filesystem...
INFO[0004] COPY . /example
INFO[0004] Taking snapshot of files...
INFO[0004] Skipping push to container registry due to --no-push flag
[nils@buks example]$ ls -la
total 4
drwxr-xr-x 2 nils nils 100 Mar 7 06:36 .
drwxrwxrwt 16 root root 360 Mar 7 06:36 ..
-rw-r--r-- 1 nils nils 28 Mar 7 06:13 Dockerfile
-rw-r--r-- 1 nils nils 0 Mar 7 06:36 missing
lrwxrwxrwx 1 nils nils 9 Mar 7 06:13 symlink -> ./missing
[nils@buks example]$ cat Dockerfile
FROM alpine
COPY . /example
Doesn't crash and doesn't resolve the symlink
[nils@buks example]$ sudo docker build --no-cache -t testimage .
Sending build context to Docker daemon 3.072kB
Step 1/2 : FROM alpine
---> e7d92cdc71fe
Step 2/2 : COPY . /example
---> 89d33df5422b
Successfully built 89d33df5422b
Successfully tagged testimage:latest
[nils@buks example]$ sudo docker run --rm -it testimage ls -la /example
total 8
drwxr-xr-x 1 root root 48 Mar 7 05:36 .
drwxr-xr-x 1 root root 148 Mar 7 05:36 ..
-rw-r--r-- 1 root root 28 Mar 7 05:13 Dockerfile
-rw-r--r-- 1 root root 0 Mar 7 05:36 missing
lrwxrwxrwx 1 root root 9 Mar 7 05:13 symlink -> ./missing
Doesn't crash but resolves the symlink:
[nils@buks example]$ sudo docker run --rm -it -v $(pwd):/workspace:ro --entrypoint='' gcr.io/kaniko-project/executor:debug-v0.18.0 /busybox/sh -c '/kaniko/executor --no-push && ls -l /example/'
INFO[0000] Resolved base name alpine to alpine
INFO[0000] Resolved base name alpine to alpine
INFO[0000] Retrieving image manifest alpine
INFO[0001] Retrieving image manifest alpine
INFO[0002] Built cross stage deps: map[]
INFO[0002] Retrieving image manifest alpine
INFO[0002] Retrieving image manifest alpine
INFO[0003] Unpacking rootfs as cmd COPY . /example requires it.
INFO[0003] Taking snapshot of full filesystem...
INFO[0003] Resolving paths
INFO[0003] COPY . /example
INFO[0003] Resolving paths
INFO[0003] Taking snapshot of files...
INFO[0003] Skipping push to container registry due to --no-push flag
total 4
-rw-r--r-- 1 1000 1000 28 Mar 7 05:39 Dockerfile
-rw-r--r-- 1 1000 1000 0 Mar 7 05:39 missing
-rw-r--r-- 1 1000 1000 0 Mar 7 05:39 symlink
Doesn't crash and doesn't resolve the symlink:
[nils@buks example]$ sudo docker run --rm -it -v $(pwd):/workspace:ro --entrypoint='' gcr.io/kaniko-project/executor:debug-v0.17.1 /busybox/sh -c '/kaniko/executor --no-push && ls -l /example/'
INFO[0000] Resolved base name alpine to alpine
INFO[0000] Resolved base name alpine to alpine
INFO[0000] Retrieving image manifest alpine
INFO[0001] Retrieving image manifest alpine
INFO[0002] Built cross stage deps: map[]
INFO[0002] Retrieving image manifest alpine
INFO[0002] Retrieving image manifest alpine
INFO[0003] Unpacking rootfs as cmd COPY . /example requires it.
INFO[0003] Taking snapshot of full filesystem...
INFO[0003] COPY . /example
INFO[0003] Taking snapshot of files...
INFO[0003] Skipping push to container registry due to --no-push flag
total 8
-rw-r--r-- 1 1000 1000 28 Mar 7 05:39 Dockerfile
-rw-r--r-- 1 1000 1000 0 Mar 7 05:39 missing
lrwxrwxrwx 1 root root 9 Mar 7 05:39 symlink -> ./missing
Let me know if I can provide any further information that may help you.
Are you unable to reproduce the two repro I gave in the original post?
yes i am. The reason why we made that change was to mimic what docker does and picked up the regression. I will work on fixing it and fixing the tests.
Thanks. Sorry for the misunderstanding. Given that you removed the regression label and since your response sounded like โnot a bugโ to me, I wasn't sure you were actually able to reproduce :smile: .
No worries at all. I fixed up my PR to revert back to old behavior.
Can you try on your end to see if that fixes your issue?
gcr.io/kaniko-project/executor:debug
gcr.io/kaniko-project/executor:debug-edge
Thank you. I can confirm this issue is fixed on debug-edge.
The integration test is failing probably due to incorrect permissions.
I will take a look tomorrow
I can also confirm debug-edge is working for me.
I'm not having much luck getting this down to a minimal repro, but here's the general idea. Dockerfile:
FROM node:10-alpine
# Take advantage of staged Docker builds by taking care of npm installs
# separately
# This is a basic NPM package with it's own dependencies
WORKDIR /usr/src/packages/@company/backend-shared
COPY ./packages/@company/backend-shared/package*.json ./
RUN npm install
# This is a Node app that ran `npm install ../../packages/@company/backend-shared`,
# which creates a symlink from
# `/usr/src/services/backend/node_modules/@company/backend-shared`
# to
# `/usr/src/packages/@company/backend-shared`
WORKDIR /usr/src/services/backend/
COPY ./services/backend/package*.json ./
RUN npm install
# Copy dependency source and build
WORKDIR /usr/src/packages/@company/backend-shared
COPY ./packages/@company/backend-shared/ ./
RUN npm run build
# Copy source and build
# Crashes here
WORKDIR /usr/src/services/backend
Here's the error:
INFO[0024] WORKDIR /usr/src/services/backend/
INFO[0024] cmd: workdir
INFO[0024] Changed working directory to /usr/src/services/backend/
INFO[0024] Creating directory /usr/src/services/backend/
INFO[0024] Resolving paths
INFO[0024] Taking snapshot of files...
error building image: error building stage: failed to take snapshot: unable to add file /usr/src/services/backend/backend to layered map: error creating hash for /usr/src/services/backend/backend: lstat /usr/src/services/backend/backend: no such file or directory
ERROR
ERROR: build step 0 "gcr.io/kaniko-project/executor:latest" failed: step exited with non-zero status: 1
I tried reducing down to very basic Node modules and wasn't able to reproduce it. It could've been that I used a different path structure, different NPM dependencies, or that I was skipping the npm run build steps. Would be time consuming to track it down, but if the logs give you a clue I'd be happy to debug further down a specific path.
@WestonThayer Thanks for confirming!
INFO[0024] Taking snapshot of files...
error building image: error building stage: failed to take snapshot: unable to add file /usr/src/services/backend/backend to layered map: error creating hash for /usr/src/services/backend/backend: lstat /usr/src/services/backend/backend: no such file or directory
ERROR
Thanks @WestonThayer The error you are seeing is due to #1112 and not due to symlinks.
Also having symlink issues with an image containing a PHP app, where its dependency manager _(Composer)_ will create symlinks to provided binaries.
When building using Docker itself, the files are โ as expected โ symlinked into vendor/bin/:
bramus in ~/repos/vbridge/pubsub-process-cctv on develop*
$ docker run -it --entrypoint /bin/bash "pubsub-process-cctv"
root@7e98449610d1:/# ls -Al /app/vendor/bin/
total 0
lrwxrwxrwx 1 www-data www-data 43 Mar 16 12:48 google-cloud-batch -> ../google/cloud/Core/bin/google-cloud-batch
lrwxrwxrwx 1 www-data www-data 50 Mar 16 12:48 google-cloud-debugger -> ../google/cloud/Debugger/bin/google-cloud-debugger
lrwxrwxrwx 1 www-data www-data 26 Mar 16 12:48 phpunit -> ../phpunit/phpunit/phpunit
lrwxrwxrwx 1 www-data www-data 30 Mar 16 12:57 pubsub-pull -> ../plonk/plonk/bin/pubsub-pull
root@7e98449610d1:/#
When building using kaniko, the files in the vendor/bin/ are no symlinks but actual copies:
bramus in ~/repos/vbridge/pubsub-process-cctv on develop*
$ docker run -it --entrypoint /bin/bash "eu.gcr.io/REDACTED/pubsub-process-cctv:develop"
root@afbe6246ea6d:/# ls -Al /app/vendor/bin/
total 16
-rwxr-xr-x 1 www-data www-data 1729 Mar 16 13:59 google-cloud-batch
-rwxr-xr-x 1 www-data www-data 2125 Mar 16 13:59 google-cloud-debugger
-rwxr-xr-x 1 www-data www-data 1443 Mar 16 13:59 phpunit
-rwxr-xr-x 1 www-data www-data 2030 Mar 16 14:51 pubsub-pull
root@afbe6246ea6d:/#
For reference, the Composer dependencies were built in a separate image, and then COPY'd into the main image (e.g. multi-stage build):
FROM composer:1.9.1 as composer
COPY composer.json composer.lock /app/
RUN composer install \
--ignore-platform-reqs \
--no-interaction \
--no-plugins \
--no-scripts \
--prefer-dist \
--optimize-autoloader \
-vvv
โฆ
FROM php:7.3-cli
COPY --chown=www-data:www-data --from=composer /app/vendor/ /app/vendor/
โฆ
EDIT: Is this this issue, or is it #437 ?
@bramus Tried your issue on latest v0.19.0 build and it does not work. i will add a fix in next release.
docker run -it --entrypoint /busybox/sh -v /usr/local/google/home/tejaldesai/.config/gcloud:/root/.config/gcloud -v /usr/local/google/home/tejaldesai/workspace/kaniko/integration:/workspace gcr.io/kaniko-project/executor:debug
/ # /kaniko/executor -f dockerfiles/Dockerfile1 --context=dir://workspace --destination=gcr.io/tejal-test/test-symlink
...
I got this minimal Dockerfile to fail on v0.18.0 and v0.19.0 when building with target two using --target two (building target one works fine though, so it seems the other target is somehow interfering). v0.17.1 is working fine so I suspect it must be related to this issue. Hope it is of any help for testing. The Dockerfile:
FROM alpine:3.11 AS base
RUN mkdir -p /dir/source /dir/target
RUN echo test > /dir/target/file.txt
WORKDIR /dir/source
RUN ln -s ../target/file.txt file.txt
FROM scratch as one
COPY --from=base /dir/ /dir/
FROM scratch as two
COPY --from=base /dir/ /dir/
Resulting log from base stage:
INFO[0001] Retrieving image manifest alpine:3.11
INFO[0002] Built cross stage deps: map[0:[/dir/ /dir/]]
INFO[0002] Retrieving image manifest alpine:3.11
INFO[0003] Retrieving image manifest alpine:3.11
INFO[0004] Unpacking rootfs as cmd RUN mkdir -p /dir/source /dir/target requires it.
INFO[0004] Taking snapshot of full filesystem...
INFO[0004] Resolving paths
INFO[0004] RUN mkdir -p /dir/source /dir/target
INFO[0004] cmd: /bin/sh
INFO[0004] args: [-c mkdir -p /dir/source /dir/target]
INFO[0004] Taking snapshot of full filesystem...
INFO[0005] Resolving paths
INFO[0005] RUN echo test > /dir/target/file.txt
INFO[0005] cmd: /bin/sh
INFO[0005] args: [-c echo test > /dir/target/file.txt]
INFO[0005] Taking snapshot of full filesystem...
INFO[0005] Resolving paths
INFO[0005] WORKDIR /dir/source
INFO[0005] cmd: workdir
INFO[0005] Changed working directory to /dir/source
INFO[0005] RUN ln -s ../target/file.txt file.txt
INFO[0005] cmd: /bin/sh
INFO[0005] args: [-c ln -s ../target/file.txt file.txt]
INFO[0005] Taking snapshot of full filesystem...
INFO[0005] Resolving paths
INFO[0005] Saving file /dir/ for later use
INFO[0005] Saving file /dir/ for later use
error building image: symlink ../target/file.txt /kaniko/0/dir/source/file.txt: file exists
I tried the dockefile in https://github.com/GoogleContainerTools/kaniko/issues/1111#issuecomment-607793510
This was a easy fix where we did not dedupe the files to saved
https://github.com/GoogleContainerTools/kaniko/pull/1252
Will try to get in next release.
The issue in https://github.com/GoogleContainerTools/kaniko/issues/1111#issuecomment-599593569 is still not fixed.
The issue in #1111 (comment) is still not fixed.
Do I create a separate issue?
Most helpful comment
@bramus Tried your issue on latest v0.19.0 build and it does not work. i will add a fix in next release.