Kaniko: Long builds fail with "UNAUTHORIZED: \"authentication required\""

[Delphinator]

steps to reproduce

mkdir -p ./kaniko-issue

cat > ./kaniko-issue/Dockerfile <<EOF
FROM debian:stable-slim
RUN sleep 1
EOF
# This will work as expected
docker run -v `pwd`/kaniko-issue/:/workspace \
    gcr.io/kaniko-project/executor:latest \
    -c /workspace -f Dockerfile \
    -d registry.example.com/does/not:matter \
    --tarPath /workspace/tarball.tar

cat > ./kaniko-issue/Dockerfile <<EOF
FROM debian:stable-slim
RUN sleep 360
EOF
# This will fail with "UNAUTHORIZED: \"authentication required\""
docker run -v `pwd`/kaniko-issue/:/workspace \
    gcr.io/kaniko-project/executor:latest \
    -c /workspace -f Dockerfile \
    -d registry.example.com/does/not:matter \
    --tarPath /workspace/tarball.tar

# rm -r ./kaniko-issue

additional obervations

I also ran tcpdump on the network interface of the container. I saw a quite a bit of traffic at the start (I assume pulling the image) and a single, short TLS connection to index.docker.io after sleep was done.

The issue seems to be gone (or at least takes substantially longer to arise) if I substitute debian:stable-slim with any image from my harbor (private docker registry) instance.

working theory

My working theory of the underlying cause based on those two observations is, that kaniko tries to fetch the image config of the base image using an expired bearer token. This config would normally be extended and included in the tarball or pushed to the registry.

[priyawadhwa]

Hey @Delphinator , I tried to repro this issue but was able to build the image as expected:

$ cat Dockerfile 
FROM debian:stable-slim
RUN sleep 360
$ ./run_in_docker.sh Dockerfile $(pwd) gcr.io/priya-wadhwa/test:test
time="2018-07-25T21:50:49Z" level=info msg="Unpacking filesystem of debian:stable-slim..."
2018/07/25 21:50:49 No matching credentials found for index.docker.io, falling back on anonymous
time="2018-07-25T21:50:49Z" level=info msg="Mounted directories: [/kaniko /var/run /proc /dev /dev/pts /sys /sys/fs/cgroup /sys/fs/cgroup/systemd /sys/fs/cgroup/net_cls,net_prio /sys/fs/cgroup/cpu,cpuacct /sys/fs/cgroup/memory /sys/fs/cgroup/perf_event /sys/fs/cgroup/cpuset /sys/fs/cgroup/blkio /sys/fs/cgroup/freezer /sys/fs/cgroup/pids /sys/fs/cgroup/devices /dev/mqueue /workspace /etc/resolv.conf /etc/hostname /etc/hosts /dev/shm /root/.config/gcloud /proc/asound /proc/bus /proc/fs /proc/irq /proc/sys /proc/sysrq-trigger /proc/kcore /proc/keys /proc/timer_list /proc/sched_debug /sys/firmware]"
time="2018-07-25T21:50:50Z" level=info msg="Unpacking layer: 0"
time="2018-07-25T21:50:50Z" level=info msg="Not adding /dev because it is whitelisted"
time="2018-07-25T21:50:50Z" level=info msg="Not adding /etc/hostname because it is whitelisted"
time="2018-07-25T21:50:50Z" level=info msg="Not adding /etc/resolv.conf because it is whitelisted"
time="2018-07-25T21:50:50Z" level=info msg="Not adding /proc because it is whitelisted"
time="2018-07-25T21:50:50Z" level=info msg="Not adding /sys because it is whitelisted"
time="2018-07-25T21:50:51Z" level=info msg="Not adding /var/run because it is whitelisted"
time="2018-07-25T21:50:51Z" level=info msg="Taking snapshot of full filesystem..."
time="2018-07-25T21:50:53Z" level=info msg="cmd: /bin/sh"
time="2018-07-25T21:50:53Z" level=info msg="args: [-c sleep 360]"
time="2018-07-25T21:56:53Z" level=info msg="Taking snapshot of full filesystem..."
time="2018-07-25T21:56:54Z" level=info msg="No files were changed, appending empty layer to config."
2018/07/25 21:56:55 mounted blob: sha256:7d1463d31d7e5ad679ea175cd72afede858628ca49598925a17410e452d5ccec
2018/07/25 21:56:55 mounted blob: sha256:b5186294b6f665381d964ff1e51910d9c03599009ca8a8a54a66607f44daf490
gcr.io/priya-wadhwa/test:test: digest: sha256:d1244e42b3de8475275900102535ca00efdc05c9bbb4882c78ff816215028ea6 size: 429

We fetch the image config of the base image before starting to run commands here, and it's downloaded along with the base image, so an expired token shouldn't be the issue.

When pushing the image after running commands, we also get a new auth token.

I'm not sure why you're hitting this error since I couldn't repro the issue, but the only thing I can think of right now is that it's an auth issue with the registry you're trying to push to?

[Delphinator]

That's odd. I just tried it again and I'm consistently getting errors when sleeping 6 minutes. I do remember 6 minutes being somewhat close to the threshold though. Maybe 10 minutes will do the job in your environment?

I'm not convinced it's an auth issue with the private registry.

• During the initial debugging the only network traffic I saw after the sleep period was a very short connection to index.docker.io, not to the registry.
• I found no http requests in the access logs of the registry during the relevant time period.
• Below I used registry.example.com (a DNS lookup returns NXDOMAIN, so this should throw errors if there is any attempt at communication) and was able to successfully build tarballs with a short sleep period and got the same error if sleeping 6 minutes.

Just to be sure, this is the exact image I used and the docker version:

# docker inspect gcr.io/kaniko-project/executor:latest
[
    {
        "Id": "sha256:20b09335e9e6883fe370dff009b33a09530c1727835a96042f6733c506fba3ca",
        "RepoTags": [
            "gcr.io/kaniko-project/executor:latest"
        ],
        "RepoDigests": [
            "gcr.io/kaniko-project/executor@sha256:501056bf52f3a96f151ccbeb028715330d5d5aa6647e7572ce6c6c55f91ab374"
        ]
# [snip]

# docker --version
Docker version 18.06.0-ce, build 0ffa825

And logs of my most recent attempts:

trying to push to a registry with valid credentials in /root/.docker/config.json

sleep 1:

+ docker run -v /tmp/kaniko-issue/:/workspace -v /root/.docker:/root/.docker:ro gcr.io/kaniko-project/executor:latest -c /workspace -f Dockerfile -d [REDACTED]/[REDACTED]/test:latest
time="2018-07-26T09:05:44Z" level=info msg="Unpacking filesystem of debian:stable-slim..."
2018/07/26 09:05:44 No matching credentials found for index.docker.io, falling back on anonymous
time="2018-07-26T09:05:45Z" level=info msg="Mounted directories: [/kaniko /var/run /proc /dev /dev/pts /sys /sys/fs/cgroup /sys/fs/cgroup/systemd /sys/fs/cgroup/perf_event /sys/fs/cgroup/freezer /sys/fs/cgroup/net_cls,net_prio /sys/fs/cgroup/blkio /sys/fs/cgroup/cpu,cpuacct /sys/fs/cgroup/memory /sys/fs/cgroup/pids /sys/fs/cgroup/devices /sys/fs/cgroup/cpuset /dev/mqueue /workspace /root/.docker /etc/resolv.conf /etc/hostname /etc/hosts /dev/shm /proc/bus /proc/fs /proc/irq /proc/sys /proc/sysrq-trigger /proc/acpi /proc/kcore /proc/keys /proc/timer_list /proc/sched_debug /sys/firmware]"
time="2018-07-26T09:05:45Z" level=info msg="Unpacking layer: 0"
time="2018-07-26T09:05:46Z" level=info msg="Not adding /dev because it is whitelisted"
time="2018-07-26T09:05:46Z" level=info msg="Not adding /etc/hostname because it is whitelisted"
time="2018-07-26T09:05:46Z" level=info msg="Not adding /etc/resolv.conf because it is whitelisted"
time="2018-07-26T09:05:46Z" level=info msg="Not adding /proc because it is whitelisted"
time="2018-07-26T09:05:47Z" level=info msg="Not adding /sys because it is whitelisted"
time="2018-07-26T09:05:49Z" level=info msg="Not adding /var/run because it is whitelisted"
time="2018-07-26T09:05:49Z" level=info msg="Taking snapshot of full filesystem..."
time="2018-07-26T09:05:50Z" level=info msg="cmd: /bin/sh"
time="2018-07-26T09:05:50Z" level=info msg="args: [-c sleep 1]"
time="2018-07-26T09:05:51Z" level=info msg="Taking snapshot of full filesystem..."
time="2018-07-26T09:05:52Z" level=info msg="No files were changed, appending empty layer to config."
2018/07/26 09:05:52 pushed blob sha256:b5186294b6f665381d964ff1e51910d9c03599009ca8a8a54a66607f44daf490
2018/07/26 09:05:54 pushed blob sha256:7d1463d31d7e5ad679ea175cd72afede858628ca49598925a17410e452d5ccec
docker.[REDACTED]/[REDACTED]/test:latest: digest: sha256:d1244e42b3de8475275900102535ca00efdc05c9bbb4882c78ff816215028ea6 size: 429

sleep 360:

+ docker run -v /tmp/kaniko-issue/:/workspace -v /root/.docker:/root/.docker:ro gcr.io/kaniko-project/executor:latest -c /workspace -f Dockerfile -d [REDACTED]/[REDACTED]/test:latest
time="2018-07-26T08:53:02Z" level=info msg="Unpacking filesystem of debian:stable-slim..."
2018/07/26 08:53:02 No matching credentials found for index.docker.io, falling back on anonymous
time="2018-07-26T08:53:03Z" level=info msg="Mounted directories: [/kaniko /var/run /proc /dev /dev/pts /sys /sys/fs/cgroup /sys/fs/cgroup/systemd /sys/fs/cgroup/perf_event /sys/fs/cgroup/freezer /sys/fs/cgroup/net_cls,net_prio /sys/fs/cgroup/blkio /sys/fs/cgroup/cpu,cpuacct /sys/fs/cgroup/memory /sys/fs/cgroup/pids /sys/fs/cgroup/devices /sys/fs/cgroup/cpuset /dev/mqueue /workspace /root/.docker /etc/resolv.conf /etc/hostname /etc/hosts /dev/shm /proc/bus /proc/fs /proc/irq /proc/sys /proc/sysrq-trigger /proc/acpi /proc/kcore /proc/keys /proc/timer_list /proc/sched_debug /sys/firmware]"
time="2018-07-26T08:53:03Z" level=info msg="Unpacking layer: 0"
time="2018-07-26T08:53:04Z" level=info msg="Not adding /dev because it is whitelisted"
time="2018-07-26T08:53:04Z" level=info msg="Not adding /etc/hostname because it is whitelisted"
time="2018-07-26T08:53:04Z" level=info msg="Not adding /etc/resolv.conf because it is whitelisted"
time="2018-07-26T08:53:04Z" level=info msg="Not adding /proc because it is whitelisted"
time="2018-07-26T08:53:04Z" level=info msg="Not adding /sys because it is whitelisted"
time="2018-07-26T08:53:06Z" level=info msg="Not adding /var/run because it is whitelisted"
time="2018-07-26T08:53:06Z" level=info msg="Taking snapshot of full filesystem..."
time="2018-07-26T08:53:08Z" level=info msg="cmd: /bin/sh"
time="2018-07-26T08:53:08Z" level=info msg="args: [-c sleep 360]"
time="2018-07-26T08:59:08Z" level=info msg="Taking snapshot of full filesystem..."
time="2018-07-26T08:59:09Z" level=info msg="No files were changed, appending empty layer to config."
2018/07/26 08:59:09 pushed blob sha256:b5186294b6f665381d964ff1e51910d9c03599009ca8a8a54a66607f44daf490
time="2018-07-26T08:59:09Z" level=error msg="UNAUTHORIZED: \"authentication required\""

just building a tarball

sleep 1:

+ docker run -v /tmp/kaniko-issue/:/workspace gcr.io/kaniko-project/executor:latest -c /workspace -f Dockerfile -d registry.example.com/does/not:matter --tarPath /workspace/tarball.tar
time="2018-07-26T09:05:56Z" level=info msg="Unpacking filesystem of debian:stable-slim..."
2018/07/26 09:05:56 No matching credentials found for index.docker.io, falling back on anonymous
time="2018-07-26T09:05:57Z" level=info msg="Mounted directories: [/kaniko /var/run /proc /dev /dev/pts /sys /sys/fs/cgroup /sys/fs/cgroup/systemd /sys/fs/cgroup/perf_event /sys/fs/cgroup/freezer /sys/fs/cgroup/net_cls,net_prio /sys/fs/cgroup/blkio /sys/fs/cgroup/cpu,cpuacct /sys/fs/cgroup/memory /sys/fs/cgroup/pids /sys/fs/cgroup/devices /sys/fs/cgroup/cpuset /dev/mqueue /workspace /etc/resolv.conf /etc/hostname /etc/hosts /dev/shm /proc/bus /proc/fs /proc/irq /proc/sys /proc/sysrq-trigger /proc/acpi /proc/kcore /proc/keys /proc/timer_list /proc/sched_debug /sys/firmware]"
time="2018-07-26T09:05:57Z" level=info msg="Unpacking layer: 0"
time="2018-07-26T09:05:57Z" level=info msg="Not adding /dev because it is whitelisted"
time="2018-07-26T09:05:57Z" level=info msg="Not adding /etc/hostname because it is whitelisted"
time="2018-07-26T09:05:57Z" level=info msg="Not adding /etc/resolv.conf because it is whitelisted"
time="2018-07-26T09:05:58Z" level=info msg="Not adding /proc because it is whitelisted"
time="2018-07-26T09:05:58Z" level=info msg="Not adding /sys because it is whitelisted"
time="2018-07-26T09:05:59Z" level=info msg="Not adding /var/run because it is whitelisted"
time="2018-07-26T09:05:59Z" level=info msg="Taking snapshot of full filesystem..."
time="2018-07-26T09:06:01Z" level=info msg="cmd: /bin/sh"
time="2018-07-26T09:06:01Z" level=info msg="args: [-c sleep 1]"
time="2018-07-26T09:06:02Z" level=info msg="Taking snapshot of full filesystem..."
time="2018-07-26T09:06:03Z" level=info msg="No files were changed, appending empty layer to config."

sleep 360:

+ docker run -v /tmp/kaniko-issue/:/workspace gcr.io/kaniko-project/executor:latest -c /workspace -f Dockerfile -d registry.example.com/does/not:matter --tarPath /workspace/tarball.tar
time="2018-07-26T08:59:11Z" level=info msg="Unpacking filesystem of debian:stable-slim..."
2018/07/26 08:59:11 No matching credentials found for index.docker.io, falling back on anonymous
time="2018-07-26T08:59:12Z" level=info msg="Mounted directories: [/kaniko /var/run /proc /dev /dev/pts /sys /sys/fs/cgroup /sys/fs/cgroup/systemd /sys/fs/cgroup/perf_event /sys/fs/cgroup/freezer /sys/fs/cgroup/net_cls,net_prio /sys/fs/cgroup/blkio /sys/fs/cgroup/cpu,cpuacct /sys/fs/cgroup/memory /sys/fs/cgroup/pids /sys/fs/cgroup/devices /sys/fs/cgroup/cpuset /dev/mqueue /workspace /etc/resolv.conf /etc/hostname /etc/hosts /dev/shm /proc/bus /proc/fs /proc/irq /proc/sys /proc/sysrq-trigger /proc/acpi /proc/kcore /proc/keys /proc/timer_list /proc/sched_debug /sys/firmware]"
time="2018-07-26T08:59:13Z" level=info msg="Unpacking layer: 0"
time="2018-07-26T08:59:14Z" level=info msg="Not adding /dev because it is whitelisted"
time="2018-07-26T08:59:14Z" level=info msg="Not adding /etc/hostname because it is whitelisted"
time="2018-07-26T08:59:14Z" level=info msg="Not adding /etc/resolv.conf because it is whitelisted"
time="2018-07-26T08:59:14Z" level=info msg="Not adding /proc because it is whitelisted"
time="2018-07-26T08:59:14Z" level=info msg="Not adding /sys because it is whitelisted"
time="2018-07-26T08:59:16Z" level=info msg="Not adding /var/run because it is whitelisted"
time="2018-07-26T08:59:16Z" level=info msg="Taking snapshot of full filesystem..."
time="2018-07-26T08:59:17Z" level=info msg="cmd: /bin/sh"
time="2018-07-26T08:59:17Z" level=info msg="args: [-c sleep 360]"
time="2018-07-26T09:05:17Z" level=info msg="Taking snapshot of full filesystem..."
time="2018-07-26T09:05:19Z" level=info msg="No files were changed, appending empty layer to config."
time="2018-07-26T09:05:19Z" level=error msg="UNAUTHORIZED: \"authentication required\""

This also fails if I mount the docker configuration as above and change the target accordingly.

[priyawadhwa]

I tried again with 10 minutes, and it still worked fine:

time="2018-07-27T19:54:37Z" level=info msg="Unpacking filesystem of debian:stable-slim..."
2018/07/27 19:54:37 No matching credentials found for index.docker.io, falling back on anonymous
time="2018-07-27T19:54:38Z" level=info msg="Mounted directories: [/kaniko /var/run /proc /dev /dev/pts /sys /sys/fs/cgroup /sys/fs/cgroup/systemd /sys/fs/cgroup/net_cls,net_prio /sys/fs/cgroup/cpu,cpuacct /sys/fs/cgroup/memory /sys/fs/cgroup/perf_event /sys/fs/cgroup/cpuset /sys/fs/cgroup/blkio /sys/fs/cgroup/freezer /sys/fs/cgroup/pids /sys/fs/cgroup/devices /dev/mqueue /workspace /etc/resolv.conf /etc/hostname /etc/hosts /dev/shm /root/.config/gcloud /proc/asound /proc/bus /proc/fs /proc/irq /proc/sys /proc/sysrq-trigger /proc/kcore /proc/keys /proc/timer_list /proc/sched_debug /sys/firmware]"
time="2018-07-27T19:54:38Z" level=info msg="Unpacking layer: 0"
time="2018-07-27T19:54:39Z" level=info msg="Not adding /dev because it is whitelisted"
time="2018-07-27T19:54:39Z" level=info msg="Not adding /etc/hostname because it is whitelisted"
time="2018-07-27T19:54:39Z" level=info msg="Not adding /etc/resolv.conf because it is whitelisted"
time="2018-07-27T19:54:39Z" level=info msg="Not adding /proc because it is whitelisted"
time="2018-07-27T19:54:39Z" level=info msg="Not adding /sys because it is whitelisted"
time="2018-07-27T19:54:40Z" level=info msg="Not adding /var/run because it is whitelisted"
time="2018-07-27T19:54:40Z" level=info msg="Taking snapshot of full filesystem..."
time="2018-07-27T19:54:41Z" level=info msg="cmd: /bin/sh"
time="2018-07-27T19:54:41Z" level=info msg="args: [-c sleep 600]"
time="2018-07-27T20:04:41Z" level=info msg="Taking snapshot of full filesystem..."
time="2018-07-27T20:04:43Z" level=info msg="No files were changed, appending empty layer to config."
2018/07/27 20:04:44 mounted blob: sha256:7d1463d31d7e5ad679ea175cd72afede858628ca49598925a17410e452d5ccec
2018/07/27 20:04:44 mounted blob: sha256:b5186294b6f665381d964ff1e51910d9c03599009ca8a8a54a66607f44daf490
gcr.io/priya-wadhwa/test:test: digest: sha256:d1244e42b3de8475275900102535ca00efdc05c9bbb4882c78ff816215028ea6 size: 429

If you could submit a PR with a Dockerfile that breaks our CI we might be able to explore this issue a bit more. You'd just have to add another Dockerfile in the dockerfiles directory

[dlorenc]

Ping @Delphinator any luck getting a repro?

[MnrGreg]

I'm experiencing the same error message after 6mins and 20seconds of a NPM container build.

INFO[0281] Adding whiteout for /app/node_modules/extract-zip
INFO[0281] Adding whiteout for /app/node_modules/es6-promise/auto.js
INFO[0320] Storing source image from stage 0 at path /kaniko/stages/0
ERRO[0364] UNAUTHORIZED: "authentication required"

NPM Docker file:

# Stage 0, "build-stage", based on Node.js, to build and compile the frontend
FROM tiangolo/node-frontend:10 as build-stage
WORKDIR /app
COPY package*.json /app/
RUN npm install
COPY ./ /app/
ARG configuration=production
RUN npm run build -- --output-path=./dist/out --configuration $configuration

# Stage 1, based on Nginx, to have only the compiled app, ready for production with Nginx
FROM nginx:1.15
COPY --from=build-stage /app/dist/out/ /usr/share/nginx/html
# Copy the default nginx.conf provided by tiangolo/node-frontend
COPY --from=build-stage /nginx.conf /etc/nginx/conf.d/default.conf

I can however build and push the following sleepy container without issue.

FROM nginx:1.15
RUN sleep 360

INFO[0009] Taking snapshot of full filesystem...
INFO[0010] Executing 0 build triggers
INFO[0010] cmd: /bin/sh
INFO[0010] args: [-c sleep 360]
INFO[0370] Taking snapshot of full filesystem...
INFO[0371] No files were changed, appending empty layer to config.
2018/08/02 20:30:52 mounted blob: sha256:32d9726baeef1d8d626fa618fdc76200248b83e6453d642c25bddee92326d5ab
2018/08/02 20:30:52 mounted blob: sha256:be8881be8156e4068e611fe956aba2b9593ebd953be14fb7feea6d0659aa3abe
2018/08/02 20:30:52 mounted blob: sha256:87e5e6f71297704d2e6724248ededfb71e994dcf2fd9568b309b2d294e6ee455
2018/08/02 20:30:55 pushed blob sha256:f823acf1d9c5e1ad81a45e1220dd5718bdeae8354f8d1fee0f2be5eb260d7d2b
2018/08/02 20:30:57 container-registry.XXX/stores/XXweb:timerv1: digest: sha256:7ac9d1c278db35acd78a1da0dc6cbe2486b5838bab7c6334ec3879786dffe591 size: 754

Destination registry is Docker Trusted Registry 2.5.3

[Delphinator]

I'm experiencing the same error message after 6mins and 20seconds of a NPM container build.

I just pushed another attempt to reproduce in #267 with a two stage build. Hopefully it breaks.

I can however build and push the following sleepy container without issue.

I'm not 100% sure if 6 minutes are enough on a fast system. I did my experiments on a somewhat busy system with spinning disks. That does slow container startup and unpacking / packing down significantly. Maybe try 10 minutes just to be extra sure?

[Delphinator]

Another thought: I first encountered the issue while building images using the CI component of my private gitlab instance. I just created a new user there, added the SSH keys of @priyawadhwa and @dlorenc (github makes SSH keys public for some reason) to that user and pushed a repo, which reproduces the issue in gitlab CI. The repo and build logs are publicly accessible at https://gitlab.jensgutermuth.de/kaniko-issue-245/reproduce. Feel free to experiment (probably best on a branch, so you don't step on each others work).

The CI jobs in that repo first build kaniko using itself to facilitate adding debugging code or testing changes. Just change the git clone command in Dockerfile-kaniko. Using kaniko to build itself unfortunately requires a bit of hacky shell scripting and strategic moving around of files, since kaniko-build images cannot have files in /kaniko. Let's hope this won't cause issues.

[Delphinator]

Well, I'm completely confused now. Two stage builds seem to work for me (ignore the -o pipefail-stuff, that's just the gitlab runner expecting things).

[MnrGreg]

I did a bit more troubleshooting and noticed that when adding the 'COPY --from' the build succeeds 1/5 times with a 185 sleep, while without the 'COPY --from' it succeeds every time 5/5.

Decreasing the sleep time to below 60 seconds with the 'COPY --from' succeeds 5/5 times.

FROM container-registry.XX/stores/node-frontend:10 as build-stage
RUN sleep 185
FROM container-registry.XX/stores/nginx:1.15
COPY --from=build-stage /nginx.conf /etc/nginx/conf.d/default.conf
RUN echo "second stage is working"

INFO[0368] Not adding /kaniko/0/var/log/fontconfig.log because it was added by a prior layer
INFO[0368] Unpacking layer: 2
ERRO[0368] UNAUTHORIZED: "authentication required"

@Delphinator could we include the 'COPY --from prior build' into you tests- seems it make play a part?

[MnrGreg]

Just got an UNAUTHORIZED with a 60 sleep and at a different unpacking stage. Seems to be quite inconsistent.

Also seem that this is during a Pull from the registry and not the final image Push.

INFO[0046] Taking snapshot of full filesystem...
INFO[0053] Executing 0 build triggers
INFO[0053] cmd: /bin/sh
INFO[0053] args: [-c sleep 60]
INFO[0113] Taking snapshot of full filesystem...
INFO[0119] No files were changed, appending empty layer to config.
INFO[0119] Storing source image from stage 0 at path /kaniko/stages/0
INFO[0282] trying to extract to /kaniko/0
INFO[0282] Mounted directories: [/kaniko /var/run /proc /dev /dev/pts /sys /sys/fs/cgroup /sys/fs/cgroup/cpuset /sys/fs/cgroup/cpu /sys/fs/cgroup/cpuacct /sys/fs/cgroup/blkio /sys/fs/cgroup/memory /sys/fs/cgroup/devices /sys/fs/cgroup/freezer /sys/fs/cgroup/net_cls /sys/fs/cgroup/perf_event /sys/fs/cgroup/net_prio /sys/fs/cgroup/hugetlb /sys/fs/cgroup/pids /sys/fs/cgroup/systemd /dev/mqueue /workspace /etc/resolv.conf /etc/hostname /etc/hosts /dev/shm /root/.docker/config.json /kaniko/ssl/certs/ca-certificates.crt/dev/console /proc/bus /proc/fs /proc/irq /proc/sys /proc/sysrq-trigger /proc/acpi /proc/kcore /proc/keys /proc/timer_list /proc/sched_debug /sys/firmware /proc /dev /dev/pts /sys /sys/fs/cgroup /sys/fs/cgroup/cpuset /sys/fs/cgroup/cpu /sys/fs/cgroup/cpuacct /sys/fs/cgroup/blkio /sys/fs/cgroup/memory /sys/fs/cgroup/devices /sys/fs/cgroup/freezer /sys/fs/cgroup/net_cls /sys/fs/cgroup/perf_event /sys/fs/cgroup/net_prio /sys/fs/cgroup/hugetlb /sys/fs/cgroup/pids /sys/fs/cgroup/systemd /dev/mqueue /workspace /etc/resolv.conf /etc/hostname /etc/hosts /dev/shm /root/.docker/config.json /kaniko/ssl/certs/ca-certificates.crt /dev/console /proc/bus /proc/fs /proc/irq /proc/sys /proc/sysrq-trigger /proc/acpi /proc/kcore /proc/keys /proc/timer_list /proc/sched_debug /sys/firmware]
INFO[0282] Unpacking layer: 12
INFO[0283] Unpacking layer: 11
INFO[0316] Unpacking layer: 10
ERRO[0316] UNAUTHORIZED: "authentication required"

second run with timer at 10 seconds:

INFO[0329] Not adding /kaniko/0/app because it was added by a prior layer
INFO[0329] Unpacking layer: 9
INFO[0352] Whiting out /kaniko/0/usr/bin/.wh.curl
INFO[0355] Whiting out /kaniko/0/usr/share/doc/.wh.curl
INFO[0373] Whiting out /kaniko/0/usr/share/man/man1/.wh.curl.1.gz
INFO[0373] Whiting out /kaniko/0/var/lib/apt/lists/.wh.deb.debian.org_debian_dists_jessie-updates_InRelease
INFO[0373] Whiting out /kaniko/0/var/lib/apt/lists/.wh.deb.debian.org_debian_dists_jessie-updates_main_binary-amd64_Packages.gz
INFO[0373] Whiting out /kaniko/0/var/lib/apt/lists/.wh.deb.debian.org_debian_dists_jessie_Release
INFO[0373] Whiting out /kaniko/0/var/lib/apt/lists/.wh.deb.debian.org_debian_dists_jessie_Release.gpg
INFO[0373] Whiting out /kaniko/0/var/lib/apt/lists/.wh.deb.debian.org_debian_dists_jessie_main_binary-amd64_Packages.gz
INFO[0373] Whiting out /kaniko/0/var/lib/apt/lists/.wh.lock
INFO[0373] Whiting out /kaniko/0/var/lib/apt/lists/partial/.wh..wh..opq
INFO[0373] Whiting out /kaniko/0/var/lib/apt/lists/.wh.security.debian.org_debian-security_dists_jessie_updates_InRelease
INFO[0373] Whiting out /kaniko/0/var/lib/apt/lists/.wh.security.debian.org_debian-security_dists_jessie_updates_main_binary-amd64_Packages.gz
INFO[0373] Whiting out /kaniko/0/var/lib/dpkg/info/.wh.curl.list
INFO[0373] Whiting out /kaniko/0/var/lib/dpkg/info/.wh.curl.md5sums
INFO[0373] Unpacking layer: 8
ERRO[0373] UNAUTHORIZED: "authentication required"

[Delphinator]

@Delphinator could we include the 'COPY --from prior build' into you tests- seems it make play a part?

Sure! Let's see what happens: https://gitlab.jensgutermuth.de/kaniko-issue-245/reproduce/pipelines/2056

[Delphinator]

Jop. A build w/o copy --from succeeds, while a build using copy --from fails.

[MnrGreg]

Lucky us! You hit the error on the first go. It seems to be somewhat sporadic for me.

[Delphinator]

It's been very consistent for me so far: 1st run 2nd run 3rd run

[sap-svetter]

Hi all,

I got the same issues after setting up some python software module (compiled with internet sources) on a base image.
docker login succeeds without issues but kaniko still refers to "unauthorized".

Looking forward to a fix 👍

If I can do anything to help, tell me and I will do my very best to help you!

[akhmerov]

Could the duration of the push itself be relevant? The difference in upload speed could explain the lack of reproducibility.

[BryanHunt]

I have just run into this problem when running as part of a Jenkins build on Kubernetes. When I run locally (on my MacBook Pro) I'm able to build the Docker image just fine, but when I run as part of a Jenkins build on Kubernetes, the Kaniko container consumes a huge amount of memory (50+ GB) and after 14 minutes, it fails with:

error building image: UNAUTHORIZED: "authentication required"

At first, I thought it was a problem with trying to push the image to our local repository, so I added the --no-push option and I still get this error. Here's my build pipeline:

properties ([
      buildDiscarder(logRotator(numToKeepStr: '3')),
      disableConcurrentBuilds()
    ])

def label = "nv-ad-docs-${UUID.randomUUID().toString()}"

podTemplate(label: label, imagePullSecrets: ['docker'], yaml: """
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: nodejs
    image: <our-image-repo>/gpuwa-node:10-alpine-9
    imagePullPolicy: Always
    command:
    - cat
    tty: true
  - name: kaniko
    image: gcr.io/kaniko-project/executor:debug
    imagePullPolicy: Always
    command:
    - /busybox/cat
    tty: true
    volumeMounts:
      - name: jenkins-docker-cfg
        mountPath: /root
    resources:
      {requests: {cpu: 1000m, memory: 100Gi}, limits: {cpu: 2000m, memory: 100Gi}}
  volumes:
  - name: jenkins-docker-cfg
    projected:
      sources:
      - secret:
          name: docker
          items:
            - key: .dockerconfigjson
              path: .docker/config.json

""")
{
  node(label) {
    stage('Build') {
      container('nodejs') {
        checkout scm
        sh 'npm install --only-production'
      }
    }

    stage("Assemble") {
      container(name: 'kaniko', shell: '/busybox/sh') {
        withEnv(['PATH+EXTRA=/busybox']) {
          sh '''#!/busybox/sh
          /kaniko/executor --dockerfile=`pwd`/Dockerfile --context=`pwd` --destination=<our-image-repo>/test-node:k8s --no-push
          '''
        }
      }
    }
  }
}

and here is my Dockerfile:

FROM node:10-alpine

WORKDIR /app
COPY . /app

EXPOSE 8080
CMD node index.js

[brandon-bethke-neudesic]

We are also experiencing this issue and it's a major roadblock. We do not know how to reproduce it consistently but it happens often. We are using kaniko in a gitlab ci pipeline.

Error: error building image: Error setting layer times: Error getting layer: UNAUTHORIZED: "authentication required"

We are using the debug image: gcr.io/kaniko-project/executor:debug
and this is how we launch kaniko:
/kaniko/executor --reproducible --dockerfile Dockerfile --context /mycontext --destination gcr.io/myplatform/myimage

[BryanHunt]

@brandon-bethke-neudesic I worked around the problem by switching to https://github.com/containers/buildah

[nicolaferraro]

I suspect the issue I'm experiencing in #372 is related. There are no "sleep" commands, just a build that is taking 5~10 mins.

[tzununbekov]

I run into this problem and figured out that "unauthorized" response is caused by expired anonymous token when build stage last longer than 5+ minutes, similar to docker/distribution#1589 though fix described there didn't help for my case

Issue may be reproduced with following Dockerfile:

FROM golang:1.10.1
WORKDIR /go/src/github.com/tzununbekov/simple-app/
COPY . .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -v -o app
# Without following line build passes successfully 
RUN sleep 360

FROM scratch
COPY --from=0 /go/src/github.com/tzununbekov/simple-app .
ENTRYPOINT ["/app"]

and main.go next to the dockerfile with any simple code:

package main

import "fmt"

func main() {
    fmt.Println("Hello world!")
}

Managed to work this around by retrieving stage source image before getting its tarball in pkg/executor/build.go. I may create PR for this but token expiry handler would be a better solution here.

[ianberinger]

I'm impacted by this as well.

Thanks @tzununbekov for the pointer.
I've debugged this further, this seems to be caused by https://github.com/google/go-containerregistry not refreshing expired Bearer tokens. Bearer tokens expire after a fixed duration (for docker.io after 300s).
If a stage takes longer than that, saving the stage (which involves getting the compressed source image layers) will fail.

I've created a PR that fixes this: https://github.com/google/go-containerregistry/pull/283.

[pieterlange]

@tstromberg can you assign this issue to someone else so this fix gets shipped without undue delay? This makes kaniko unusable for building containers that take >5m to build.

[dlorenc]

@pieterlange it looks like the fix to the underlying issue is being reviewed now in go-containeregistry.

Once that's merged we can get the dependency updated here fairly easily.

[priyawadhwa]

Update: #388 was just merged, but I'll keep this issue open for a few days in case anyone continues to see this error.

Thanks @ianberinger for the fix!!

[akhmerov]

Is the fix included in the latest kaniko-project/executor?

[priyawadhwa]

yup, it should be in gcr.io/kaniko-project/executor:latest

[yurrriq]

388 fixed my long builds.

Edit: It looks like I spoke too soon...

error building image: getting stage builder for stage 1: no token in bearer response:
{"errors":[{"code":"DENIED","message":"access forbidden"}],"http_status":403}

Update: Extending the _authorization token duration_, per @pieterlange's suggestion in GitLab's container registry settings fixed it. Thanks, everyone!

[pieterlange]

I can confirm this fixed it for me!

@yurrriq make sure the token doesn't expire serverside either - gitlab-ci's expiry also happens to be 5 minutes, but you can easily up that in the admin console.

[slavoutich]

Works for me, thanks everyone!

[kennethaasan]

I can confirm the same. Good job guys!

[AndreasBieber]

Hi,

Unfortunately we still have this problem. We use Artifactory as a Docker Registry.
When caching layers we still get a 401. Sometimes when pushing the 3rd or 4th layer. Sometimes, however, only when pushing the final layer. Also a deactivation of the cache does not help.

This problem occurs only on long running builds.

error building image: error building stage: failed to push to destination private-registry/service/cache:7ecab1d646fdcf96a55cdfb66938bb1f5c565e510729a480a26b992b309e6069: no token in bearer response:
{
  "errors" : [ {
    "status" : 401,
    "message" : "Authentication is required"
  } ]
}

[yurrriq]

@AndreasBieber, perhaps you need to replace /root/ with /kaniko/, as described on this potentially relevant GitLab issue.

[AndreasBieber]

@yurrriq /kaniko is already in place, but do not work.

[AndreasBieber]

Forgotten to mention: This problem only occurs with multistage builds.

[bobcatfish]

Sorry to hear this is still happening @AndreasBieber :(

Is there any chance you could provide us with a instructions we could use to reproduced the issue you are still seeing, e.g. maybe similar to the example in this issue's description?

[AndreasBieber]

@bobcatfish: Sorry, it was my bad. We are building our own image of kaniko for the GitLab CI Runner with custom scripts.

...
FROM gcr.io/kaniko-project/executor:v0.6.0 as kaniko
...
FROM base
COPY --from=kaniko /kaniko/executor /kaniko/

Since we also build this custom image with kaniko, the docker auth config was accidentally overwritten during the build process.

Now it works like charm.
Thanks anyway.

[priyawadhwa]

Awesome, I'm going to go ahead and close this issue since it seems like #388 fixed it. If anyone experiences this again please comment on this thread or open another issue!

[akhmerov]

I seem to observe a regression between debug-v0.7.0 and debug-v0.8.0, debug-v0.8.0 fails with the same error message (authentication is required).

[slavoutich]

@akhmerov Replace /root/.docker -> /kaniko/.docker in .gitlab-ci.yml, this issue has nothing to do with your issue.

[Caligatio]

I hate to re-open old bugs but recently just encountered this with a private registry. Quick builds (< 5 minutes) are uploaded just fine but longer builds (> 5 minutes) hit a 401 on upload. Unfortunately I don't run the registry so I cannot confirm the bearer token lifetime.

Reading the comments above, I'm confused whether this bug was actually fixed or people just extended the server-side token lifetimes.

[Delphinator]

I'm fairly certain this was actually fixed. The original issue was reproducible with the central registry at registry.docker.io and I don't think they changed their token lifetimes. https://github.com/GoogleContainerTools/kaniko/issues/245#issuecomment-427678387 explains the fix and links to the relevant PR.

[culdev]

@Caligatio I kept getting this error in multi stage docker builds when running in GitLab CI.
Solved it by moving the .docker folder like slavoutich suggested, e.g:

Build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  variables:
    DOCKER_CONFIG: /kaniko/.docker
  script:
    - mkdir -p /kaniko/.docker
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json
    - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:latest --destination $CI_REGISTRY_IMAGE/builds:$(date +'%Y-%m-%d')

[Caligatio]

I'm currently hitting this with all my builds that take longer than 5-ish minutes. I am confused by yurrriq's response as it seems like the fix didn't work for them so they just extended the token lifetime.

@culdev: I can confirm I have the /kaniko/.docker/config.json file in place and it works for quick builds.

EDIT: If it somehow matters, I'm also sitting behind a proxy.

EDIT2: Turns out my issue was caused by a misbehaving Registry auth service. The symptoms looked similar but was a completely different problem. Sorry all!