Kaniko: [0.21.0] Permission denied when using GCR caching

[Mokto]

Actual behavior
After updating to 0.21.0, I get access denied (I'm using GCR).
I haven't tried 0.20.0, because of the bugs present on this one.
But it was working with 0.19, 0.18, 0.17, 0.16.

Expected behavior
Being able to push layers through GCR.

To Reproduce
Steps to reproduce the behavior:

1. Try to build an image with GCR

Additional Information

• Dockerfile

FROM python:3.7.5

ENV PATH /usr/local/nvidia/bin/:$PATH
ENV LD_LIBRARY_PATH /usr/local/nvidia/lib:/usr/local/nvidia/lib64

ENV NVIDIA_VISIBLE_DEVICES all
ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
LABEL com.nvidia.volumes.needed="nvidia_driver"

ENV CUDA_VISIBLE_DEVICES=0

ENV PACKAGES="\
build-essential \
libssl-dev \
liblzma-dev \
libmagickwand-dev \
curl \
zip \
"
RUN apt -y update \
    && apt -y upgrade \
    && apt -y install ${PACKAGES}

INFO[0157] Pushing layer eu.gcr.io/{PROJECT}/{REPO}/cache:aa9e858784087f68c07788a90e987a85f0e102442d2cc080745fe19dbee270e0 to cache now 
E0506 05:32:34.870621      11 metadata.go:154] while reading 'google-dockercfg' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg
E0506 05:32:34.873013      11 metadata.go:166] while reading 'google-dockercfg-url' metadata: http status code: 404 while fetching url http://metadata.google.internal./computeMetadata/v1/instance/attributes/google-dockercfg-url
WARN[0159] error uploading layer to cache: failed to push to destination eu.gcr.io/{PROJECT}/{REPO}/cache:4e242b84587a99fb7635dd599ce64c320231736fa6738bbe3f7942b267a5c928: PATCH https://eu.gcr.io/v2/{PROJECT}/{REPO}/cache/blobs/uploads/AJnulEH5k9X4qLOJKDk-D0QqRVMYP61rQ4XPeRewSOT23RDJNQ68Ne_FHk-iacL9uA8FxhruPQ2iC12YXQYywPc: DENIED: Access denied. 
error pushing image: failed to push to destination eu.gcr.io/{PROJECT}/{REPO}:deploy-cache-fix-4e98c20555c3b6459ad511637b771a2cda57b460: PATCH https://eu.gcr.io/v2/{PROJECT}/{REPO}/blobs/uploads/AJnulEEKHdOiEGXQKIq8VaS2hKHziCksXlEa7qer2NOQFUImIWOjkZgEW9jzBmTs8hz-aCOM72rnVm87OjlE5N8: DENIED: Access denied.

Triage Notes for the Maintainers

| Description | Yes/No |
|----------------|---------------|
| Please check if this a new feature you are proposing |

• - [ ]

|
| Please check if the build works in docker but not in kaniko |

• - [ ]

|
| Please check if this error is seen when you use --cache flag |

• - [ ]

|
| Please check if your dockerfile is a multistage dockerfile |

• - [ ]

|

Please let me know if you need anything else !

[ozonep]

Yep, just had the same issue while trying to build in Google Cloud Build.
But I didn't have cache enabled.
Just building & pushing image.

Works fine with Docker.
And image is not multi-stage.

[mariuz]

Also got issues with latest release , worked yesterday

Step #0: gcr.io/kaniko-project/executor:latest
Step #0: error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "gcr.io/project ...": creating push check transport for gcr.io failed: GET https://gcr.io/v2/token?scope=repository .... %3Apush%2Cpull&service=gcr.io: UNAUTHORIZED: You don't haveFinished Step #0

Works with
name: "gcr.io/kaniko-project/executor:v0.19.0"

[ryanprater]

Our systems are also impacted by this, with the same permission denied error. I'm assuming this issue will become very popular in the next hours :)

If you're encountering this, you are probably not specifying a version of Kaniko in your cloudbuild.yml, and therefore, always using the latest version.
As a quick fix, just specify yesterday's version as so:

# Please update kaniko version occasionally
- name: 'gcr.io/kaniko-project/executor:v0.20.0'

[cdupuis]

v0.21.0 works fine for us (after v0.20.0 not working). Caching and image push to gcr.io work fine here. But we aren't using GCP's built-in auth and instead have a Docker cred file mounted in our GKE clusters.

[cdupuis]

Out of interest of what is going wrong here, could you cat your /kaniko/.docker/config.json and check that you get the GCR credHelpers installed?

I have the suspicion that Google Cloud Build mounts a config.json and kaniko isn't adding the credHelpers to that any longer.

You could also try v0.21.0 with a DOCKER_CONFIG env var pointing at any empty directory.

[orkaplan]

We had the same issue using kaniko executor on google cloud build, downgrade to v0.20.0 works fine

[cdupuis]

Here's another work-around that let's you run v0.21.0 on GCB:

steps:
    - name: 'gcr.io/kaniko-project/executor:v0.21.0'
      args:
      - --destination=gcr.io/$PROJECT_ID/hackathon-starter
      - --cache=true
      env:
      - "DOCKER_CONFIG=/kaniko/.docker-test"

I just tested that and confirmed it works.

[ghatdev]

We had the exact same issue with @orkaplan.
Works fine with v0.20.0, too.

[Andor]

https://github.com/GoogleContainerTools/kaniko/compare/v0.20.0...v0.21.0
Only two pull-requests were merged there: https://github.com/GoogleContainerTools/kaniko/pull/1229 and https://github.com/GoogleContainerTools/kaniko/pull/1230

[andeplane]

I'm using kaniko to build in our k8s cluster. Also affected by this without Cloud build.

[ddgenome]

This is on me. I have found the bug and will submit another PR.

In short, /kaniko/.docker != /kaniko/.docker/config.json.

[ddgenome]

I figured out why this did not turn up in my local testing. While the bug was introduced in #1229, it only manifested itself after #1230 was also merged.