Kaniko: v0.21.0 breaks in cloudbuild for private GCR image

[victortrac]

Actual behavior
Builds using gcr.io/kaniko-project/executor:latest started failing with a GCR authentication error today when the kaniko is trying to build a private image hosted in GCR. Changing the kaniko executer tag to v0.20.0 fixes the problem.

gcr.io/kaniko-project/executor:latest
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "gcr.io/<my-project-id>/<my-docker-image>:<my-docker-tag>": creating push check transport for gcr.io failed: GET https://gcr.io/v2/token?scope=repository%3A<my-project-id>%2F<my-docker-image>%3Apush%2Cpull&service=gcr.io: UNAUTHORIZED: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication

Expected behavior
Kaniko executor should be able to read the service account assigned to cloudbuild to pull the docker image from GCR just like in previous versions.

To Reproduce
Steps to reproduce the behavior:

1. Use kaniko executor v0.21.0 in a cloudbuild that uses a private GCR image

Additional Information

• Kaniko Image:

Digest: sha256:fee59f1fc71e70b3a0f4d93be747ff94a81e8079dcccef735005a29890b18a5e
Status: Downloaded newer image for gcr.io/kaniko-project/executor:latest

Triage Notes for the Maintainers

| Description | Yes/No |
|----------------|---------------|
| Please check if this a new feature you are proposing |

• - [ ]

|
| Please check if the build works in docker but not in kaniko |

• - [ ]

|
| Please check if this error is seen when you use --cache flag |

• - [ ]

|
| Please check if your dockerfile is a multistage dockerfile |

• - [ ]

|

[tejal29]

@victortrac Looks like this got fixed in https://github.com/GoogleContainerTools/kaniko/pull/1238

Would you up for trying the edge build here to confirm?

gcr.io/kaniko-project/executor:debug-edge
gcr.io/kaniko-project/executor:edge

[deepapanwar]

Had same issue when using gcr.io/kaniko-project/executor:latest. Runs correctly when using gcr.io/kaniko-project/executor:debug-edge.

[dinvlad]

Could you please keep this issue open, until this fix gets released in the new Kaniko version? Thanks!

[ALTELMA]

@dinvlad +1 and Let me know if it fixes asap.

[ctison]

gcr.io/kaniko-project/executor:debug-edge fixed error checking push permissions [...] while using GOOGLE_APPLICATION_CREDENTIALS. Thanks.

[tejal29]

hey folks, I was able to verify the gcr.io/kaniko-project/executor:edge works with GCB.
I created an issue to add integration test for GCB.
Would love some contributions~
https://github.com/GoogleContainerTools/kaniko/issues/1247

[tejal29]

Release branch is out https://github.com/GoogleContainerTools/kaniko/pull/1248

[tejal29]

https://github.com/GoogleContainerTools/kaniko/releases/tag/v0.22.0 is out. Can someone please try this

[zoran15]

https://github.com/GoogleContainerTools/kaniko/releases/tag/v0.22.0 is out. Can someone please try this

It appears to be working for us (compute-image-tools)

[TimShilov]

https://github.com/GoogleContainerTools/kaniko/releases/tag/v0.22.0 is out. Can someone please try this

Worked for me. Thanks! 👍

[tejal29]

Fixed on latest version v0.22.0

[ejose19]

I'm getting this on executor:debug-v0.22.0

WARN[0184] error uploading layer to cache: failed to push to destination us.gcr.io/... GET https://us.gcr.io/v2/token?scope=repository...: UNAUTHORIZED: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication 

Edit: Also since v0.20 kaniko is neither finding nor pushing to cache, using 0.19 still works