Kaniko: HTTP 401 returned when trying to push to Dockerhub with correct credentials

[srkiNZ84]

Actual behavior
While running a build with correct and valid docker credentials mounted under /kaniko/.docker/config.json and the environment variable DOCKER_CONFIG=/kaniko/.docker, Kaniko fails to pass the "push" permission test to Dockerhub, throwing the error:

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "index.docker.io/myname/kaniko-test:latest": UNAUTHORIZED: authentication required; [map[Type:repository Class: Name:myname/kaniko-test Action:pull] map[Type:repository Class: Name:myname/kaniko-test Action:push]]

NOTE: The "config.json" has confirmed to be valid by running docker --config docker/ build -t myname/testing-push . (with the config pointed to the same file) and confirming that this authenticated successfully against Dockerhub.

Expected behavior
That the Build successfully authenticates against Dockerhub

NOTE: This might be related to an issue in an upstream dependency as per: https://github.com/GoogleContainerTools/kaniko/issues/614

To Reproduce
Run the following command to build an image and push to a Dockerhub repository you have permissions to, with the Dockerhub "config.json" file under the "docker" directory:

docker run \
  -v /path/to/code/kaniko-test:/workspace \
  -v docker:/kaniko/.docker \
  --env DOCKER_CONFIG=/kaniko/.docker \
  gcr.io/kaniko-project/executor:debug \
  --dockerfile /workspace/Dockerfile \
  --destination yourname/kaniko-test:latest \
  --context dir:///workspace/ \
  --cache=false

Additional Information

• Dockerfile

FROM nginx

RUN ls -al
RUN ls -al /root/
#RUN ls -al /root/.docker/
#RUN cat /root/.docker/config.json
RUN ls -al /kaniko
RUN ls -al /kaniko/.docker/
RUN cat /kaniko/.docker/config.json

• Build Context
  sample config.json (secret removed):

{
    "auths": {
        "https://index.docker.io/v1/": {
      "email": "[email protected]",
      "auth": "ABCDEFGHIJKLMNOP"
    }
    }
}

• Kaniko Image (fully qualified with digest)
  Not sure. Latest debug one?

[fdlk]

Thanks for filing! I thought I must be doing something wrong. Can confirm that docker config authentication works fine in 0.7.0 and is broken in 0.8.0 and 0.9.0.

[alexellis]

I have the same issue with knative build and Kaniko, regressing to the v0.6.0 tag seems to "fix' it.

[alexellis]

v0.7.0 also works for me with v0.8.0 and v0.9.0 also failing.

[alexellis]

Hi @mattmoor would you know if anyone is looking at this on the Kaniko team?

[ImJasonH]

I'm looking into this today, sorry for the delay I don't normally subscribe to Kaniko issues.

[srkiNZ84]

If it helps, when I build the above Dockerfile and get it to print out the contents of /kaniko/.docker/config.json it looks like the contents of the file don't match what's being mounted:

INFO[0023] RUN cat /kaniko/.docker/config.json
INFO[0023] cmd: /bin/sh
INFO[0023] args: [-c cat /kaniko/.docker/config.json]
{
    "auths": {},
    "credHelpers": {
        "asia.gcr.io": "gcr",
        "eu.gcr.io": "gcr",
        "gcr.io": "gcr",
        "staging-k8s.gcr.io": "gcr",
        "us.gcr.io": "gcr"
    }

It's my first time using Kaniko and I'm not a Docker "layer" expert, but I would expect that the volume mount would replace this file, regardless of the contents of any of the layers.

[jonjohnsonjr]

I can't repro this, but none of the PRs between v0.7.0 and v0.8.0 look that suspicious... possibly https://github.com/GoogleContainerTools/kaniko/pull/522 or https://github.com/GoogleContainerTools/kaniko/pull/515.

[fdlk]

@jonjohnsonjr for me it fails locally on MacBook using docker and on Kubernetes cluster. I am pushing to a private registry and expecting it to pick up base64 encoded credentials from /root/.docker/config.json.

[jonjohnsonjr]

@srkiNZ84 in that docker run command, can you try changing:

-v docker:/kaniko/.docker

to

-v $PWD/docker:/kaniko/.docker

[jonjohnsonjr]

Also I'm curious how you logged in. Is the auth entry your base64-encoded username:password? I'm wondering if there's a legacy way to do a docker login that encodes the credentials differently than we're expecting, since the email flag for docker login has been deprecated for a really long time.

[fdlk]

@jonjohnsonjr Our build script retrieves the base64-encoded username:password from the secret vault and echoes it to the json file as follows:

echo '{"auths": {"registry.example.org": {"auth": "${NEXUS_AUTH}"}}}' > /root/.docker/config.json

This works in 0.7.0 but not in later versions.

[srkiNZ84]

Hi!

in that docker run command, can you try changing ....

So, yes... adding $PWD to the start of the volume mount command fixed the issue for me. When the build runs and it prints out the contents of that docker config it's printing out the correct file contents and then proceeds to successfully push to the remote registry. Yay! Thank you so much for this :-)

Also I'm curious how you logged in ...

So, I'm running on OS X and I had to turn off the "store credentials in keychain" thing and then i logged out, deleted the credentials file under my ~/.docker/config.json, then re-created it by running docker login again and then copied that credential file over (taking out a bunch of other registries). It does look like this file does not include an email field and that I added this in manually, possibly when trying to debug this issue. My apologies for the confusion this caused. FWIW, I can confirm that the docker CLI client does seem to be able to handle the legacy version of this config file, with the email included. This was tested by running:

docker --config docker/ push yourname/kaniko-test:latest

[alexellis]

I believe this issue needs to stay open or a new issue needs to be logged. This clearly stopped working with knative build after 0.7.0.

[jonjohnsonjr]

@alexellis do you mind opening another issue with more details of what you're doing/seeing? I just tried pushing to dockerhub using kaniko v0.8.0 on knative/build and it seems to work for me :/

[bryanhuntesl]

If it helps, when I build the above Dockerfile and get it to print out the contents of /kaniko/.docker/config.json it looks like the contents of the file don't match what's being mounted:

INFO[0023] RUN cat /kaniko/.docker/config.json
INFO[0023] cmd: /bin/sh
INFO[0023] args: [-c cat /kaniko/.docker/config.json]
{
  "auths": {},
  "credHelpers": {
      "asia.gcr.io": "gcr",
      "eu.gcr.io": "gcr",
      "gcr.io": "gcr",
      "staging-k8s.gcr.io": "gcr",
      "us.gcr.io": "gcr"
  }

It's my first time using Kaniko and I'm not a Docker "layer" expert, but I would expect that the volume mount would replace this file, regardless of the contents of any of the layers.

Maybe from a UX perspective it would be helpful if the default file didn't exist, then kaniko would die with an error indicating the file was missing if it wasn't mounted correctly. I also screwed up the volume mount and struggled to figure out what was wrong - the kaniko image doesn't support shell exec so hard to have a poke around.

[dinvlad]

We've just experienced this many times (on a different GCR repo each time, so seems random) inside Cloud Build, using the latest v0.21.0. It got resolved after I downgraded to v0.20.0.