Kaniko: Kaniko supporting multiple auths

[webmutation]

Actual behavior
Currently using A private pull and a private push registry. So user needs to be logged in to both private registries.
Currently when building i get
"No matching credentials were found, falling back on anonymous"

Expected behavior
What is the proper usage for pulling images from docker hub.

To Reproduce
Steps to reproduce the behavior:

1. create private registry for pull
2. create private registry for push
3. create auth for pushing
4. build image
5. see notification "No matching credentials were found, falling back on anonymous" 3x

does the kaniko debug image perform a docker login under the hood... how should I pass it the information?

[jubel-han]

Depending on where you are performing the executor, normally, you will need to mount the docker auth config.json under the directory of /kaniko/.docker/ according to here.

[webmutation]

It is mounted I even inject the same config file I use locally
cat config.json > /kaniko/.docker/config.json

The problem seems to be that it does not recognize more than one registry at a time... so if you have a pull and a push registry that are different say you want to pull from docker hub (as a logged in user, anonymous works) and push to your private registry it will not work... at least this has been my experience using Kaniko with a gitlab docker runner.

I worked around by pulling from a mirror repository that contained docker hub and my own private registries, so I don't have a blocking issue, its the behavior from kaniko that does not seem correct to me. It should be able to take all the configuration passed, meaning all auth tokens for the respective registries.

[MCyprien]

I have the same kind of problem, I try to publish my container to the gitlab registry and gcr, individually it work, but not with both of them.

Build Docker image:
  stage: build
# Using debug image for gitlab-ci purpose https://docs.gitlab.com/ee/ci/docker/using_kaniko.html
  image:
    name: gcr.io/kaniko-project/executor:debug-v0.10.0
    entrypoint: [""]
  script:
    - export GOOGLE_APPLICATION_CREDENTIALS="$(pwd)/my_gcr_creds.json"
    - echo "{\"auths\":{\"$CI_REGISTRY\":{\"username\":\"$CI_REGISTRY_USER\",\"password\":\"$CI_REGISTRY_PASSWORD\"}}}" > /kaniko/.docker/config.json 
    - /kaniko/executor --context $(pwd) --dockerfile ./Dockerfile --destination $GCR_REPO --destination $GITLAB_DOCKER_REPO

[AASanchezA]

any update on this issue, I need to pull from private registry and need to push to aws

[raphaelLacerda]

any status on this feature request?

[edvald]

@AASanchezA I'd expect you could mount or otherwise write the /kaniko/.docker/config.json JSON file as suggested, and just have two keys under auths, one for each registry. Did you try that? I'm working on supporting this in https://github.com/garden-io/garden and until I find out otherwise, I'm going on the assumption you can put multiple auths in that file, same as when you configure docker locally.

[AASanchezA]

@edvald at the end i solved running two separate jobs on my gitlab-ci.yml one for registry that i need to push the container image, this is working quit well but the image is been build two times

[gawi]

If the goal if to be able to publish to GCR and to something else, this is absolutely feasible by providing two auth entries in config.json. You would need to authenticate to GCR using a JSON key file.

What you can't do is to publish to multiple GCR projects having different credentials because the authentication key is the hostname. You would have more than one entry under gcr.io. It wouldn't work. As a work-around, you could try to use CNAMEs for gcr.io such as us.gcr.io.

[Shanuson]

@gawi how does the config.json looks like in this case? What would be the content of the auth field for gcr:

{
  "auths": {
    "some docker registry url": {
      "auth": "a key"
    },{
    "eu.gcr.io":{
    ?? what should be here ??
    }
  }
}

[gawi]

@Shanuson Assuming you use the JSON key file method to authenticate to GCR:

{
  "auths": {
    "https://asia.gcr.io": {
      "username": "_json_key",
      "password": "<JSON service account key #1 properly escaped>"
    },
    "https://eu.gcr.io": {
      "username": "_json_key",
      "password": "<JSON service account key #2 properly escaped>"
    },
    "https://gcr.io": {
      "username": "_json_key",
      "password": "<JSON service account key #3 properly escaped>"
    },
    "https://us.gcr.io": {
      "username": "_json_key",
      "password": "<JSON service account key #4 properly escaped>"
    }
  }
}

And the repositories can then be:

• asia.gcr.io/project1/xyz
• eu.gcr.io/project2/xyz
• gcr.io/project3/xyz
• us.gcr.io/project4/xyz

Each of them can have a different account key. Obviously, this is a hack, it doesn't scale and you are at the mercy of the CNAME entries on the Google side. This is a last resort solution.

[webmutation]

As far as I can tell this works. I have two registries I would like to push the same image to both

In config.json

{"auths": "registrya.org": {"username":"usera","password":"[MASKED]"} }, "registryb.com": {"username":"userb","password":"[MASKED]"} } }

/kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile \ --build-arg IMAGE_DATE="$(date --utc +%Y-%m-%dT%H:%M:%SZ)" \ --destination $CIA_REGISTRY/$CI_REGISTRY_IMAGE:mock \ --destination $CIB_REGISTRY/$CI_REGISTRY_IMAGE:mock
This prevents the dual build. I did not test with tokens or other types of registries in this case both CIA and CIB are Nexus registries.

[seboudry]

If you want to push to GCR using a JSON key file and also to other registries, you must define config.json file with gcr credential helper alongside with other auths registry credentials.

Example with https://docs.gitlab.com/ee/ci/docker/using_kaniko.html and also GCR:

build:
  stage: build
  image:
    name: gcr.io/kaniko-project/executor:debug
    entrypoint: [""]
  variables:
    GOOGLE_APPLICATION_CREDENTIALS: /path/to/key.json
  script:
    - |
      cat <<EOF > /kaniko/.docker/config.json
      {
        "auths": {
          "$CI_REGISTRY": {
            "username": "$CI_REGISTRY_USER",
            "password": "$CI_REGISTRY_PASSWORD"
          }
        },
        "credHelpers": {
          "eu.gcr.io": "gcr"
        }
      }
      EOF
    - >
      /kaniko/executor
      --context $CI_PROJECT_DIR
      --dockerfile $CI_PROJECT_DIR/Dockerfile
      --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
      --destination eu.gcr.io/gcp-project/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME:$CI_COMMIT_SHORT_SHA

[tmshn]

Yeah @seboudry's solution looks right; the kaniko docker image has docker-credentials-gcr and it will be configured only if no config file exists.

Maybe this should work too:

cat <<EOF > /kaniko/.docker/config.json
{
  "auths": {
    "$CI_REGISTRY": {
      "auth": "$BASE64_ENCODED_USERNAME_AND_PASSWORD"
    }
  }
}
EOF
docker-credential-gcr configure-docker

---

Note that the above example is for GitLab CI. If you use Google Cloud Build, the config file should be something like:

steps:
- name: gcr.io/kaniko-project/executor:debug
  entrypoint: sh
  args:
  - -cx
  - |
    cat <<EOF > /kaniko/.docker/config.json
    {
      "auths": {
        "$CI_REGISTRY": {
          "auth": "$BASE64_ENCODED_USERNAME_AND_PASSWORD"
        }
      }
    }
    EOF
    docker-credential-gcr configure-docker
  volumes:
  - name: kaniko-docker-config
    path: /kaniko/.docker
- name: gcr.io/kaniko-project/executor:latest
  args: [ ... ] # as usual
  volumes:
  - name: kaniko-docker-config
    path: /kaniko/.docker

[sneko]

Hi,

Sorry if that's not the right place to post but does anyone have successfully authenticated Kaniko to GCR with the JSON key?

Before I was using Docker CLI... then img (that works great, except for a specific pipeline that times out so I need to move just this one to Kaniko). And it was pretty easy to make things authenticated:

cat $IMAGE_REGISTRY_KEYFILE | img login -u _json_key --password-stdin $IMAGE_REGISTRY_URL

But with Kaniko, impossible to succeed for now. I followed https://github.com/GoogleContainerTools/kaniko#pushing-to-google-gcr and my service account is mounted in my CI/CD, and even by doing:

                    sh 'export GOOGLE_APPLICATION_CREDENTIALS=$IMAGE_REGISTRY_KEYFILE'
                    sh '/kaniko/executor --context `pwd` --destination $IMAGE_TAG:latest'

I'm always facing an "Access denied" whereas with img it works.

I guess I missed something but I don't understand what. I still have the solution of creating the /kaniko/.docker/config.json but since I'm using _json_key as a user, how to easily format the file while escaping the service account quotes into the shell of the Kanifo image? (all above examples of config.json creation are done with user/password way, so nothing to escape in theory)

Thank you,

[sneko]

It was a mistake from me...

Exporting the environment variable in a different step was not persisted to the Kaniko build step. The solution was to directly populate GOOGLE_APPLICATION_CREDENTIALS as I did for IMAGE_REGISTRY_KEYFILE : by setting this variable on the pipeline container.

Sorry for disturbing,

[Reinkaos]

Has there been any update on this issue?

With the recent change to the Docker Pull Limit I am needing to be able to pull the base image from Docker Hub as an authenticated user so that I can build my Docker Image which is then pushed up to our own private registry in ECR.