Kaniko: Failed to push image to Docker Hub

[qianzhangxa]

Actual behavior
I am running Kubernetes v1.15.5 cluster and I created a Kaniko pod to build and push an image to Docker Hub, here is the pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:latest
    args: ["--dockerfile=/workspace/Dockerfile",
           "--context=dir://workspace",
           "--destination=zhq527725/test-kaniko"]
    volumeMounts:
      - name: kaniko-secret
        mountPath: /root
      - name: dockerfile-storage
        mountPath: /workspace
  restartPolicy: Never
  volumes:
    - name: kaniko-secret
      secret:
        secretName: docker-hub
        items:
          - key: .dockerconfigjson
            path: .docker/config.json
    - name: dockerfile-storage
      persistentVolumeClaim:
        claimName: dockerfile-claim

And I created a secret for Kaniko to login Docker Hub:

kubectl create secret docker-registry docker-hub --docker-server=https://index.docker.io/v1/ --docker-username=zhq527725 --docker-password=<my-password> --docker-email=<my-email>

But I found Kaniko pod failed to push to Docker Hub, here is its logs:

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "index.docker.io/zhq527725/test-kaniko:latest": UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:zhq527725/test-kaniko Type:repository] map[Action:push Class: Name:zhq527725/test-kaniko Type:repository]]

And the I changed the --destination arg of Kaniko pod from zhq527725/test-kaniko to registry.docker.io/zhq527725/test-kaniko, but it still failed with a different error:

error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "registry.docker.io/zhq527725/test-kaniko:latest": creating push check transport for registry.docker.io failed: unrecognized HTTP status: 503 Service Unavailable

I confirm that I can use docker push with the same username/password to successfully push image into Docker Hub.

Expected behavior
Successfully push images to Docker Hub.

So far, I have tested Kaniko in my Kubernetes cluster against Harbor, Docker Hub and gcr, and I found it only works with gcr.

[tejal29]

At @qianzhangxa Can you please follow the instructions here
https://github.com/GoogleContainerTools/kaniko#pushing-to-docker-hub

You need to volume mount the secret in /kaniko/.docker/config.json

[qianzhangxa]

Thanks @tejal29, I already tried it before, if I volume mount the secret to /kaniko, the container will fail to start due to:

  Warning  Failed     24s   kubelet, qzhang-worker-pool0-1  Error: failed to create containerd task: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"/kaniko/executor\": stat /kaniko/executor: no such file or directory": unknown

And if I volume mount the secret to /kaniko/.docker/config.json, the container will fail to start due to:

  Warning  Failed     11s   kubelet, qzhang-worker-pool0-1  Error: failed to create containerd task: OCI runtime create failed: container_linux.go:345: starting container process caused "process_linux.go:430: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/kubelet/pods/83eb27c9-1f5b-47f6-985b-8df0f8b1a064/volumes/kubernetes.io~secret/kaniko-secret\\\" to rootfs \\\"/run/containerd/io.containerd.runtime.v1.linux/k8s.io/kaniko/rootfs\\\" at \\\"/run/containerd/io.containerd.runtime.v1.linux/k8s.io/kaniko/rootfs/kaniko/.docker/config.json\\\" caused \\\"not a directory\\\"\"": unknown

Anyway, I have fixed it by setting the env var DOCKER_CONFIG to /root/.docker/ as I mentioned here, and actually in https://github.com/GoogleContainerTools/kaniko/blob/master/examples/pod.yaml, you will see that the secret will be volume mounted in /root.

[gebi]

we just had our kaniko setup stopped working with the same problem error checking push permissions as reported in #1209 .

Our fix was to just use mgit/base:kaniko-executor-debug-stable as image which also fixes #656

Is there anything we can help with regarding stabilizing kaniko?

[aroq]

we just had our kaniko setup stopped working with the same problem error checking push permissions as reported in #1209 .

Our fix was to just use mgit/base:kaniko-executor-debug-stable as image which also fixes #656

Is there anything we can help with regarding stabilizing kaniko?

Same issue, same fix, and the same question.

[gebi]

@aroq JFYI we updated kaniko to version v0.16.0 in mgit/base:kaniko-executor-debug-stable our integration tests run through without issues and both problems #1209 and #656 are "still" fixed (we have dedicated tests for both problems).

(sorry we are using a mono repo on our side so we can skip releases pretty much on the base images (there are just -latest and -stable, and i'd really like to rather fix that here in upstream, than paper over bugs over bugs in additional layers)