Kind: Kind can't pull Docker images from Github's pkg registry

[tamalsaha]

I am using kind v0.5.1 to pull Docker image from Github's new pkg registry. But this is failing. The exact same commands are working in minikube v1.4.0.

export DOCKER_TOKEN=**my-personal-token-with-pkg-read-write-permission**
export USERNAME=stashed
kubectl create secret docker-registry regcred --namespace=kube-system --docker-server=docker.pkg.github.com --docker-username=${USERNAME} --docker-password=${DOCKER_TOKEN}

Now, I create the following pod.

apiVersion: v1
kind: Pod
metadata:
  name: private-reg
  namespace: kube-system
spec:
  restartPolicy: Never
  containers:
  - name: private-reg-container
    image: docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-25-g1aa27c95_linux_amd64
  imagePullSecrets:
  - name: regcred

If I describe the pod, I see the following error:

Events:
  Type     Reason     Age                  From                         Message
  ----     ------     ----                 ----                         -------
  Normal   Scheduled  2m27s                default-scheduler            Successfully assigned kube-system/private-reg to kind-control-plane
  Normal   Pulling    62s (x4 over 2m25s)  kubelet, kind-control-plane  Pulling image "docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-25-g1aa27c95_linux_amd64"
  Warning  Failed     62s (x4 over 2m24s)  kubelet, kind-control-plane  Failed to pull image "docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-25-g1aa27c95_linux_amd64": rpc error: code = Unknown desc = failed to resolve image "docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-25-g1aa27c95_linux_amd64": no available registry endpoint: unexpected status code https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-25-g1aa27c95_linux_amd64: 406 Not Acceptable
  Warning  Failed     62s (x4 over 2m24s)  kubelet, kind-control-plane  Error: ErrImagePull
  Warning  Failed     49s (x6 over 2m24s)  kubelet, kind-control-plane  Error: ImagePullBackOff
  Normal   BackOff    38s (x7 over 2m24s)  kubelet, kind-control-plane  Back-off pulling image "docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-25-g1aa27c95_linux_amd64"

I followed the steps in https://github.com/kubernetes-sigs/kind/issues/817 and I get the same error:

root@kind-control-plane:/# crictl pull docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-25-g1aa27c95_linux_amd64
FATA[0000] pulling image failed: rpc error: code = Unknown desc = failed to resolve image "docker.pkg.github.com/stashed/stash/stash:v0.9.0-rc.0-25-g1aa27c95_linux_amd64": no available registry endpoint: unexpected status code https://docker.pkg.github.com/v2/stashed/stash/stash/manifests/v0.9.0-rc.0-25-g1aa27c95_linux_amd64: 406 Not Acceptable 

So, the error is 406 Not Acceptable . I am able to pull this image on my desktop's docker daemon. So, this seems like a kind specific issue.

[tamalsaha]

406 error code seems to be related to Accept- header.

This issue seems same as : https://github.com/sylabs/singularity/issues/868

[BenTheElder]

https://github.com/containerd/containerd/issues/3291

[tamalsaha]

@BenTheElder , it seems containerd just fixed this in master which will be part of their 1.13.0 release. Will kind be able to use that version or this fix needs to be ported back to containerd 1.2.x ?

[BenTheElder]

Will kind be able to use that version or this fix needs to be ported back to containerd 1.2.x ?

We'd like to get on 1.3 or newer anyhow, but we need binaries for architectures besides amd64 ideally, the upstream containerd release tarballs are only linux/amd64 but we nominally support ppc64le and arm / arm64 as well.

We're planning to get either upstream containerd releases to have tarballs for more platforms or start building our own release tarballs.

In the meantime we've been on Ubuntu's packages. It would be great if this landed in whatever Ubuntu packages (currently 1.2.9) but I'm not sure ubuntu doing patch updates within Ubuntu releases anyhow :(

https://github.com/kubernetes-sigs/kind/pull/875 gets us on 1.2.9 at least.

@aojea was looking into the cross compile for that today.

see also https://github.com/kubernetes-sigs/kind/issues/874

[BenTheElder]

xref: https://github.com/containerd/containerd/pull/3702

[zimmertr]

Why is this closed?

[BenTheElder]

@zimmertr please see https://github.com/containerd/containerd/pull/3702
there's nothing for us to do, it's tracked there and github is rolling out fixes.

[BenTheElder]

er sorry see: https://github.com/containerd/containerd/issues/3291#issuecomment-685190574