Containerd: image pull fails with "failed to resolve reference [...] could not resolve digest for [...]"

[timogriese]

Can not pull an image from registry.access.redhat.com, even though it works with standard docker.

Steps to reproduce the issue:

1. Install microk8s
   snap install microk8s --classic --channel=1.16/stable
2. Install docker
   snap install docker
3. Pull image with containerd
   microk8s.ctr --debug image pull registry.access.redhat.com/ubi8-minimal:8.0-213
4. Pull image with docker
   docker pull registry.access.redhat.com/ubi8-minimal:8.0-213

Describe the results you received:
Result of 3.

microk8s.ctr --debug image pull registry.access.redhat.com/ubi8-minimal:8.0-213
DEBU[0000] fetching                                      image="registry.access.redhat.com/ubi8-minimal:8.0-213"
DEBU[0000] resolving                                    
DEBU[0000] do request                                    request.headers=map[Accept:[application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, *]] request.method=HEAD url="https://registry.access.redhat.com/v2/ubi8-minimal/manifests/8.0-213"
DEBU[0000] fetch response received                       response.headers=map[Accept-Ranges:[bytes] Connection:[keep-alive] Content-Length:[1471] Content-Type:[application/vnd.docker.distribution.manifest.list.v2+json] Date:[Sat, 07 Dec 2019 10:27:19 GMT] Etag:["cf60afcdc3295005a1b9fbc24cb417b5:1568711792"] Last-Modified:[Tue, 17 Sep 2019 08:54:45 GMT] Server:[Apache] X-Docker-Size:[-1]] status="200 OK" url="https://registry.access.redhat.com/v2/ubi8-minimal/manifests/8.0-213"
ctr: failed to resolve reference "registry.access.redhat.com/ubi8-minimal:8.0-213": could not resolve digest for registry.access.redhat.com/ubi8-minimal:8.0-213

Result of 4.

docker pull registry.access.redhat.com/ubi8-minimal:8.0-213
8.0-213: Pulling from ubi8-minimal
57de4da701b5: Pull complete 
cf0f3ebe9f53: Pull complete 
Digest: sha256:c505667389712dc337986e29ffcb65116879ef27629dc3ce6e1b17727c06e78f
Status: Downloaded newer image for registry.access.redhat.com/ubi8-minimal:8.0-213

Describe the results you expected:
I would expect the image to be pulled correctly like docker does.
Am I missing some sort of necessary config?

Output of containerd --version:

microk8s.ctr --version
ctr github.com/containerd/containerd v1.2.5

Any other relevant information:
Ubuntu 18.04.3 LTS (GNU/Linux 4.15.0-70-generic x86_64)

[ktock]

This seems to be related to https://github.com/containerd/containerd/issues/3238 (the patch is https://github.com/containerd/containerd/pull/3245 which is available since v1.2.9).

[dmcgowan]

@ktock thanks for linking the fix.

Tested that it does work, closing this since it already resolved in supported versions.

[bountin]

Hey @dmcgowan! I'm having the same problem, just with github packages.
Did you only verify it with containerd directly, or did you also manage to get it working in microk8s?

[thejettanderson]

+1 for microk8s not being able to pull github packages with docker. It can't seem to resolve the digest