Containerd: unable to fetch manifest by Docker-Content-Digest during pull

[fuweid]

Description

The ctr pulls some internal image, like

# vagrant @ ubuntu-xenial in ~ [9:54:02]
$ sudo ctr image pull acs-reg.alipay.com/acs/networkplugins:2.0.0-243ddfb
acs-reg.alipay.com/acs/networkplugins:2.0.0-243ddfb: resolving      |--------------------------------------|
elapsed: 0.3 s                                       total:   0.0 B (0.0 B/s)
ctr: httpReaderSeeker: failed open: could not fetch content descriptor sha256:b7e2a895fe87ffef8256181002547f8b6ef785574bfca309b005dc9b45970021 (application/vnd.docker.distribution.manifest.v1+prettyjws) from remote: not found

The resolver can fetch the manifest description by https://acs-reg.alipay.com/v2/acs/networkplugins/manifests/2.0.0-243ddfb. But it failed to fetch data by the manifest digest value. Is it the bug in registry?

body for the https://acs-reg.alipay.com/v2/acs/networkplugins/manifests/2.0.0-243ddfb

{
   "schemaVersion": 1,
   "name": "acs/networkplugins",
   "tag": "2.0.0-243ddfb",
   "architecture": "amd64",
    ....
}

header for the https://acs-reg.alipay.com/v2/acs/networkplugins/manifests/2.0.0-243ddfb ->

Server →Tengine/2.1.0
Date →Thu, 29 Nov 2018 01:52:18 GMT
Content-Type →application/vnd.docker.distribution.manifest.v1+prettyjws
Content-Length →4863
Connection →keep-alive
Docker-Content-Digest →sha256:b7e2a895fe87ffef8256181002547f8b6ef785574bfca309b005dc9b45970021
Docker-Distribution-Api-Version →registry/2.0
Etag →"sha256:b7e2a895fe87ffef8256181002547f8b6ef785574bfca309b005dc9b45970021"
Set-Cookie →spanner=1qDOoMqYoyX+9zP9nwUxG9QxFml9Ic/NXt2T4qEYgj0=;path=/;secure;
Via →spanner-office-2.et15[200]

registry ->
/bin/registry github.com/docker/distribution v2.3.1.m

Steps to reproduce the issue:

internal image, no open.

Describe the results you received:

failed to fetch image

Describe the results you expected:

success to fetch image

Output of containerd --version:

containerd github.com/containerd/containerd v1.2.0-65-g55baf50 55baf50b5af476a489284679b299a97ae9ad1f54

NOTE: maybe I should file the issue to github.com/docker/distribution

[dmcgowan]

This is an issue with the Accept header. In containerd we moved to a single Accept header with comma separated values as defined by HTTP https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html while Docker used multiple Accept headers, one per media type. There were a few registries that needed to be fixed to properly support the header, but I am unaware of which version of the open source registry had this bug. We should more clearly define which version of the open source registry are supported for push/pull with containerd since it seems in this case an older version of the registry may be being used.

[dmcgowan]

Looks like this was fixed in https://github.com/docker/distribution/pull/1782 which means registry 2.4 and earlier could run into this issue.

[fuweid]

Thanks @dmcgowan ! good to know this.