Containerd: GCR authentication error with multiple scopes

[schrodit]

Description
OCI image resolver fails when multiple images are fetched from gcr(google container registry) (see code example below).
This error only occurs if the second image is lexically ordered after the first image.

_Our Observations_:
The first image can always be correctly fetched as the docker authorizers handlers are empty.

During the retry a new handler for the current request is added in the authorizers AddResponse method .
But with the common scope set to scope=repository:my-project/a:xxx.

When the second image should be fetched it uses the previously created handler. This handler contains the old scope as common scope which is then also added to the eu.gcr.io/v2/token request (see here for the post request and here for the gcr get request).
The scopes are sorted here which means the scopes are always supplied in the http query parameters based on their lexical order.

This behavior together with our observation that the error only occurs if the second image is lexically ordered after the first image, let us suspect that gcr only validates the first available scope value but ignores the second.

Steps to reproduce the issue:
The issue occurs when multiple oci images were pulled from the registry as in this code example.
Be aware that this example only fails if image a is alphabetically sorted before image b.

import (
    "context"
    "errors"
    "fmt"
    "net/http"
    "os"

    "github.com/containerd/containerd/remotes/docker"
)

var jsonkey = "here comes my gcr serviceaccount key"

func main() {
    ctx := context.Background()
    defer ctx.Done()

    auth := docker.NewDockerAuthorizer(docker.WithAuthCreds(func(string) (string, string, error){
        return "_json_key", jsonkey, nil
    }))
    resolver := docker.NewResolver(docker.ResolverOptions{
        Hosts: func(host string) ([]docker.RegistryHost, error) {
            if host == "eu.gcr.io" {
                return []docker.RegistryHost{{
                    Client:     http.DefaultClient,
                    Authorizer: auth,
                    Host:       "eu.gcr.io",
                    Scheme:     "https",
                    Path:       "/v2",
                    Capabilities: docker.HostCapabilityResolve|docker.HostCapabilityPull|docker.HostCapabilityPush,
                }}, nil
            }
            return []docker.RegistryHost{}, errors.New("not found")
        },
    })

        // this image will be successfully fetched
    _, _, err := resolver.Resolve(ctx, "eu.gcr.io/myproj/a:0.1.0")
    check(err)
    fmt.Println("Fetched 1")

         // will fail with "unexpected status code [manifests 0.x.x]: 401 Unauthorized" 
    _, _, err = resolver.Resolve(ctx, "eu.gcr.io/myproj/b:0.1.0")
    check(err)
    fmt.Println("Fetched 2")

    _, _, err = resolver.Resolve(ctx, "eu.gcr.io/myproj/c:0.x.x")
    check(err)
    fmt.Println("Fetched 3")
}

Describe the results you received:
Second resolve is unauthorized.

Describe the results you expected:
Should correctly get the oauth token from the registry.

Output of containerd --version:
v1.3.6

Any other relevant information:

[clarkbw]

👋 We pushed out a fix for this today. If you could try this again and let us know I'd appreciate it. Thanks!

[schrodit]

@clarkbw is the fix already in master?
I checked the above script with github.com/containerd/containerd v1.4.1-0.20201029224352-abe1807a131f but the second call is still unauthorized?

Or is it only fixed in GHCR?

[clarkbw]

Or is it only fixed in GHCR?

Sorry, only in GHCR. Updated the wrong issue while watching both. 😬