Amazon-vpc-cni-k8s: 401 Unauthorized on image pull

[smcquay]

I followed instructions here and when the daemonset pods start up I get the following in the event log:

  Normal   SuccessfulMountVolume  19s               kubelet, k8sworker-0-006.infra.poc.aun1.i.wish.com  MountVolume.SetUp succeeded for volume "log-dir"
  Normal   SuccessfulMountVolume  19s               kubelet, k8sworker-0-006.infra.poc.aun1.i.wish.com  MountVolume.SetUp succeeded for volume "cni-net-dir"
  Normal   SuccessfulMountVolume  19s               kubelet, k8sworker-0-006.infra.poc.aun1.i.wish.com  MountVolume.SetUp succeeded for volume "cni-bin-dir"
  Normal   SuccessfulMountVolume  19s               kubelet, k8sworker-0-006.infra.poc.aun1.i.wish.com  MountVolume.SetUp succeeded for volume "default-token-mww6r"
  Normal   BackOff                18s               kubelet, k8sworker-0-006.infra.poc.aun1.i.wish.com  Back-off pulling image "602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:latest"
  Warning  Failed                 18s               kubelet, k8sworker-0-006.infra.poc.aun1.i.wish.com  Error: ImagePullBackOff
  Normal   Pulling                5s (x2 over 19s)  kubelet, k8sworker-0-006.infra.poc.aun1.i.wish.com  pulling image "602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:latest"
  Warning  Failed                 5s (x2 over 19s)  kubelet, k8sworker-0-006.infra.poc.aun1.i.wish.com  Failed to pull image "602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:latest": rpc error: code = Unknown desc = failed to resolve image "602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:latest": unexpected status code https://602401143452.dkr.ecr.us-west-2.amazonaws.com/v2/amazon-k8s-cni/manifests/latest: 401 Unauthorized
  Warning  Failed                 5s (x2 over 19s)  kubelet, k8sworker-0-006.infra.poc.aun1.i.wish.com  Error: ErrImagePull

where the salient information is:

Failed to pull image "602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:latest": rpc error: code = Unknown desc = failed to resolve image "602401143452.dkr.ecr.us-west-2.amazonaws.com/amazon-k8s-cni:latest": unexpected status code https://602401143452.dkr.ecr.us-west-2.amazonaws.com/v2/amazon-k8s-cni/manifests/latest: 401 Unauthorized

I can reproduce with curl from in my VPC, and from outside of AWS:

$ curl https://602401143452.dkr.ecr.us-west-2.amazonaws.com/v2/amazon-k8s-cni/manifests/latest                                                                                                                                                                              
Not Authorized

Also when I try to get to it through a browser it asks for HTTP Basic Auth.

Have I missed some configuration somewhere? Have these images moved?

[liwenwu-amazon]

@smcquay can you check if your instance role have permission to pull image from ECR registry?
for example,

        {
            "Sid": "kopsK8sECR",
            "Effect": "Allow",
            "Action": [
                "ecr:GetAuthorizationToken",
                "ecr:BatchCheckLayerAvailability",
                "ecr:GetDownloadUrlForLayer",
                "ecr:GetRepositoryPolicy",
                "ecr:DescribeRepositories",
                "ecr:ListImages",
                "ecr:BatchGetImage"
            ],
            "Resource": [
                "*"
            ]
        },

[smcquay]

@liwenwu-amazon I missed that the first time around, but I've added those permissions (and also the L-IPAM perms while I was at it), but I still cannot seem to hit that url successfully.

I'm trying from us-west-1, but I get the same error when I change to point at a registry in that region, but I get the same Not Authorized results.

Any ideas?

[liwenwu-amazon]

@smcquay , can I take a look at your instance IAM policy? You can send them to me directly at [email protected].
Also, the CNI plugin docker image is ONLY available at us-west-2 region.

[smcquay]

Information sent; thanks for looking into this!

[smcquay]

I was able to build the image myself and push it to my own container registry. I'm going to call this resolved.

[WarpRat]

If anybody else happens to come across this issue, we faced a similar problem. Adding the proper IAM role permissions fixed it, but not until rolling the masters. Kubelet only pulls the authorization token from ECR at initialization, so adding these permissions to your masters' IAM roles won't allow kubelet to get the proper token until the masters have been reinitialized.

[jdsantiagojr]

Ran into the same issue. I was able to get around it by doing the same thing @smcquay did.