Eksctl: kube-proxy update is using a non-existing image tag

[Surian]

What happened?
After upgrading a cluster from EKS 1.16 to 1.17 (platform version eks.2), I updated kube-proxy with eksctl utils update-kube-proxy --approve.
The DaemonSet has been updated with image tag v1.17.9 (following the master version). However, this image tag does not exist in AWS ECR repository for EKS.

docker pull 602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/kube-proxy:v1.17.9
Error response from daemon: manifest for 602401143452.dkr.ecr.us-east-1.amazonaws.com/eks/kube-proxy:v1.17.9 not found: manifest unknown: Requested image not found

Only v1.17.6 (released with EKS 1.17 eks.1) and v1.17.7 (released with EKS 1.17 eks.2) are available.

What you expected to happen?
I expect that kube-proxy version is using a version made available by AWS (AWS doc https://docs.aws.amazon.com/fr_fr/eks/latest/userguide/update-cluster.html is reporting that kube-proxy version is actually v1.17.7 for EKS 1.17 cluster)

How to reproduce it?
Create an EKS cluster on 1.17 eks.2.
Execute eksctl utils update-kube-proxy --approve.

Anything else we need to know?
AWS documentation says that EKS 1.17 eks.2 is using Kubernetes 1.17.9 (https://docs.aws.amazon.com/eks/latest/userguide/platform-versions.html) but they are using 1.17.7 for kube-proxy (https://docs.aws.amazon.com/fr_fr/eks/latest/userguide/update-cluster.html).
eksctl kube-proxy update is broken as eksctl is using k8s version used by the control plane as image tag for kube-proxy

Versions
Please paste in the output of these commands:

$ eksctl version
0.25.0
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.9-eks-4c6976", GitCommit:"4c6976793196d70bc5cd29d56ce5440c9473648e", GitTreeState:"clean", BuildDate:"2020-07-17T19:00:19Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17+", GitVersion:"v1.17.9-eks-4c6976", GitCommit:"4c6976793196d70bc5cd29d56ce5440c9473648e", GitTreeState:"clean", BuildDate:"2020-07-17T18:46:04Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}

Logs

$ cat cluster.yml | eksctl utils update-kube-proxy -f - --approve
[ℹ]  eksctl version 0.25.0
[ℹ]  using region eu-west-1
[ℹ]  "kube-proxy" is now up-to-date

$ kubectl -n kube-system get ds kube-proxy -o yaml
apiVersion: apps/v1                                  
kind: DaemonSet
metadata:
    spec:
      containers:
      - command:
        - kube-proxy
        - --v=2
        - --config=/var/lib/kube-proxy-config/config
        image: 602401143452.dkr.ecr.eu-west-1.amazonaws.com/eks/kube-proxy:v1.17.9
        imagePullPolicy: IfNotPresent
        name: kube-proxy

[mrnonz]

I'm faced this issue.

For workaround you need to manual patch to using version v1.17.7 instead

kubectl set image daemonset.apps/kube-proxy \
    -n kube-system \
    kube-proxy=602401143452.dkr.ecr.{AWS_ZONE}.amazonaws.com/eks/kube-proxy:v1.17.7

[mrnonz]

I think, we need to hardcode the version of kube-proxy follow AWS document instead using the same version of cluster.

https://github.com/weaveworks/eksctl/blob/master/pkg/ctl/utils/update_kube_proxy.go#L66

and here document from AWS https://docs.aws.amazon.com/eks/latest/userguide/update-cluster.html

[tomhuang12]

Faced the same issue... can confirm that @mrnonz 's command fixed it.

[javiertoledos]

I can confirm this is happening with kubernetes version 1.17.9 (eks.2) and eksctl 0.25. The workaround @mrnonz suggested worked. This is causing also that new nodes (for example when upgrading a cluster) to be not ready because the CNI plugin in the workers cannot be initialized (since kube-proxy is not available).

[michaelbeaumont]

Duplicate of https://github.com/weaveworks/eksctl/issues/1088

[michaelbeaumont]

@javiertoledos @mrnonz @Surian @tomhuang12 I'm going to close this issue for now as the images and the docs are now up to date.

[tomhuang12]

Not really. The image tag is now v1.17.9-eksbuild.1 which doesn't follow what eksctl expects as v.17.9. Here is the command suggested by AWS:

kubectl set image daemonset.apps/kube-proxy \
    -n kube-system \
    kube-proxy=602401143452.dkr.ecr.us-west-2.amazonaws.com/eks/kube-proxy:v1.17.9-eksbuild.1

[normalfaults]

I'm seeing the same issue as of today as @tomhuang12 . Tag is expecting eksbuild.1

[michaelbeaumont]

Indeed, please update to version 0.26-rc.1