Amazon-eks-ami: Need EKS control plane and AMI with matching k8s version 1.13.8

[llamahunter]

What happened:
EKS worker nodes and control plane running different version

What you expected to happen:
Latest AWS EKS AMI supports same version of k8s as EKS control plane

How to reproduce it (as minimally and precisely as possible):
Run 1.11 eks.v2 control plan, and official eks ami for workers

Anything else we need to know?:
Change appears to have been checked in on 2019-03-14, but no new AMI has been released since then.

Environment:

• AWS Region: us-east-1
• Instance Type(s): m3.large
• EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion): eks.2
• Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version): 1.11
• AMI Version: v20190220
• Kernel (e.g. uname -a): ??
• Release information (run cat /tmp/release on a node):
  see above

[whereisaaron]

The AMI's are in progress, should be announced here:
https://github.com/aws/containers-roadmap/issues/188#issuecomment-474010632

@tabern:

We updated the EKS control plane to kubernetes 1.11.8 last week to deliver a fix for the recent Golang CVE (kubernetes/kubernetes#73379) to the Kubernetes API server. We are working on delivering updated AMIs for EKS worker nodes and will update this issue when they are available.

[llamahunter]

ok.... subscribed to the other issue... closing this one.

[llamahunter]

The EKS control plane and worker nodes are still out of sync. The control plane is at 1.12.6, but the workers are at 1.12.7. This causes prometheus version mismatch alerts to fire. Please keep the control plane and workers in sync.

[max-rocket-internet]

@llamahunter I think it's OK: https://kubernetes.io/docs/setup/release/version-skew-policy/

kubelet must not be newer than kube-apiserver, and may be up to two minor versions older.

Example:

kube-apiserver is at 1.13
kubelet is supported at 1.13, 1.12, and 1.11

1.12.7 and 1.12.6 are the same minor and major version but at different patch levels.

[tabern]

This is correct. You can run the control plane and worker nodes safely with any patch version skew, and as @max-rocket-internet mentioned, up to two minor versions apart. Is alerting based on patch version skew a standard Prom alerting rule configuration?

[llamahunter]

The problem is that default kube-prometheus configurations report alerts if the versions don’t match. I can suppress the alerts, but I’d prefer to know when there was a version mismatch (as sometimes eks auto updates the control plane), so that I can get the workers back in sync. How hard is it for amazon to just keep the worker ami they supply in sync with the control plane they provide?

[max-rocket-internet]

How hard is it for amazon to just keep the worker ami they supply in sync with the control plane they provide?

Not simple 😅

[llamahunter]

Update description to note that now need AMI matching 1.13.8, since the control plane just auto updated.

[ttyS0]

It would also be nice if the binaries would be released along side the AMIs. I use this repository as a base for building my EKS AMIs, and even though a release was cut a few days ago, the S3 bucket hasn't been updated yet. I realize this isn't exactly the problem described by this issue, but it seems like it would be lessened if the binaries in the amazon-eks bucket were updated in tandem with the control plane updates. Then, even if the amazon AMIs weren't built there would at least be a facility for people to build their own.

aws s3 ls s3://amazon-eks/
                           PRE 1.10.11/
                           PRE 1.10.13/
                           PRE 1.10.3/
                           PRE 1.11.10/
                           PRE 1.11.5/
                           PRE 1.11.8/
                           PRE 1.11.9/
                           PRE 1.12.7/
                           PRE 1.12.9/
                           PRE 1.13.7/
                           PRE cloudformation/

[llamahunter]

fwiw, there's now a 1.13.8 AMI available.