Amazon-eks-ami: Duplicate AMI versions for v20190614

[marcoabreu]

What happened:
We filter our AMIs not by AMI ID but instead use the version string "amazon-eks-gpu-node-1.13-v20190614" to allow a single filter across regions. On June 19th, a bunch of new AMIs was released but they didn't update the version name.

What you expected to happen:
I expected each version to exist only once. Instead, there are multiple images with the same version.

How to reproduce it (as minimally and precisely as possible):
https://us-west-2.console.aws.amazon.com/ec2/v2/home?region=us-west-2#Images:visibility=public-images;ownerAlias=602401143452;search=amazon-eks-gpu-node-1.13-v20190614;sort=desc:creationDate

Anything else we need to know?:
It seems like somebody pressed upload too early, since multiple AMIs have been pushed on that date (1.11, 1.12, 1.13, etc) without updating the version.

  | amazon-eks-gpu-node-1.13-v20190614 | ami-08e5329e1dbf22c6a | amazon/amazon-eks-gpu-node-1.13-v20190614 | amazon | Public | available | June 19, 2019 at 11:17:21 PM UTC+2 | Other Linux | ebs | hvm
  | amazon-eks-gpu-node-1.13-v20190614 | ami-081361ac5fd4eb9a1 | amazon/amazon-eks-gpu-node-1.13-v20190614 | amazon | Public | available | June 15, 2019 at 8:58:39 AM UTC+2 | Other Linux | ebs | hvm

Environment:

• AWS Region: us-west-2
• Instance Type(s): GPU, doesn't matter
• EKS Platform version (use aws eks describe-cluster --name <name> --query cluster.platformVersion): Doesn't matter
• Kubernetes version (use aws eks describe-cluster --name <name> --query cluster.version): Doesn't matter
• AMI Version: v20190614
• Kernel (e.g. uname -a): Doesn't matter
• Release information (run cat /etc/eks/release on a node): Doesn't matter

Doesn't matter

[max-rocket-internet]

Wow, that's really annoying! In the Hashicorp EKS Terraform module we also filter by name for the exact same reason: https://github.com/terraform-aws-modules/terraform-aws-eks/blob/master/data.tf#L22

[M00nF1sh]

Hi.
I'm really sorry for caused the trouble.
It's due to an incident that the older one of "amazon-eks-gpu-node-1.13-v20190614" is incorrectly configured with Nvidia driver kernel module.(The kernel module is not auto-compiled due to missing kernel headers at the time of build).
We will clean up the old one of amazon-eks-gpu-node-1.13-v20190614 shortly after verified it's not in use by customers.

BTW, we are improving our automated test process to ensure such error will be captured before publish AMIs.

[marcoabreu]

Hi, thanks a lot for the quick response!

Could you, in future, also ensure that there are no AMIs released with multiple names? I think the users have the expectation that they are unique. Is that assumption correct?

[Jeffwan]

@marcoabreu Thanks for the suggestion. We will make sure one to one mapping from AMI name to AMI Id. This has been improved a lot.

Resolve the issue. Feel free to reopen if you have further questions.