Amazon-eks-ami: Support NVMe SSD root volumes

[phillycheeze]

What would you like to be added:

The AWS EKS AMIs should support the root volume of NVMe SSD instance types within AWS.
When going through the _Launch Instance_ flow of an eks-ami in the AWS console, it requires you to attach a second EBS volume on instance types that have an NVMe SSD root volume.

Why is this needed:

For workloads that require high disk-throughput, these instance types provide higher bandwidth over EBS volumes.

[Jeffwan]

/cc @Jeffwan

[leakingtapan]

If you uses Nitro instances (c5, m5, etc) the root volume will be NVMe by default:

[ec2-user@ip-172-31-21-94 ~]$ df
Filesystem     1K-blocks    Used Available Use% Mounted on
devtmpfs         3890372       0   3890372   0% /dev
tmpfs            3903832       0   3903832   0% /dev/shm
tmpfs            3903832     292   3903540   1% /run
tmpfs            3903832       0   3903832   0% /sys/fs/cgroup
/dev/nvme0n1p1  20959212 1467484  19491728   8% /
tmpfs             780768       0    780768   0% /run/user/1000

Why do you need a second EBS volume?

[phillycheeze]

I'm talking about the _d_ family instances specifically. Like the _m5d.large_ instance type. These types don't use an EBS volume as a root device, instead they use a hardware-based SSD that is directly attached to the instance.

To be more specific, we use EKS for our Gitlab server inside of AWS. Our gitlab CI/CD runners use a dedicated EKS cluster. Since we are spinning up and down nodes many times throughout the day to match load of our CI jobs, we need only a small amount of disk space but very high throughput (docker pull/push and npm install being ran in parallel for dozens of jobs uses a ton of IOPs). It's more cost efficient for us to use the instance types that provide hardware NVMe SSDs as a root volume, and it seems like it's an easy thing to implement given that the Amazon Linux 2 OS provides driver support for these SSDs by default and the EKS AMIs are based off of them.

Please let me know if I'm missing something. From my testing, the AMIs require an EBS volumes with at least 20GB no matter the instance type.

[phillycheeze]

With that said, I imagine that using these instances probably already works without any configuration changes. This may simply be an AMI requirement for an EBS volumes that isn't necessarily required. If someone knows how to get around the requirement that the AMI have an EBS volume attached, I can test it and maybe use it without a problem.

[phillycheeze]

I've attached a screenshot that hopefully helps describe what I said above. It forces me to add an EBS volume on the 1.13.11 eks ami on an m5d.large instance type.

[phillycheeze]

So now I'm realizing that this isn't specific to EKS at all, that amazon doesn't allow the NVMe SSD to be the root device? If this is true, I'm probably just really dumb and you can close this issue :)

[leakingtapan]

With m5d family, NVMe volume is attached to each instance as ephemeral volume (non EBS volume).

With M5d instances, local NVMe-based SSDs are physically connected to the host server and provide block-level storage that is coupled to the lifetime of the M5 instance

See https://aws.amazon.com/ec2/instance-types/

Yep, and for this case NVMe won't be root device.

[phillycheeze]

Thanks so much!

[donotpush]

@phillycheeze we are facing the same issue, we are trying to use NVMe as root device to improve the performance of our Gitlab runners. Our current bottleneck is the IOPs. Did you manage to find a solution?

[johan13]

I had the same idea to put the root fs on NVMe for my GitLab runner. I haven't found a way to do it, but instead I just put /var/lib/docker on the NVMe volume. That is where all the disk access is anyway.

That can be done by adding "amazonec2-userdata=/usr/share/gitlab-runner/mount-nvme.sh" to MachineOptions in config.toml and use this mount-nvme.sh:

#!/bin/sh
mkfs.ext4 /dev/nvme0n1
mkdir -p /var/lib/docker
mount /dev/nvme0n1 /var/lib/docker

I use a Debian Buster AMI, the paths may be different on other distros.

[jae-63]

Thanks for this important clue @johan13 . We found that the following code fragment (based upon your suggestion) works for us, when inserted into our userdata, e.g. for r5d AWS instances. As you can see, in particular we found that we must stop/start the docker daemon.

if ( lsblk | fgrep -q nvme1n1 ); then
   mkfs.ext4 /dev/nvme1n1
   systemctl stop docker
   mkdir -p /var/lib/docker
   mount /dev/nvme1n1 /var/lib/docker
   chmod 711 /var/lib/docker
   systemctl start docker
fi