Terraform: i3.8xlarge: failure to launch i3.8xlarge instance with four nvme instance store SSDs using auto-scaling group

[wjiang-qc]

@qnu
We are trying to launch i3.8xlarge instance using auto-scaling group to use its four nvme instance store SSDs, but failed to do so. Could you help take a look to see if we are using a wrong configuration or is this a Terraform bug?

Configuration:

    ephemeral_block_device {
        device_name = "/dev/nvme0n1"
        virtual_name = "ephemeral0"
    }
    ephemeral_block_device {
        device_name = "/dev/nvme1n1"
        virtual_name = "ephemeral1"
    }
    ephemeral_block_device {
        device_name = "/dev/nvme2n1"
        virtual_name = "ephemeral2"
    }
    ephemeral_block_device {
        device_name = "/dev/nvme3n1"
        virtual_name = "ephemeral3"
    }

gives the error:

Failed
Launching a new EC2 instance
2017 March 14 15:05:32 UTC-7
Description:DescriptionLaunching a new EC2 instance. Status Reason: Invalid device name /dev/nvme1n1. Launching EC2 instance failed.
Cause:CauseAt 2017-03-14T22:05:31Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 0 to 1.

[danellis]

@tanner-bruce, the numbering is:

Instance types that support two NVMe instance store volume have /dev/nvme0n1 and /dev/nvme1n1, and so on.

I think it's supposed to be like this:

  ephemeral_block_device {
    device_name = "xvdb"
    virtual_name = "nvme0n1"
  }

Unfortunately, that gives this error:

InvalidParameterValue: Value (nvme0n1) for parameter virtualName is invalid. Expected format: 'ephemeral'

That's coming back from the EC2 API, so I'm not sure whether it's Terraform not supporting something or if I'm just specifying it wrong.

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html#StorageOnInstanceTypes

[allenbenz-kordata]

I had luck using

ephemeral_block_device {
    device_name = "/dev/sdf"
    virtual_name = "nvme0n1"
  }

according to: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html#available-ec2-device-names

NVMe volumes are automatically enumerated and assigned a device name. There is no need to specify NVMe volumes in your block device mapping.

[STRML]

It appears you should use a device_name of the format /dev/sd*, and a virtual_name of the format ephemeral*.

These drives aren't listed in the EC2 console nor are they actually mounted at /dev/sd*, but they do show up with this configuration under /dev/nvme*n1.

For clarity:

ephemeral_block_device {
  device_name = "/dev/sdb" # This won't actually get mounted here, it's /dev/nvme0n1
  virtual_name = "ephemeral0"
}

[evilezh]

from aws documentation:
NVMe volumes are automatically enumerated and assigned a device name. There is no need to specify NVMe volumes in your block device mapping.

which means - this is bug, and for nvme devices device_name shall be optional.

[STRML]

Aha - I can confirm, no ephemeral_block_device declaration is necessary at all. The instance will launch with the expected local drives at /dev/nvme*n1 without any configuration.

[urbanski]

I believe this is broken. When not specifying an ephemeral_block_device struct when launching an i3.8xlarge only the first NVMe device is accessible. I've tried several of the solutions proposed here and none of them can get an i3.8xlarge to boot with all four NVMe drives. Launching the instance directly in the AWS console works fine but for some reason the NVMe drives don't come up when launched via terraform.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.