Terraform: Add support for Enhanced Networking to aws_instance resource.

[kwilczynski]

Amazon at the moment supports two flavours of Enhanced Networking feature:

Overview and documentation.

• SR-IOV (Intel 82599 VF)
  http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/sriov-networking.html
• Elastic Network Adapter (ENA)
  http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking-ena.html

For any VPC-enabled and HVM-compatible EC2 instances.

We should allow users to enable Enhanced Network support and choose which flavour they would like to use. A very common use case, would be to run and span a Docker overlay network layer across multiple instances participating as hosts in the Docker Swarm. Such nodes do benefit from more linear and consistent performance which Enhanced Networking be bring, especially when latency is concerned and momentary spikes in such and "choking up" network throughput (e.g. exhausting the packer per-second rate, etc.) is not desired.

For this to work an AMI (Amazon Machine Image) needs to have moderately up-to-date Linux kernel and include supported version of both the ixgbevf[1] and ena[2] kernel driver, thus it would be prudent to to include latest drives and to enable support for the Enhanced Networking when creating (with Packer, etc.) custom AMIs for both the EBS and Instance Store type.

The Elastic Network Adapter is quite new (see the announcement) and it is not yet supported on Windows due to lack of available drivers for this platform (an on-going development effort by Amazon), plus the only instance type that supports ENA at the moment is X1, but hopefully Amazon would add ENA support to others later.

Worth noting is, that the Amazon Linux HVM-enabled AMIs already ship with both of the drivers included and have Enhanced Networking support enabled, thus are ready to use from the get-go.

API documentation:

• https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstanceAttribute.html
• https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_ModifyInstanceAttribute.html

Conceptually, this very similar to the way how the "SourceDestCheck" check is now supported in Terraform, as per: resource_aws_instance.go#L595-L603

Related to https://github.com/mitchellh/packer/issues/3688 in Packer.

1. Intel 82599 VF (SR-IOV) drivers.
2. Amazon Elastic Network Adapter (ENA) drivers.

[kwilczynski]

I propose adding new a optional attribute called "enhanced_networking", which would be a schema.TypeString and take either "sriov" (or perhaps "simple" would be more appropriate name?) or "ena", and then set things up appropriately. Whereas when not specified the Enhanced Networking would remain what the EC2 instance originally had set in its attributes.

We need to check whether instance is an HVM-compatible one (since it is a requirement), but I would not advocate checking (and thus hard-coding) for which instance type it was enabled. I would leave the due diligence to the user.

What do you think, @stack72 and @phinze?

[jantman]

I'm trying to deploy Consul with Terraform, and would really like to minimize latency between nodes... which enhanced networking is supposed to help with. Having this feature is pretty important to me, as there's no clean way to currently handle this otherwise...

[kwilczynski]

@jantman leave it with me. I will try to close this one as soon as possible.

[ashleydw]

any news @kwilczynski ?

i'm looking into my instances and seeing that although the instance type is c4.large and the mod ixgbevf is installed, ethtool -n still shows vif as the driver.

[kwilczynski]

@ashleydw hi there!

Sincere apologies for the delay (to @jantman as well) - I got busy with some personal matters.

@ashleydw how soon do you need it?

[ashleydw]

No worries. I'm wondering if you can't point me towards where the network driver would be set so i can have a go at modifying it myself?

[kwilczynski]

@ashleydw hi there again!

You mean, you want to send a Pull Request? If so, then the key would be to utilise the ModifyInstanceAttribute API call (and other relevant; see above in the initial message) hidden behind an attribute like e.g. enhanced_networking which would be either a string (might be more future-proof?) or a boolean, etc. Under the hood, one would need to handle both ways of setting SR-IOV and ENA accordingly (these are slightly different).

Having said that, I would envision this to be added to all of the resources that have anything to do with an instance (since the code is not shared), to make sure that spot instances, etc., can benefit from this too (this, a multiple PRs might be required).

[emmanuel]

We would very much like to be able to specify this attribute in the aws_autoscaling_group resource (or aws_launch_configuration).

[mikljohansson]

+1

We're utilizing the new R4 instances that became available a few weeks back and which use ENA. For the Elasticsearch cluster we're building the network latency/performance really matters and we'd really love for this flag to be added so we can easily enable ENA on the instances

[c4milo]

I'm not a 100% sure but I believe this issue belongs more to Packer than Terraform as the AMI images supporting the ENA driver have to be registered in AWS with the ena-support flag. In Terraform one would just configure the AMI image accordingly.

[emmanuel]

@c4milo You're right. I have learned (since my comment here) that this is an attribute on the AMI and needs no opt-in on a per-instance basis. No change needed in Terraform.

[nathanhruby]

It's unclear if this is even something you can do at launch time or on an existing AMI? AFAICT, this is an instance-only attribute, and even then requires the machine instance to be in a stopped state.

[rgs1]

so to be clear, this can be closed per @c4milo comments?

[kwilczynski]

@rgs1 what makes you think so?

[rgs1]

@kwilczynski the fact that it's an AMI level property, not an instance property? or can it be managed at both levels?

[riteshhadawani]

Hi All,

Here in this page, http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking-ena.html#test-enhanced-networking-ena

It states that, even after you AMI has the ena module installed, still you need to enable the ena support in the instance. And that has been done though below command.

We are using AMI which is already having the ENS support enabled with module installed. Still ENA network interface is not enabled.

Can't we have the terraform property which creates instance with ena-support enabled. see below instance level details supported by AWS.

http://docs.aws.amazon.com/cli/latest/reference/ec2/modify-instance-attribute.html

$ aws ec2 modify-instance-attribute --instance-id instance_id --ena-support

[emmanuel]

I'm going to walk back my earlier assertion a bit. What I have observed is that "Enhanced Networking" (SR-IOV) support is enabled for an instance that is launched from an AMI that has the attribute set and an appropriate driver installed/enabled in the OS (ixbegf on Linux, IIRC). I have seen this work without setting any instance-level attributes (i.e., without using modify-instance-attribute or the console).

I have no experience with ENA and withdraw any assertions about what changes might be required (or not) in Terraform to support it.

[emmanuel]

Correction: the Linux kernel module for SR-IOV on EC2 is called ixgbevf. Source: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/sriov-networking.html

See here for more on ENA: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking-ena.html. In the case of ENA, the docs seem to state that a modify-instance-attributes call is not required to enable ENA support, if the attribute is set on the AMI; if not set on the AMI, then modifying the instance looks necessary. Quote from the ENA doc:

The latest Amazon Linux HVM AMIs have the module required for enhanced networking with ENA installed and have the required enaSupport attribute set. Therefore, if you launch an instance with the latest Amazon Linux HVM AMI on a supported instance type, enhanced networking with ENA is already enabled for your instance.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.