Terraform: Network Interface private_ips in different order

[Shaiou]

• aws_network_interface

Hi there,
I'm running into some issue when creating a network interface, I specified a list of private_ips and made sure the one I wanted as primary was listed first, however it does respect that order and the second one appears as primary:

Here's my ressource:

resource "aws_network_interface" "waf_1_pub" {
    subnet_id = "${aws_subnet.public.1.id}"
    security_groups = ["${aws_security_group.waf_prv.id}"]
    # We put the basic ip for sync and all and one IP per VS
    private_ips = ["${cidrhost(aws_subnet.public.1.cidr_block, var.waf_ip_offset)}", "${join(",",template_file.waf_1_vs_ips.*.rendered)}"]
    security_groups = [ "${aws_security_group.waf_pub.id}" ]
    attachment {
        instance = "${aws_instance.waf.1.id}"
        device_index = 1
    }
}

And the output of the terraform show after the apply:

aws_network_interface.waf_1_pub:
  id = eni-8ba85cf3
  attachment.# = 1
  attachment.3009111673.attachment_id = eni-attach-2baa9ae8
  attachment.3009111673.device_index = 1
  attachment.3009111673.instance = i-18be9790
  description = 
  private_ips.# = 2
  private_ips.3297441504 = 10.220.1.210
  private_ips.3717318561 = 10.220.1.200 => that was the first one in my list and
  security_groups.# = 2
  security_groups.1563906976 = sg-9a10b6fd
  security_groups.973784294 = sg-9910b6fe
  source_dest_check = true
  subnet_id = subnet-e2c50e86
  tags.# = 0

Can you help ?

[freimer]

I looked at the code, and this appears to be because the list is actually defined as a set, and then converted to a List(). There is no order preserved. I propose that we add a field called private_ip that indicates the single private IP address for the interface, and keep private_ips to indicate any secondary IP addresses. The only distinction is which one is primary, which can be very important. Until then, I plan on only specifying one address in private_ips and then using a local-exec provisioner to add the secondary IPs to the network interface, using the AWS CLI.
👍

[freimer]

Hmm, my plan to use local-exec to add additional secondary IPs won't work, because when we re-apply it deletes the additional private IPs! So this really needs to be fixed...

[freimer]

See #6986

[freimer]

@Shaiou you might want to check out https://github.com/freimer/terraform/tree/freimer/aws_network_interfaces also. It's not "done" yet, but I extended it to be able to do:

resource "aws_instance" "instance" {
  network_interface {
    network_interface_id = "${aws_network_interface.ENI0.id}"
    device_index = 0
  }
  network_interface {
    network_interface_id = "${aws_network_interface.ENI1.id}"
    device_index = 1
  }
  network_interface {
    network_interface_id = "${aws_network_interface.ENI2.id}"
    device_index = 2
  }
}

You wouldn't put any availability_zone, security_groups, vpc_security_group_ids, subnet_id, private_ip, source_dest_check, or associate_public_ip_address. The idea is that if you used network_interface you would be creating all of the ENIs as separate resources, and not using attachment, and instead specifying those ENIs in the instance. So the ENIs would already have the proper security group, subnet, etc, all set, and you wouldn't specify any of that in the instance. All tests run, and I ran it on Mac OS X and created three instances this way. I need to figure out the resourceAwsInstanceRead, resourceAwsInstanceUpdate, and possibly others, so that state is read correctly. There are some hard-coded things in there as far as IP addresses and such that probably shouldn't be. And, I don't know Go, yet. But it is a start to fixing this, I believe the correct way.

[freimer]

Still not "done" but this now works. In PR #7096

[KashifSaadat]

+1, also encountered this issue on versions 0.7 - 0.7.4 for some of my AWS instances that require a specific primary IP.

Thanks @freimer for your efforts!

[freimer]

Unfortunately, the PR was not merged, and now there are conflicts. I will have to see when I will get a chance to redo this on top of the latest release.

[RupertExact]

This would be very helpful in reliably setting up Windows clusters and MS SQL Availabilty Groups, otherwise IP address conflicts are generated when the "wrong" IP address ends up as the primary address for that ENI. Multiple ENI's doesn't help.

[stevenewey]

+1

[darrylmendez]

I get a strange error

Error: aws_instance.ar-instance: "network_interface": conflicts with subnet_id. I am adding primary ENI on instance creation. Subnets are the same for both the instance and the ENI.

resource "aws_instance" "ar-instance" {

---

---

network_interface {
device_index = 0
interface_id = "${var.primary_eni_id}"
delete_on_termination = false
}
}

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.