Terraform-provider-aws: Updating network_interface_id in aws_route fails

[lethalpaga]

Terraform Version

Terraform v0.10.8
provider.aws 1.2.0

Affected Resource(s)

• aws_route

Terraform Configuration Files

Config 1:

provider "aws" {
  version             = "~>1.2.0"
}

resource "aws_route" "route_to_eni" {
  route_table_id         = "rtb-ba9acbbb"
  destination_cidr_block = "1.2.3.0/24"

  network_interface_id = "eni-35cadddd"
}

Config 2 (updated network_interface_id):

provider "aws" {
  version             = "~>1.2.0"
}

resource "aws_route" "route_to_eni" {
  route_table_id         = "rtb-ba9acbbb"
  destination_cidr_block = "1.2.3.0/24"

  network_interface_id = "eni-35cadddd"
}

Debug Output

https://gist.github.com/lethalpaga/28714fe7180b2debce09489c77ba1556

Expected Behavior

The route should have been updated with the new ENI id

Actual Behavior

The apply fails with

* aws_route.route_to_eni: InvalidInstanceID: There are multiple interfaces attached to instance 'i-0ec77fe2cdde1'. Please specify an interface ID for the operation instead.
    status code: 400, request id: 060b4661-c5e8-4254-92c0-ac1d13f21df7

Steps to Reproduce

1. terraform apply with config1 to create the route
2. terraform apply with config2 to attempt updating it

Important Factoids

• The route creation works fine. Updating it fails.
• In both config1 and config2 the ENI is attached to an instance that has 2 network interfaces attached.

References

• Possibly related to GH-1426

[raylu-stripe]

this bug is caused by https://github.com/hashicorp/terraform/pull/7686

[YakDriver]

I was able to successfully update a route using new aws_route import ability in PR #5657 . The steps I took were as follows:

1. Create route
2. Import the route with a new config
3. Apply

Terraform properly updated the route resource and I verified that the change occurred in AWS.

Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  ~ aws_route.internal-default-route
      network_interface_id: "eni-0b5793f41da64217d" => "eni-07b3859117af14835"

Plan: 0 to add, 1 to change, 0 to destroy.
...
aws_route.internal-default-route: Modifying... (ID: r-rtb-091e53b115437114d_124.0.0.0/16)
  network_interface_id: "eni-0b5793f41da64217d" => "eni-07b3859117af14835"
aws_route.internal-default-route: Modifications complete after 1s (ID: r-rtb-091e53b115437114d_124.0.0.0/16)

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

[bflad]

Sorry this was set to close with a recently merged pull request. I think we should specifically write an acceptance test that ensures updating an aws_route network_interface_id that points to an aws_instance with two network interfaces updates correctly to verify this as closed.

[kellersyf]

I'm not sure why this was closed with the PRs affecting route imports. That appears to be an unrelated issue?

This problem still exists. If I create a route with a next-hop of an ENI, and then later change the ENI that the route points to, I get the error mentioned in this issue.

Original resource:

resource "aws_route" "my_route" {
    route_table_id = aws_route_table.my_route_table.id
    destination_cidr_block = "0.0.0.0/0"
    network_interface_id = aws_network_interface.primary_eni.id

Updated resource:

resource "aws_route" "my_route" {
    route_table_id = aws_route_table.my_route_table.id
    destination_cidr_block = "0.0.0.0/0"
    network_interface_id = aws_network_interface.secondary_eni.id

Results in this:

Error: InvalidInstanceID: There are multiple interfaces attached to instance 'i-xxxxxxxxxxxxxxxx'. Please specify an interface ID for the operation instead.

I can manually taint the resource and it replaces without issue.

Let me know what other information I can provide to help remedy this. Thanks!

[jonathanhle]

I'm not sure why this was closed with the PRs affecting route imports. That appears to be an unrelated issue?

This problem still exists. If I create a route with a next-hop of an ENI, and then later change the ENI that the route points to, I get the error mentioned in this issue.

Original resource:

resource "aws_route" "my_route" {
    route_table_id = aws_route_table.my_route_table.id
    destination_cidr_block = "0.0.0.0/0"
    network_interface_id = aws_network_interface.primary_eni.id

Updated resource:

resource "aws_route" "my_route" {
    route_table_id = aws_route_table.my_route_table.id
    destination_cidr_block = "0.0.0.0/0"
    network_interface_id = aws_network_interface.secondary_eni.id

Results in this:

Error: InvalidInstanceID: There are multiple interfaces attached to instance 'i-xxxxxxxxxxxxxxxx'. Please specify an interface ID for the operation instead.

I can manually taint the resource and it replaces without issue.

Let me know what other information I can provide to help remedy this. Thanks!

Yeah - I'm getting the same thing with 11.14 TFE. I opened a support ticket. Maybe they'll reopen this.

[nywilken]

hi @kellersyf @jonathanhle I'm going to reopen this issue as I don't see any evidence of the fix in the merged PRs, along with the fact that folks are still seeing this error. In order to best help could we get a sample configuration with the instance resource included that is generating this error?

Have you tired running your configuration in 0.12.10 with the latest provider? If so are you running into the same issue?

[sorah]

I just face this issue on Terraform v0.12.10 + provider.aws v2.32.0.

1. Create aws_route with network_interface_id, apply
2. Have a different EC2 instance with _multiple_ ENIs
3. Change network_interface_id of aws_route (1) with an ENI of instance (2), apply

(3) fails with the error mentioned in this issue.

---

https://github.com/terraform-providers/terraform-provider-aws/commit/c572134bb890a694b83a4eb459ded40159392851 changed instance_id to always take precedence over network_instance_id and guessing this is the cause. The list differs between resourceAwsRouteUpdate and resourceAwsRouteCreate. But reverting this doesn't fix (then it starts failing updating routes with instance_id).

[sorah]

I expect terraform sends UpdateRoute request with NetworkInterfaceID, but actually it sends InstanceID always.

[kellersyf]

Is it possible to just do a straight-up replace of the routes when they're updated? Is there any benefit in doing an in-place change vs a remove/re-create of the route? That would solve pretty much all your cases for updating, and the API call is very quick.

[jonathanhle]

hi @kellersyf @jonathanhle I'm going to reopen this issue as I don't see any evidence of the fix in the merged PRs, along with the fact that folks are still seeing this error. In order to best help could we get a sample configuration with the instance resource included that is generating this error?

Have you tired running your configuration in 0.12.10 with the latest provider? If so are you running into the same issue?

Unfortunately, can't move my code base to 0.12.10 yet. Have only tried the above on 0.11.14.

[don-code]

Switching routes over to another ENI is part of an automated failover process I'm working on, so as a workaround I run a one-liner before doing so that taints every route that's changing. In the interest of sharing:

terraform plan | grep '~ .*aws_route\.' | sed -E -e 's/  ~ /terraform taint /' -e 's/module\.(.*).aws_route/-module \1 aws_route/' -e 's/\[/./' -e 's/]//'

...which produces output like:

terraform taint -module cisco aws_route.public.0 
terraform taint -module cisco aws_route.public.1 
terraform taint aws_route.vpn-public.1 
terraform taint aws_route.vpn-public.2 
terraform taint aws_route.vpn-private

Explanation:

1. Make a plan.
2. Filter out only the routes that are changing.
3. Change the move sigil to terraform taint.
4. If the route is part of a module, add it to the -module flag. (Note that this will not work for routes in nested modules. I don't have any, so someone else can cross that bridge when they come to it.)
5. If the route is looped over using count, change the [n] suffix of the resource identifier to the .n suffix that taint wants.