Terraform-provider-aws: aws_route_tables change every time

Created on 26 Mar 2018 · 4Comments · Source: hashicorp/terraform-provider-aws

_This issue was originally opened by @edgan as hashicorp/terraform#17696. It was migrated here as a result of the provider split. The original body of the issue is below._

Terraform Version

Terraform v0.11.5

...

Terraform Configuration Files

resource "aws_nat_gateway" "gw-a" {
  allocation_id = "${aws_eip.gw-a.id}"
  subnet_id     = "${aws_subnet.a-external.id}"
  depends_on = ["aws_internet_gateway.development"]

  tags {
    Name = "NAT gateway a"
  }
}

resource "aws_nat_gateway" "gw-b" {
  allocation_id = "${aws_eip.gw-b.id}"
  subnet_id     = "${aws_subnet.b-external.id}"
  depends_on = ["aws_internet_gateway.development"]

  tags {
    Name = "NAT gateway b"
  }
}

resource "aws_nat_gateway" "gw-c" {
  allocation_id = "${aws_eip.gw-c.id}"
  subnet_id     = "${aws_subnet.c-external.id}"
  depends_on = ["aws_internet_gateway.development"]

  tags {
    Name = "NAT gateway c"
  }
}

resource "aws_route_table" "internal-a" {
  vpc_id = "${aws_vpc.development.id}"

  tags {
    Name = "${var.account}-internal-a"
  }

  route {
    cidr_block  = "0.0.0.0/0"
    gateway_id = "${aws_nat_gateway.gw-a.id}"
  }
}

resource "aws_route_table" "internal-b" {
  vpc_id = "${aws_vpc.development.id}"

  tags {
    Name = "${var.account}-internal-b"
  }

  route {
    cidr_block  = "0.0.0.0/0"
    gateway_id = "${aws_nat_gateway.gw-b.id}"
  }
}

resource "aws_route_table" "internal-c" {
  vpc_id = "${aws_vpc.development.id}"

  tags {
    Name = "${var.account}-internal-c"
  }

  route {
    cidr_block  = "0.0.0.0/0"
    gateway_id = "${aws_nat_gateway.gw-c.id}"
  }
}

resource "aws_route_table_association" "a-internal" {
  subnet_id      = "${aws_subnet.a-internal.id}"
  route_table_id = "${aws_route_table.internal-a.id}"
}

resource "aws_route_table_association" "b-internal" {
  subnet_id      = "${aws_subnet.b-internal.id}"
  route_table_id = "${aws_route_table.internal-b.id}"
}

resource "aws_route_table_association" "c-internal" {
  subnet_id      = "${aws_subnet.c-internal.id}"
  route_table_id = "${aws_route_table.internal-c.id}"
}

Debug Output

Terraform will perform the following actions:

  ~ aws_route_table.internal-a
      route.2699986813.cidr_block:                "" => "0.0.0.0/0"
      route.2699986813.egress_only_gateway_id:    "" => ""
      route.2699986813.gateway_id:                "" => "nat-011e5ea2e407b2333"
      route.2699986813.instance_id:               "" => ""
      route.2699986813.ipv6_cidr_block:           "" => ""
      route.2699986813.nat_gateway_id:            "" => ""
      route.2699986813.network_interface_id:      "" => ""
      route.2699986813.vpc_peering_connection_id: "" => ""
      route.3651061347.cidr_block:                "0.0.0.0/0" => ""
      route.3651061347.egress_only_gateway_id:    "" => ""
      route.3651061347.gateway_id:                "" => ""
      route.3651061347.instance_id:               "" => ""
      route.3651061347.ipv6_cidr_block:           "" => ""
      route.3651061347.nat_gateway_id:            "nat-011e5ea2e407b2333" => ""
      route.3651061347.network_interface_id:      "" => ""
      route.3651061347.vpc_peering_connection_id: "" => ""

  ~ aws_route_table.internal-b
      route.3093487912.cidr_block:                "" => "0.0.0.0/0"
      route.3093487912.egress_only_gateway_id:    "" => ""
      route.3093487912.gateway_id:                "" => "nat-094697e03eca6378b"
      route.3093487912.instance_id:               "" => ""
      route.3093487912.ipv6_cidr_block:           "" => ""
      route.3093487912.nat_gateway_id:            "" => ""
      route.3093487912.network_interface_id:      "" => ""
      route.3093487912.vpc_peering_connection_id: "" => ""
      route.74691776.cidr_block:                  "0.0.0.0/0" => ""
      route.74691776.egress_only_gateway_id:      "" => ""
      route.74691776.gateway_id:                  "" => ""
      route.74691776.instance_id:                 "" => ""
      route.74691776.ipv6_cidr_block:             "" => ""
      route.74691776.nat_gateway_id:              "nat-094697e03eca6378b" => ""
      route.74691776.network_interface_id:        "" => ""
      route.74691776.vpc_peering_connection_id:   "" => ""

  ~ aws_route_table.internal-c
      route.2120357975.cidr_block:                "0.0.0.0/0" => ""
      route.2120357975.egress_only_gateway_id:    "" => ""
      route.2120357975.gateway_id:                "" => ""
      route.2120357975.instance_id:               "" => ""
      route.2120357975.ipv6_cidr_block:           "" => ""
      route.2120357975.nat_gateway_id:            "nat-0b1617bb4ca1a5245" => ""
      route.2120357975.network_interface_id:      "" => ""
      route.2120357975.vpc_peering_connection_id: "" => ""
      route.675060784.cidr_block:                 "" => "0.0.0.0/0"
      route.675060784.egress_only_gateway_id:     "" => ""
      route.675060784.gateway_id:                 "" => "nat-0b1617bb4ca1a5245"
      route.675060784.instance_id:                "" => ""
      route.675060784.ipv6_cidr_block:            "" => ""
      route.675060784.nat_gateway_id:             "" => ""
      route.675060784.network_interface_id:       "" => ""
      route.675060784.vpc_peering_connection_id:  "" => ""

aws_route_table.internal-b: Modifying... (ID: rtb-5468022c)
  route.3093487912.cidr_block:                "" => "0.0.0.0/0"
  route.3093487912.egress_only_gateway_id:    "" => ""
  route.3093487912.gateway_id:                "" => "nat-094697e03eca6378b"
  route.3093487912.instance_id:               "" => ""
  route.3093487912.ipv6_cidr_block:           "" => ""
  route.3093487912.nat_gateway_id:            "" => ""
  route.3093487912.network_interface_id:      "" => ""
  route.3093487912.vpc_peering_connection_id: "" => ""
  route.74691776.cidr_block:                  "0.0.0.0/0" => ""
  route.74691776.egress_only_gateway_id:      "" => ""
  route.74691776.gateway_id:                  "" => ""
  route.74691776.instance_id:                 "" => ""
  route.74691776.ipv6_cidr_block:             "" => ""
  route.74691776.nat_gateway_id:              "nat-094697e03eca6378b" => ""
  route.74691776.network_interface_id:        "" => ""
  route.74691776.vpc_peering_connection_id:   "" => ""
aws_route_table.internal-c: Modifying... (ID: rtb-a16802d9)
  route.2120357975.cidr_block:                "0.0.0.0/0" => ""
  route.2120357975.egress_only_gateway_id:    "" => ""
  route.2120357975.gateway_id:                "" => ""
  route.2120357975.instance_id:               "" => ""
  route.2120357975.ipv6_cidr_block:           "" => ""
  route.2120357975.nat_gateway_id:            "nat-0b1617bb4ca1a5245" => ""
  route.2120357975.network_interface_id:      "" => ""
  route.2120357975.vpc_peering_connection_id: "" => ""
  route.675060784.cidr_block:                 "" => "0.0.0.0/0"
  route.675060784.egress_only_gateway_id:     "" => ""
  route.675060784.gateway_id:                 "" => "nat-0b1617bb4ca1a5245"
  route.675060784.instance_id:                "" => ""
  route.675060784.ipv6_cidr_block:            "" => ""
  route.675060784.nat_gateway_id:             "" => ""
  route.675060784.network_interface_id:       "" => ""
  route.675060784.vpc_peering_connection_id:  "" => ""
aws_route_table.internal-a: Modifying... (ID: rtb-5368022b)
  route.2699986813.cidr_block:                "" => "0.0.0.0/0"
  route.2699986813.egress_only_gateway_id:    "" => ""
  route.2699986813.gateway_id:                "" => "nat-011e5ea2e407b2333"
  route.2699986813.instance_id:               "" => ""
  route.2699986813.ipv6_cidr_block:           "" => ""
  route.2699986813.nat_gateway_id:            "" => ""
  route.2699986813.network_interface_id:      "" => ""
  route.2699986813.vpc_peering_connection_id: "" => ""
  route.3651061347.cidr_block:                "0.0.0.0/0" => ""
  route.3651061347.egress_only_gateway_id:    "" => ""
  route.3651061347.gateway_id:                "" => ""
  route.3651061347.instance_id:               "" => ""
  route.3651061347.ipv6_cidr_block:           "" => ""
  route.3651061347.nat_gateway_id:            "nat-011e5ea2e407b2333" => ""
  route.3651061347.network_interface_id:      "" => ""
  route.3651061347.vpc_peering_connection_id: "" => ""
aws_route_table.internal-b: Modifications complete after 0s (ID: rtb-5468022c)
aws_route_table.internal-a: Modifications complete after 0s (ID: rtb-5368022b)
aws_route_table.internal-c: Modifications complete after 0s (ID: rtb-a16802d9)

Expected Behavior

No change

Steps to Reproduce

terraform apply -var-file terraform.tfvars -var-file sensitive.tfvars

Additional Context

I switched our routes from using NAT instances to NAT gateways, and they cause a change every time now.

question servicec2

Source

hashibot[bot]

Most helpful comment

Hi @edgan 👋 Sorry you're running into trouble here.

The aws_route_table resource supports two different attributes inside route configuration blocks: gateway_id and nat_gateway_id, which map to how the underlying AWS API works with routes. In your plan output above you can see Terraform trying to switch which attribute the NAT gateway is associated with:

aws_route_table.internal-a: Modifying... (ID: rtb-5368022b)
...
  route.2699986813.gateway_id:                "" => "nat-011e5ea2e407b2333"
...
  route.3651061347.nat_gateway_id:            "nat-011e5ea2e407b2333" => ""
...

(Aside: the middle number is just an internal hash that Terraform is using to determine two different route configurations. We have plans to fix this output to be easier to read in future versions of Terraform.)

We do note this in the resource documentation, however maybe we could implement better validation:

NOTE on gateway_id and nat_gateway_id: The AWS API is very forgiving with these two attributes and the aws_route_table resource can be created with a NAT ID specified as a Gateway ID attribute. This will lead to a permanent diff between your configuration and statefile, as the API returns the correct parameters in the returned route table. If you're experiencing constant diffs in your aws_route_table resources, the first thing to check is whether or not you're specifying a NAT ID instead of a Gateway ID, or vice-versa.

Does it work if you update your Terraform configuration to use nat_gateway_id instead of gateway_id?

bflad on 26 Mar 2018

👍2

All 4 comments

Hi @edgan 👋 Sorry you're running into trouble here.

aws_route_table.internal-a: Modifying... (ID: rtb-5368022b)
...
  route.2699986813.gateway_id:                "" => "nat-011e5ea2e407b2333"
...
  route.3651061347.nat_gateway_id:            "nat-011e5ea2e407b2333" => ""
...

We do note this in the resource documentation, however maybe we could implement better validation:

NOTE on gateway_id and nat_gateway_id: The AWS API is very forgiving with these two attributes and the aws_route_table resource can be created with a NAT ID specified as a Gateway ID attribute. This will lead to a permanent diff between your configuration and statefile, as the API returns the correct parameters in the returned route table. If you're experiencing constant diffs in your aws_route_table resources, the first thing to check is whether or not you're specifying a NAT ID instead of a Gateway ID, or vice-versa.

Does it work if you update your Terraform configuration to use nat_gateway_id instead of gateway_id?

bflad on 26 Mar 2018

👍2

That was it, but I would say it shouldn't be so liberal about what it accepts. If it works, but makes noise, it is bad.

edgan on 27 Mar 2018

👍1

Closing as this is already documented in the resource documentation. If someone feels inclined to submit a pull request to add the relevant ValidateFunc: validation.StringMatch(regexp.MustCompile(^XXX-), "message"), for each of the resource attributes, we'll happily take a look.

bflad on 22 Jun 2018

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

hashibot[bot] on 5 Apr 2020

Was this page helpful?

0 / 5 - 0 ratings

Related issues

AWS CodeBuild using environment variables from EC2 Parameter Store

blaltarriba · 3Comments

aws_cloudfront_distribution - "Only one viewer certificate change may be in progress at a time"

hashibot · 3Comments

aws_security_group: revoke_rules_on_delete conflict with 'terraform plan'

carmas · 3Comments

aws_alb_target_group_attachment does not support list for target_id

hashibot · 3Comments

aws_elasticsearch_domain: vpc_options - subnet_ids asks for list but can only be a single value.

ccslamstack · 3Comments