Terraform-provider-aws: aws_lb keeps flopping when allocation_id is not present in subnet_mappings

Created on 31 Jan 2019  ·  8Comments  ·  Source: hashicorp/terraform-provider-aws

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

v0.11.10
v0.11.11

Affected Resource(s)

  • aws_lb

Terraform Configuration Files

resource "aws_lb" "platform_int" {
  count = "${var.is_nlb_internal == true ? 1 : 0}"
  name  = "nlb-platform-${var.environment}"

  internal           = true
  load_balancer_type = "network"
  ip_address_type    = "ipv4"

  subnet_mapping {
    subnet_id = "${var.subnet_id_lb}"
  }

  tags {
    Name       = "nlb-platform-${var.environment}"
    env        = "${var.environment}"
    team       = "${var.tag_team}"
    live       = "${var.tag_live}"
    visibility = "int"
    managed_by = "Terraform"
  }
}

Expected Behavior

The NLB should be left untouched as there are no changes to the subnet allocation.

Actual Behavior


The NLB is to be deleted and re-created every time as per the output. It seems this is related to the otherwise "optional" attribute allocation_id in the subnet mapping. When the NLB is created as internet-facing with an EIP pre-allocated there is no flopping.


An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create
-/+ destroy and then create replacement

Terraform will perform the following actions:

-/+ module.lendable_platform.aws_lb.platform_int (new resource required)
      id:                                       "arn:aws:elasticloadbalancing:eu-west-1:<redacted>:loadbalancer/net/nlb-platform-prestg/75f75ac883f141a6" => <computed> (forces new resource)
      arn:                                      "arn:aws:elasticloadbalancing:eu-west-1:<redacted>:loadbalancer/net/nlb-platform-prestg/75f75ac883f141a6" => <computed>
      arn_suffix:                               "net/nlb-platform-prestg/75f75ac883f141a6" => <computed>
      dns_name:                                 "nlb-platform-prestg-75f75ac883f141a6.elb.eu-west-1.amazonaws.com" => <computed>
      enable_cross_zone_load_balancing:         "false" => "false"
      enable_deletion_protection:               "false" => "false"
      internal:                                 "true" => "true"
      ip_address_type:                          "ipv4" => "ipv4"
      load_balancer_type:                       "network" => "network"
      name:                                     "nlb-platform-prestg" => "nlb-platform-prestg"
      security_groups.#:                        "0" => <computed>
      subnet_mapping.#:                         "0" => "1" (forces new resource)
      subnet_mapping.3090889211.allocation_id:  "" => ""
      subnet_mapping.3090889211.subnet_id:      "" => "subnet-0f3817005d5e35ad7" (forces new resource)
      subnets.#:                                "1" => <computed>
      tags.%:                                   "6" => "6"
      tags.Name:                                "nlb-platform-prestg" => "nlb-platform-prestg"
      tags.env:                                 "prestg" => "prestg"
      tags.live:                                "no" => "no"
      tags.managed_by:                          "Terraform" => "Terraform"
      tags.team:                                "appdev" => "appdev"
      tags.visibility:                          "ext" => "int"
      vpc_id:                                   "vpc-<redacted>" => <computed>
      zone_id:                                  "<redacted>" => <computed>

.
.
.

Steps to Reproduce

  1. terraform apply

Important Factoids

Bog standard stuff

bug servicelbv2
Source
picture zskulcsar
👍7

Most helpful comment

The fix to prevent the difference when allocation_id is omitted has been merged and will release with version 2.8.0 of the Terraform AWS Provider, likely later this week. 👍

picture bflad on 25 Apr 2019
🎉2

All 8 comments

This in fact is an old issue: https://github.com/terraform-providers/terraform-provider-aws/issues/3797#issuecomment-386677567

This is happening to me since some time. I was investigating and the issue is on when getting the data to save in the state.

If you define the subnet_mapping with the allocation id, then the availabilityZones object looks like (Faked values to avoid exposing them):

2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4: 2019/01/31 20:07:02 [DEBUG] SUBNET MAPPING: [{
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:   LoadBalancerAddresses: [{
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:       AllocationId: "eipalloc-xxxxxxx1",
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:       IpAddress: "1.1.1.1"
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:     }],
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:   SubnetId: "subnet-yyyyy1",
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:   ZoneName: "us-east-1a"
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4: } {
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:   LoadBalancerAddresses: [{
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:       AllocationId: "eipalloc-xxxxxxx2",
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:       IpAddress: "1.1.1.2"
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:     }],
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:   SubnetId: "subnet-yyyyy2",
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:   ZoneName: "us-east-1c"
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4: } {
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:   LoadBalancerAddresses: [{
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:       AllocationId: "eipalloc-xxxxxxx3",
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:       IpAddress: "1.1.1.3"
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:     }],
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:   SubnetId: "subnet-yyyyy3",
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:   ZoneName: "us-east-1e"
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4: } {
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:   LoadBalancerAddresses: [{
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:       AllocationId: "eipalloc-xxxxxxx4",
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:       IpAddress: "1.1.1.4"
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:     }],
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:   SubnetId: "subnet-yyyyy4",
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:   ZoneName: "us-east-1d"
2019-01-31T20:07:02.776-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4: }]

But, if the subnet_mapping does not contain the allocation id, then it looks as you would have defined the subnets using subnets:

2019-01-31T28:02:02.171-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4: 2019/01/31 20:07:02 [DEBUG] SUBNET MAPPING: [{
2019-01-31T28:02:02.171-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:   SubnetId: "subnet-yyyyy1",
2019-01-31T28:02:02.171-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:   ZoneName: "us-east-1a"
2019-01-31T28:02:02.171-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4: } {
2019-01-31T28:02:02.171-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:   SubnetId: "subnet-yyyyy2",
2019-01-31T28:02:02.171-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:   ZoneName: "us-east-1c"
2019-01-31T28:02:02.171-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4: } {
2019-01-31T28:02:02.171-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:   SubnetId: "subnet-yyyyy3",
2019-01-31T28:02:02.171-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:   ZoneName: "us-east-1e"
2019-01-31T28:02:02.171-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4: } {
2019-01-31T28:02:02.171-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:   SubnetId: "subnet-yyyyy4",
2019-01-31T28:02:02.171-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4:   ZoneName: "us-east-1d"
2019-01-31T28:02:02.171-0300 [DEBUG] plugin.terraform-provider-aws_v1.57.0_x4: }]

The data to the state is flatten and recovered by the availabilityZones structure for both subnets and subnet_mapping. The value for subnet_mapping is recovered specifically from availabilityZone.LoadBalancerAddresses:

func flattenSubnetsFromAvailabilityZones(availabilityZones []*elbv2.AvailabilityZone) []string {
    var result []string
    for _, az := range availabilityZones {
        result = append(result, aws.StringValue(az.SubnetId))
    }
    return result
}

func flattenSubnetMappingsFromAvailabilityZones(availabilityZones []*elbv2.AvailabilityZone) []map[string]interface{} {
    l := make([]map[string]interface{}, 0)
    for _, availabilityZone := range availabilityZones {
        for _, loadBalancerAddress := range availabilityZone.LoadBalancerAddresses {
            m := make(map[string]interface{})
            m["subnet_id"] = aws.StringValue(availabilityZone.SubnetId)

            if loadBalancerAddress.AllocationId != nil {
                m["allocation_id"] = aws.StringValue(loadBalancerAddress.AllocationId)
            }

            l = append(l, m)
        }
    }
    return l
}

Thus, as availabilityZone.LoadBalancerAddresses is empty when using subnet_mapping without the allocation id, it seems to not be a way to differentiate between the following state results:

subnets = [ "subnet-xxxx1", "subnet-xxxx2" ]

or

subnet_mapping {
    subnet_id = "subnet-xxxx1"
}
subnet_mapping {
    subnet_id = "subnet-xxxx2"
}

Ignoring it would be an ugly workaround.. but I don't see another better approach so far:

  lifecycle {
    ignore_changes = ["subnet_mapping"]
  }
facundovictor picture facundovictor on 1 Feb 2019

I've created the PR #7434 that should ensure subnet_mapping is saved on creation/update. Let's wait for feedback from collaborators and owners.

facundovictor picture facundovictor on 4 Feb 2019

@facundovictor thanks for the reply. I'm, in fact, ignoring the changes till this is not fixed. As workarounds go it is fine.

zskulcsar picture zskulcsar on 4 Feb 2019

@zskulcsar how are you ignoring the changes? This is preventing me from updating production, as the 'flapping' causes an unnecessary rebuild of my network load-balancers, which would take all 21 product clusters out-of-service and create issues related to DNS replication for long-cache customers. I am literally dead in the water due to the issue.

johntoups picture johntoups on 5 Feb 2019

@johntoups see lifecycle section at the end:

resource "aws_lb" "platform_int" {
  count = "${var.is_nlb_internal == true ? 1 : 0}"
  name  = "nlb-platform-${var.environment}"

  internal           = true
  load_balancer_type = "network"
  ip_address_type    = "ipv4"

  subnet_mapping {
    subnet_id = "${var.subnet_id_lb}"
  }

  lifecycle {
    ignore_changes = ["subnet_mapping"]
  }
}

For reference: https://www.terraform.io/docs/configuration/resources.html#lifecycle

zskulcsar picture zskulcsar on 5 Feb 2019

The fix to prevent the difference when allocation_id is omitted has been merged and will release with version 2.8.0 of the Terraform AWS Provider, likely later this week. 👍

bflad picture bflad on 25 Apr 2019
🎉2

This has been released in version 2.8.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

nywilken picture nywilken on 27 Apr 2019

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

hashibot[bot] picture hashibot[bot] on 30 Mar 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

blaltarriba picture blaltarriba  ·  3Comments
modax picture modax  ·  3Comments
hashibot picture hashibot  ·  3Comments
ccslamstack picture ccslamstack  ·  3Comments
hashibot picture hashibot  ·  3Comments