Terraform-provider-aws: EIP scope change incorrectly detected

Created on 25 Jul 2018  ·  16Comments  ·  Source: hashicorp/terraform-provider-aws

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

$ terraform version
Terraform v0.11.7
+ provider.aws v1.28.0

Affected Resource(s)

-/+ aws_eip.sandbox_infrastructure_zappi_it_us_east_1b (new resource required)
      id:                                                     "eipalloc-8b296e83" => <computed> (forces new resource)
      allocation_id:                                          "" => <computed>
      association_id:                                         "" => <computed>
      domain:                                                 "standard" => <computed>
      instance:                                               "i-ac10f68c" => <computed>
      network_interface:                                      "" => <computed>
      private_ip:                                             "" => <computed>
      public_ip:                                              "23.21.180.181" => <computed>
      tags.%:                                                 "0" => "4"
      tags.Environment:                                       "" => "sandbox"
      tags.Infrastructure:                                    "" => "kubernetes"
      tags.Name:                                              "" => "us-east-1b.sandbox.infrastructure.zappi.it"
      tags.Role:                                              "" => "operations"
      vpc:                                                    "false" => "true" (forces new resource)

Terraform Configuration Files

resource "aws_eip" "sandbox_infrastructure_zappi_it_us_east_1b" {
  vpc = true

  depends_on = [
    "aws_internet_gateway.sandbox_infrastructure_zappi_it"
  ]

  tags = {
    Name            = "us-east-1b.sandbox.infrastructure.zappi.it"
    Environment     = "sandbox"
    Infrastructure  = "kubernetes"
    Role            = "operations"
  }
}

Debug Output

Expected Behavior

Terraform should not incorrectly detect that the VPC scope has changed when it has not, as per the screenshot below:

image

Actual Behavior

Terraform incorrectly detects that the VPC scope has changed from false to true.

The example above was not the first incident, but just a more concrete example. Below is the first case we identified:

-/+ aws_eip.us-east-1b-uat-kubernetes-zappi-it (new resource required)
      id:                                                 "eipalloc-855eacb4" => <computed> (forces new resource)
      allocation_id:                                      "" => <computed>
      association_id:                                     "" => <computed>
      domain:                                             "standard" => <computed>
      instance:                                           "i-ac10f68c" => <computed>
      network_interface:                                  "" => <computed>
      private_ip:                                         "" => <computed>
      public_ip:                                          "23.21.180.181" => <computed>
      tags.%:                                             "0" => "3"
      tags.KubernetesCluster:                             "" => "uat.kubernetes.zappi.it"
      tags.Name:                                          "" => "us-east-1b.uat.kubernetes.zappi.it"
      tags.kubernetes.io/cluster/uat.kubernetes.zappi.it: "" => "owned"
      vpc:                                                "false" => "true" (forces new resource)

-/+ aws_nat_gateway.us-east-1b-uat-kubernetes-zappi-it (new resource required)
      id:                                                 "nat-050bee284fbf24f25" => <computed> (forces new resource)
      allocation_id:                                      "eipalloc-855eacb4" => "${aws_eip.us-east-1b-uat-kubernetes-zappi-it.id}" (forces new resource)
      network_interface_id:                               "eni-8ff31264" => <computed>
      private_ip:                                         "10.81.2.116" => <computed>
      public_ip:                                          "34.193.141.134" => <computed>
      subnet_id:                                          "subnet-68d74754" => "subnet-68d74754"
      tags.%:                                             "3" => "3"
      tags.KubernetesCluster:                             "uat.kubernetes.zappi.it" => "uat.kubernetes.zappi.it"
      tags.Name:                                          "us-east-1b.uat.kubernetes.zappi.it" => "us-east-1b.uat.kubernetes.zappi.it"
      tags.kubernetes.io/cluster/uat.kubernetes.zappi.it: "owned" => "owned"

  ~ aws_route.private-us-east-1b-0-0-0-0--0
      nat_gateway_id:                                     "nat-050bee284fbf24f25" => "${aws_nat_gateway.us-east-1b-uat-kubernetes-zappi-it.id}"

Error during apply:

Applying uat terraform plan...
aws_nat_gateway.us-east-1b-uat-kubernetes-zappi-it: Destroying... (ID: nat-050bee284fbf24f25)
aws_nat_gateway.us-east-1b-uat-kubernetes-zappi-it: Still destroying... (ID: nat-050bee284fbf24f25, 10s elapsed)
aws_nat_gateway.us-east-1b-uat-kubernetes-zappi-it: Still destroying... (ID: nat-050bee284fbf24f25, 20s elapsed)
aws_nat_gateway.us-east-1b-uat-kubernetes-zappi-it: Still destroying... (ID: nat-050bee284fbf24f25, 30s elapsed)
aws_nat_gateway.us-east-1b-uat-kubernetes-zappi-it: Still destroying... (ID: nat-050bee284fbf24f25, 40s elapsed)
aws_nat_gateway.us-east-1b-uat-kubernetes-zappi-it: Still destroying... (ID: nat-050bee284fbf24f25, 50s elapsed)
aws_nat_gateway.us-east-1b-uat-kubernetes-zappi-it: Destruction complete after 50s
aws_eip.us-east-1b-uat-kubernetes-zappi-it: Destroying... (ID: eipalloc-855eacb4)

Error: Error applying plan:

1 error(s) occurred:

* aws_eip.us-east-1b-uat-kubernetes-zappi-it (destroy): 1 error(s) occurred:

* aws_eip.us-east-1b-uat-kubernetes-zappi-it: InvalidParameterValue: Invalid value 'eipalloc-855eacb4' for PublicIp. Not a valid IPv4 address.
    status code: 400, request id: 9a2c55d3-aa28-4503-b4f0-51e607f5dd07

This behaviour started today at ~08:45 AM UTC for 1 of 3 availability zones for one environment initially.

However, in subsequent terraform plan executions a few minutes later, all of our environments are now affected.

Steps to Reproduce

We use the following during our CI runs:

  1. terraform init
  2. terraform plan -no-color --out="${environment}.plan" | tee "${environment}.report"
  3. terraform apply "${environment}.plan"

Don't have an isolated reproducible case yet, but will provide if necessary.

Important Factoids

Running in VPC.
Terraform is run in CI.

bug servicec2 upstream
Source
picture zacblazic
👍11

Most helpful comment

@rv-aburdine certainly wouldn't hurt!

picture bflad on 25 Jul 2018
👍3

All 16 comments

Ran into the same problem this morning, with an interesting difference that I think may (hopefully) shed some light on the issue:

Terraform plan output

aws_eip.bastion (new resource required)
      id:                   "eipalloc-64c1c253" => <computed> (forces new resource)
      allocation_id:        "" => <computed>
      association_id:       "" => <computed>
      domain:               "standard" => <computed>
      instance:             "" => "i-0f09aa63bde8abf6d"
      network_interface:    "" => <computed>
      private_ip:           "" => <computed>
      public_ip:            "54.163.241.54" => <computed>
      vpc:                  "false" => "true" (forces new resource)

-/+ aws_eip.choose_bastion (new resource required)
      id:                   "eipalloc-a7967aac" => <computed> (forces new resource)
      allocation_id:        "" => <computed>
      association_id:       "" => <computed>
      domain:               "standard" => <computed>
      instance:             "" => "i-005bbe6d945152a2e"
      network_interface:    "" => <computed>
      private_ip:           "" => <computed>
      public_ip:            "54.163.241.54" => <computed>
      vpc:                  "false" => "true" (forces new resource)

Terraform Configuration

resource "aws_eip" "bastion" {
  instance = "${aws_instance.bastion.id}"
  vpc      = true
}

resource "aws_eip" "choose_bastion" {
  instance = "${aws_instance.choose_bastion.id}"
  vpc      = true
}

Current Terraform State

"aws_eip.bastion": {
                    "type": "aws_eip",
                    "depends_on": [
                        "aws_instance.bastion"
                    ],
                    "primary": {
                        "id": "eipalloc-64c1c253",
                        "attributes": {
                            "association_id": "eipassoc-8fc12584",
                            "domain": "vpc",
                            "id": "eipalloc-64c1c253",
                            "instance": "i-0f09aa63bde8abf6d",
                            "network_interface": "eni-801a3a7b",
                            "private_ip": "10.110.164.31",
                            "public_ip": "35.170.90.79",
                            "tags.%": "0",
                            "vpc": "true"
                        },
                        "meta": {},
                        "tainted": false
                    },
                    "deposed": [],
                    "provider": "provider.aws"
                },
                "aws_eip.choose_bastion": {
                    "type": "aws_eip",
                    "depends_on": [
                        "aws_instance.choose_bastion"
                    ],
                    "primary": {
                        "id": "eipalloc-a7967aac",
                        "attributes": {
                            "association_id": "eipassoc-eb232f31",
                            "domain": "vpc",
                            "id": "eipalloc-a7967aac",
                            "instance": "i-005bbe6d945152a2e",
                            "network_interface": "eni-f065ca97",
                            "private_ip": "10.0.0.79",
                            "public_ip": "18.211.114.115",
                            "tags.%": "0",
                            "vpc": "true"
                        },
                        "meta": {
                            "e2bfb730-ecaa-11e6-8f88-34363bc7c4c0": {
                                "delete": 180000000000,
                                "read": 900000000000,
                                "update": 300000000000
                            }
                        },
                        "tainted": false
                    },
                    "deposed": [],
                    "provider": "provider.aws"
                },

The key thing to notice is that although both eips are different, terraform seems to think they have the exact same starting configuration (except for the ids) that needs to be updated - same public IP, same "vpc: false" status. There are also several other eips in the same account/configuration that are showing the same problem, terraform thinks each one has the same public IP / vpc: false configuration.

My guess is that the AWS api is returning things differently than previously (either correctly or incorrectly) that's causing terraform to get the same eip every time it refreshes the state for different eips.

additional notes

  • Run via terraform enterprise
  • AWS provider version: latest

Update (after looking into it a bit more)

After running aws ec2 describe-addresses on the command line for the account - it looks like terraform is taking the first returned eip and using that for _all_ of the eips in the account:

$ aws ec2 describe-addresses
{
  "Addresses": [{
    "InstanceId": "",
    "PublicIp": "54.163.241.54",
    "Domain": "standard"
  }, {
    // additional eips returned, including those that _should_ be used for the existing ones in configuration
  }]
}
rv-aburdine picture rv-aburdine on 25 Jul 2018

Can we determine if this is affecting just one region or multiple? It may be an EC2 API update rolling out.

If someone could provide the EC2 API responses from the AWS SDK logging by enabling debug logging, that would be immensely helpful in troubleshooting this issue.

e.g.

2018/07/24 07:51:05 [DEBUG] [aws-sdk-go] DEBUG: Response ec2/DescribeAddresses Details:
---[ RESPONSE ]--------------------------------------
HTTP/1.1 200 OK
Connection: close
Transfer-Encoding: chunked
Content-Type: text/xml;charset=UTF-8
Date: Tue, 24 Jul 2018 07:51:06 GMT
Server: AmazonEC2
Vary: Accept-Encoding


-----------------------------------------------------
2018/07/24 07:51:05 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
<DescribeAddressesResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
    <requestId>b9ec2e2d-a851-44f0-8397-6877e418f31f</requestId>
    <addressesSet>
        <item>
            <publicIp>35.164.116.119</publicIp>
            <allocationId>eipalloc-d86847e4</allocationId>
            <domain>vpc</domain>
            <instanceId>i-008ec6fd6d75e6e15</instanceId>
            <associationId>eipassoc-8f331942</associationId>
            <networkInterfaceId>eni-9a39a191</networkInterfaceId>
            <networkInterfaceOwnerId>187416307283</networkInterfaceOwnerId>
            <privateIpAddress>172.31.41.160</privateIpAddress>
        </item>
    </addressesSet>
</DescribeAddressesResponse>

Thanks!

bflad picture bflad on 25 Jul 2018

@bflad Here's a plan I did a few minutes ago with trace logging level:

2018-07-25T15:34:19.148+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4: 2018/07/25 15:34:19 [DEBUG] [aws-sdk-go] DEBUG: Response ec2/DescribeAddresses Details:
2018-07-25T15:34:19.148+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4: ---[ RESPONSE ]--------------------------------------
2018-07-25T15:34:19.148+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4: HTTP/1.1 200 OK
2018-07-25T15:34:19.148+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4: Connection: close
2018-07-25T15:34:19.148+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4: Transfer-Encoding: chunked
2018-07-25T15:34:19.148+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4: Content-Type: text/xml;charset=UTF-8
2018-07-25T15:34:19.148+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4: Date: Wed, 25 Jul 2018 13:34:18 GMT
2018-07-25T15:34:19.148+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4: Server: AmazonEC2
2018-07-25T15:34:19.148+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4: Vary: Accept-Encoding
2018-07-25T15:34:19.148+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:
2018-07-25T15:34:19.148+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:
2018-07-25T15:34:19.148+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4: -----------------------------------------------------
2018-07-25T15:34:19.148+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4: 2018/07/25 15:34:19 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
2018-07-25T15:34:19.148+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4: <DescribeAddressesResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
2018-07-25T15:34:19.148+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:     <requestId>9b3baee8-013d-4051-96ae-fdf4fd303a6f</requestId>
2018-07-25T15:34:19.148+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:     <addressesSet>
2018/07/25 15:34:19 [TRACE] root: eval: *terraform.EvalWriteState
2018-07-25T15:34:19.148+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:         <item>
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:             <publicIp>23.21.180.181</publicIp>
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:             <domain>standard</domain>
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:             <instanceId>i-ac10f68c</instanceId>
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:         </item>
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:         <item>
2018/07/25 15:34:19 [TRACE] [walkRefresh] Exiting eval tree: aws_eip.sandbox_infrastructure_zappi_it_us_east_1b
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:             <publicIp>23.22.39.66</publicIp>
2018/07/25 15:34:19 [TRACE] dag/walk: walking "aws_nat_gateway.sandbox_infrastructure_zappi_it_us_east_1b"
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:             <allocationId>eipalloc-8b296e83</allocationId>
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:             <domain>vpc</domain>
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:             <associationId>eipassoc-51c56f9c</associationId>
2018/07/25 15:34:19 [TRACE] vertex 'root.aws_nat_gateway.sandbox_infrastructure_zappi_it_us_east_1b': walking
2018/07/25 15:34:19 [TRACE] root: eval: *terraform.EvalSequence
2018/07/25 15:34:19 [TRACE] root: eval: *terraform.EvalGetProvider
2018/07/25 15:34:19 [TRACE] root: eval: *terraform.EvalReadState
2018/07/25 15:34:19 [TRACE] root: eval: *terraform.EvalRefresh
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:             <networkInterfaceId>eni-f7b932cf</networkInterfaceId>
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:             <networkInterfaceOwnerId>[redacted]</networkInterfaceOwnerId>
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:             <privateIpAddress>10.43.1.27</privateIpAddress>
2018/07/25 15:34:19 [TRACE] vertex 'root.aws_nat_gateway.sandbox_infrastructure_zappi_it_us_east_1b': evaluating
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:             <tagSet>
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:                 <item>
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:                     <key>Name</key>
2018/07/25 15:34:19 [TRACE] [walkRefresh] Entering eval tree: aws_nat_gateway.sandbox_infrastructure_zappi_it_us_east_1b
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:                     <value>us-east-1b.sandbox.infrastructure.zappi.it</value>
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:                 </item>
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:                 <item>
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:                     <key>Infrastructure</key>
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:                     <value>kubernetes</value>
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:                 </item>
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:                 <item>
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:                     <key>Role</key>
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:                     <value>operations</value>
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:                 </item>
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:                 <item>
2018-07-25T15:34:19.149+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:                     <key>Environment</key>
2018-07-25T15:34:19.150+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:                     <value>sandbox</value>
2018-07-25T15:34:19.150+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:                 </item>
2018-07-25T15:34:19.150+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:             </tagSet>
2018-07-25T15:34:19.150+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:         </item>
2018-07-25T15:34:19.150+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4:     </addressesSet>
2018-07-25T15:34:19.150+0200 [DEBUG] plugin.terraform-provider-aws_v1.27.0_x4: </DescribeAddressesResponse>

Hopefully that helps.

zacblazic picture zacblazic on 25 Jul 2018

Seems like there's an additional item node being returned now? 🤔

zacblazic picture zacblazic on 25 Jul 2018

We ended up fixing our issue by removing the unallocated (i.e. "standard") eips in our account - with debug logging we saw the exact same problem - additional items _without_ the correct allocation id being returned the the AWS DescribeAddresses request. Ideally AWS shouldn't be returning the extra items if you pass the allocation id to the request, but the logic could possibly be changed here to maybe search in all returned items to find the right one? : https://github.com/terraform-providers/terraform-provider-aws/blob/4df232fc4f1b62d0ce7d3a4ce3fb70503574ac60/aws/resource_aws_eip.go#L185-L193

trace:

2018/07/25 09:59:14 [DEBUG] [aws-sdk-go] DEBUG: Response ec2/DescribeAddresses Details:
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: ---[ RESPONSE ]--------------------------------------
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: HTTP/1.1 200 OK
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Connection: close
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Transfer-Encoding: chunked
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Content-Type: text/xml;charset=UTF-8
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Date: Wed, 25 Jul 2018 13:59:13 GMT
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Server: AmazonEC2
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Vary: Accept-Encoding
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: -----------------------------------------------------
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 2018/07/25 09:59:14 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: <DescribeAddressesResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:     <requestId>23e15c76-9365-4d84-849c-d0550c9d58f0</requestId>
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:     <addressesSet>
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         <item>
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <publicIp>54.163.241.54</publicIp>
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <domain>standard</domain>
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <instanceId/>
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         </item>
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         <item>
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <publicIp>54.204.5.68</publicIp>
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <domain>standard</domain>
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <instanceId/>
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         </item>
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         <item>
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <publicIp>18.211.114.115</publicIp>
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <allocationId>eipalloc-a7967aac</allocationId>
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <domain>vpc</domain>
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <instanceId>i-005bbe6d945152a2e</instanceId>
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <associationId>eipassoc-eb232f31</associationId>
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <networkInterfaceId>eni-f065ca97</networkInterfaceId>
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <networkInterfaceOwnerId>078577932248</networkInterfaceOwnerId>
2018/07/25 09:59:14 [TRACE] root: eval: *terraform.EvalWriteState
2018/07/25 09:59:14 [TRACE] [walkRefresh] Exiting eval tree: aws_eip.choose_bastion
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <privateIpAddress>10.0.0.79</privateIpAddress>
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         </item>
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:     </addressesSet>
2018/07/25 09:59:14 [TRACE] dag/walk: walking "provider.aws (close)"
2018/07/25 09:59:14 [TRACE] vertex 'root.provider.aws (close)': walking
2018-07-25T09:59:14.393-0400 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: </DescribeAddressesResponse>
rv-aburdine picture rv-aburdine on 25 Jul 2018

That sounds like a potential fix we might need to implement if the server side filtering isn't working as expected. For anyone following this issue, I would suggest contacting AWS support if you can about this issue so the EC2 team can potentially rollback/fix this behavior.

bflad picture bflad on 25 Jul 2018

@bflad would it be worth making a PR anyways for it? (I am volunteering 😄) seems like it would make the logic in that spot a bit more clear anyways.

rv-aburdine picture rv-aburdine on 25 Jul 2018

@rv-aburdine certainly wouldn't hurt!

bflad picture bflad on 25 Jul 2018
👍3

Unfortunately we haven't got a support contract currently, so if anyone else has and can report on our behalf that would be much appreciated.

In the interim we've had to manually re-create the destroyed resources as we can't remove our standard EIP at this moment. Though, we're looking into re-creating associated resources within a VPC.

Thanks for the assistance thus far everyone!

zacblazic picture zacblazic on 25 Jul 2018

@zacblazic I just sent in a support request to AWS to hold off on their rollout and pass this bug to their engineering (for whatever that's worth).

Given the severity of the issue, would this warrant cutting a release today? Our infra nearly took a nosedive - thankfully, we don't trust the CI to do EIP deallocation ;-)

dafyddcrosby picture dafyddcrosby on 25 Jul 2018

@zacblazic I just sent in a support request to AWS to hold off on their rollout and pass this bug to their engineering (for whatever that's worth).

Thanks @dafyddcrosby!

thankfully, we don't trust the CI to do EIP deallocation

This was a scary lesson for us actually. We have the following workflow:

  1. Create PR.
  2. Wait for CI to run a terraform plan and report it back to GitHub as a comment.
  3. Get approval from reviewers.
  4. Land change on master.
  5. CI runs terraform plan & terraform apply on master.

We were unlucky enough to have the terraform plan in step 2 not catch the issue. It only appeared during the master run.

In light of what happened today we're updating our workflow to require manual approval between the terraform plan & terraform apply on master runs.

zacblazic picture zacblazic on 25 Jul 2018

Terraform Plan (DEBUG)

Request

2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 2018/07/25 10:44:11 [DEBUG] [aws-sdk-go] DEBUG: Request ec2/DescribeAddresses Details:
2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: POST / HTTP/1.1
2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Host: ec2.us-east-1.amazonaws.com
2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: User-Agent: aws-sdk-go/1.14.26 (go1.9.2; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.8-dev
2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Content-Length: 76
2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Authorization: <REDACTED>
2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Content-Type: application/x-www-form-urlencoded; charset=utf-8
2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: X-Amz-Date: 20180725T164411Z
2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Accept-Encoding: gzip
2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 
2018-07-25T10:44:11.975-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Action=DescribeAddresses&AllocationId.1=eipalloc-de3a31d6&Version=2016-11-15

Note the allocation ID above: eipalloc-de3a31d6

Response

2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 2018/07/25 10:44:12 [DEBUG] [aws-sdk-go] DEBUG: Response ec2/DescribeAddresses Details:
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: ---[ RESPONSE ]--------------------------------------
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: HTTP/1.1 200 OK
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Connection: close
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Transfer-Encoding: chunked
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Content-Type: text/xml;charset=UTF-8
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Date: Wed, 25 Jul 2018 16:44:12 GMT
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Server: AmazonEC2
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: Vary: Accept-Encoding
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: -----------------------------------------------------
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: 2018/07/25 10:44:12 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: <DescribeAddressesResponse xmlns="http://ec2.amazonaws.com/doc/2016-11-15/">
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:     <requestId>b2b333cd-e839-47b7-9fba-a27f20d61c2d</requestId>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:     <addressesSet>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         <item>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <publicIp>23.21.219.184</publicIp>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <domain>standard</domain>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <instanceId/>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         </item>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         <item>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <publicIp>54.225.190.133</publicIp>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <domain>standard</domain>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <instanceId/>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         </item>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         <item>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <publicIp>107.22.209.166</publicIp>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <domain>standard</domain>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <instanceId>i-ff4e8d8e</instanceId>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         </item>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         <item>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <publicIp>174.129.10.84</publicIp>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <domain>standard</domain>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <instanceId>i-6761af1a</instanceId>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         </item>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         <item>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <publicIp>34.192.126.190</publicIp>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <allocationId>eipalloc-de3a31d6</allocationId>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <domain>vpc</domain>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <associationId>eipassoc-14144abf</associationId>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <networkInterfaceId>eni-b7c2a786</networkInterfaceId>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <networkInterfaceOwnerId>670359441688</networkInterfaceOwnerId>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:             <privateIpAddress>10.105.0.76</privateIpAddress>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:         </item>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4:     </addressesSet>
2018-07-25T10:44:12.572-0600 [DEBUG] plugin.terraform-provider-aws_v1.28.0_x4: </DescribeAddressesResponse>

Note the allocation ID eipalloc-de3a31d6 is returned, but it's the last element in the response XML.
Note the first element in the response XML is IP 23.21.219.184

Plan Output

-/+ module.vpc.aws_eip.nat2 (new resource required)
      id:                                        "eipalloc-de3a31d6" => <computed> (forces new resource)
      allocation_id:                             "" => <computed>
      association_id:                            "" => <computed>
      domain:                                    "standard" => <computed>
      instance:                                  "" => <computed>
      network_interface:                         "" => <computed>
      private_ip:                                "" => <computed>
      public_ip:                                 "23.21.219.184" => <computed>
      vpc:                                       "false" => "true" (forces new resource)

Note the public_ip property is showing a current value of 23.21.219.184 -- the first element in the ec2/DescribeAddresses response.

AWS CLI

aws ec2 describe-addresses --allocation-ids eipalloc-de3a31d6
{
    "Addresses": [
        {
            "InstanceId": "",
            "PublicIp": "23.21.219.184",
            "Domain": "standard"
        },
        {
            "InstanceId": "",
            "PublicIp": "54.225.190.133",
            "Domain": "standard"
        },
        {
            "InstanceId": "i-ff4e8d8e",
            "PublicIp": "107.22.209.166",
            "Domain": "standard"
        },
        {
            "InstanceId": "i-6761af1a",
            "PublicIp": "174.129.10.84",
            "Domain": "standard"
        },
        {
            "PublicIp": "34.192.126.190",
            "AllocationId": "eipalloc-de3a31d6",
            "AssociationId": "eipassoc-14144abf",
            "Domain": "vpc",
            "NetworkInterfaceId": "eni-b7c2a786",
            "NetworkInterfaceOwnerId": "670359441688",
            "PrivateIpAddress": "10.105.0.76"
        }
    ]
}

After executing the AWS CLI command several times, I was able to get one correct response:

aws ec2 describe-addresses --allocation-ids eipalloc-de3a31d6
{
    "Addresses": [
        {
            "PublicIp": "34.192.126.190",
            "AllocationId": "eipalloc-de3a31d6",
            "AssociationId": "eipassoc-14144abf",
            "Domain": "vpc",
            "NetworkInterfaceId": "eni-b7c2a786",
            "NetworkInterfaceOwnerId": "670359441688",
            "PrivateIpAddress": "10.105.0.76"
        }
    ]
}

This seems to indicate there is a rollout in progress.

adamrbennett picture adamrbennett on 25 Jul 2018
👍1

I got a response back from AWS support

The internal team has rolled a fix for the issue that you were facing.
Could you please confirm from your end if you are still facing the issue?

It looks fixed on my end, but we might still want to make this code more robust in case something like this happens again in the future.

dafyddcrosby picture dafyddcrosby on 25 Jul 2018

I got a response back from AWS support

The internal team has rolled a fix for the issue that you were facing.
Could you please confirm from your end if you are still facing the issue?

That's great news! Thanks for the update.

I'm testing on my end now.

but we might still want to make this code more robust in case something like this happens again in the future.

Agreed.

zacblazic picture zacblazic on 25 Jul 2018

Since it appears this was resolved upstream in the API, going to close this issue out. We'll still followup with the code adjustments of #5331 though 👍

bflad picture bflad on 26 Jul 2018

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!

hashibot[bot] picture hashibot[bot] on 4 Apr 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

andywirv picture andywirv  ·  3Comments
gothrek22 picture gothrek22  ·  3Comments
reedloden picture reedloden  ·  3Comments
dvishniakov picture dvishniakov  ·  3Comments
blaltarriba picture blaltarriba  ·  3Comments