Terraform: aws_customer_gateway updates existing manually created cgw when it has same IP address

[kiich]

Runing terraform on centos 6.6 with Terraform v0.6.15

And using aws_customer_gateway resource like this:

resource "aws_customer_gateway" "cgw" {
   bgp_asn = 65000
   ip_address = "192.1.1.1"
   type = "ipsec.1"
   tags {
      Name                = "terraform test"
      Terraform           = "true"
   }
}

The above all works fine when i do not have any other customer gateway in my AWS. However, if i create a cgw manually from AWS console with the SAME IP first and run the above, it updates the manually created one instead where as i was expecting it to fail like it does when i try to do the same (create another cgw with already existing IP).

This isn't such a big deal but it did confuse me when i couldn't do terraform destroy because it thought the cgw was in use (which it was by another manually created VPN i have) when i expected the destroy to succeed as i assumed the terraform run created its own version of cgw.

Probably not a bug but would be good to get it clarified and some how mentioned in doc.

Expected Behavior

terraform plan to some how show the cgw resource can not be created because it already sees another cgw with the same IP which was created OUTSIDE of terraform with same IP.

Actual Behavior

terraform UPDATES existing customer gateway resource which was not created by terraform.

terraform plan:
[...]

• module.vpn.aws_customer_gateway.cgw
  bgp_asn: "" => "65000"
  ip_address: "" => "192.1.1.1"
  tags.#: "" => "4"
  tags.Name: "" => "terraform test"
  tags.Terraform: "" => "true"
  type: "" => "ipsec.1"

terraform apply:
module.vpn.aws_customer_gateway.cgw: Creating...
bgp_asn: "" => "65000"
ip_address: "" => "192.1.1.1"
tags.#: "" => "4"
tags.Name: "" => "terraform test"
tags.Terraform: "" => "true"
type: "" => "ipsec.1"

module.vpn.aws_customer_gateway.cgw: Creation complete

but destory fails:
Error applying plan:

1 error(s) occurred:

• aws_customer_gateway.cgw: IncorrectState: The customer gateway is in use.
  status code: 400, request id: 551be6a7-cf76-45aa-b4f4-8958416ef465

somehow even more dangerous, if the cgw in question is not in use, terraform actually destroys it even though it was not initially created by terraform.

[stack72]

Hi @kiich

I agree with you that Terraform isn't behaving correct in this scenario - I will try and address that now. when i create a gateway using your config and then add another gateway manually (via the console), I get the following:

I am going to try and make Terraform respect the same behaviour

Paul

[stack72]

Hi @kiich

Ok, on further investigation here, this isn't actually a Terraform bug. I have just followed the following steps:

1. Create Customer Gateway with the following config:

resource "aws_customer_gateway" "cgw" {
   bgp_asn = 65000
   ip_address = "192.1.1.1"
   type = "ipsec.1"
   tags {
      Name                = "terraform test"
      Terraform           = "true"
   }
}

I can see that this has been created as per the console:

On trying to use the console to add another gateway with the same IP and BGP, I get the error above. But when I use the AWS cli, I get the following:

% aws ec2 create-customer-gateway --type ipsec.1 --bgp-asn 65000 --public-ip 192.1.1.1                                                                                                                  2 ↵
{
    "CustomerGateway": {
        "CustomerGatewayId": "cgw-cd528bd3",
        "IpAddress": "192.1.1.1",
        "State": "available",
        "Type": "ipsec.1",
        "BgpAsn": "65000"
    }
}

As you can see, that customer gateway id returned from the CREATE on the CLI is the _same_ as that in the Terraform created configuration (As per the screenshot!)

I believe the SDK is behaving in the same way as the cli and thus why Terraform is taking control of the gateway. I am going to raise this with the SDK team now

Thanks

Paul

[stack72]

FYI, I raised this issue https://github.com/aws/aws-sdk-go/issues/855

[kiich]

Hey @stack72 - cheers for looking into this and raising the bug against the sdk! Will keep an eye on that as well.

[johananl]

Hi @stack72, @kiich,

Looking at the answer from AWS on aws/aws-sdk-go#855, adding a CGW using the CLI or the AWS API is idempotent. I've also verified this myself.

So, this indeed creates a problem since Terraform might "take over" a resource it didn't create and potentially overwrite its tags and / or destroy it by accident. This is exactly what happened to me, which brought me here :-)

I'm not sure now what the expected Terraform behavior should be. Perhaps the existing CGW should not be added to the state and simply "ignored" somehow when it already exists.
This is quite a unique case on AWS since normally you would get an error in these situations.

What do you think?

[kashook]

I recently ran into this using Terraform 0.8.6. In my case, the Customer Gateway had been created with another Terraform. I didn't realize it until I tried to destroy the second Terraform, and it (fortunately) failed when trying to delete the gateway because it was actually in use by a VPN connection. The second Terraform modified the Name tag of the original resource in AWS (which was quite confusing at first).

The AWS documentation does clearly state that calling create-customer-gateway with the same VPN type, IP address, and BGP ASN parameter values will return the existing gateway. However, consider this Terraform example. (The IP is of course not actually a VPN device, but that doesn't matter for this example).

provider "aws" {
  region = "us-east-1"
}

resource "aws_customer_gateway" "my_customer_gateway" {
  bgp_asn    = "65530"
  ip_address = "8.8.8.8"
  type       = "ipsec.1"

  tags {
    Name = "test"
  }
}

resource "aws_customer_gateway" "my_customer_gateway2" {
  bgp_asn    = "65530"
  ip_address = "8.8.8.8"
  type       = "ipsec.1"

  tags {
    Name = "test2"
  }
}

When I run that, only one gateway is actually created. However, Terraform _acts_ like it created two separate resources. Whichever one had the create call issued last "wins". (The Name tag on the one actual resource is modified). The tfstate file has two different gateways in it, each with the Name as specified in the tf file, but they happen to have the same id.

E:\hg\ca\terraform-testcg>terraform apply
aws_customer_gateway.my_customer_gateway2: Creating...
  bgp_asn:    "" => "65530"
  ip_address: "" => "8.8.8.8"
  tags.%:     "" => "1"
  tags.Name:  "" => "test2"
  type:       "" => "ipsec.1"
aws_customer_gateway.my_customer_gateway: Creating...
  bgp_asn:    "" => "65530"
  ip_address: "" => "8.8.8.8"
  tags.%:     "" => "1"
  tags.Name:  "" => "test"
  type:       "" => "ipsec.1"
aws_customer_gateway.my_customer_gateway2: Still creating... (10s elapsed)
aws_customer_gateway.my_customer_gateway: Still creating... (10s elapsed)
aws_customer_gateway.my_customer_gateway2: Creation complete
aws_customer_gateway.my_customer_gateway: Creation complete

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Now, when I run a plan, Terraform shows that it is going to modify the resource that "lost" because the name tag doesn't match. I am running the exact same Terraform twice in a row with _no changes_, yet it wants to modify something (and always will).

E:\hg\ca\terraform-testcg>terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but
will not be persisted to local or remote state storage.

aws_customer_gateway.my_customer_gateway: Refreshing state... (ID: cgw-9a6188f3)
aws_customer_gateway.my_customer_gateway2: Refreshing state... (ID: cgw-9a6188f3)

The Terraform execution plan has been generated and is shown below.
Resources are shown in alphabetical order for quick scanning. Green resources
will be created (or destroyed and then created if an existing resource
exists), yellow resources are being changed in-place, and red resources
will be destroyed. Cyan entries are data sources to be read.

Note: You didn't specify an "-out" parameter to save this plan, so when
"apply" is called, Terraform can't guarantee this is what will execute.

~ aws_customer_gateway.my_customer_gateway2
    tags.Name: "test" => "test2"


Plan: 0 to add, 1 to change, 0 to destroy.

When I destroy, Terraform once again acts like there are two resources:

E:\hg\ca\terraform-testcg>terraform destroy -force
aws_customer_gateway.my_customer_gateway2: Refreshing state... (ID: cgw-9a6188f3)
aws_customer_gateway.my_customer_gateway: Refreshing state... (ID: cgw-9a6188f3)
aws_customer_gateway.my_customer_gateway: Destroying...
aws_customer_gateway.my_customer_gateway2: Destroying...
aws_customer_gateway.my_customer_gateway: Destruction complete
aws_customer_gateway.my_customer_gateway2: Destruction complete

Destroy complete! Resources: 2 destroyed.

The fact that Terraform is giving the impression that there are two resources when there really aren't seems weird to me.

This example is a bit contrived, but I think it shows odd behavior. If this behavior was addressed, it would also happen to fix the real life scenario myself and the OP described where some customer gateway already exists and was created externally to the terraform.

If Terraform _is_ actually supposed to behave this way, then at the very least I think the docs should have a warning similar to the warning in the AWS documentation IMHO.

[kiich]

@johananl Ignoring existing resource TF didn't create sounds like an option and somehow catch that the call to create CGW returned with already existing resource. I guess when I initially created this issue, i was more concerned about the deletion of cgw that TF didn't create.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.