Terraform-provider-aws: aws_vpn_connection is always destroyed and recreated when tunnelx_preshared_key is set

[hashibot[bot]]

_This issue was originally opened by @wenwolf as hashicorp/terraform#17336. It was migrated here as a result of the provider split. The original body of the issue is below._

---

Terraform Version

Terraform v0.11.3
+ provider.aws v1.9.0

Terraform Configuration Files

resource "aws_vpn_connection" "main" {
  vpn_gateway_id      = "<myvpcid>"
  customer_gateway_id = <mycgwid>"
  type                = "ipsec.1"
  tunnel1_inside_cidr = "<X.X.X.X/30>"
  tunnel2_inside_cidr = "<X.X.X.X/30>"
  tunnel1_preshared_key = "<myPreSharedKey1>"
  tunnel2_preshared_key = "<myPreSharedKey2>"
}

Debug Output

Terraform will perform the following actions:

-/+ aws_vpn_connection.main (new resource required)
     ...
     tunnel1_preshared_key:          <sensitive> => <sensitive> (forces new resource)
     ...
     tunnel2_preshared_key:          <sensitive> => <sensitive> (forces new resource)
     ...

Expected Behavior

When there is no change to the resource the VPN should be left untouched.

Actual Behavior

Without any change to the resource the aws_vpn_connection is destroyed and a new one is created.
The tunnel1_preshared_key and tunnel2_preshared_key attributes state that they "forces a new resource"

Steps to Reproduce

Create an AWS VPN connection with the aws_vpn_connection resource with the tunnel1_preshared_key and tunnel2_preshared_key attributes set to use your own keys. Try to apply twice.

Comment

The value of the keys is stored in clear in the tfstate so it's available for terraform to check before forcing the creation of a new resource no?

Thank you for your support.
Kind regards

[bflad]

@wenwolf are the tunnel#_preshared_key flipped with each other (tunnel1_preshared_key wants to be in tunnel2_preshared_key and vice versa)? How about the tunnel1_inside_cidr and tunnel2_inside_cidr?

We have an interesting problem as the EC2 API doesn't necessarily return the tunnel configurations in the same order, but we currently sort by ipsec_tunnel>vpn_gateway>tunnel_outside_address>ip_address to determine which is "tunnel1" and "tunnel2" and return them consistently in the Terraform configuration. If you can confirm the above is true where the two values are flipped, we should be able to pin the correct tunnels to match the given Terraform configuration rather than resorting to the sorting logic.

[wenwolf]

@bflad thanks for the initial answer.
There is indeed an issue with the tunnel1_inside_cidr as the tunnel1_cgw_inside_address that is generated doesn't match to the tunnel1_inside_cidr but to the one of the tunnel2_inside_cidr and vice versa.

According to my terraform show and the aws cli I can confirm that the ipsec_tunnel>vpn_gateway>tunnel_outside_address>ip_address x-path to determine the tunnel from the output of the AWS cli should be fine.
TF tunnel1_address and TF tunnel1_preshared_key are aligned between TF and AWS. What is wrong is the tunnel1_inside_cidr in TF that actually matches the tunnel2.
As you can see below, the tunnel1_inside_cidr doesn't match the tunnel1_cgw_inside_address (xxx.0.10 is out of the xxx.0.4/30 range):

tunnel1_cgw_inside_address = 169.254.0.10
tunnel1_inside_cidr = 169.254.0.4/30

 tunnel2_cgw_inside_address = 169.254.0.6
 tunnel2_inside_cidr = 169.254.0.8/30

So what is strange is that the preshared keys, the tunnel address, cgw inside address etc... are all good, it's really the inside_cidr that look swapped and if the sorting was done as stated on the ip_address we shouldn't have that issue.

One temporary workaround for me is to run terraform apply initially with the presharedkeys in and then comment these out so that it doesn't identify any change and thus not destroying my tunnel.

Thanks for your help.

[bflad]

Our daily acceptance testing finally ran into this issue itself. Here is the test error and full Terraform state of the resource:

=== RUN   TestAccAWSVpnConnection_tunnelOptions
--- FAIL: TestAccAWSVpnConnection_tunnelOptions (230.09s)
    testing.go:513: Step 14 error: Check failed: Check 4/6 error: aws_vpn_connection.foo: Attribute 'tunnel1_preshared_key' expected "12345678", got "abcdefgh"


aws_vpn_connection.foo:
  ID = vpn-57826642
  provider = provider.aws
  customer_gateway_configuration = <?xml version="1.0" encoding="UTF-8"?>
<vpn_connection id="vpn-57826642">
  <customer_gateway_id>cgw-9fd77081</customer_gateway_id>
  <vpn_gateway_id>vgw-fd69cde3</vpn_gateway_id>
  <vpn_connection_type>ipsec.1</vpn_connection_type>
  <ipsec_tunnel>
    <customer_gateway>
      <tunnel_outside_address>
        <ip_address>178.0.0.1</ip_address>
      </tunnel_outside_address>
      <tunnel_inside_address>
        <ip_address>169.254.9.2</ip_address>
        <network_mask>255.255.255.252</network_mask>
        <network_cidr>30</network_cidr>
      </tunnel_inside_address>
      <bgp>
        <asn>928</asn>
        <hold_time>30</hold_time>
      </bgp>
    </customer_gateway>
    <vpn_gateway>
      <tunnel_outside_address>
        <ip_address>34.215.162.49</ip_address>
      </tunnel_outside_address>
      <tunnel_inside_address>
        <ip_address>169.254.9.1</ip_address>
        <network_mask>255.255.255.252</network_mask>
        <network_cidr>30</network_cidr>
      </tunnel_inside_address>
      <bgp>
        <asn>7224</asn>
        <hold_time>30</hold_time>
      </bgp>
    </vpn_gateway>
    <ike>
      <authentication_protocol>sha1</authentication_protocol>
      <encryption_protocol>aes-128-cbc</encryption_protocol>
      <lifetime>28800</lifetime>
      <perfect_forward_secrecy>group2</perfect_forward_secrecy>
      <mode>main</mode>
      <pre_shared_key>abcdefgh</pre_shared_key>
    </ike>
    <ipsec>
      <protocol>esp</protocol>
      <authentication_protocol>hmac-sha1-96</authentication_protocol>
      <encryption_protocol>aes-128-cbc</encryption_protocol>
      <lifetime>3600</lifetime>
      <perfect_forward_secrecy>group2</perfect_forward_secrecy>
      <mode>tunnel</mode>
      <clear_df_bit>true</clear_df_bit>
      <fragmentation_before_encryption>true</fragmentation_before_encryption>
      <tcp_mss_adjustment>1379</tcp_mss_adjustment>
      <dead_peer_detection>
        <interval>10</interval>
        <retries>3</retries>
      </dead_peer_detection>
    </ipsec>
  </ipsec_tunnel>
  <ipsec_tunnel>
    <customer_gateway>
      <tunnel_outside_address>
        <ip_address>178.0.0.1</ip_address>
      </tunnel_outside_address>
      <tunnel_inside_address>
        <ip_address>169.254.8.2</ip_address>
        <network_mask>255.255.255.252</network_mask>
        <network_cidr>30</network_cidr>
      </tunnel_inside_address>
      <bgp>
        <asn>928</asn>
        <hold_time>30</hold_time>
      </bgp>
    </customer_gateway>
    <vpn_gateway>
      <tunnel_outside_address>
        <ip_address>52.26.164.247</ip_address>
      </tunnel_outside_address>
      <tunnel_inside_address>
        <ip_address>169.254.8.1</ip_address>
        <network_mask>255.255.255.252</network_mask>
        <network_cidr>30</network_cidr>
      </tunnel_inside_address>
      <bgp>
        <asn>7224</asn>
        <hold_time>30</hold_time>
      </bgp>
    </vpn_gateway>
    <ike>
      <authentication_protocol>sha1</authentication_protocol>
      <encryption_protocol>aes-128-cbc</encryption_protocol>
      <lifetime>28800</lifetime>
      <perfect_forward_secrecy>group2</perfect_forward_secrecy>
      <mode>main</mode>
      <pre_shared_key>12345678</pre_shared_key>
    </ike>
    <ipsec>
      <protocol>esp</protocol>
      <authentication_protocol>hmac-sha1-96</authentication_protocol>
      <encryption_protocol>aes-128-cbc</encryption_protocol>
      <lifetime>3600</lifetime>
      <perfect_forward_secrecy>group2</perfect_forward_secrecy>
      <mode>tunnel</mode>
      <clear_df_bit>true</clear_df_bit>
      <fragmentation_before_encryption>true</fragmentation_before_encryption>
      <tcp_mss_adjustment>1379</tcp_mss_adjustment>
      <dead_peer_detection>
        <interval>10</interval>
        <retries>3</retries>
      </dead_peer_detection>
    </ipsec>
  </ipsec_tunnel>
</vpn_connection>
  customer_gateway_id = cgw-9fd77081
  routes.# = 0
  static_routes_only = false
  tags.% = 0
  tunnel1_address = 34.215.162.49
  tunnel1_bgp_asn = 7224
  tunnel1_bgp_holdtime = 30
  tunnel1_cgw_inside_address = 169.254.9.2
  tunnel1_inside_cidr = 169.254.8.0/30
  tunnel1_preshared_key = abcdefgh
  tunnel1_vgw_inside_address = 169.254.9.1
  tunnel2_address = 52.26.164.247
  tunnel2_bgp_asn = 7224
  tunnel2_bgp_holdtime = 30
  tunnel2_cgw_inside_address = 169.254.8.2
  tunnel2_inside_cidr = 169.254.9.0/30
  tunnel2_preshared_key = 12345678
  tunnel2_vgw_inside_address = 169.254.8.1
  type = ipsec.1
  vgw_telemetry.# = 2
  vgw_telemetry.1621335275.accepted_route_count = 0
  vgw_telemetry.1621335275.last_status_change = 2018-02-23 08:53:41 +0000 UTC
  vgw_telemetry.1621335275.outside_ip_address = 34.215.162.49
  vgw_telemetry.1621335275.status = DOWN
  vgw_telemetry.1621335275.status_message = IPSEC IS DOWN
  vgw_telemetry.4167421053.accepted_route_count = 0
  vgw_telemetry.4167421053.last_status_change = 2018-02-23 08:53:09 +0000 UTC
  vgw_telemetry.4167421053.outside_ip_address = 52.26.164.247
  vgw_telemetry.4167421053.status = DOWN
  vgw_telemetry.4167421053.status_message = 
  vpn_gateway_id = vgw-fd69cde3

[danielnitz]

for me, this issue seems to be fixed with the following versions:

Terraform v0.11.7 and provider.aws v1.16.0

[zkupu]

I am still encountering this bug.

Attempts to deploy against an environment where there are existing AWS VPN connections with defined tunnel_preshared_keys that were originally deployed with the same terraform scripts results in those resources being forcefully recreated.

I have tried using the following Docker Container versions:

Terraform latest
Terraform v0.11.7

And the following provider.aws versions:

provider.aws v1.16.0
provider.aws v1.19.0

Here is a sample output of a terraform plan where no changes have been made to the deploy or the existing AWS resource:

customer_gateway_id: "xxx-xxxxxxxx" => "${element(aws_customer_gateway.cgw1.*.id,count.index)}" (forces new resource) routes.#: "2" => <computed> static_routes_only: "true" => "true" tags.%: "1" => "1" tags.Name: "xxxx" => "xxxxxx" tunnel1_address: "256.256.256.256" => <computed> tunnel1_bgp_asn: "" => <computed> tunnel1_bgp_holdtime: "0" => <computed> tunnel1_cgw_inside_address: "322.322.322.322" => <computed> tunnel1_inside_cidr: "" => <computed> tunnel1_preshared_key: <sensitive> => <sensitive> (attribute changed) tunnel1_vgw_inside_address: "322.322.322.322" => <computed> tunnel2_address: "322.322.322.322" => <computed> tunnel2_bgp_asn: "" => <computed> tunnel2_bgp_holdtime: "0" => <computed> tunnel2_cgw_inside_address: "275.275.275.275" => <computed> tunnel2_inside_cidr: "" => <computed> tunnel2_preshared_key: <sensitive> => <sensitive> (attribute changed) tunnel2_vgw_inside_address: "555.555.555.555" => <computed> type: "ipsec.1" => "ipsec.1" vgw_telemetry.#: "2" => <computed> vpn_gateway_id: "xxx-xxxxxxxx" => "xxx-xxxxxxxx"

[danielnitz]

@zkupu this worked for me:

resource "aws_vpn_connection" "con1" {
  vpn_gateway_id        = "${aws_vpn_gateway.main.id}"
  customer_gateway_id   = "${aws_customer_gateway.con1.id}"
  type                  = "ipsec.1"
  static_routes_only    = false
  tunnel1_preshared_key = "${var.con1_key1}"
  tunnel2_preshared_key = "${var.con1_key2}"
  tunnel1_inside_cidr   = "169.254.40.16/30"
  tunnel2_inside_cidr   = "169.254.41.16/30"
}

With Terraform v0.11.7 and provider.aws v1.16.0

[zkupu]

I'm creating 5 VPN connections at once, so my code is a bit different, but basically the same. I guess it is possible that adding count causes a problem for the preshared key.

When deploying with my existing config, without the tunnel_preshared keys defined, terraform completes the changes without destroying and re-creating those existing resources.

This still fails when using Terraform v0.11.7 and provider.aws v1.16.0.

Here's a copy of my current config:

resource "aws_vpn_connection" "conn1" {
  count      = "${var.create_vpn ? length(split(",", var.customer_gateway_ips)) : 0}"
  customer_gateway_id = "${element(aws_customer_gateway.cgw1.*.id, count.index)}"
  vpn_gateway_id      = "${aws_vpn_gateway.vpn_gw.id}"
  type                = "ipsec.1"
  tunnel1_preshared_key = "${var.tunnel1_preshared_key}"
  tunnel2_preshared_key = "${var.tunnel2_preshared_key}"
  static_routes_only  = true
}

[ppieprzycki]

Hi.
I got similar problem with swapping data between tunnelX_inside_cidr
Terraform v0.11.8 / provider.aws v1.35.0

described in issue
https://github.com/terraform-providers/terraform-provider-aws/issues/5809

Any idea how to resolve this? Or maybe anyone created issue inside amazon regarding this behavior?

[tlindsay42]

Issue occurs every time for me with terraform v0.11.8 / terraform-provider-aws v1.41.0 with or without specifying inside CIDRs and static routes only set to either enabled or disabled.

[grealish]

This creates a major blocker for us, as our internal IT (firewall team) demands that we defined the tunnel pre-shared key as they wish to rotate it, for now we have an exception to get the AWS generated one, but this won't last long, is there we can even had an "ignore resource" passed to the module as ignoring changes doesn't work outside of resources.

current versions:
Terraform v0.11.10

• provider.aws v1.42.0

[patrykk2252]

The same here:

Terraform v0.11.11

• provider.aws v1.31.0

I had to comment out following:
'#' Once tunnels are created you MUST comment out vpn-gateway/main.tf
'#' resource "aws_vpn_connection" "tunnel_preshared" -
'#' Issue 3359 - https://github.com/terraform-providers/terraform-provider-aws/issues/3359
'#'tunnel1_preshared_key = "${var.tunnel1_preshared_key}"
'#'tunnel2_preshared_key = "${var.tunnel2_preshared_key}"

Then it works fine. Thanks for the tip. Waiting for permanent solution.

[jmvizcainoio]

I have the same problem using:

• Terraform v0.11.13
• provider.aws: version = "~> 2.4"

I fix the problem adding
````
tunnel1_inside_cidr = "169.254.40.16/30"
tunnel2_inside_cidr = "169.254.41.16/30"
`````

and now works fine. Thanks by the solution

[craigday]

It's definitely a problem with the values for tunnel1_preshared_key and tunnel2_preshared_key getting flipped. The evidence for this is that my build outputs VPN creation commands for our SRX firewall, so I have the keys in the output. When I compare those keys and VPN script to the scripts created by the AWS console, the PSKs are reversed.

Hence every time you do a terraform apply, it flips the PSKs again and again.

Simple workaround is to make the PSK the same for each tunnel :)

[stephan2012]

Same issue for Terraform v0.12.

[andreaswittig]

Still not fixed. Terraform 0.12 and AWS provider 2.53.0.

[snovikov]

Same issue.

Terraform v0.12.24
+ provider.aws v2.64.0

Solution was:

1. Deploy VPN tunnel
2. Swap PSK values for tunnel1 & tunnel2

[michz]

Same problem here, "sometimes".

One observation:

Had this "sometimes" and "sometimes not". After 5 tries it looked like the Tunnels are sorted by the "Outside IP Address" (which can obviously not be changed and are assigned by AWS automatically). In my (not representative) tests it looked like that always the Tunnel with the "lower" Outside IP Address (if sorting alphanumerically as strings) was Tunnel 1, regardless of my specification in terraform code.