Terraform-provider-aws: aws_vpn_connection - inside_tunnel_cidr swapped between tunnel interfaces

[ppieprzycki]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.8

• provider.aws v1.35.0

Affected Resource(s)

"aws_vpn_connection"

Terraform Configuration Files

We create resource by using module

module "vpn_gateway_main" {
  source = "../terraform-aws-vpn-gateway.git"
  create_vpn_connection = "${ var.create_vpn  ? 1 : 0}"   

  vpn_gateway_id      = "${module.vpc.vgw_id}"
  customer_gateway_id = "${local.vpn_main_cgw_id}"

  tunnel1_inside_cidr   = "${local.vpn1_tunnel1_cidr}"
  tunnel2_inside_cidr   = "${local.vpn1_tunnel2_cidr}"
  vpc_id                       = "${module.vpc.vpc_id}"
  create_vpn_gateway_attachment = false 
}

from module resource is created
https://github.com/terraform-aws-modules/terraform-aws-vpn-gateway/blob/master/main.tf

resource "aws_vpn_connection" "tunnel" {
  count = "${var.create_vpn_connection && length(var.tunnel1_inside_cidr) > 0 && length(var.tunnel2_inside_cidr) > 0 && length(var.tunnel1_preshared_key) == 0 && length(var.tunnel2_preshared_key) == 0 ? 1 : 0}"

  vpn_gateway_id      = "${var.vpn_gateway_id}"
  customer_gateway_id = "${var.customer_gateway_id}"
  type                = "ipsec.1"

  static_routes_only = "${var.vpn_connection_static_routes_only}"

  tunnel1_inside_cidr = "${var.tunnel1_inside_cidr}"
  tunnel2_inside_cidr = "${var.tunnel2_inside_cidr}"

  tags = "${merge(
    map(
      "Name", "VPN Connection between VPC ${var.vpc_id} and Customer Gateway ${var.customer_gateway_id}"
    ),
    var.tags
  )}"
}

Debug Output

#terraform apply

module.account_vpc1.module.vpn_gateway_main.aws_vpn_connection.tunnel: Creating...
  customer_gateway_configuration: "" => "<computed>"
  customer_gateway_id:            "" => "cgw-07eb19a67ffa83222"
  routes.#:                       "" => "<computed>"
  static_routes_only:             "" => "false"
  tags.%:                         "" => "1"
  tags.Name:                      "" => "VPN Connection between VPC vpc-070021af585778888 and Customer Gateway cgw-07eb19a67ffa83222"
  tunnel1_address:                "" => "<computed>"
  tunnel1_bgp_asn:                "" => "<computed>"
  tunnel1_bgp_holdtime:           "" => "<computed>"
  tunnel1_cgw_inside_address:     "" => "<computed>"
  tunnel1_inside_cidr:            "" => "169.254.10.0/30"
  tunnel1_preshared_key:          "<sensitive>" => "<sensitive>"
  tunnel1_vgw_inside_address:     "" => "<computed>"
  tunnel2_address:                "" => "<computed>"
  tunnel2_bgp_asn:                "" => "<computed>"
  tunnel2_bgp_holdtime:           "" => "<computed>"
  tunnel2_cgw_inside_address:     "" => "<computed>"
  tunnel2_inside_cidr:            "" => "169.254.10.4/30"
  tunnel2_preshared_key:          "<sensitive>" => "<sensitive>"
  tunnel2_vgw_inside_address:     "" => "<computed>"
  type:                           "" => "ipsec.1"
  vgw_telemetry.#:                "" => "<computed>"
  vpn_gateway_id:                 "" => "vgw-024f5f47d937a4adb"

module.account_vpc1.module.vpn_gateway_secondary.aws_vpn_connection.tunnel: Creating...
  customer_gateway_configuration: "" => "<computed>"
  customer_gateway_id:            "" => "cgw-0f5f408a10dda83111"
  routes.#:                       "" => "<computed>"
  static_routes_only:             "" => "false"
  tags.%:                         "" => "1"
  tags.Name:                      "" => "VPN Connection between VPC vpc-070021af585778888 and Customer Gateway cgw-0f5f408a10dda83111"
  tunnel1_address:                "" => "<computed>"
  tunnel1_bgp_asn:                "" => "<computed>"
  tunnel1_bgp_holdtime:           "" => "<computed>"
  tunnel1_cgw_inside_address:     "" => "<computed>"
  tunnel1_inside_cidr:            "" => "169.254.10.8/30"
  tunnel1_preshared_key:          "<sensitive>" => "<sensitive>"
  tunnel1_vgw_inside_address:     "" => "<computed>"
  tunnel2_address:                "" => "<computed>"
  tunnel2_bgp_asn:                "" => "<computed>"
  tunnel2_bgp_holdtime:           "" => "<computed>"
  tunnel2_cgw_inside_address:     "" => "<computed>"
  tunnel2_inside_cidr:            "" => "169.254.10.12/30"
  tunnel2_preshared_key:          "<sensitive>" => "<sensitive>"
  tunnel2_vgw_inside_address:     "" => "<computed>"
  type:                           "" => "ipsec.1"
  vgw_telemetry.#:                "" => "<computed>"
  vpn_gateway_id:                 "" => "vgw-024f5f47d937a4adb"

Panic Output

#terraform state show module.account_vpc1.module.vpn_gateway_main.aws_vpn_connection.tunnel

VPN connection 1 
customer_gateway_id                           = cgw-07eb19a67ffa83222
routes.#                                      = 0
static_routes_only                            = false
tags.%                                        = 1
tags.Name                                     = VPN Connection between VPC vpc-070021af58578888 and Customer Gateway cgw-07eb19a67ffa83222
tunnel1_address                               = 18.184.98.78
tunnel1_bgp_asn                               = 64512
tunnel1_bgp_holdtime                          = 30
tunnel1_cgw_inside_address                    = 169.254.10.6
tunnel1_inside_cidr                           = 169.254.10.0/30
tunnel1_preshared_key                         = password
tunnel1_vgw_inside_address                    = 169.254.10.5
tunnel2_address                               = 35.158.222.155
tunnel2_bgp_asn                               = 64512
tunnel2_bgp_holdtime                          = 30
tunnel2_cgw_inside_address                    = 169.254.10.2
tunnel2_inside_cidr                           = 169.254.10.4/30
tunnel2_preshared_key                         = password
tunnel2_vgw_inside_address                    = 169.254.10.1

#terraform state show module.account_vpc1.module.vpn_gateway_secondary.aws_vpn_connection.tunnel

customer_gateway_id                           = cgw-0f5f408a10dda8111
routes.#                                      = 0
static_routes_only                            = false
tags.%                                        = 1
tags.Name                                     = VPN Connection between VPC vpc-070021af585778888 and Customer Gateway cgw-0f5f408a10dda8111
tunnel1_address                               = 18.196.49.49
tunnel1_bgp_asn                               = 64512
tunnel1_bgp_holdtime                          = 30
tunnel1_cgw_inside_address                    = 169.254.10.10
tunnel1_inside_cidr                           = 169.254.10.8/30
tunnel1_preshared_key                         = password
tunnel1_vgw_inside_address                    = 169.254.10.9
tunnel2_address                               = 35.158.82.179
tunnel2_bgp_asn                               = 64512
tunnel2_bgp_holdtime                          = 30
tunnel2_cgw_inside_address                    = 169.254.10.14
tunnel2_inside_cidr                           = 169.254.10.12/30
tunnel2_preshared_key                         = password
tunnel2_vgw_inside_address                    = 169.254.10.13
type                                          = ipsec.1

Expected Behavior

We created two VPN connections called in output "main" and "secondary" to separate Customer gateways. Both IPSEC Connection have inside two tunnels.
To prevent conflicts we manually assign CIDR inside each tunnel.

First ipsec connection has
Connection "main":
tunnel1_cidr: 169.254.10.0/30
tunnel2_cidr: 169.254.10.4/30

Connection "secondary":
tunnel1_cidr: 169.254.8.0/30
tunnel2_cidr: 169.254.12.4/30

Actual Behavior

From time to time I can see that IP addresses are swapped between tunnel interfaces within a single connection.
It doesn't occur each time but quite often

As we can see in terraform state show command we have the problem in first connection.
We assigned CIDR 169.254.10.0/30 and is visible while we are creating resource but we at the end obtain addresses for second interfaces

tunnel1_cgw_inside_address = 169.254.10.6
tunnel1_inside_cidr = 169.254.10.0/30
tunnel1_vgw_inside_address = 169.254.10.5

tunnel2_cgw_inside_address = 169.254.10.2
tunnel2_inside_cidr = 169.254.10.4/30
tunnel2_vgw_inside_address = 169.254.10.1

Steps to Reproduce

1. terraform apply

Important Factoids

References

• #0000

[jmanteau]

@a-teisseire

[selslack]

Can this be looked at by anybody?

[selslack]

The root cause, from my understanding, is that cgw_inside_address and vgw_inside_address are being sourced from customer_gateway_configuration XML while inside_cidr is sourced from the input.

Please correct me if I'm wrong.