Terraform-provider-aws: Apply forces new aws_ec2_client_vpn_network_association

[orkenstein]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.11

• provider.aws v1.59.0
• provider.tls v1.2.0

Affected Resource(s)

• aws_vpc

Terraform Configuration Files

# Specify the provider and access details

# We use vault to get credentials, but you can use variables to achieve the same thing
# data "vault_generic_secret" "aws_creds" {
#   path = "aws/sts/manage-${var.aws_account_id}"
# }

provider "aws" {
  # access_key = "${data.vault_generic_secret.aws_creds.data["access_key"]}"
  # secret_key = "${data.vault_generic_secret.aws_creds.data["secret_key"]}"
  # token      = "${data.vault_generic_secret.aws_creds.data["security_token"]}"
  region     = "${var.aws_region}"
}

### Network

# Fetch AZs in the current region
data "aws_availability_zones" "available" {}

resource "aws_vpc" "main" {
  cidr_block = "172.17.0.0/16"

  tags = {
    Name = "tf-ecs-vpc"
  }
}

# Create var.az_count private subnets, each in a different AZ
resource "aws_subnet" "private" {
  count             = "${var.az_count}"
  cidr_block        = "${cidrsubnet(aws_vpc.main.cidr_block, 8, count.index)}"
  availability_zone = "${data.aws_availability_zones.available.names[count.index]}"
  vpc_id            = "${aws_vpc.main.id}"
}

# Create var.az_count public subnets, each in a different AZ
resource "aws_subnet" "public" {
  count                   = "${var.az_count}"
  cidr_block              = "${cidrsubnet(aws_vpc.main.cidr_block, 8, var.az_count + count.index)}"
  availability_zone       = "${data.aws_availability_zones.available.names[count.index]}"
  vpc_id                  = "${aws_vpc.main.id}"
  map_public_ip_on_launch = true
}

# IGW for the public subnet
resource "aws_internet_gateway" "gw" {
  vpc_id = "${aws_vpc.main.id}"
}

# Route the public subnet traffic through the IGW
resource "aws_route" "internet_access" {
  route_table_id         = "${aws_vpc.main.main_route_table_id}"
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = "${aws_internet_gateway.gw.id}"
}

# Create a NAT gateway with an EIP for each private subnet to get internet connectivity
resource "aws_eip" "gw" {
  count      = "${var.az_count}"
  vpc        = true
  depends_on = ["aws_internet_gateway.gw"]
}

resource "aws_nat_gateway" "gw" {
  count         = "${var.az_count}"
  subnet_id     = "${element(aws_subnet.public.*.id, count.index)}"
  allocation_id = "${element(aws_eip.gw.*.id, count.index)}"
}

# Create a new route table for the private subnets
# And make it route non-local traffic through the NAT gateway to the internet
resource "aws_route_table" "private" {
  count  = "${var.az_count}"
  vpc_id = "${aws_vpc.main.id}"

  route {
    cidr_block = "0.0.0.0/0"
    nat_gateway_id = "${element(aws_nat_gateway.gw.*.id, count.index)}"
  }
}

# Explicitely associate the newly created route tables to the private subnets (so they don't default to the main route table)
resource "aws_route_table_association" "private" {
  count          = "${var.az_count}"
  subnet_id      = "${element(aws_subnet.private.*.id, count.index)}"
  route_table_id = "${element(aws_route_table.private.*.id, count.index)}"
}

### Security

# ALB Security group
# This is the group you need to edit if you want to restrict access to your application
resource "aws_security_group" "lb" {
  name        = "tf-ecs-alb"
  description = "controls access to the ALB"
  vpc_id      = "${aws_vpc.main.id}"

  ingress {
    protocol    = "tcp"
    from_port   = 80
    to_port     = 80
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port = 0
    to_port   = 0
    protocol  = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

# Traffic to the ECS Cluster should only come from the ALB
resource "aws_security_group" "ecs_tasks" {
  name        = "tf-ecs-tasks"
  description = "allow inbound access from the ALB only"
  vpc_id      = "${aws_vpc.main.id}"

  ingress {
    protocol        = "tcp"
    from_port       = "${var.app_port}"
    to_port         = "${var.app_port}"
    security_groups = ["${aws_security_group.lb.id}"]
  }

  egress {
    protocol    = "-1"
    from_port   = 0
    to_port     = 0
    cidr_blocks = ["0.0.0.0/0"]
  }
}

### ALB

resource "aws_alb" "main" {
  name            = "tf-ecs-chat"
  subnets         = ["${aws_subnet.private.*.id}"]
  security_groups = ["${aws_security_group.lb.id}"]
}

resource "aws_alb_target_group" "app" {
  name        = "tf-ecs-chat"
  port        = 80
  protocol    = "HTTP"
  vpc_id      = "${aws_vpc.main.id}"
  target_type = "ip"
}

# Redirect all traffic from the ALB to the target group
resource "aws_alb_listener" "front_end" {
  load_balancer_arn = "${aws_alb.main.id}"
  port              = "80"
  protocol          = "HTTP"

  default_action {
    target_group_arn = "${aws_alb_target_group.app.id}"
    type             = "forward"
  }
}

### ECS

resource "aws_ecs_cluster" "main" {
  name = "tf-ecs-cluster"
}

data "aws_iam_role" "ecs_task_execution_role" {
  name = "ecsTaskExecutionRole"
}

resource "aws_ecs_task_definition" "app" {
  family                   = "app"
  network_mode             = "awsvpc"
  requires_compatibilities = ["FARGATE"]
  cpu                      = "${var.fargate_cpu}"
  memory                   = "${var.fargate_memory}"
  execution_role_arn       = "${data.aws_iam_role.ecs_task_execution_role.arn}"

  container_definitions = <<DEFINITION

  [
    {
      "cpu": ${var.fargate_cpu},
      "image": "${var.app_image}",
      "memory": ${var.fargate_memory},
      "name": "app",
      "networkMode": "awsvpc",
      "portMappings": [
        {
          "containerPort": ${var.app_port},
          "hostPort": ${var.app_port}
        }
      ]
    }
  ]
DEFINITION
}

resource "aws_ecs_service" "main" {
  name            = "tf-ecs-service"
  cluster         = "${aws_ecs_cluster.main.id}"
  task_definition = "${aws_ecs_task_definition.app.arn}"
  desired_count   = "${var.app_count}"
  launch_type     = "FARGATE"

  network_configuration {
    security_groups = ["${aws_security_group.ecs_tasks.id}"]
    subnets         = ["${aws_subnet.private.*.id}"]
  }

  load_balancer {
    target_group_arn = "${aws_alb_target_group.app.id}"
    container_name   = "app"
    container_port   = "${var.app_port}"
  }

  depends_on = [
    "aws_alb_listener.front_end",
  ]
}

resource "tls_private_key" "ca_key" {
  algorithm = "RSA"
}

resource "tls_private_key" "client_key" {
  algorithm = "RSA"
}

resource "tls_private_key" "server_key" {
  algorithm = "RSA"
}

resource "tls_self_signed_cert" "ca_cert" {
  key_algorithm   = "RSA"
  private_key_pem = "${tls_private_key.ca_key.private_key_pem}"

  subject {
    common_name  = "Filmustage Cert Authority"
    organization = "Filmustage, Inc"
  }

  validity_period_hours = 12

  allowed_uses = [
    "key_encipherment",
    "digital_signature",
    "server_auth",
  ]
}

resource "tls_cert_request" "client_request" {
  key_algorithm   = "RSA"
  private_key_pem = "${tls_private_key.client_key.private_key_pem}"

  subject {
        common_name  = "filmustage.vpn.client"
        organization = "Filmustage, Inc"
  }
}

resource "tls_cert_request" "server_request" {
  key_algorithm   = "RSA"
  private_key_pem = "${tls_private_key.server_key.private_key_pem}"

  subject {
        common_name  = "filmustage.vpn.server"
        organization = "Filmustage, Inc"
  }
}

resource "tls_locally_signed_cert" "client_cert" {
  cert_request_pem   = "${tls_cert_request.client_request.cert_request_pem}"
  ca_key_algorithm   = "RSA"
  ca_private_key_pem = "${tls_private_key.ca_key.private_key_pem}"
  ca_cert_pem        = "${tls_self_signed_cert.ca_cert.cert_pem}"

  validity_period_hours = 12

  allowed_uses = [
    "key_encipherment",
    "digital_signature",
    "server_auth",
  ]
}

resource "tls_locally_signed_cert" "server_cert" {
  cert_request_pem   = "${tls_cert_request.server_request.cert_request_pem}"
  ca_key_algorithm   = "RSA"
  ca_private_key_pem = "${tls_private_key.ca_key.private_key_pem}"
  ca_cert_pem        = "${tls_self_signed_cert.ca_cert.cert_pem}"

  validity_period_hours = 12

  allowed_uses = [
    "key_encipherment",
    "digital_signature",
    "server_auth",
  ]
}

resource "aws_acm_certificate" "server_acm" {
  private_key       = "${tls_private_key.server_key.private_key_pem}"
  certificate_body  = "${tls_locally_signed_cert.server_cert.cert_pem}"
  certificate_chain = "${tls_self_signed_cert.ca_cert.cert_pem}"
}

resource "aws_acm_certificate" "client_acm" {
  private_key       = "${tls_private_key.client_key.private_key_pem}"
  certificate_body  = "${tls_locally_signed_cert.client_cert.cert_pem}"
  certificate_chain = "${tls_self_signed_cert.ca_cert.cert_pem}"
}

resource "aws_cloudwatch_log_group" "vpn_lg" {
  name = "vpn"
}

resource "aws_cloudwatch_log_stream" "vpn_ls" {
  name           = "vpn-usage"
  log_group_name = "${aws_cloudwatch_log_group.vpn_lg.name}"
}

resource "aws_ec2_client_vpn_endpoint" "CVPN_endpoint" {
  description = "Terraform controlled CVPN. Bleep-bloooop-bop!"
  server_certificate_arn = "${aws_acm_certificate.server_acm.arn}"
  client_cidr_block = "172.66.0.0/22"

  authentication_options {
    type = "certificate-authentication"
    root_certificate_chain_arn = "${aws_acm_certificate.client_acm.arn}"
  }

  connection_log_options {
    enabled = true
    cloudwatch_log_group = "${aws_cloudwatch_log_group.vpn_lg.name}"
    cloudwatch_log_stream = "${aws_cloudwatch_log_stream.vpn_ls.name}"
  }
}

resource "aws_ec2_client_vpn_network_association" "pub_assoc" {
  client_vpn_endpoint_id = "${aws_ec2_client_vpn_endpoint.CVPN_endpoint.id}"
  count         = "${var.az_count}"
  subnet_id     = "${element(aws_subnet.public.*.id, count.index)}"
}

Debug Output

On first apply resources are created correctly, each next run causes:
https://gist.github.com/orkenstein/c5f868adfd16830d4550118ccccdbc19

Panic Output

Expected Behavior

No actions needed

Actual Behavior

Terraform wants to recreate associations

Steps to Reproduce

1. Make aws vpn setup
2. Add couple of asscoations
3. apply multiple time

Important Factoids

References

[kabo]

I'm having the exact same issue. My workaround:

resource "aws_ec2_client_vpn_network_association" "vpn-subnet-assoc-2" {
  client_vpn_endpoint_id = "${aws_ec2_client_vpn_endpoint.client-vpn.id}"
  subnet_id = "${var.vpn_subnet_assoc_2}"
  lifecycle {
    ignore_changes = "subnet_id" # I don't know why it tries to change this every time
  }
}

[lukerosser]

Seems like this API call isn't behaving as expected and returns the full list of associations:

    result, err := conn.DescribeClientVpnTargetNetworks(&ec2.DescribeClientVpnTargetNetworksInput{
        ClientVpnEndpointId: aws.String(d.Get("client_vpn_endpoint_id").(string)),
        AssociationIds:      []*string{aws.String(d.Id())},
})

https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_ec2_client_vpn_network_association.go#L83

So when the first item in the array is read, expecting it to only return one item, it's always hitting the same association regardless of which AssociationID is sent.

[lukerosser]

I tested this with the aws cli and had the same problem. Raised a case with AWS to see if this is something wrong with the client VPN service API.

Their response:

Thanks again for the update concerning the describe-client-vpn-target-networks command thats failing to filter the associated IDs.

I have also tested this and got the same results.

Please note that I have opened an internal ticket with our service team and will update you as soon as I have more information.

[denibertovic]

I have a general question about setting up ec2 client vpn and found this thread so hopefully it's a good place to ask.

Specifically I'm confused by the root_certificate_chain_arn = "${aws_acm_certificate.client_acm.arn}" line. The name and the value don't make sense in my mind. On one hand we're saying this is the root certificate chain arn, which in my mind means a CA, but we're passing it the client certificate arn?

Am I misunderstanding what client certificate means here? My understanding was that I would generate a client certificate for each user that has access to the VPN using the same CA. This way each user has their own unique cert/key combo that they're using for accessing the VPN and not one shared cert/key. Also it would allow me to revoke access per user. Is this not the case?
Is there an example where I can use multiple client certificates in this manner?

ping @slapula (since I think you're the most likely person to know the answer to this).

[slapula]

@denibertovic I see what you are saying here and it would be good to check the API docs. Here's the comment I found associated with that param:

    // The ARN of the client certificate. The certificate must be signed by a certificate
    // authority (CA) and it must be provisioned in AWS Certificate Manager (ACM).
    ClientRootCertificateChainArn *string `type:"string"`
    // contains filtered or unexported fields

I did my best here to do a one-to-one naming scheme with these parameters but I seem to have missed this one (it should be client_root_certificate_chain_arn). You are right, it's not super clear initially but that comment does clear it up a bit. I'd also recommend checking out the documentation

[denibertovic]

Thanks @slapula . I think it's not your fault as the official documentation is also confusing quite a bit. Especially the easy-rsa examples. I've gone over the link you posted a couple of times but just now I noticed this part:

If the client certificate has been issued by the same Certificate Authority (Issuer) as the server certificate, then you can continue to use the server certificate ARN for the client certificate ARN. The client certificate must be provisioned in AWS Certificate Manager (ACM).

(who writes these confusing docs)

This now makes sense to me. I was able to point root_certificate_chain_arn to "${aws_acm_certificate.server_acm.arn}" and it worked just fine as it extract the CA from that (I imagine). The client cert actually doesn't have to be imported into ACM at this point.

I've also found the example above has allowed_uses set incorrectly for ca, server and client certs. The following should be correct:

ca:

allowed_uses = [
   "cert_signing",
   "crl_signing",
]

client:

allowed_uses = [
    "key_encipherment",
    "digital_signature",
    "client_auth",
  ]

server:

allowed_uses = [
    "key_encipherment",
    "digital_signature",
    "server_auth",
  ]

Also, for the ca_cert one needs to set is_ca_certificate = true.

[denibertovic]

On topic: I should note that I'm not seeing the associations getting recreated on terraform plan.

[lukerosser]

AWS provided a workaround:

aws ec2 describe-client-vpn-target-networks --client-vpn-endpoint-id <CLIENT_VPN_ENDPOINT_ID> --region <REGION> --filters Name=association-id,Values=<ASSOCIATION_ID>

No news on them fixing the issue through their API as of yet.

Might be worth updating the code to use this method instead.

[maxgio92]

Thanks to all! For me ignoring the subnet_id of the association via the _lifecycle_ conf works (as a workaround) only to avoid that the associations will change after the first apply.

But a next plan apply - after the original one that causes the reported error message on the expected association's state - does not notice that an association is removed or in _disassociating_ state, and won't try to recreate it.

[kevinchabreck]

@denibertovic thank you for these replies - I found myself asking the same question you did regarding the naming of the root_certificate_chain_arn field, and @slapula's response provided some helpful context. I also appreciate the nugget of info you dug out of the docs stating that the server cert could be reused here. However, I have a question about this discrepancy between the docs and the statement you made about importing the cert into ACM:

If the client certificate has been issued by the same Certificate Authority (Issuer) as the server certificate, then you can continue to use the server certificate ARN for the client certificate ARN. The client certificate must be provisioned in AWS Certificate Manager (ACM).

(who writes these confusing docs)

This now makes sense to me. I was able to point root_certificate_chain_arn to "${aws_acm_certificate.server_acm.arn}" and it worked just fine as it extract the CA from that (I imagine). The client cert actually doesn't have to be imported into ACM at this point.

Have you confirmed that the client certs do not need to be uploaded to ACM? If this is true, how are we supposed to revoke VPN access to a specific user? Will we need to regenerate a new CA and an all new set of client and server certs, and redistribute them to all users?

[Emptyless]

Could be wrong but I believe the issue to be here: https://github.com/terraform-providers/terraform-provider-aws/blob/master/aws/resource_aws_ec2_client_vpn_network_association.go#L112

It always uses ClientVpnTargetNetworks[0] instead of checking which one of the returned target networks corresponds with the current d.Id()

[pjaol]

This is definitely a headache
I had an existing Client VPN Endpoint I was bringing into terraform and the missing import on network associations is really making it hard
There is now an AssociationId which you should be able to use

$ aws ec2 describe-client-vpn-target-networks --client-vpn-endpoint-id cvpn-endpoint-ccccccc

{
    "ClientVpnTargetNetworks": [
        {
            "AssociationId": "cvpn-assoc-xxxxxx",
            "VpcId": "vpc-aaaaaaaa",
            "TargetNetworkId": "subnet-bbbbbbb",
            "ClientVpnEndpointId": "cvpn-endpoint-ccccccc",
            "Status": {
                "Code": "associated"
            },
            "SecurityGroups": [
                "sg-dddddddd"
            ]
        }
    ]
}

The following I was able to hack into the tfstate to get it to work

resources : [
........
{
      "mode" :"managed",
      "type" :"aws_ec2_client_vpn_network_association",
      "name" :"vpn-prod",
      "provider": "provider.aws",
      "instances":[
        {
          "schema_version": 0,
          "attributes" :{
            "client_vpn_endpoint_id" : "cvpn-endpoint-ccccccc",
            "id": "cvpn-assoc-xxxxxx",
            "security_groups": ["sg-dddddddd"],
            "status" :"associated",
            "subnet_id":"subnet-bbbbbbb",
            "vpc_id": "vpc-aaaaaaaa"
          }
        }
      ]
    }
]

in an ideal world you should be able to do a

$ terraform import aws_ec2_client_vpn_network_associatio.prod cvpn-assoc-xxxxxx

[OliverBailey]

@pjaol - That worked perfectly. Was trying to solve this headache, as it's an absolute pain without being able to import. Implementing the above worked perfectly; so thank you for that :)

[eBulldog]

Just to confirm this issue still exists in Terraform v0.12.28, each subsequent plan/apply attempts to change 2 of 3 associations in the for_each block. The following work around has mitigated the impact of the issue for me and re-sharing for those who arrive here with the same problem:

resource "aws_ec2_client_vpn_network_association" "postgres_vpn_endpoint" {
  for_each = data.aws_subnet_ids.database.ids
  client_vpn_endpoint_id = aws_ec2_client_vpn_endpoint.postgres_vpn_endpoint.id
  subnet_id              = each.key
  lifecycle {
    ignore_changes = [subnet_id] # This is a hack to fix a bug: https://github.com/terraform-providers/terraform-provider-aws/issues/7597
  }
}

[OliverBailey]

Just the life cycle block?

[mrvladis]

Just to confirm this issue still exists in Terraform v0.12.28, each subsequent plan/apply attempts to change 2 of 3 associations in the for_each block. The following work around has mitigated the impact of the issue for me and re-sharing for those who arrive here with the same problem:

Thanks a lot. That worked well for me.

[stephenmcmaster-work]

Perfect, worked for me - many thanks @maxgio92

[ryan-roemer]

We've found this issue not only for aws_ec2_client_vpn_network_association, but also aws_ec2_client_vpn_route in wanting to use subnet_id[0] instead of subnet_id[count.index] in our situation of two subnets.

I think that @Emptyless is on to something in https://github.com/terraform-providers/terraform-provider-aws/issues/7597#issuecomment-576223147 that there may be some hard-coding of "first" of some resources that might be multiple.

We've had to apply a similar patch to aws_ec2_client_vpn_network_association and manually create the correct subnetted resources in raw AWS outside of TF to make it correct and not have terraform plan try to blow it away with a resource replacement with the wrong subnet, e.g.:

resource "aws_ec2_client_vpn_route" "route" {
  count                  = "2"
  description            = "tf-${var.service_name}-${var.client}-route-${count.index}"
  client_vpn_endpoint_id = aws_ec2_client_vpn_endpoint.endpoint.id
  destination_cidr_block = "0.0.0.0/0"
  target_vpc_subnet_id   = aws_ec2_client_vpn_network_association.association[count.index].subnet_id

  # Apply same baid-aid as for issue "Apply forces new aws_ec2_client_vpn_network_association"
  # https://github.com/terraform-providers/terraform-provider-aws/issues/7597#issuecomment-467193628
  lifecycle {
    ignore_changes = [target_vpc_subnet_id]
  }
}

[kkotowick]

Confirming that this is still an issue (with ws_ec2_client_vpn_network_association) with Terraform v0.13.2. It keeps trying to re-create the association in a for_each.