Terraform-provider-aws: using aws_vpc_endpoint_subnet_association with count for multiple subnet hangs

[GElkayam]

Hello,

I've created a vpc endpoint for using ec2 api, and tried to add aws_vpc_endpoint_subnet_association so it will be in multiple AZs.

I've configured the aws_vpc_endpoint as a resource without any subnets, and added a aws_vpc_endpoint_subnet_association resource with a list of subnets with count, after subnets are in place.

Terraform Configuration Files

resource "aws_vpc_endpoint_subnet_association" "ec2_private-elb-main" {
  count           = "${length(module.subnet__private-elb.subnet_ids)}"
  vpc_endpoint_id = "${module.vpc__main.ep_ec2_id}"
  subnet_id       = "${module.subnet__private-elb.subnet_ids[count.index]}"
}

in two different accounts that I've tried that it created only one aws_vpc_endpoint_subnet_association, and hanged for the rest.

aws_vpc_endpoint_subnet_association.ec2_private-elb-main[2]: Creating...
subnet_id: "" => "subnet-12341ec2"
vpc_endpoint_id: "" => "vpce-12343dccc075156d3"
aws_vpc_endpoint_subnet_association.ec2_private-elb-main[0]: Creating...
subnet_id: "" => "subnet-1234b585"
vpc_endpoint_id: "" => "vpce-12343dccc075156d3"
aws_vpc_endpoint_subnet_association.ec2_private-elb-main[1]: Creating...
subnet_id: "" => "subnet-1234d612"
vpc_endpoint_id: "" => "vpce-12343dccc075156d3"
aws_vpc_endpoint_subnet_association.ec2_private-elb-main[2]: Creation complete after 2s (ID: a-vpce-12343dccc075156d31244978879)
aws_vpc_endpoint_subnet_association.ec2_private-elb-main.0: Still creating... (10s elapsed)
aws_vpc_endpoint_subnet_association.ec2_private-elb-main.1: Still creating... (10s elapsed)
aws_vpc_endpoint_subnet_association.ec2_private-elb-main.0: Still creating... (20s elapsed)
aws_vpc_endpoint_subnet_association.ec2_private-elb-main.1: Still creating... (20s elapsed)
aws_vpc_endpoint_subnet_association.ec2_private-elb-main.0: Still creating... (30s elapsed)
.
.
.
aws_vpc_endpoint_subnet_association.ec2_private-elb-main.0: Still creating... (12m40s elapsed)
aws_vpc_endpoint_subnet_association.ec2_private-elb-main.1: Still creating... (12m50s elapsed)
aws_vpc_endpoint_subnet_association.ec2_private-elb-main.0: Still creating... (12m50s elapsed)
aws_vpc_endpoint_subnet_association.ec2_private-elb-main.1: Still creating... (13m0s elapsed)
aws_vpc_endpoint_subnet_association.ec2_private-elb-main.0: Still creating... (13m0s elapsed)
aws_vpc_endpoint_subnet_association.ec2_private-elb-main.1: Still creating... (13m10s elapsed)
aws_vpc_endpoint_subnet_association.ec2_private-elb-main.0: Still creating... (13m10s elapsed)
aws_vpc_endpoint_subnet_association.ec2_private-elb-main.1: Still creating... (13m20s elapsed)
aws_vpc_endpoint_subnet_association.ec2_private-elb-main.0: Still creating... (13m20s elapsed)

the endpoint itself got only one subnet, but looking in Network Interfaces, I see a large amount of network interfaces created for that endpoint and subnet, while most are not in use, but 3 (as requested) are (28 ENIs which are in "available" state).

Terraform Version

terraform -v
Terraform v0.11.3

• provider.aws v1.9.0

Affected Resource(s)

• aws_vpc_endpoint_subnet_association
• aws_vpc_endpoint

Debug Output

Small sample:
https://gist.github.com/GElkayam/f3d5854d096b1162a4ce6c39325c4626

Expected Behavior

all subnets should be assigned to the endpoint.

Actual Behavior

only one subnet was assigned, multiple ENIs were created.
Unable to add subnets manually, as AWS return HTTP 500.

Steps to Reproduce

1. terraform apply

Important Factoids

Trying to apply object by object, and waiting for them to finish seems successful.

[GElkayam]

talking to AWS support, seems this is a known issue. I was able to workaround it by applying with target, one by one. for safty, waited for the endpoint to be available before moving on.
terraform apply -target=aws_vpc_endpoint_subnet_association.ec2_private-elb-main[0]
terraform apply -target=aws_vpc_endpoint_subnet_association.ec2_private-elb-main[1]
terraform apply -target=aws_vpc_endpoint_subnet_association.ec2_private-elb-main[2]

[ewbankkit]

@GElkayam I noticed similar behaviour when implementing the functionality and in my case narrowed it down to trying to associate one VPC endpoint with multiple subnets that were in the same AZ - Are all your subnets in distinct AZs?

[ewbankkit]

We may have to use awsMutexKV to prevent parallel association attempts.

[GElkayam]

In my case each subnet is in an AZ (dedicated for internal ELBs and other AWS allocated IP addresses)

[ewbankkit]

I can reproduce this with an additional acceptance test for the aws_vpc_endpoint_subnet_association resource; I'll fix.

[dirkcjelli]

I worked around this bug with

resource "aws_vpc_endpoint_subnet_association" "test-private-shared-1" {
  vpc_endpoint_id = "${aws_vpc_endpoint.this.id}"
  subnet_id       = "${data.aws_subnet_ids.private-shared.ids[0]}"
  depends_on      = ["aws_vpc_endpoint.this"]   
}

resource "aws_vpc_endpoint_subnet_association" "test-private-shared-2" {
  vpc_endpoint_id = "${aws_vpc_endpoint.this.id}"
  subnet_id       = "${data.aws_subnet_ids.private-shared.ids[1]}"
  depends_on      = ["aws_vpc_endpoint_subnet_association.test-private-shared-1"] 
}

resource "aws_vpc_endpoint_subnet_association" "test-private-shared-3" {
  vpc_endpoint_id = "${aws_vpc_endpoint.this.id}"
  subnet_id       = "${data.aws_subnet_ids.private-shared.ids[2]}"
  depends_on      = ["aws_vpc_endpoint_subnet_association.test-private-shared-2"] 
}

but I hope to get rid of this abomination soon :)
UPDATE: For some reason, it seems to be non-deterministic when this works

[ewbankkit]

Yes, it would be nice for AWS to fix the underlying issue 😄.

[dirkcjelli]

Another update: the depending resources also stopped working for some reason, also, setting parallelism=1 did not help.
I resorted to using subnet_ids as in

resource "aws_vpc_endpoint" "endpoint" {
  vpc_id            = "${var.vpc_id}"
  service_name      = "${data.aws_vpc_endpoint_service.selected.service_name}"
  vpc_endpoint_type = "Interface"

  security_group_ids = ["${local.endpoint_sg_ids}"]
  subnet_ids         = ["${data.aws_subnet_ids.selected.ids}"]
}

for the time being. Will debug this further tomorrow.

[GMartinez-Sisti]

Same issue here, I found that adding subnets to the endpoint doesn't work, you need to create it with the subnets already defined. This is true also if adding subnets to the already created endpoint in the AWS Console, it throws an error when applying.

@dirkcjelli 's solution works when creating from scratch, the only difference is that I'm not creating the list from a data resource:

resource "aws_vpc_endpoint" "ec2" {
  vpc_id              = "${aws_vpc.main.id}"
  service_name        = "com.amazonaws.${var.region}.ec2"
  vpc_endpoint_type   = "Interface"
  security_group_ids  = ["${aws_security_group.private.id}"]
  private_dns_enabled = true
  subnet_ids          = ["${formatlist("%s", aws_subnet.private.*.id)}"]
}

[bflad]

A fix for this should be merged into master and will release with version 1.28.0 of the AWS provider, likely later today.

[bflad]

This has been released in version 1.28.0 of the AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!