Terraform-provider-aws: Provider produced inconsistent result after apply - aws_vpc_endpoint_route_table_association.s3_vpc_endpoint

[ffran09]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version 0.12.23 aws provider version 2.51.0

Affected Resource(s)

aws_vpc_endpoint_route_table_association.s3_vpc_endpoint

Terraform Configuration Files

resource "aws_route_table" "private_subnet_rtb" {
  count  = length(var.availability_zones) * var.flag
  vpc_id = aws_vpc.vpc[0].id
}

resource "aws_vpc_endpoint_route_table_association" "s3_vpc_endpoint" {
  vpc_endpoint_id = aws_vpc_endpoint.s3[0].id
  count           = length(var.availability_zones) * var.flag
  route_table_id  = element(aws_route_table.private_subnet_rtb.*.id, count.index)
}

resource "aws_vpc_endpoint" "s3" {
  count        = var.flag
  vpc_id       = aws_vpc.vpc[0].id
  service_name = "com.amazonaws.${data.aws_region.current.name}.s3"
} 

Debug Output

Error: Provider produced inconsistent result after apply vpc.aws_vpc_endpoint_route_table_association.s3_vpc_endpoint[0], provider "registry.terraform.io/-/aws" produced an unexpected new value for was present, but now absent. This is a bug in the provider, which should be reported in the provider's own issue tracker..

Expected Behavior

Terraform apply should have been successful

Actual Behavior

Terraform apply fails

Steps to Reproduce

This does not happen all the time, running apply multiple times

[ewbankkit]

@ffran09 If you add count = var.flag to resource "aws_vpc_endpoint_route_table_association" "s3_vpc_endpoint" do you still get the problem?

[ffran09]

Yes, we still get the same error with the count variable.
I have updated the tf code, we are using var.flag in resource "aws_vpc_endpoint_route_table_association" "s3_vpc_endpoint" as well.

[tbugfinder]

I just run into this error message, too.
I'm using the terraform-registry module terraform-aws-vpc.

       Error: Provider produced inconsistent result after apply

       When applying changes to
       module.aws_vpc_test.aws_vpc_endpoint_route_table_association.private_s3[0],
       provider "registry.terraform.io/-/aws" produced an unexpected new value for
       was present, but now absent.

       This is a bug in the provider, which should be reported in the provider's own
       issue tracker.

Terraform v0.12.24

       Initializing provider plugins...
       - Checking for available provider plugins...
       - Downloading plugin for provider "null" (hashicorp/null) 2.1.2...
       - Downloading plugin for provider "template" (hashicorp/template) 2.1.2...
       - Downloading plugin for provider "aws" (hashicorp/aws) 2.54.0...
       - Downloading plugin for provider "random" (hashicorp/random) 2.2.1...
       - Downloading plugin for provider "local" (hashicorp/local) 1.4.0...

[ewbankkit]

Relevant:

• https://github.com/hashicorp/terraform/issues/20688

This is most likely an eventual consistency issue. The association is successfully created but when the provider then turns around and reads back the state, the AWS EC2 service can't find it.
The fix will be to add a retry loop in resourceAwsVpcEndpointRouteTableAssociationCreate.

[queglay]

I also see it with aws_network_interface_sg_attachment

3776 [23:29:14] Error: timeout - last error: ssh: rejected: connect failed (Connection timed out)
3777 [23:29:14] 
3778 [23:29:14] 
3779 [23:29:14] 
3780 [23:29:14] Error: Provider produced inconsistent result after apply
3781 [23:29:14] 
3782 [23:29:14] When applying changes to
3783 [23:29:14] module.softnas.aws_network_interface_sg_attachment.sg_attachment_vpn[0],
3784 [23:29:14] provider "registry.terraform.io/-/aws" produced an unexpected new value for
3785 [23:29:14] was present, but now absent.
3786 [23:29:14] 
3787 [23:29:14] This is a bug in the provider, which should be reported in the provider's own
3788 [23:29:14] issue tracker.

[spicysomtam]

I just suffered this; it seemed to have setup the route associations. I re-ran the apply and it needed to add the network associations which worked (even though they are already there). Then another plan showed all was good. So I guess the answer is just do another apply?

[tbugfinder]

Well, yes and no.
In most cases it applies changes in a single step. In few cases this error is raised....

[bashims]

Seeing this problem with Terraform v0.12.27 and AWS provider v2.70.0

This issue seems to be non-deterministic.

[drexler]

Same issue. Re-applying the plan does resolve it.

Terraform v0.12.24
provider.aws v3.1.0

[mwheeler-aicradle]

Same issue. Encountered with TF v0.12.28, and provider v3.11.0.

[fractos]

Seeing this for TF v0.12.29 and provider 2.70.0. Generally seems to be the s3 or dynamoDB endpoints. Non-deterministic, probably happens 1 time in 20.

[lifelofranco]

Same issue with TF 0.13.5 and provider v3.12.0.

Error: Provider produced inconsistent result after apply

When applying changes to
module.vpc.aws_vpc_endpoint_route_table_association.public_dynamodb[0],
provider "registry.terraform.io/hashicorp/aws" produced an unexpected new
value: Root resource was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

[atsalolikhin-spokeo]

We are on Terraform 0.13.3 and on hashicorp/aws v2.70.0.

Apply failed with

Error: Provider produced inconsistent result after apply

When applying changes to
module.my_module_name.aws_network_interface_sg_attachment.my_server[2],
provider "registry.terraform.io/hashicorp/aws" produced an unexpected new
value: Root resource was present, but now absent.

This is a bug in the provider, which should be reported in the provider's own
issue tracker.

The next apply failed with

Error: security group [id redacted] already attached to interface ID [id redacted]

  on [filename] line 22, in resource "aws_network_interface_sg_attachment" "my_server":
  22: resource "aws_network_interface_sg_attachment" "my_server" {

So did the one after it.

We had to manually detach the Security Group, then apply again. That resolved it.

[ewbankkit]

I have noticed that the aws_vpc_endpoint_subnet_association resource which makes very similar AWS API calls to the aws_vpc_endpoint_route_table_association resource has a call to vpcEndpointWaitUntilAvailable

https://github.com/hashicorp/terraform-provider-aws/blob/63df631903b9f3c452d2cdc8aebd1976f870e7d3/aws/resource_aws_vpc_endpoint.go#L458-L471

whereas the route table association does not wait for the VPC endpoint to reach available state.
I can see no harm in adding such a call.

[ewbankkit]

Given that this is happening for Gateway VPC endpoints it is likely best to use the same logic as used in aws_route and retry after creating the association to ensure it can be read back correctly.

[ewbankkit]

aws_vpc_endpoint_route_table_association issue address in #16200.
aws_network_interface_sg_attachment issue raised in #16201.