Terraform-provider-aws: EC2 Transit Gateway VPC Attachment "Transit gateway route table was deleted or no longer exists" when attaching to Shared Transit Gateway from another AWS Account

[amall015]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform: 0.11.10
AWS Provider: 1.50.0

Affected Resource(s)

• aws_ec2_transit_gateway_vpc_attachment

Terraform Configuration Files

resource "aws_ec2_transit_gateway_vpc_attachment" "VPC-Attachment" {
  vpc_id                                          = "${var.vpc_id}"
   ##value from shared transit gateway in another AWS account.
  transit_gateway_id                       = "${var.transit_gw_id}"
  subnet_ids                                    = ["${var.subnet_ids}"]
  transit_gateway_default_route_table_association = false
  transit_gateway_default_route_table_propagation = false
}

Debug Output

https://gist.github.com/amall015/01e86a35645824f9fd6e035fc05c0483

Panic Output

N/A

Expected Behavior

Transit gateway VPC attachment should be created using shared Transit gateway from another account without describing the default route table, since the rtb information is not retrievable in the context of the shared account.

Actual Behavior

Error is produced stating:

error determining EC2 Transit Gateway Attachment (tgw-attach-#######) association to Route Table (tgw-rtb-#######): InvalidRouteTableID.NotFound: Transit Gateway Route Table tgw-rtb-####### was deleted or does not exist.
    status code: 400, request id: 69239b45-69b2-475e-ae74-3e3c72baad0f

Steps to Reproduce

1. terraform apply
2. All subsequent terraform plans

Important Factoids

The VPC attachment resource IS created successfully. However the error always appears when trying to refresh the information about the resource. So the first plan works, the apply creates the resource and then fails, and then all plans after fail.

References

Spawned from comment in:
EC2 Transit Gateway attachment associations cannot be non-default #6664
https://github.com/terraform-providers/terraform-provider-aws/issues/6664#issuecomment-443349234

• #6664

[sethbacon]

still a problem in Terraform Version .11.11 using TFE (assume provider is 1.54)

[congaframe]

Hi,
i have the same setup with Shared Transit Gateway and experience the same issue with v1.54. I'm wondering if anyone has any workaround ?
It looks like the Transit Gateway Attachment is associated to a route table - which belongs to the main account that owns the Shared Transit Gateway
I can't create new Transit Gateway Route Tables on the shared account either , as the dropdown doesn't list any Transit Gateway ID.

[mxmxx]

Seeing same as @sethbacon

[kcchiefdawg]

I am experiencing the same issues with Terraform v0.11.11 and provider.aws v1.54.0.

[EFeit]

Experiencing the same issue with Terraform v0.11.11 and provider.aws v1.55.0.

Any update?

[mskucherawy]

Also still present in v0.11.10/v1.56.0. And, during the same apply:

* aws_ec2_transit_gateway_route_table.tgw_route_table: error creating EC2 Transit Gateway Route Table: InvalidTransitGatewayID.NotFound: Transit Gateway tgw-### was deleted or does not exist.
[...]
* aws_ec2_transit_gateway_vpc_attachment.tgw_attachment: error creating EC2 Transit Gateway VPC Attachment: DuplicateTransitGatewayAttachment: tgw-### has non-deleted Transit Gateway Attachments with same VPC ID.

The first error says the TGW is gone; the second says it's there with attachments. How can both be true?

[dbektas]

Same here.
using v0.11.11/v1.56.0

VPC attachment get created no issues there, but then error comes up with a route table:

* aws_ec2_transit_gateway_route_table.this: error creating EC2 Transit Gateway Route Table: InvalidTransitGatewayID.NotFound: Transit Gateway tgw-### was deleted or does not exist.

[ghost]

I'm also having this issue v0.11.11 + v1.56.0. After applying, refreshing state fails with the Transit Gateway Route Table was deleted or does not exist. message.

[yuriydee]

Same issue as everyone else above even with:

• Terraform v0.11.11
• provider.aws v1.57.0

Any updates on a fix?

[anupkandpile]

@bflad Not sure if you are aware of this. Can you please take a look? Thanks!

[jmisharp]

@bflad Having the exact same issue. Do you need to create a route table and a route on each account and link them to the Transit Gateway to make it work? Not sure if it solves this issue, but I will try it out.

[bflad]

Hi Folks 👋 (and thanks to @andrewsuperbrilliant for an initial fix implementation 🥇), sorry for the hassle and delays here! When the EC2 Transit Gateway resources were initially under development prior to its general availability release around AWS re:Invent last year, the Resource Access Manager (RAM) service was announced around the same time, so its functionality and interaction with Transit Gateways was unknown at the time. 😅

Generally in Terraform, we do not have Terraform resources for ensuring infrastructure _removal_ beyond cases unless we are completely managing the whole configuration for a piece of infrastructure. Managing the full EC2 Transit Gateway Default Route Table seemed problematic given how Routes are inserted automatically in many scenarios. As a compromise, the intention of the transit_gateway_default_route_table_association and transit_gateway_default_route_table_propagation arguments on the aws_ec2_transit_gateway_vpc_attachment resource was to help operators target removing EC2 Transit Gateway Default Route Table Routes for the specific VPC attachment, which works great if you have access to the Default Route Table.

As outlined in this issue, trying to read and manage the EC2 Transit Gateway Default Route Table configuration for a RAM shared EC2 Transit Gateway is not possible. The fix as started by #7513 is on the correct path, but is missing covering acceptance testing to ensure this functionality works with Terraform resource import and prevent regressions. Historically we have not setup the Terraform AWS Provider acceptance testing with multiple AWS accounts, so we have spent some time getting an implementation of that going that is easy to implement across potentially hundreds of acceptance tests.

The good news here is that #7513 will get some review items today just for the resource changes and adding some additional documentation. After those are addressed, we will merge in the fixes for this issue. After a certain degree of testing complexity, especially for new acceptance testing concepts that may need additional vetting, we do not fully expect _every_ contributor to be able to implement the testing to the maintainer's desires so we will accept the contribution after manually verifying it one last time using placeholder testing with our new setup (to be merged in separately after the fact).

We should be able to get the fixes released either middle of this upcoming week or the next. The timing and release numbers are a little indeterminate at the moment as we are looking to merge in Terraform 0.12 provider support, which is still being finalized, so apologies for being more vague than usual on the timing and version.

[bflad]

The fix for this has been merged and will release with version 2.2.0 of the Terraform AWS Provider, likely later today.

[bflad]

This has been released in version 2.2.0 of the Terraform AWS provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

[sestary]

@bflad Thank you for fixing/releasing the fix for this issue.

Please note that if you have an existing transit_gateway_vpc_attachment that already has the flags for the default_route_table set to "true" it fails when trying to update them to "false". I had to delete the resource, run a terraform apply to delete it and then re-add it.

```+ terraform apply terraform.out -no-color
module.logging.aws_ec2_transit_gateway_vpc_attachment.default: Modifying... (ID: tgw-attach-07df9ed4c61daf11e)
transit_gateway_default_route_table_association: "true" => "false"
transit_gateway_default_route_table_propagation: "true" => "false"

Error: Error applying plan:

1 error(s) occurred:

• module.logging.aws_ec2_transit_gateway_vpc_attachment.default: 1 error(s) occurred:

• aws_ec2_transit_gateway_vpc_attachment.default: error updating EC2 Transit Gateway Attachment (tgw-attach-07df9ed4c61daf11e) Route Table (tgw-rtb-0b0a7c2ca85fe316e) association: error determining EC2 Transit Gateway Attachment Route Table (tgw-rtb-0b0a7c2ca85fe316e) association (tgw-attach-07df9ed4c61daf11e): InvalidRouteTableID.NotFound: Transit Gateway Route Table tgw-rtb-0b0a7c2ca85fe316e was deleted or does not exist.
  status code: 400, request id: 34a61496-ec44-409b-baa1-46f92af94671```

[sestary]

There also still seems to be an issue if you run another apply since just like the previous comment it's trying to modify it again from true to false.

Creation of attachment:
```module.logging.aws_ec2_transit_gateway_vpc_attachment.default: Creating...
dns_support: "" => "enable"
ipv6_support: "" => "disable"
subnet_ids.#: "" => "2"
subnet_ids.2783765677: "" => "subnet-943c26df"
subnet_ids.31365102: "" => "subnet-c51c4eea"
tags.%: "" => "4"
tags.Name: "" => "logging-dev-teamsystem"
tags.Namespace: "" => "logging"
tags.Stage: "" => "dev"
tags.Workspace: "" => "default"
transit_gateway_default_route_table_association: "" => "false"
transit_gateway_default_route_table_propagation: "" => "false"
transit_gateway_id: "" => "tgw-0e3b783e3c17f7350"
vpc_id: "" => "vpc-1d71a766"
vpc_owner_id: "" => ""
module.logging.aws_ec2_transit_gateway_vpc_attachment.default: Creation complete after 1s (ID: tgw-attach-074e67efc6590257f)

Next run of Terraform plan/apply:
```module.logging.aws_ec2_transit_gateway_vpc_attachment.default: Modifying... (ID: tgw-attach-074e67efc6590257f)
  transit_gateway_default_route_table_association: "true" => "false"
  transit_gateway_default_route_table_propagation: "true" => "false"

Error: Error applying plan:

1 error(s) occurred:

* module.logging.aws_ec2_transit_gateway_vpc_attachment.default: 1 error(s) occurred:

* aws_ec2_transit_gateway_vpc_attachment.default: error updating EC2 Transit Gateway Attachment (tgw-attach-074e67efc6590257f) Route Table (tgw-rtb-0b0a7c2ca85fe316e) association: error determining EC2 Transit Gateway Attachment Route Table (tgw-rtb-0b0a7c2ca85fe316e) association (tgw-attach-074e67efc6590257f): InvalidRouteTableID.NotFound: Transit Gateway Route Table tgw-rtb-0b0a7c2ca85fe316e was deleted or does not exist.
    status code: 400, request id: 220606a4-ecd0-4cfa-b6d8-ded50a3c886a

Edit - I am using the newly released 2.2.0 version:

```Initializing provider plugins...

• Checking for available provider plugins on https://releases.hashicorp.com...
• Downloading plugin for provider "null" (2.1.0)...
• Downloading plugin for provider "aws" (2.2.0)...
• Downloading plugin for provider "template" (2.1.0)...
• Downloading plugin for provider "tls" (1.2.0)...
• Downloading plugin for provider "local" (1.1.0)..```

[bflad]

@sestary are you explicitly configuring those two arguments to false? e.g.

resource "aws_ec2_transit_gateway_vpc_attachment" "default" {
  # ... other configuration ...

  transit_gateway_default_route_table_association = false
  transit_gateway_default_route_table_propagation = false
}

Configuring that argument for shared EC2 Transit Gateways is not supported as noted in the updated resource documentation:

• transit_gateway_default_route_table_association - (Optional) Boolean whether the VPC Attachment should be associated with the EC2 Transit Gateway association default route table. This cannot be configured or perform drift detection with Resource Access Manager shared EC2 Transit Gateways. Default value: true.
• transit_gateway_default_route_table_propagation - (Optional) Boolean whether the VPC Attachment should propagate routes with the EC2 Transit Gateway propagation default route table. This cannot be configured or perform drift detection with Resource Access Manager shared EC2 Transit Gateways. Default value: true.

Removing those two arguments should resolve your issue. If that's not the case here or the documentation is not clear, please create a new GitHub issue for further triage. Thanks.

[sestary]

Removing those two arguments did resolve my issue. Should be error message be updated to reflect that they are not valid in this case instead of "route table does not exist"?

Thank you for your help.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!