Terraform-provider-aws: aws_dx_gateway_association_proposal recreated

[saliceti]

Terraform bug

It happens when associating a virtual private gateway with a direct connect gateway in a separate account (See AWS doc). There is a terraform resource for the proposal. But this resource is not permanent, so terraform will try to recreate it.

Terraform Version

• terraform 0.11.10
• terraform-provider-aws 2.12.0

Affected Resource(s)

• aws_dx_gateway_association_proposal
• aws_dx_gateway_association

Terraform Configuration Files

resource "aws_dx_gateway_association_proposal" "dcgw_association_proposal" {
  depends_on                  = ["aws_vpn_gateway_attachment.vpn_attachment"]
  dx_gateway_id               = "${local.dcgw_id}"
  dx_gateway_owner_account_id = "${local.shared_services_account_id}"
  associated_gateway_id       = "${aws_vpn_gateway.vpn_gw.id}"
}

resource "aws_dx_gateway_association" "dcgw_association" {
  provider = "aws.shared_services"

  proposal_id                         = "${aws_dx_gateway_association_proposal.dcgw_association_proposal.id}"
  dx_gateway_id                       = "${local.dcgw_id}"
  associated_gateway_owner_account_id = "${local.account_id}"

  timeouts {
    delete = "30m"
    create = "30m"
  }
}

Expected Behavior

• Apply: create proposal, associate in other account, networks are connected
• Any apply after that: No change

Actual Behavior

• Apply: create proposal, associate in other account
• Apply immediately: No change
• Apply after a few days once the proposal has been removed by AWS: create proposal, force recreation of association, networks are disconnected for up to 20min -> DOWNTIME

Steps to Reproduce

1. terraform apply
2. Wait for a few days or delete the proposal (Example)
3. terraform plan: shows creation of the proposal and recreation of the association

Important Factoids

Raised a support case with AWS to understand the behaviour. Here is the response:

I relayed the same question to the service team and they have informed me that the proposal will be available as long as the "accepter" has not accepted the proposal. Once the proposal is accepted, it will disappear after some time. There is no defined time duration as to how long the proposal is available after acceptance through CLI. As we use a distributed and eventual consistency model, I would highly recommend you to not use the time the proposal remains in the CLI output after acceptance to make any design, architectural, or application decisions.

[ewbankkit]

A solution here may be to modify the behavior of the aws_dx_gateway_association.proposal_id attribute so that on update if the new proposal was for the same Direct Connect gateway/associated gateway pair that no new association is created.
Maybe even allow the new value to be empty on update.
I think this could be achieved via a CustomizeDiffFunc.

[ewbankkit]

@saliceti How urgent is this to address? Thanks.

[saliceti]

We stopped using this resource and replaced it with boto code triggered by a provisionner.
It’s ugly but it works. So it’s not urgent anymore. Thanks.

[saliceti]

To be honest this resource is not useful in the current state. If no plan is made to fix it soon it should be removed in the meantime.

[josephschadlick]

I've just run into this problem as well. I tend to agree with @saliceti - this resource should be fixed or removed, as this bug is not obvious and has the potential to cause outages.

[acarrasquillo]

Just ran into this issue too. Terraform created a new proposal and detached the VPC from the Direct Connect Gateway.

[SureshGoli123]

I have also ran into this issue. It should be fixed as soon as possible. Anywork around for now?

[huddy]

We're also seeing this issue, we would prefer to avoid any messy work arounds obviously :)

[ewbankkit]

Looking at this some more the possibility I see is to change ForceNew to false for the proposal_id attribute of the aws_dx_gateway_association.
This would mean that once a DX Gateway Association has been created, any change to the proposal_id would trigger a recreation of the resource.

[sporokh]

Same issue for me using the latest terraform version 0.12.24. Basically aws_dx_gateway_association_proposal removed by AWS itself within 1-3 days, so the actual id of the resource is no longer in AWS, which forces new creation.
Lifecycle ignore_changes is not helping either. The resource is literally unusable. Any estimates on the fix?

[hhh0505]

I'm experiencing the same issue with terraform version 0.12.24. We need a fix ASAP, this is causing downtime if we need to add or change any resoucres.

[piyat]

Encountered this too, going to omit until fixed.