Terraform-provider-aws: TGW multi-provider aws_ram_resource_share_accepter cannot find invitation

[hashibot[bot]]

_This issue was originally opened by @mitucsaki as hashicorp/terraform#25023. It was migrated here as a result of the provider split. The original body of the issue is below._

---

Creating a TGW across 4 accounts. I am using the provider mark to switch accounts. organization RAM sharing is off and TGW auto-accept is also turned off for security. RAM external principals = true:

Account 1 creates the TGW and RAM association. Associates the TGW and creates the ram principals with account 2 and account 3
Account 2 uses aws_ram_resource_share_accepter to accept the ram invitation
Account 3 uses aws_ram_resource_share_accepter to accept the ram invitation

resource "aws_ram_resource_association" "ram_tgw_associate" {
  provider = aws.account_1

  depends_on = [aws_ec2_transit_gateway.tgw]

  resource_arn       = aws_ec2_transit_gateway.tgw.arn
  resource_share_arn = aws_ram_resource_share.ram.id
}

resource "aws_ram_principal_association" "tgw_ram_principal_account_2" {
  provider = aws.account_1

  depends_on = [aws_ram_resource_association.ram_tgw_associate]

  principal          = data.aws_caller_identity.aws.account_2.account_id
  resource_share_arn = aws_ram_resource_share.ram.id
}
resource "aws_ram_principal_association" "tgw_ram_principal_account_3" {
  provider = aws.account_1

  depends_on = [aws_ram_resource_association.ram_tgw_associate]

  principal          = data.aws_caller_identity.aws.account_3.account_id
  resource_share_arn = aws_ram_resource_share.ram.id
}

resource "aws_ram_resource_share_accepter" "tgw_ram_accept_account_2" {
  provider = aws.account_2

  share_arn = aws_ram_principal_association.tgw_ram_principal_account_2.resource_share_arn
}

resource "aws_ram_resource_share_accepter" "tgw_ram_accept_account_3" {
  provider = aws.account_3

  share_arn = aws_ram_principal_association.tgw_ram_principal_account_3.resource_share_arn
}

Expected Behaviour

Both Accounts successfully accept the invitation and TF continues

Actual behavior:

Account 2 successfully accepts and has the invitation approved:
aws_ram_resource_share_accepter. tgw_ram_accept_account_2: Creation complete after 1s [id=arn:aws:ram:ca-central-1:OMITTED:resource-share/resource-id-here-omitted

Account 3 gives an error saying it cannot find the resource share id: however in the AWS Console the invitation is accepted and valid and I can see it being shared from terraform. Looks like TF fails to acknowledge that the creation is completed.

Error retrieving resource shares: UnknownResourceException: ResourceShare arn:aws:ram:ca-central-1:OMITTED:resource-share/resource-id-here-omitted could not be found

Other Notes

I've gone through the TGW Example of sharing, however here I am using more providers and my organization does not allow RAM sharing directly, therefore I have to include this extra step + the external principals

[raut02a]

I am facing the similar issue, but I think it's more of a issue with aws resource share than providers. In my case I am doing :

1. Create Transit Gateway and RAM resource share with this TG in parent account. (This Works Fine!!)
2. Share RAM resource with children accounts and attach VPCs of children accounts to this TG in parent account.
   I referred to AWS and Terraform documentation where they mentioned that resource share propagation takes some time, hence I added time_sleep resource of 2 mins after invitation is sent and before invitation accepted.
   Just after 2 mins of wait, invitation is accepted and I can see in child account that it is moved from Pending to Active status but next moment terraform plan fails with error : _Error: error retrieving resource shares: UnknownResourceException: ResourceShare arn:aws:ram:eu-west-1:11111111111:resource-share/abc-123-xyz could not be found._ even though arn is valid and TF code accepted it.
   If same code is rerun, then apply fails with error that there is no invitation to accept, which is valid as in first run itself invitation was accepted.
   Hence looks like some glitch Terraform that it is not able to see that valid invitation and arn is accepted but still it is failing for invalid resource.
   Request you to have a look at it on priority. version : 12.20

[raut02a]

Looks like there are so many recent issues logged for aws ram resource share accepter :
https://github.com/terraform-providers/terraform-provider-aws/issues/13458
https://github.com/terraform-providers/terraform-provider-aws/issues/13357
https://github.com/terraform-providers/terraform-provider-aws/issues/13365

[mitucsaki]

Still experiencing the same issue on 12.26. I’ve updated to the new release, looks like there is a new wait timer now added just for these scenarios.

I’ve implemented the time_sleep resource with a 60 second duration to ensure the invite flows through the accounts. Both providers still get the invite issue.

[raut02a]

Due to security reason, we can not enable resource sharing at Org level, then Is there a work around or alternative to aws_ram_resource_share_accepter till Terraform fix this issue ???

[mitucsaki]

Been trying all sorts of things. I set the time_sleep as a trigger for the resource arn, but it still does not work, giving me the same issue.

The resource arn terraform creates is the proper one, it just cant seem to fetch it properly. I don’t have a work around, currently my code is blocked due to this

[nyalavarthi]

I am having same issue.

account 1 - TGW created and resource is shared.
account 2 - Creating Principal association and resource share accepter ( in a single repo / multiple repos')

end up getting this error "error retrieving resource shares: UnknownResourceException: ResourceShare arn:aws:ram:eu-central-1:000000000:resource-share/xxxx-xxxx-xxxx could not be found"

and when i looked at console in account 2, the invitation is successfully accepted but the terraform apply is failing with above error. I have tried adding dependency and timer etc without successfully

[mitucsaki]

A potential workaround that I’ve yet to try could be using local-exec to accept the ram share Via CLI and just adding a dependency for the tgw VPC acceptance on that.

[nyalavarthi]

@mitucsaki I have tried the cli command but seems like my arn is wrong.

code

resource "null_resource" "cli-command" {
  provisioner "local-exec" {
    command = "aws ram accept-resource-share-invitation --resource-share-invitation-arn arn:aws:ram:eu-central-1:$111111111:resource-share-invitation/arn:aws:ram:eu-central-1:000000000:resource-share-invitation/XXXXXXXXX"
  }
}

where 000000000 - is owner account of resource shared and 111111111 is the receiver account and XXXXX is the ID of resource share

Error: Error running command 'aws ram accept-resource-share-invitation --resource-share-invitation-arn arn:aws:ram:eu-central-1:000000000:resource-share-invitation/arn:aws:ram:eu-central-1:111111111:resource-share-invitation/XXXXXX': exit status 255. Output: 
 An error occurred (MalformedArnException) when calling the AcceptResourceShareInvitation operation: The specified resource share invitation ARN arn:aws:ram:eu-central-1:111111111:resource-share-invitation/arn:aws:ram:eu-central-1:000000000:resource-share-invitation/XXXXX is not valid. Verify the ARN and try again.

[raut02a]

I think @nyalavarthi need to have access to ARN hence it is giving this error. You need to have assumed_role or aws secrets set before executing ram commands.

[raut02a]

The below code seems to be working fine. Tested in Production for more than 30 account. You need to have var.region and var.account_id set in Terraform code. Basically it is doing aws cli to do ram action in null_resource using local_exec :

resource "null_resource" "acceptInvite" {
provisioner "local-exec" {
command = < sleep 90
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
aws sts get-caller-identity
temp_role=$(aws sts assume-role --role-arn arn:aws:iam::${var.account_id}:role/R_Role --role-session-name cli_session --duration-seconds 900)
AWS_ACCESS_KEY_ID=$(echo $${temp_role}|jq .Credentials.AccessKeyId|xargs)
AWS_SECRET_ACCESS_KEY=$(echo $${temp_role}|jq .Credentials.SecretAccessKey|xargs)
AWS_SESSION_TOKEN=$(echo $${temp_role}|jq .Credentials.SessionToken|xargs)
aws configure set region ${var.region}
getInvite=$(aws ram get-resource-share-invitations --region ${var.region} --output text --query 'resourceShareInvitations[? \
status==PENDING].{ARN:resourceShareInvitationArn}')
if [ ! -z $${getInvite} ]; then
aws ram accept-resource-share-invitation --region ${var.region} --resource-share-invitation-arn $${getInvite};
sleep 30
else
echo "No invitation to accept..."
fi
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
EOF
}
}

[nyalavarthi]

This seems to be working , no issues so far. I have added sleep timer along with trigger.

following code is executed in account B where we are receiving RAM invitation . this account ( B ) has cross account permissions in Account A to associate Principal.

# resource share via RAM
resource "aws_ram_principal_association" "ram_principal" {
  principal                   = var.account_number
  resource_share_arn  = var.tgw_ram_share_arn
  provider                    = aws.shared_account

}

#wait until the association is complete
resource "time_sleep" "wait_120_seconds" {
  #depends_on = [aws_ram_principal_association.ram_principal]

  create_duration = "120s"
  triggers = {
    # This sets up a proper dependency on the RAM association
    share_arn = aws_ram_principal_association.ram_principal.id
  }
}

# Accept the shared resource.
resource "aws_ram_resource_share_accepter" "receiver_accept" {
  share_arn  = element(split(",", time_sleep.wait_120_seconds.triggers["share_arn"]),0)
}

[raut02a]

Any specific reason, you have used aws_ram_principal_association.ram_principal.id instead of aws_ram_principal_association.ram_principal.resource_share_arn in triggers ?
If you have used resource_share_arn then you don't need element,split function in accept.
But I think, that does not work, gives UnknowResourceException ?

[nyalavarthi]

aws_ram_principal_association only returns an "id" attribute which is a combination of arn separated by a comma . i have no other way to get ARN unless i hard code it.

[nyalavarthi]

above solution is not working consistently, i tried the same code for a different account and now i am getting original errors that I got initially . any other solutions ?

[mitucsaki]

I don’t have any solution working at the moment. I’ve tried setting your code as well, and oddly enough one provider was able to accept and find the ram share, but the second one wasn’t.

[raut02a]

@apparentlymart , sorry to tag you in this issue too. I have 100+ accounts to handle and their vpcs.
Could you please have a look.

[fredsmith1234]

+1 here. anyone had any luck with a workaround or fix?

[moises-panisello]

another +1 here
we are also experimenting with the same problem and we are using a separate project to minimize the handle of this error that happens about the 20% of the executions both creating and destroying transit gateway attachments

[mitucsaki]

@moises-panisello what do u mean using a separate project?

I'm sharing with two accounts, sometimes one of the accounts accepts the invite, but I've never had a case where the error did not occur.

[raut02a]

I refined my solution above. Let me know if above solution does not work : https://github.com/terraform-providers/terraform-provider-aws/issues/13494#issuecomment-644881581

[mitucsaki]

provider 3.00 still having the same bug. shame, I would really like to use the terraform resources aws_ram_resource_share_accepter and not a local command. looks like I'll have to find another solution for this.

[junaid18183]

Same issue 20% of time it does not work.

[raut02a]

I have shared TGW with more than 65 accounts (LZ implementation) and did not face any issue with above code/solution https://github.com/terraform-providers/terraform-provider-aws/issues/13494#issuecomment-644881581
What errors you are getting ?

[junaid18183]

13494 (comment)

I am not using the CLI , the generic TF

[dlyubarskiy]

I am seeing this issue as well. Share accepter works ok, in the console I can see that the share did get accepted, but on the TF side I get an error saying the share hasn't been found. Share and accepter are defined in separate TF blocks that don't run at the same time, so it's not a timing issue for me.

[bee-keeper]

Can also confirm am experiencing this problem with latest AWS provider and TF version. I see the invitation, if I accept I can then import the resource. Strange because the invitation is clearly there but TF is reporting that it can't find it.

[dbuhtiyarov]

I have the same issue, latest TF and AWS library

[datacenter-dylan]

Same issue, fails ~20% of the time.

[soupdiver]

I also run into this issue. I even tried to workaround it with a local-exec block that manually calls aws ram accept-resource-share-invitation.
However this also does not work because Terraform does not expose the share invitation ARN.
The argue for not exposing the invitation ARN is that we have the wonderful aws_ram_resource_share_accepter which is broken 😁
See: https://github.com/hashicorp/terraform-provider-aws/issues/9432#issuecomment-521406397

@bflad

[mitucsaki]

I still ran into the issue. The only workaround I got working was actually separating it into modules.

VPC and TGW VPC attachment was in module where I specified the provider
Outside of the module is where the actual TGW and TGW attachment was created. Doing this method allowed me to attach the VPC to the TGW using the vpc_attachment

Outside of the module is where it put the tgw acceptance resource. This method worked, although its not ideal.

[2stacks]

Jumping on the wagon. I have confirmed this issue exists when deploying to GovCloud. I have added dependencies timers etc. Nothing seems to work. One in ever 10 attempts seems to work fine for two accounts. The aws_ram_resource_share_accepter completes with no issues but Terraform immediately fails with the following error.

Error: error retrieving resource shares: UnknownResourceException: ResourceShare arn:aws-us-gov:ram:us-gov-east-1:119621994737:resource-share/b082136d-f0d8-472b-a660-8938c07d3c90 could not be found.

Workaround

If I run terraform untaint 'aws_ram_resource_share_accepter.poclab[0]' and then run terraform apply again everything completes as expected.

Code used

resource "aws_ram_resource_share" "rrs" {
  provider = aws.root
  count    = var.create_tgw && var.share_tgw ? 1 : 0

  name                      = coalesce(var.ram_name, var.name)
  allow_external_principals = var.ram_allow_external_principals

  tags = merge(
    {
      Name  = format("%s", coalesce(var.ram_name, var.name))
      owner = local.root_caller
    },
    var.tags,
    var.ram_tags,
  )
}

resource "aws_ram_resource_association" "rra" {
  provider = aws.root
  count    = var.create_tgw && var.share_tgw ? 1 : 0

  resource_arn       = aws_ec2_transit_gateway.tgw[0].arn
  resource_share_arn = aws_ram_resource_share.rrs[0].arn
}

resource "aws_ram_principal_association" "rpa" {
  provider = aws.root
  count = var.create_tgw && var.share_tgw ? length(var.ram_principals) : 0

  principal          = var.ram_principals[count.index]
  resource_share_arn = aws_ram_resource_share.rrs[0].arn
}

# AWS resources shared via Resource Access Manager can take a few seconds to
# propagate across AWS accounts after RAM returns a successful association.
resource "time_sleep" "ram_resource_propagation" {
  count           = var.create_tgw && var.share_tgw ? 1 : 0
  create_duration = "60s"

  triggers = {
    # This sets up a proper dependency on the RAM association
    tgw_arn   = aws_ram_resource_association.rra[0].resource_arn
    share_arn = aws_ram_principal_association.rpa[0].resource_share_arn
  }
}

// Resource Share Accepter required per account
resource "aws_ram_resource_share_accepter" "poclab" {
  provider = aws.poclab
  count    = var.create_tgw && var.share_tgw && var.ram_allow_external_principals ? 1 : 0

  share_arn = time_sleep.ram_resource_propagation[0].triggers["share_arn"]
}

[2stacks]

aws_ram_principal_association only returns an "id" attribute which is a combination of arn separated by a comma . i have no other way to get ARN unless i hard code it.

Use resource_share_arn