Terraform: CloudTrail resource creation failure when creating aws_iam_role and aws_iam_role_policy in single run

[tmclaugh]

There appears to be an issue with creating cloudtrails when the iam role for cloudwatch logs
is created in the same run. Terraform says the resource creation has completed but when creating the CloudTrail an InvalidCloudWatchLogsRoleArnException error occurs. A subsequent run completes just fine. This leads me to believe that the issue is with the IAM role policy change not being propagated across AWS fast enough for Terraform.

Terraform Version

Terraform v0.9.3

Affected Resource(s)

Please list the resources as a list, for example:

• aws_cloudtrail
• aws_iam_role_policy
• aws_iam_role

Terraform Configuration Files

https://gist.github.com/0dd20ed6bc2a9dcd5c316c9d5193cdc2

Debug Output

https://gist.github.com/690acefc73b6509d89e53f389981cdd0

Expected Behavior

aws_cloudtrail is successful created

Actual Behavior

It appears that the role policy is not properly propagated by AWS before attempting to create the cloudtrail resource. A second run successfully creates wht aws_cloudtrail resource.

* aws_cloudtrail.ct: InvalidCloudWatchLogsRoleArnException: Access denied. Check the trust relationships for your role.
        status code: 400, request id: 4a209c1f-206b-11e7-bb0a-0d6cd06b0119

Steps to Reproduce

1. terraform apply

[tmclaugh]

I was able to confirm that this is a timing issue. I created another terraform configuration that has more resources defined in it. At first it would encounter this error but by using depends_on heavily I was able to draw the run out longer and this error does not occur.

[tmclaugh]

The issue is hit or miss in my new module. This holds us back from supporting Terraform based setup. :(

https://github.com/threatstack/tf_threatstack_aws_integration

[willejs]

I can also confirm this, its very easy to reproduce. I guess there needs to be a retry built into creating the cloudtrail...
I'm contemplating putting a sleep somewhere to fix this in the meantime 🙈 😿

[gobbx]

I'm experiencing the same issue. The configuration file fails the first time but successfully completes when I run it again. CloudTrail needs to wait longer for the other resources to be created.

I can also confirm that this is a timing issue. I was able to get around this problem by adding the following code to my configuration file:

provisioner "local-exec" {
  command = "sleep 10"
}

[willejs]

I think this is an issue that is present in lambda as well, as cloudtrail is run on lambda?

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.