Terraform-provider-aws: aws_s3_bucket_object eventual consistency error after object creation

[jpgu-epam]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

0.12.20

Aws Provider Version

2.54

Affected Resource(s)

• aws_s3_bucket_object

Terraform Configuration Files

resource "aws_s3_bucket_object" "foo-archive" {
  key                    = "foo.sh"
  bucket                 = aws_s3_bucket.foo_bucket.bucket
  source                 = foo.path
  server_side_encryption = "aws:kms"
}

Expected Behavior

Resource creation is successful.

Actual Behavior

aws_s3_bucket_object.foo: Failed to get object tags (bucket: some-bucket, key: /foo): NoSuchKey: The specified key does not

As is generally the case with eventual consistency issues, this only happens sometimes.

Steps to Reproduce

Just define a standard S3 bucket object like so and run apply:

resource "aws_s3_bucket_object" "foo" {
bucket = "some-bucket"
key = "/foo"
source = "${path.module}/foo"
}

Eventually, you get:

Error: error listing tags for S3 Bucket (foo-test-1f0mzi-foo-cache) Object (/foo): NoSuchKey: The specified key does not exist.
    status code: 404

References

• #5841

[borfig]

It looks like the usual "eventual consistency" in AWS API.
For example, in my CloudTrail log, I see that the between the bucket creation and probing its tagging there were 0.014 seconds... a retry should be added
(I have seen similar AWS API behaviour in other services as well)