Terraform-provider-aws: Unable to create/manage two S3 buckets in separate regions from the same .tf file

[fabiendelpierre]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform v0.11.8
+ provider.aws v1.38.0

Affected Resource(s)

• aws_s3_bucket

Terraform Configuration Files

provider "aws" {
  region  = "us-east-1"
  version = "~> 1.38"
}

resource "aws_s3_bucket" "a" {
  bucket  = "s3-access-denied-us-east-1"
  acl     = "private"
  region  = "us-east-1"
}

resource "aws_s3_bucket" "b" {
  bucket  = "s3-access-denied-us-east-2"
  acl     = "private"
  region  = "us-east-2"
}

Notice how my provider bit is set for us-east-1. I'm trying to create a bucket in us-east-1 and another bucket in us-east-2. The idea is to set up replication from the former to the latter. This'll be used for Elasticsearch backups, and my employer wants backups of the backups in a different region.

Debug Output

https://gist.github.com/fabiendelpierre/a29eb9d9d1c2fb43484405b0841faa19
Note: there are two parts in there. The first part is for the initial terraform apply when neither bucket exists. The second part is for a subsequent terraform apply to show that I get an error every time I run TF after the first run. More on that farther below.

Panic Output

N/A

Expected Behavior

Create two buckets and don't complain.

Actual Behavior

TF actually creates both buckets just fine, but I get this error for the second bucket that is set to be created in us-east-2:

Error: Error applying plan:

1 error(s) occurred:

* aws_s3_bucket.b: 1 error(s) occurred:

* aws_s3_bucket.b: error reading S3 Bucket (s3-access-denied-us-east-2): Forbidden: Forbidden
        status code: 403, request id: AFCA75884CEBF5C9, host id: MU9pTAgmdnYo3XAy6ijcH+VPeC1amQlfq4BQbC4bvmOqbpnLFr51sOFj/lfEGkqhyKrwrKJnUvk=

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

Subsequent runs of Terraform will yield a slightly different error, even after removing the aws_s3_bucket.b resource from my code:

aws_s3_bucket.b: Refreshing state... (ID: s3-access-denied-us-east-2)
aws_s3_bucket.a: Refreshing state... (ID: s3-access-denied-us-east-1)

Error: Error refreshing state: 1 error(s) occurred:

* aws_s3_bucket.b: aws_s3_bucket.b: error reading S3 Bucket (s3-access-denied-us-east-2): Forbidden: Forbidden
        status code: 403, request id: 434289028F1CBCBE, host id: 4AoUH4Z/YrRbUpGx9aO9eaQD6lNIDz8zyRRX5mQzNfIyQQht55VuA/g4ufZYsG8HKKGZI17ZEdU=

The only way to get TF running again is to remove the resource from my code, remove the bucket manually through the AWS API, and run TF to remove the resource from my state file.

Like I mentioned, I would like to create two buckets in different regions, and set up replication between the two. I could easily set it up manually, but it would be better if I could do it with TF.

I briefly looked at the debug log and it looks like Terraform gets a 403 when sending s3:HeadBucket against the bucket in the different region... but from my shell I can do an aws s3api head-bucket --bucket s3-access-denied-us-east-2 with no issue using the same credentials. In fact I can interact with the bucket through the GUI or the API in whatever way I see fit, it's just Terraform that seems to have a problem with this.

I'm not sure if what I'm trying to do is actually supported, but I thought I'd check. Let me know if you have any questions!

Steps to Reproduce

1. Put the code above on its own somewhere.
2. terraform apply

Important Factoids

N/A

References

N/A

[bflad]

Hi @fabiendelpierre 👋 Nice to see you around these parts! Hope things are well.

We certainly do support this situation, although admittedly, we should probably cross reference our documentation on setting up multiple providers in a few more places. 😅

Each provider configuration supports an alias argument, which can then be used in conjunction with a resource configuration provider argument. By default, a provider without an alias argument becomes the "default" provider of that type, which is what the resource provider argument defaults to when omitted from resource configurations.

An additional oddity about the aws_s3_bucket resource is that region argument, which is very unlike all other AWS provider resources. It defaults to the region of the provider being used, and we will probably get rid of it at some point.

I believe this should help get you on the right track. 😄

provider "aws" {
  alias      = "a"
  region  = "us-east-1"
  version = "~> 1.38"
}

provider "aws" {
  alias      = "b"
  region  = "us-east-2"
  version = "~> 1.38"
}

resource "aws_s3_bucket" "a" {
  provider = "aws.a"

  bucket  = "s3-access-denied-us-east-1"
  acl     = "private"
}

resource "aws_s3_bucket" "b" {
  provider = "aws.b"

  bucket  = "s3-access-denied-us-east-2"
  acl     = "private"
}

You can choose to omit one of the provider aliases (letting it be the default) if you so desire, but my personal preference when working with multi-region/account configurations is that its nice to be explicit. If this code is going to live within a Terraform module, you might also want to check out the Providers within Modules documentation as well which can provide some guidance with passing provider configurations around so you only need to define them once in the top level configuration.

Let us know how it goes, thanks! 🚀

[fabiendelpierre]

Hi Brian! Been seeing you a lot around here, is this your new full-time gig? All is well here, thanks!

And this looks like it works fine, thanks much 👍 Tried it in my sandbox and set it up in the module I'm working on, no issues.

[nywilken]

Looks like this is safe to close. If not please let me know and I will gladly reopen and label appropriately.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!