Terraform: Getting "access denied" when switching S3 backends in multiple AWS account setup

[cgswong]

Terraform Version

$ terraform --version
Terraform v0.11.7
+ provider.aws v1.17.0

Terraform Configuration Files

terraform.tf:
terraform {
  backend "s3" {}
}

backend-prod.tfvars:
bucket = "terraform-111111111111"

key = "r53cwlog/prod/terraform.tfstate"

region = "us-east-1"

encrypt = "true"

dynamodb_table = "r53cwlog_tf_state_lock"


backend-non-prod.tfvars:
bucket = "terraform-222222222222"

key = "r53cwlog/non-prod/terraform.tfstate"

region = "us-east-1"

encrypt = "true"

dynamodb_table = "r53cwlog_tf_state_lock"

Debug Output

https://gist.github.com/cgswong/40ab841741137b3631c5bef02762f905

Expected Behavior

Terraform switching backends from S3 in account A to account B successfully.

terraform init -backend-config=env/backend-prod.tfvars

Actual Behavior

Terraform backend switch results in "Access Denied" error using account with AWS Administrator privileges:

$ terraform init -backend-config=env/backend-prod.tfvars

Initializing the backend...
Backend configuration changed!

Terraform has detected that the configuration specified for the backend
has changed. Terraform will now check for existing state in the backends.


Error inspecting states in the "s3" backend:
    AccessDenied: Access Denied
    status code: 403, request id: 809BFEAC6F189586, host id: oR6bxLZCwV/wB3u3HZXE1bXgeGnZTwJD7Z4EINGtrl1f4ea+qtUEFfiJT/Uwxc4CfGB3onyr4ow=

Prior to changing backends, Terraform inspects the source and destination
states to determine what kind of migration steps need to be taken, if any.
Terraform failed to load the states. The data in both the source and the
destination remain unmodified. Please resolve the above error and try again.

Steps to Reproduce

1. With AWS environment variables set for account A run:

   terraform init -backend-config=env/backend-non-prod.tfvars
   terraform plan -var-file=env/non-prod.tfvars -out=non-prod.tfplan
   terraform apply non-prod.tfplan
   

2. With AWS environment variables set for account B run:

   terraform init -backend-config=env/backend-prod.tfvars
   

Additional Context

Error occurs with and without Makefile. Confirmed role being used has administrator privileges, and can read/write to S3 bucket hosting state.

[jbardin]

Hi @cgswong,

Sorry this is tripping you up, trying to handle multiple accounts in the backend can be confusing.

The access denied is because when you run init and change the backend config, terraform's default behavior is to migrate the state from previous backend to the new backend. So you new configuration may be correct, but you don't probably have the credentials loaded to access the previous state.

This is what the -reconfigure flag was added to support, which ignores the previous configuration altogether.

[cgswong]

Thanks for the quick reply!

That is what I figured. So considering I need to switch between my two accounts, is it okay to do so using the -reconfigure flag each time, and on each such switch Terraform will read the last known state for that account/environment from the bucket/prefix and be fine?

Is there a better/recommended approach?

Thanks.

[jbardin]

@cgswong,

Nope, that is perfectly fine, as it appears you have policies in place to keep the wrong credentials from accessing the incorrect state. As far as the remote state is concerned, using -reconfigure is the same as running init for the first time.

Now that the legacy remote states are a couple releases behind us, I've been thinking about change the defaults for this in a future major release. I'll open a separate issue for that.

Thanks!

[izlam]

Hi,

I found the issue is the order of your providers, if your backend uses a non-default provider, ensure they have the same order.

```hcl
module "sample" {
providers = {
aws.enterprise = "aws.enterprise"
aws.default = "aws.default"
}

[AJVA2580]

Hi,

Are you guys able to share why am I getting this error?

please see below -

_

Terraform has detected that the configuration specified for the backend
has changed. Terraform will now check for existing state in the backends.

Error inspecting states in the "s3" backend:
NoSuchBucket: The specified bucket does not exist
status code: 404, request id: xxxxxxx host id: =xxxxxxxxxxxxxxxxxxxxxx

Prior to changing backends, Terraform inspects the source and destination
states to determine what kind of migration steps need to be taken, if any.
Terraform failed to load the states. The data in both the source and the
destination remain unmodified. Please resolve the above error and try again.

_

I try re creating the bucket in different region but still not working.

Thanking you in advances for your help

[shawnmclean]

AWS_PROFILE=secondary terraform init worked for me.

[chadasapp]

I'm getting hit with the same bug, sadly.

[christhomas]

I think this is a permission issue. The documentation does not correctly nor adequately describe how to apply the correct permissions. It just states the permissions to apply. But not the full process, so I'm stumped how this is supposed to work.

[chadasapp]

From what little I've dug out, the role in the new account tries to hit the old bucket, or vice versa, at some point.

I'd rather not have to setup temporary cross-account-cross-bucket permissions when Terraform has credentials available to the source & destination.

[christhomas]

Yeah, I tried to fiddle around a bit just now but it seems to not make any sense why terraform itself would need those permissions since, in both "infrastructure" and "app" projects, I'm using the same credentials to access resources in aws.

[christhomas]

This is what I got when I tried with TF_LOG=TRACE

2018/11/29 20:59:04 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
<Error>
    <Code>AccessDenied</Code>
    <Message>Access Denied</Message>
    <RequestId>CENSORED</RequestId>
    <HostId>CENSORED</HostId>
</Error>
2018/11/29 20:59:04 [DEBUG] [aws-sdk-go] DEBUG: Validate Response s3/ListObjects failed, not retrying, error AccessDenied: Access Denied
    status code: 403, request id: CENSORED, host id: CENSORED
Error inspecting states in the "s3" backend:
    AccessDenied: Access Denied
    status code: 403, request id: CENSORED, host id: CENSORED

[christhomas]

I've found that my problem was that I am not using a default profile, so I needed to add the profile = "xxxx" to the backend s3 credentials block and then it works :( I should have realised that earlier.

see: https://github.com/hashicorp/terraform/issues/13589

[arturmartins]

in my case, I had added the following lines to my main.tf:

terraform {
  [...]
  backend "s3" {
+    role_arn       = "arn:aws:iam::[REDACTED]:role/[ROLENAME]"
     [...]
  }
}

provider "aws" {
+  assume_role {
+    role_arn = "arn:aws:iam::[REDACTED]:role/[ROLENAME]"
+  }
[...]
}

I needed to run
terraform init -reconfigure in order to update .terraform/terraform.tfstate

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.