Terraform: s3 backend should validate that the selected bucket exists during its initialization

[chaffees]

Terraform Version

Terraform v0.11.10

terraform -version

Terraform Configuration Files

terraform {
    backend "s3" {
        encrypt = true
        bucket = "some-remote-state-storage-s3"
        dynamodb_table = "some-tf-state-lock-dynamo"
        region = "us-east-2"
        key = "terraform.tfstate"
    }
}

Debug Output

Error inspecting states in the "s3" backend:
NoSuchBucket: The specified bucket does not exist
status code: 404, request id: 6DCCC569BD3C199D, host id: qKSeieYGUoUIcinEl7KmQU8yMuFz/5zRRzIRJqHQ3aIQEksgdL/Kx4JY8k6YcietWHA0VM2oAeg=

Prior to changing backends, Terraform inspects the source and destination
states to determine what kind of migration steps need to be taken, if any.
Terraform failed to load the states. The data in both the source and the
destination remain unmodified. Please resolve the above error and try again.

Crash Output

Expected Behavior

terraform state file should be stored in S3 bucket.

Actual Behavior

Error was produced during terraform initialization.

Steps to Reproduce

1. terraform init

Additional Context

S3 bucket was created manually and the user running terraform has AmazonS3FullAccess.

The backend.tf file is in the main directory with the main.tf file.

References

[builtinnya]

I've got the same error for simpler configuration (confirmed with Terraform v0.11.10 and v0.11.11.)

terraform {
  backend "s3" {
    bucket = "my-bucket-name"
    key = "terraform.tfstate"
    region = "ap-northeast-1"
    profile = "my-profile"
  }
}

I've also confirmed that the bucket name and profile name should be correct by running:

$ AWS_PROFILE=my-profile aws s3 ls
<BUCKET CREATION TIME> my-bucket-name

(and the bucket region matches the configuration.)

It's very strange because I didn't get any error using similar configuration to set up a different environment a few weeks ago (with Terraform v0.11.10.)

[builtinnya]

I solved the issue in my case.

Just remove .terraform/ directory and run terraform init again.

Once you run terraform init with a wrong bucket name, .terraform/terraform.tfstate will be created and it will contain the wrong bucket name. Fixing the name in your .tf configuration doesn't help because Terraform keeps using the name from .terraform/terraform.tfstate.

[mildwonkey]

I'm glad you were able to figure this out @builtinnya, thank you for updating the issue!

[mildwonkey]

Yikes, sorry about that @chaffees - I should not have closed _your_ issue.

We most frequently see messages like yours when the s3 bucket is in a different aws account or region from the API keys currently in use by terraform. Can you confirm that the bucket is in "us-east-2"?

[ziczhu]

I encountered this problem too, it's actually a bug like @builtinnya pointed out. @mildwonkey Any plan to fix this? Thank you.

[apparentlymart]

I think the best thing we could do here is to make the s3 backend perform validation that the bucket exists during its own configuration, rather than waiting until a later step that actually accesses the bucket. That would prevent it from initializing at all, whereas today it seems to be initializing but then encountering an error at runtime when trying to use the bucket.

I'm going to relabel this issue to reflect that goal.

[MoBattah]

No update? Running into this issue.

[salmanwaheed]

I solved the issue in my case.

Just remove .terraform/ directory and run terraform init again.

Once you run terraform init with a wrong bucket name, .terraform/terraform.tfstate will be created and it will contain the wrong bucket name. Fixing the name in your .tf configuration doesn't help because Terraform keeps using the name from .terraform/terraform.tfstate.

awesome 👍