Terraform: Incorrect region S3 bucket data source

[Jonnymcc]

Terraform Version

Terraform v0.7.1-dev (85c0105c9ec71db451ae53ba3b76e6a1eb802fa1)

Affected Resource(s)

• s3
  

  Terraform Configuration Files

data "aws_s3_bucket_object" "aws-remote-state" {
    bucket = "terraform-remote-state"
    key = "/aws/aws.tfstate"
}

Output

Error refreshing state: 1 error(s) occurred:

* data.aws_s3_bucket_object.aws-remote-state: Failed getting S3 object: BucketRegionError: incorrect region, the bucket is not in 'us-west-2' region
        status code: 301, request id:

Expected Behavior

Should have read data from the S3 bucket.

Actual Behavior

Fails because bucket is in another region.

Steps to Reproduce

1. terraform plan
   

   Important Factoids

The AWS provider resource is configured to use us-west-2 as that is where we would like to provision. The S3 bucket is in us-east-1 and it appears as though the aws_s3_bucket_object does not have a region parameter.

References

https://www.terraform.io/docs/providers/aws/d/s3_bucket_object.html
https://github.com/hashicorp/terraform/blob/f257895d67bac63df250161272ff717b249df442/builtin/providers/aws/data_source_aws_s3_bucket_object.go#L107-L115

[kwilczynski]

@Jonnymcc hello there! I have to say that this one is a bit tricky one.

TL;DR

Have you tried using another provider with an alias as per the following providers documentation. You could then pass the alias into the data source, perhaps that would solve the problem for you.

There exist GetBucketLocation function in the SDK, which would work for getting the region of the bucket, but sadly there seem to be no way to pass region to any of the calls which can be used to query object details e.g. HeadObject.

API calls in as above:

• GetBucketLocation
• HeadObject

The S3 client supports ability to setup region as per New, but we already get an instance of S3 client which is being passed down to the resource. It might require to use completely new session and client instance, which is definitely not the way to go for may reasons.

If we could re-use the session then it might be possible to get new instance of the S3 client easily with the same parameters, etc., but I don't see the underlying session being exposed internally outside of the initial configuration of the provider.

[radeksimko]

Hi @Jonnymcc
if your use case is exactly as shown in the example then there's more idiomatic way of referencing remote state which also allows you to specify a region:

https://www.terraform.io/docs/state/remote/s3.html#region

Have you considered that option?

Admittedly the mentioned data source does _not_ have the region field which may be treated as a feature request, but it's easy to work around that as @kwilczynski mentioned already:

provider "aws" {
    region = "desired-region"
    alias = "regional"
}

data "aws_s3_bucket_object" "aws-remote-state" {
    provider = "aws.regional"
    bucket = "terraform-remote-state"
    key = "/aws/aws.tfstate"
}

[Jonnymcc]

@radeksimko Thanks, this is what I was looking for. I was mislead by the docs when looking up data sources after seeing this deprecation warning,
* terraform_remote_state.aws-tve-remote-state: using terraform_remote_state as a resource is deprecated; consider using the data source instead.
Maybe the deprecation warning could be more explicit,
* terraform_remote_state.aws-tve-remote-state: using terraform_remote_state as a resource is deprecated; consider using the terraform_remote_state data source instead.

While reading about data sources I was under the impression that terraform_remote_state was entirely replaced by provider specific data sources.

[kwilczynski]

@Jonnymcc hello there! I was wondering, have you resolve the issue you were having? Do you still need help?

[Jonnymcc]

Nope, I resolved the issue I was having. Thanks!

[cdimitroulas]

I am still having a similar issue with Terraform v0.11.2

I have this code which references the remote state

data "terraform_remote_state" "state" {
  backend = "s3"
  config = {
    name = "${var.terraform_state_bucket}"
    key = "global/s3/${var.environment}/terraform.tfstate"
    region = "${var.region}"
  }
}

data "aws_subnet_ids" "subnets" {
  vpc_id = "${data.terraform_remote_state.state.vpc.id}"
}

module "security_groups" {
  source = "github.com/segmentio/stack/security-groups"
  name = "hs"
  vpc_id = "${data.terraform_remote_state.state.vpc.id}"
  environment = "${var.environment}"
  cidr = "0.0.0.0/0"
}

module "api" {
  source = "../../modules/api"

  environment = "${var.environment}"
  ami_id = "ami-acb020d5"
  image = "hs-api-${var.environment}"
  log_bucket = "${var.load_balancer_logs_bucket}"
  vpc_id = "${data.terraform_remote_state.state.vpc.id}"
  subnet_ids = "${data.aws_subnet_ids.subnets.ids}"
  instance_type = "t2.micro"
  load_balancer_security_groups = ["${module.security_groups.external_elb}"]
}

And I get the following error:

Error: Error refreshing state: 1 error(s) occurred:

* module.api.data.terraform_remote_state.state: 1 error(s) occurred:

* module.api.data.terraform_remote_state.state: data.terraform_remote_state.state: BucketRegionError: incorrect region, the bucket is not in 'eu-west-1' region
        status code: 301, request id: , host id:

My backend initialisation looks like this in another file:

/**
 * State remote storage
 */
terraform {
  backend "s3" {
    bucket = "hs-terraform-dev"
    key = "global/s3/staging/terraform.tfstate"
    region = "eu-west-1"
    encrypt = true
    profile = "dev"
  }
}

/**
 * Provider settings
 */
provider "aws" {
  region = "${var.region}"
}

Any advice would be much appreciated :)

[boeboe]

I had a similar issue and discovered that my .terraform/abc.tfstate file was incorrect (probably from a previous run). Removing the .terraform folder completely solved the issue.

The original version was pointing to another bucket in another region (copy paste from another project in another region... facepalm), but updating the back-end configuration did not fix this. Removing the local state file was the only solution.

[Jonnymcc]

@cdimitroulas Is your terraform_remote_state datasource bucket in eu-west-1? Did you try creating an aliased provider like suggested by @radeksimko?

[cdimitroulas]

I did not try the alias suggestion - right now I am not referencing the state bucket in the same way so I have avoided the problem.
If I come across this again I will make sure to try that though, thanks :+1:

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.