Terraform: Terraform Remote state organisation

[linuxbsdfreak]

Hi,

Terraform version: v0.9.11

I have seen https://github.com/hashicorp/terraform/issues/15131 and found interpolation is not possible when performing terraform init. Is there any other way to accomplish the same? So that the state files are organised correctly for different projects. Ideally i would like to have the state files in seperate buckets as defined below.

tf-state-bucket-us-east-1/environment/non_immutable/terraform.state
tf-state-bucket-us-east-1/environment//terraform.state
/environment//terraform.state

tf-state-bucket-us-east-2/environment/non_immutable/terraform.state
tf-state-bucket-us-east-2/environment//terraform.state
/environment//terraform.state

The non_immutable will have state info about VPC.etc. In short resource which do not change once created.

Is this possible instead of configuring a single bucket in the config? Is yes can someone share the process on how to accomplish it cleanly.

Regards,
Kevin

[nbering]

Two things that might help you. For your "immutable" stuff, use Remote State Data Provider to reference another state in a read-only manner to be shared between other states.

If you want multiple "instances" of the same config without shared state, use workspaces (formerly env). They allow you use use the same configuration but deploy multiple times without sharing state. They do have to be in the same bucket, though.

[nbering]

By the way... if your concern is about availability of your state during a regional S3 outage, check out bucket mirroring. Has nothing to do with Terraform, but it would allow you to access your state info from another region during an outage. That's the only reason I can think of to use multiple buckets aside from segregating user access (which can also be done with IAM policies and bucket prefixes).

[linuxbsdfreak]

Hi @nbering

Thanks for the info. I have few comments

1. I am able to use Remote State Data provider for refrencing the values and it works as expected.
2. I have used the workspaces features and it works good and its a nice feature. However for each env i have to keep a workspace and for each app. Which is quite a pain eg
   app1useast1dev
   app1useast1prd

My requirements are as follows.

1. Create isolated state files based on the structure above to reduce the blast radius in case of issues while applying terraform.
2. Create independant s3 buckets for storing the state files so that i can update a tfstate files without worrying about the other regions. In short not putting all the eggs in the same basket OR each region is completely isolated with respect to the state of the state files

Regards,
Kevin

[nbering]

@linuxbsdfreak If you think switching environments is a pain now, keeping separate state in separate buckets is going to be even more work.

I had similar ideas when I first started working with Terraform, but realistically swimming against the current wasn't worth the effort. Terraform workspaces effectively provides the isolation you're looking for because it's a completely separate state file, just organized in a way that Terraform can use to provide some help in managing the multiple states.

I'm not a team member so I can't comment on whether multiple backend configurations in the same project will ever be supported, but at this point it is not. If it ever does arrive, I imagine it will be to support refactoring state with terraform mv. So, all I can suggest for your second requirement is writing your configuration as a submodule and then sharing it between multiple "root" projects.

./root1/main.tf

terraform {
  backend "s3" {
    bucket = "mybucket1"
    key    = "path/to/my/key"
    region = "us-east-1"
  }
}

module "project" {
  source = "../shared"
  somevar = "foo"
}

./root2/main.tf

terraform {
  backend "s3" {
    bucket = "mybucket2"
    key    = "path/to/my/key"
    region = "us-east-2"
  }
}

module "project" {
  source = "../shared"
  somevar = "bar"
}

./shared/main.tf

variable "somevar" {}
...

Use like this:

$ cd root1

$ terraform init

$ terraform plan

$ terraform apply

$ cd ../root2

$ terraform init

$ terraform plan

$ terraform apply

As I've said before though, doing this type of thing is a lot of work for something that is a mostly theoretical gain. One can rarely eliminate complexity with software, you just end up moving it around. If a feature were implemented to support multiple remote state buckets, the complexity would be handled by Terraform at the cost of maintaining another feature. If you do something like the above, you take on the complexity yourself. If you use workspaces, no new feature is written, but you compromise on not being able to use multiple buckets.

[apparentlymart]

Hi @linuxbsdfreak, @nbering,

It is indeed true that right now the "smooth path" is to use the same backend configuration for all environments and allow Terraform to manage the different states via _workspaces_ (or "state environments", as they were called in 0.9).

Having multiple root modules that all just reference a shared child module is another path that allows more flexibility, at the expense of a slightly-tricker workflow since Terraform itself can't help switch between the environments.

One longer-term idea we've talked about is to have a special "multiplex backend" that acts as a wrapper around a number of different backends, making them behave as one. We don't have any immediate plans to implement that since it's got a lot of complexities to navigate around how it would manage credentials, etc, and so for the moment we're recommending either of the above strategies.

For more complex scenarios, it's possible to add additional arguments to the terraform init command to override certain elements of the backend configuration; we call this "partial configuration", since only part of the necessary config actually lives in the configuration files. This is not an ideal workflow since it requires careful management of which settings are currently active when running specific commands, but several users have reported success with this when running Terraform in automation such that it's easier to ensure that the right steps are always followed to keep things consistent.

As part of the 0.10 release we updated some documentation in this area that hopefully clarifies somewhat the different options here:

• terraform init: Backend Initialization covers the init options for backend configuration, and links to some more detailed docs on the partial configuration idea.
• Running Terraform in Automation is a broader guide that covers a few different aspects of running Terraform from inside wrapper scripts and automation tools, which has a section on Multi-environment Deployment.

Neither of these resources specifically discusses @nbering's suggestion of multiple top-level configs and a shared child module, since of course that's not something that Terraform _itself_ manages, but based on this discussion here I would like to find somewhere to mention that in the documentation, since it _is_ a good pattern particularly when there are also some non-trivial structural differences between environments.

[hashibot]

Hello again!

We didn't hear back from you, so I'm going to close this in the hope that a previous response gave you the information you needed. If not, please do feel free to re-open this and leave another comment with the information my human friends requested above. Thanks!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.