Terraform: offer to create missing bucket and lock_table for s3 backend

[dmrzzz]

Terraform Version

Terraform v0.9.0

Affected Resource(s)

• s3 backend

Terraform Configuration Files

terraform {
  required_version = ">= 0.9"

  backend "s3" {
    region = "us-east-2"
    lock_table = "table-xi0s3p59prd9mubqsinc"
    encrypt = "true"
    bucket = "bucket-xi0s3p59prd9mubqsinc"
    key = "test/terraform.tfstate" 
  }
}

provider "aws" {
  region = "us-east-2"
}

resource "aws_sns_topic" "foo" {
    name = "foo-topic"
}

Feature Request

I'm very excited about the new backend configuration in Terraform 0.9, but it's inconvenient that I have to manually create both an S3 bucket and a DynamoDB table before I can get started. (If I enjoyed manually creating resources, I wouldn't need Terraform!)

Terragrunt (with Terraform <= 0.8.x) handles this extremely conveniently by:

1. Prompting the user to automatically create the S3 bucket if it doesn't exist.
2. If user says yes, creating the bucket and enabling versioning.
3. Automatically creating the DynamoDB lock table if it doesn't exist.

I would love to see Terraform 0.9+ do the same (perhaps during terraform init?)

Here's how the equivalent configuration behaves using Terragrunt, starting from a blank slate:

$ terragrunt apply
[terragrunt] 2017/03/16 14:57:20 Reading Terragrunt config file at /home/dmrz/aws/dmrz-test/bar/terraform.tfvars
[terragrunt] 2017/03/16 14:57:20 Initializing remote state for the s3 backend
[terragrunt]  Remote state S3 bucket bucket-xi0s3p59prd9mubqsinc does not exist or you don't have permissions to access it. Would you like Terragrunt to create it? (y/n) y
[terragrunt] 2017/03/16 14:57:22 Creating S3 bucket bucket-xi0s3p59prd9mubqsinc
[terragrunt] 2017/03/16 14:57:22 S3 bucket bucket-xi0s3p59prd9mubqsinc created.
[terragrunt] 2017/03/16 14:57:22 Enabling versioning on S3 bucket bucket-xi0s3p59prd9mubqsinc
[terragrunt] 2017/03/16 14:57:23 Configuring remote state for the s3 backend
[terragrunt] 2017/03/16 14:57:23 Running command: /usr/local/libexec/terraform-0.8.7 remote config -backend s3 -backend-config=encrypt=true -backend-config=bucket=bucket-xi0s3p59prd9mubqsinc -backend-config=key=test/terraform.tfstate -backend-config=region=us-east-2
Initialized blank state with remote state enabled!
Remote state configured and pulled.
[terragrunt] 2017/03/16 14:57:24 Attempting to acquire lock for state file state in DynamoDB
[terragrunt] 2017/03/16 14:57:25 Lock table table-xi0s3p59prd9mubqsinc does not exist in DynamoDB. Will need to create it just this first time.
[terragrunt] 2017/03/16 14:57:25 Creating table table-xi0s3p59prd9mubqsinc in DynamoDB
[terragrunt] 2017/03/16 14:57:25 Table table-xi0s3p59prd9mubqsinc is not yet in active state. Will check again after 10s.
[terragrunt] 2017/03/16 14:57:35 Success! Table table-xi0s3p59prd9mubqsinc is now in active state.
[terragrunt] 2017/03/16 14:57:35 Attempting to create lock item for state file state in DynamoDB table table-xi0s3p59prd9mubqsinc
[terragrunt] 2017/03/16 14:57:35 Lock acquired!
[terragrunt] 2017/03/16 14:57:35 Running command: /usr/local/libexec/terraform-0.8.7 apply
aws_sns_topic.foo: Creating...
  arn:    "" => "<computed>"
  name:   "" => "foo-topic"
  policy: "" => "<computed>"
aws_sns_topic.foo: Creation complete

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

The state of your infrastructure has been saved to the path
below. This state is required to modify and destroy your
infrastructure, so keep it safe. To inspect the complete state
use the `terraform show` command.

State path: .terraform/terraform.tfstate
[terragrunt] 2017/03/16 14:57:39 Attempting to release lock for state file state in DynamoDB
[terragrunt] 2017/03/16 14:57:39 Lock released!

[adilnaimi]

@dmrzzz

just curious about the terraform {} block, as per the documentation https://www.terraform.io/docs/configuration/terraform.html

The terraform block configures the behavior of Terraform itself.
The currently only allowed configuration within this block is required_version. 

assuming the configuration block works for you (regardless the manual bucket and table creation) I think the documentation needs to be updated to include the backend configuration.

[dmrzzz]

@adilnaimi I agree the documentation page you reference is out of date, but that has nothing to do with the enhancement I am requesting/suggesting in this issue. Perhaps you should open a separate issue for the documentation page?

[adilnaimi]

@dmrzzz documentation out of date issue created https://github.com/hashicorp/terraform/issues/13064

[jyoungs]

This is a bit hacky, but as a workaround-- if you run apply without the backend settings (just comment them out temporarily) and specify the bucket/table in your TF, it should run and create them (along with everything else)-- then you can uncomment the settings and it will transition your state into the now existing bucket pretty seamlessly.

[stavxyz]

In the same pattern @jyoungs mentioned, I put together a terraform module along with a guide for dealing with particular issue. It is a work in progress, but supports most AWS Backend use cases and includes instructions for bootstrapping your project.

https://github.com/samstav/terraform-aws-backend

[dmrzzz]

@jyoungs and @samstav That's clever, but not really what I'm looking for.

To clarify my feature request, I specifically would like to avoid managing this S3 bucket as a resource "aws_s3_bucket" within my config, because I never want Terraform to destroy it.

That's in part because I like to keep all of my Terraform state for many many different configs in just one S3 bucket per AWS account (carefully making sure the keys are unique so they don't step on each other), which means that if any one of those configs decided it was king of the bucket and destroyed the bucket, all the other ones would suddenly be up the proverbial creek.

My current solution is to create the state-management resources manually with

aws s3api create-bucket --create-bucket-configuration LocationConstraint=us-east-2 \
  --bucket $BUCKET
aws s3api put-bucket-versioning --versioning-configuration Status=Enabled \
  --bucket $BUCKET
aws dynamodb create-table --region us-east-2 --table-name $TABLE \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1

and that accomplishes precisely what I need. Since I only do it once per AWS account, it's not even that hard. I just want Terraform to do it for me because I'm lazy and writing a separate tool for this feels like massive overkill. :)

[sandipdivekar1]

Any update on this? In the latest docs bucket_name is still required field.
https://www.terraform.io/docs/backends/types/s3.html