Terragrunt: Terragrunt Remote Backend S3 "Backend reinitialization required"

[bobhenkel]

I have setup my terraform as the readme described. I'm making use of remote S3 backend. I keep getting this error message and nothing is created.

> terraform version                                                                                                                                                       
Terraform v0.9.4
> terragrunt -version                                                                                                                                                     
terragrunt version v0.12.16

When I do a terragrunt apply I get this:

Backend reinitialization required. Please run "terraform init".
Reason: Initial configuration of the requested backend "s3"

The "backend" is the interface that Terraform uses to store state,
perform operations, etc. If this message is showing up, it means that the
Terraform configuration you're using is using a custom configuration for
the Terraform backend.

Changes to backend configurations require reinitialization. This allows
Terraform to setup the new configuration, copy existing state, etc. This is
only done during "terraform init". Please run that command now then try again.

If the change reason above is incorrect, please verify your configuration
hasn't changed and try again. At this point, no changes to your existing
configuration or state have been made.

I have this at the top of my main.tf file.

terraform {
  # The configuration for this backend will be filled in by Terragrunt
  backend "s3" {}
}

I have this is my root terraform.tfvars file in the modules directory.

terragrunt = {
  remote_state {
    backend = "s3"
    config {
      bucket  = "removed-terraform-state"
      key     = "${path_relative_to_include()}/terraform.tfstate"
      region  = "us-west-2"
      encrypt = true
      # Tell Terraform to do locking using DynamoDB. Terragrunt will automatically create this table for you if
      # it doesn't already exist.
      lock_table = "bob-terraform-lock-table"
    }
  }
}

[bobhenkel]

Ok so one thing of interest that I did was this to point to my local module folder

terragrunt = {
  terraform {
    source = "../../../modules/vpc/"
  }
}

I noticed this in my terragrunt apply output.

[terragrunt] 2017/05/08 15:36:08 WARNING: no double-slash (//) found in source URL /Users/bob/git-repos/github/bob/aws-framework/modules/vpc. Relative paths in downloaded Terraform code may not work.

I'll see if I don't point locally if things start working.

[bobhenkel]

That got rid of the warning, but still getting init error.

> terragrunt apply                                                                                                                                                        16:04:48
[terragrunt] [/Users/bhenkel/git-repos/github/peoplenet/pn-aws-streamingframework/dev-bhenkel2/fabric/vpc] 2017/05/08 16:05:07 Running command: terraform --version
[terragrunt] 2017/05/08 16:05:07 Reading Terragrunt config file at /Users/bhenkel/git-repos/github/peoplenet/pn-aws-streamingframework/dev-bhenkel2/fabric/vpc/terraform.tfvars
[terragrunt] 2017/05/08 16:05:07 Cleaning up existing *.tf files in /var/folders/5g/cwn11y_s6_v168gpxq82qhgr0000gp/T/terragrunt-download/FcRIXYY4Fl9PeJpplyuueZ-acoA/VelXBNiARgFwhirkJjOJ6ZIzaqQ
[terragrunt] 2017/05/08 16:05:07 Downloading Terraform configurations from file:///Users/bhenkel/git-repos/github/peoplenet/pn-aws-streamingframework/modules into /var/folders/5g/cwn11y_s6_v168gpxq82qhgr0000gp/T/terragrunt-download/FcRIXYY4Fl9PeJpplyuueZ-acoA/VelXBNiARgFwhirkJjOJ6ZIzaqQ
[terragrunt] 2017/05/08 16:05:07 Running command: terraform init -backend=false -get=false file:///Users/bhenkel/git-repos/github/peoplenet/pn-aws-streamingframework/modules /var/folders/5g/cwn11y_s6_v168gpxq82qhgr0000gp/T/terragrunt-download/FcRIXYY4Fl9PeJpplyuueZ-acoA/VelXBNiARgFwhirkJjOJ6ZIzaqQ
Initializing configuration from: "file:///Users/bhenkel/git-repos/github/peoplenet/pn-aws-streamingframework/modules"...
Terraform initialized in an empty directory!

The directory has no Terraform configuration files. You may begin working
with Terraform immediately by creating Terraform configuration files.
[terragrunt] 2017/05/08 16:05:08 Copying files from /Users/bhenkel/git-repos/github/peoplenet/pn-aws-streamingframework/dev-bhenkel2/fabric/vpc into /var/folders/5g/cwn11y_s6_v168gpxq82qhgr0000gp/T/terragrunt-download/FcRIXYY4Fl9PeJpplyuueZ-acoA/VelXBNiARgFwhirkJjOJ6ZIzaqQ/vpc
[terragrunt] 2017/05/08 16:05:08 Setting working directory to /var/folders/5g/cwn11y_s6_v168gpxq82qhgr0000gp/T/terragrunt-download/FcRIXYY4Fl9PeJpplyuueZ-acoA/VelXBNiARgFwhirkJjOJ6ZIzaqQ/vpc
[terragrunt] 2017/05/08 16:05:08 Running command: terraform apply
Backend reinitialization required. Please run "terraform init".
Reason: Initial configuration of the requested backend "s3"

The "backend" is the interface that Terraform uses to store state,
perform operations, etc. If this message is showing up, it means that the
Terraform configuration you're using is using a custom configuration for
the Terraform backend.

Changes to backend configurations require reinitialization. This allows
Terraform to setup the new configuration, copy existing state, etc. This is
only done during "terraform init". Please run that command now then try again.

If the change reason above is incorrect, please verify your configuration
hasn't changed and try again. At this point, no changes to your existing
configuration or state have been made.

Failed to load backend: Initialization required. Please see the error message above.
[terragrunt] 2017/05/08 16:05:08 exit status 1

[dpetzel]

Is it possible you ran the command once before adding the backend configuration? Do you have a .terraform directory? If so are you able to delete it and try again?

[bobhenkel]

Perhaps I've fallen off the rails, but I don't even get a .terraform directory. I'm going back through the readme to what I don't understand. I tried to follow the DRY examples up to this point.

I have been a user of vanilla terraform for over 2 years.

Thanks for the help!

[dpetzel]

Not having it is actually good in this case. I think you may also want to check the tmp directory though to. In your console output it should print out something about copying files to temp. Look for .terraform there (or just blow away the entire directory).

[brikis98]

What does your folder structure look like? Could you post the full log output?

[josh-padnick]

I've actually run into this exact issue myself, and I think I resolved it by manually deleting the temp folder entirely and then re-running terragrunt apply. I believe there is a bug where Terragrunt does not handle changes to your terraform.tfvars file correctly. Can you confirm that Terragrunt started to act up after you changed your terraform.tfvars file?

Here's how I deleted the temp folder:

1. Find the temp folder in log output. For example, here's the relevant snippet of your log output:

   The directory has no Terraform configuration files. You may begin working
   with Terraform immediately by creating Terraform configuration files.
   [terragrunt] 2017/05/08 16:05:08 Copying files from /Users/bhenkel/git-repos/github/peoplenet/pn-aws-streamingframework/dev-bhenkel2/fabric/vpc into /var/folders/5g/cwn11y_s6_v168gpxq82qhgr0000gp/T/terragrunt-download/FcRIXYY4Fl9PeJpplyuueZ-acoA/VelXBNiARgFwhirkJjOJ6ZIzaqQ/vpc
   [terragrunt] 2017/05/08 16:05:08 Setting working directory to /var/folders/5g/cwn11y_s6_v168gpxq82qhgr0000gp/T/terragrunt-download/FcRIXYY4Fl9PeJpplyuueZ-acoA/VelXBNiARgFwhirkJjOJ6ZIzaqQ/vpc
   [terragrunt] 2017/05/08 16:05:08 Running command: terraform apply
   Backend reinitialization required. Please run "terraform init".
   Reason: Initial configuration of the requested backend "s3"
   

2. I see the temporary folder is /var/folders/5g/cwn11y_s6_v168gpxq82qhgr0000gp/T/terragrunt-download/FcRIXYY4Fl9PeJpplyuueZ-acoA/VelXBNiARgFwhirkJjOJ6ZIzaqQ/vpc so I'll delete it with:

   rm -Rf "/var/folders/5g/cwn11y_s6_v168gpxq82qhgr0000gp/T/terragrunt-download/FcRIXYY4Fl9PeJpplyuueZ-acoA/VelXBNiARgFwhirkJjOJ6ZIzaqQ/vpc"
   

Note that in some cases I had to delete multiple temp folders. In one case, I believe I fixed this issue by clearing out my entire /var/folders directory, however that action is considered risky business.

I realize this fix suffers from all kinds of issues, so if you can confirm that this issue happened after you ran terragrunt apply, updated your terraform.tfvars file, and re-ran terragrunt apply, then we have a reproducible bug for which we can open a GitHub issue.

[brikis98]

You can have Terragrunt delete the folder for you by just adding the --terragrunt-source-update flag.

[josh-padnick]

@brikis98 Is there any reason why Terragrunt shouldn't update the source folder on every run? Perhaps that's the more sensible default with an option to leave the existing source folder in place.

[tomstockton]

I always run it with --terragrunt-source-update so making this the default behaviour makes more sense to me as I always want the 'freshest' code from the source module.

[dpetzel]

FWIW, this is also the default for me as well, to the extent I setup an alias so I didn't have to type it every time

[brikis98]

The reason --terragrunt-source-update isn't the default is that it blows away the whole temp directory for your code. As a result, for every single Terraform code you run, you would have to wait for Terragrunt to:

1. Download your source code.
2. Run terraform get to download your module code.
3. Run terraform init to configure your remote state.

This can add 15 - 75 seconds for every single command. I found that intolerable, so Terragrunt by default keeps the directory around. Note that:

• If the source parameter points to a local file path, Terragrunt re-copies your source code to the temp folder every time to support rapid iterative development.
• If the source parameter changes, Terragrunt redownloads the source code to the temp folder.
• If your remote state configuration in terraform.tfvars changes, Terragrunt reruns terraform init.

[josh-padnick]

@brikis98 Thanks for the clarification, and this makes sense. The one use case that isn't listed above that bit me is when I'm doing development on the module pointed to by the source property. If I make a change to the module without changing, say, the branch, and then run terragrunt get -update, behavior can be unexpected.

I suppose a solution would be to compute a checksum based on the remote module and only download it if necessary, but this would require you to download the module code in the first place, so it's probably not worth it.

Perhaps the best option here is to update docs to call out --terragrunt-source-update more explicitly.

[brikis98]

The one use case that isn't listed above that bit me is when I'm doing development on the module pointed to by the source property. If I make a change to the module without changing, say, the branch, and then run terragrunt get -update, behavior can be unexpected.

If you're doing local dev on a module, then you shouldn't be relying on versions or tags, but instead you should be overriding the source parameter by passing the --terragrunt-source flag with a local file path. Since it's a local file path, Terragrunt will copy the latest code you have locally every time you run it.

[josh-padnick]

Ah, that's a good call. Problem resolved! Looks like this is already in docs so hopefully this GitHub Issue highlights the solution.

@bobhenkel Does any of this discussion resolve your original problem?

[rkr-kununu]

I've gotten bitten by this as well, but my use case is slightly different.

We've got a CI system, that'll provision a new deployment for each development-branch. They are all using the same Terraform/Terragrunt code... and we're using Terragrunts get_env() functionality to "switch" between deployments (ie: TF_VAR_project).

The problem is that since we're always changing the S3 backend, Terraform prompts us with "Would you like to copy the state from "s3"?" especially for new development-branches. I've mitigated this by using --terragrunt-source-update, but an alternative solution would be to have Terragrunt's internal terraform-init honor any "extra_arguments". Then I could do something like:

terragrunt = {
  ...
  terraform = {
    ...
    extra_arguments "supress_prompt" {
       arguments = [ "-reconfigure" ]
       commands = [ "init" ]
    }
  }
}

[brikis98]

OK, there are several issues listed here, all of which, I believe, have been fixed:

• #301 should ensure Terragrunt properly downloads code into the tmp folder
• Use --terragrunt-source-update to clean up the tmp folder if anything goes wrong
• Use --terragrunt-source XXX to point to a local checkout at XXX
• #261 allows you to use extra_arguments with the init command