Terragrunt: How Terragrunt Will Handle Terraform 0.9.0 Compatibility

[josh-padnick]

Hello Terragrunt Community,

As of today, Terragrunt is not compatible with Terraform 0.9.x. We want to fix that. This GitHub Issue will outline our current thinking on how we plan to do that.

• Handling Remote State in Terraform 0.9: Unfortunately, Terraform 0.9.x removed the terraform remote config commands, which is why Terragrunt is not usable with Terraform 0.9.x. To resolve this, we’ve been exploring the best way to configure remote state in Terraform 0.9.x.

  Ideally, Terragrunt could simply retire this feature altogether, as Terraform 0.9 allows you to define your remote state configuration in your .tf files. Unfortunately, Terraform does not support all the features Terragrunt had, such as path_relative_to_include(), manual lock controls, and automatic retry when a lock cannot be obtained.

  Our current thinking is to move to Terraform’s remote state configuration, but with variables where you’d like to use Terragrunt features:

  # In some .tf file
  terraform {
    backend "s3" {
       bucket = "my-bucket"
       key    = "${var.remote_state_key}"
       region = "us-east-1"
    }
  }
  

  You can then configure Terragrunt to pass in the variables, using the interpolation helpers as before:

  # In terraform.tfvars or .terragrunt
  terragrunt = {
    terraform {
       extra_arguments "remote_state" {
          arguments = [
              "-var", "remote_state_key=${path_relative_to_include()}",
          ]
          commands = [
             "apply",
             "plan",
             "import",
             "push",
             "refresh"
          ]
        }
     }
  }
  

• User Migration Strategy: Many of our users (and Gruntwork clients) have multiple folders configured to use Terragrunt. We plan to explicitly think through a recommended migration strategy. This could be a simple recommendation to manually make updates, or some kind of automation. We’re still thinking through this.

• Handling Retries When a Lock Cannot be Obtained: When Terragrunt cannot obtain a lock, it keeps trying to get one until it times out. Currently, Terraform 0.9.x fails immediately if a lock cannot be obtained. Some CI scripts may depend on having an automatic retry, so we intend to wrap the native Terraform backends code with our own lock-retry mechanism.

Timeline

In an ideal world, we’d begin these changes right away, but we are currently at capacity with other work. We will start making fixes to Terragrunt as soon as we can, but that may not be for another week or two. If anyone wishes to submit a PR, we would be grateful!

Feedback

Feedback is always welcome. Please feel free to leave additional comments in this thread. Thanks for using Terragrunt, and for your patience while we add support for Terraform 0.9.x!

[antonbabenko]

Hi guys!

I have few comments on these (in reversed order).

First, handling retries when a lock cannot be obtained:
I think Terraform should exit with more specific exit-codes if it can't obtain lock (as it is already possible to do using plan -detailed-exitcode). This way CI scripts can be improved to implement retry logic, if they need to.

User Migration Strategy:
I think it is not very complicated task to update multiple folders, so I don't think automation is very much necessary there. At least I don't feel the pain to consider any automation for this.

Handling Remote State in Terraform 0.9:
From what I experience so far the missing feature in Terraform 0.9 related to remote states is that it is not possible to use interpolation helpers and I think if Terragrunt can implement it it would be great. And the proposed solution seems good to me.

[brikis98]

@antonbabenko Thanks for the feedback! Good to know we're on the right track :)

[tamsky]

Hi @josh-padnick & Jim, here's my interim hack that allows me to use terraform 0.9.x with terragrunt 0.11.0:

With it, I'm able to continue to use my existing remote state declarations.

terragrunt { remote_state { config { key = "${path_relative_to_include()}/terraform.tfstate" ...

https://gist.github.com/tamsky/8a685a971170031042d421afb276fc50

TL;DR: a script intercepts terragrunt's calls to terraform remote and converts them to terraform init + proper args.

[brikis98]

@tamsky Ah, I didn't realize the init command supported that. That may be a much simpler way to implement this. Thank you!

[josh-padnick]

After playing around with Terraform 0.9 and its backends, I wanted to document some missing features that Terragrunt currently has so we can explicitly decide how to address these:

• Terraform does not auto-create the remote state S3 Bucket
• Terraform does not auto-create the DynamoDB lock table
• It's not possible to specify a separate region for the DynamoDB lock table and S3 Bucket.

[mtb-xt]

@tamsky thanks for your workaround - could you tell how do you configure terraform's remote state in the .tf file and in terraform.tfvars file?
In my case, your workaround for some reason makes terraform think it's backend configuration changes every time, and it runs init every time, forcing terraform to ask me questions whether I want to overwrite my state or not.

I had to remove backend config from terragrunt and move it into terraforms main.tf, and only use terragrunt for locking.

[tmclaugh]

:+1: to being able to use interpolation with remote state configuration. The lack of interpolation was a rude shock this afternoon. Defining different state backends via terragrunt, stored in different .tfvars files, was a feature I relied on. I wouldn't care as much if S3 didn't have a single global namespace and needed to keep state bucket names different across accounts which is how we separate environments.

[codekoala]

Terraform 0.9.3 was just cut with some features we've been eagerly awaiting. Terragrunt is too great to give up right now though.

I noticed https://github.com/gruntwork-io/terragrunt/pull/160, but it seems like the discussion has kind of stalled over there. How are we looking on the 0.9.x remote state magic?

[mitchellh]

Hi Terragrunt users. :)

I want to first say that we're sorry we broke Terragrunt functionality. We really like Terragrunt and didn't purposely seek in any way to break it. The motivations for removing terraform remote config were entirely because when we designed the "remote backends" concept (which is far more than to support locking) remote config didn't fit into the workflow in addition to remote config having many other issues users were hitting.

In hindsight, we probably should've deprecated it because the functionality still exists and is exactly replicated with init now. Sorry about that. I'm not sure that alone would've kept Terragrunt working... but even for non Terragrunt users it caused some strife.

We've made some improvements since this issue was first posted (and probably more in the future so only take this as a point-in-time snapshot for 0.9.3):

• -lock-timeout allows locks to be retried for a period of time. (0.9.3)
• init accepts key/value configs on the CLI just like remote config used to. This makes init a pure superset of what remote config used to do. (0.9.1)

If there are things we can do to help you we'll do them to the extent they make sense for the project (such as the above). One area that is difficult for us is variable interpolation in backend configurations. Its something people want, and its something that is technically not impossible, but its not super simple either. I'm sure we'll support it but it may not be very soon. We're recommending people parameterize using init scripting (similar to remote config).

From what I've heard there are a handful of features that we don't support that Terragrunt supported. We'd be more interested in the short term of helping get Terragrunt working rather than try to replace those features. Let us know how we can help.

[brikis98]

@mitchellh Thanks for the info! We're obviously huge fans of Terraform and have been using Terragrunt as a way to quickly try out relatively major extra functionality. Would be interesting to brainstorm how that can be done in a way that is more compatible with future versions of Terraform.

-lock-timeout allows locks to be retried for a period of time. (0.9.3)

Nice!

init accepts key/value configs on the CLI just like remote config used to. This makes init a pure superset of what remote config used to do. (0.9.1)

Excellent. Our current thinking is to piggyback on the init command to make Terragrunt work with Terraform 0.9, so this should do the trick.

From what I've heard there are a handful of features that we don't support that Terragrunt supported. We'd be more interested in the short term of helping get Terragrunt working rather than try to replace those features. Let us know how we can help.

The major things we'd like to do in the near term are:

1. Fix basic compatibility by switching the use of the remote config command to the init command as discussed in this issue.
2. Continue to support automatically creating the S3 bucket (for remote state) and the DynamoDB (for locking) if either one doesn't exist already. Not sure if this is best handled by keeping the relevant config within the Terragrunt config files or trying to parse it out of Terraform itself...
3. Support a ${path.module} style interpolation so that paths work properly for remote Terraform configs downloaded into a temporary folder, as discussed in #143. I think this is straightforward to do and doesn't need any changes in Terraform.
4. Considering a way to read Terraform variables in the terragrunt = { ... } configuration, as discussed in #132. This one is potentially messy, as it's not obvious all values are available (e.g. default values) and it is getting progressively weird to use interpolation syntax in some parts of the .tfvars file and not others.

[brikis98]

Update: I finally made time to put together a PR that adds compatibility for Terraform 0.9! You can find it here: https://github.com/gruntwork-io/terragrunt/pull/167. Feedback is very welcome. Thank you everyone for your patience.

[brikis98]

I also just put up a fix for #143 which adds a helper similar to ${path.module} that will make it possible to use relative file paths in Terragrunt configurations: https://github.com/gruntwork-io/terragrunt/pull/168.