Terragrunt: Use ${var.} in terragrunt config

[krjensen]

Hi,

I have a lot of environments that all havs it's own .tfvars file for setting up base infrastructure vars like:

I define my environment in a variable called environment_name and i dont really want to define that more that one time, but my problem is that I want to reuse that variable in the terragrunt config as a path to my state in consul

environment_name = "dev"
environment_id = "01"
external_network = "external-vlan1"
user_data_linux = "../../Environments/global/user_data_linux_dev.yaml"
user_data_windows = "../../Environments/global/user_data_windows_dev.yaml"

terragrunt = {
  remote_state {
    backend = "consul"
    config {
    address = "my-consul.dev.local:8500"
        path = "terraform/state/**${var.environment_name }**"
    datacenter = "dus2-shared"
    }
  }
}

[brikis98]

Sharing variables between Terragrunt and Terraform has come up before, but to be honest, I'm not sure we know the best way to handle it. The problem is that Terraform can load variables from many places:

1. default values in the variable definition.
2. TF_VAR_XXX environment variables.
3. terraform.tfvars, if it exists.
4. -var argument.
5. -var-file argument.

We could certainly read and parse all the variables in the .tfvars file you use to configure Terragrunt, but that would only give us access to a small subset of the variables Terraform can access. So var.environment_name might work, but var.foo would not if foo was set via some other means (e.g. env var or default). I worry that inconsistency would get very confusing.

If someone has ideas on a clean solution to this, I'm open to ideas.

[JohnEmhoff]

I think I'm more interested in variables in the terragrunt file being available to terraform. The workaround is break them out and specify a couple-var-file arguments. At the very least, the documentation is a bit misleading here:

Terraform also uses terraform.tfvars as a place where you can set values for your variables, but so long as your Terraform code doesn't define any variables named terragrunt, Terraform will safely ignore this value.

This pretty strongly implies that non-terragrunt variables are passed to terraform, which doesn't seem to be the case in include-ed files.

[brikis98]

I think I'm more interested in variables in the terragrunt file being available to terraform.

Are you still putting your configs in .terragrunt or are you using a terraform.tfvars file? If it's the latter, then Terraform will read variables from that file. If you are using a .tfvars file with some other name (e.g. foo.tfvars), then you have to pass that to Terraform explicitly using the -var-file argument, as documented here.

[JohnEmhoff]

I'm putting variables in terraform.tfvars. You're correct that variables in
that file are being picked up, but not variables in my terraform.tfvars
file that is getting include-ed. If this isn't expected to be the case
I'll send over something that will reproduce it later this weekend.

On Feb 18, 2017 9:10 AM, "Yevgeniy Brikman" [e-mail]
wrote:

I think I'm more interested in variables in the terragrunt file being
available to terraform.

Are you still putting your configs in .terragrunt or are you using a
terraform.tfvars file? If it's the latter, then Terraform will read
variables from that file. If you are using a .tfvars file with some other
name (e.g. foo.tfvars), then you have to pass that to Terraform explicitly
using the -var-file argument, as documented here
https://github.com/gruntwork-io/terragrunt#config-file-search-paths.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/gruntwork-io/terragrunt/issues/132#issuecomment-280847886,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABnTd9jVxO15VfT5OcRRs9_ZZmo3tXrGks5rdvu8gaJpZM4MD8T0
.

[brikis98]

Ah, I see what you're saying. Yes, include just pulls in the terragrunt = { ... } block from the parent file, so it only affects Terragrunt and has no effect on Terraform at all. That's definitely confusing.

I'd welcome a PR to clarify the docs around this.

[JohnEmhoff]

Great, will do! Is the expected workaround to use a var file passed on the
command line? This works but is a bit awkward due to the precedence rules:
those will override variables in the child terraform.tfvars.

Anyway, great tool and thanks for writing and sharing it!

On Feb 18, 2017 9:22 AM, "Yevgeniy Brikman" [e-mail]
wrote:

Ah, I see what you're saying. Yes, include just pulls in the terragrunt =
{ ... } block from the parent file, so it only affects Terragrunt and has
no effect on Terraform at all. That's definitely confusing.

I'd welcome a PR to clarify the docs around this.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/gruntwork-io/terragrunt/issues/132#issuecomment-280848634,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABnTd7vo_RQLaFPLhASINOBO84ip4TGiks5rdv67gaJpZM4MD8T0
.

[brikis98]

Yes, you can pass the parent .tfvars file using the -var-file argument. Alternatively, you can tell Terragrunt to do that automatically using the extra_arguments configuration, as documented here.

[dhoer]

This approach is workable, but unintuitive to new users. It would be nice to be able to daisy chain terraform.tfvars together automatically. Maybe using find_in_parent_folders() function in each terraform.tfvars file found in parent directories? This would allow users to keep the env vars laid out in a way that make sense to their project.

Example:

terraform.tfvars
myenv/
  terraform.tfvars
  mymodule/
    terraform.tfvars

The env vars declared in root terraform.tfvars would merged with /myenv/terraform.tfvars and any repeated vars in the latter would overwrite the former vars and so on.

BTW, I want to thank you for putting together a great tool! Terragrunt allowed for easy simplification of an unmanageable terraform project quickly.

[brikis98]

Automatically ("magically") inheriting from parent .tfvars files is problematic:

1. It's error-prone: you may pick up random files you weren't expecting.
2. It's hard to reason about: you can't just read the config files themselves to understand the behavior; you also have to scan the entire hard drive.
3. It makes reproducible environments harder: you may have a file in a parent directory on your system that I don't have on mine.
4. It encourages more magic. Once you're inheriting automatically from parents, you immediately get requests like "well, I want to inherit from parent files... except in these cases..." or "in dev, we also want to (magically) inherit from this special extra file". This exacerbates all the problems above.

Therefore, I prefer to keep config explicit, such as adding -var-file arguments.

Using find_in_parent_folders() is an interesting idea that is a bit of a blend: it makes it explicit that you're pulling in other files, but doesn't specify exactly which file. I haven't tried it, but I suppose you could do something like:

terragrunt = {
  terraform {
    extra_arguments "parent-configs" {
      arguments = [
        "-var-file=${find_in_parent_folders()}"
      ]
      commands = [
        "apply",
        "plan",
        "import",
        "push",
        "refresh"
      ]
    }
  }
}

We may even want to extend find_in_parent_folders to accept an argument, so you can specify the file name to look for: e.g. find_in_parent_folders("secrets.tfvars").

[JohnEmhoff]

Is extra_arguments respected in files that have been include-ed? It doesn't seem to be doing anything, even when I add a garbage flag:

terraform = {
    extra_arguments "vars" {
      arguments = ["-bad-flag=asdfasaf"]
      commands = ["apply", "plan", "refresh", "output"]
    }
  }

[brikis98]

@JohnEmhoff That sounds like a potential bug. Could you file a separate issue for it?

[notbrain]

This is definitely not the can of worms :bug: I expected to find for wanting such a simple thing: that if a .tfvars variable assignment is made above a terragrunt {} block _in the same file_, to use whatever that value is _inside_ the terragrunt {} block so you don't have N various places to change the value in that file. Wouldn't the rules of precedence "just work" in that case? No other variable value would override whatever is local to the file?

Regarding -var-file technique described above: would this allow us to set up values inside a parent .tfvars file so that a terragrunt {} block like this would get the aws_region variable inserted in 2 places?

terragrunt {
  # lock stanza removed

  # Configure Terragrunt to automatically store tfstate files in an S3 bucket
  remote_state = {
    backend = "s3"
    config = {
      encrypt = "true"

      # aws_region = X, from explicit .tfvars file passed with -var-file?
      bucket = "terraform-state.${aws_region}"

      key = "${path_relative_to_include()}/terraform.tfstate"

      # (same here)
      region = "${aws_region}"
    }
  }
}

The other plan I thought of was to just use get_env() for all of the variable values that are shared between terragrunt and terraform, in which case I could set up a TF_VAR_aws_region style variable, then use "terraform-state.${get_env("TF_VAR_aws_region", "us-east-1")}".

But this was confusing, since I still needed to use get_env to get a TF_VAR_* that would otherwise be "automatically" picked up by terraform itself? It would at least allow us to use a uniform way to set up this "combo" terraform.tfvars with the terragrunt {} block in it.

# no longer needed if using TF_VAR_aws_region, downstream vars will pick it up
# aws_region = "us-east-1"
# other non-shared variables would still be ok here, e.g.:
aws_account_ids = ["0303930390"]

terragrunt {
  # lock stanza removed

  # Configure Terragrunt to automatically store tfstate files in an S3 bucket
  remote_state = {
    backend = "s3"
    config = {
      encrypt = "true"
      # only need these explicit lines in terragrunt block for any shared vars
      bucket = "haven-terraform-state.${get_env("TF_VAR_aws_region", "us-east-1")}"
      key = "${path_relative_to_include()}/terraform.tfstate"
      region = "${get_env("TF_VAR_aws_region", "us-east-1")}"
    }
  }
}

Am I missing anything?

[beanderson]

Using find_in_parent_folders() does not work when you are also specifying a different repo as the terraform source. find_in_parent_folders() returns a relative path, but since terraform runs in a tmp directory, it needs the absolute path in order to properly find the file.

terraform {
    source = "git::ssh://[email protected]/network.git//vpc?ref=${get_env("VPC_INFRA_VERSION", "master")}"

    extra_arguments "variables" {
      arguments = [
        "-var-file=${find_in_parent_folders()}"
      ]
      commands = [
        "apply",
        "destroy",
        "plan",
        "import",
        "push",
        "refresh"
      ]
    }
  }

Apply returns, error reading ../../variables.tfvars: no such file or directory

[dhoer]

I'm still finding extra_arguments "variables" approach hard as well. The following example works, but only because its using relative paths with // to pull in the entire directory:

  terraform {
    source = "../../../..//modules/app"

    extra_arguments "vars" {
      arguments = ["-var-file=../../env/dev/environment.tfvars"]
      commands = ["apply", "plan", "refresh", "output"]
    }
  }

[dmrzzz]

Two thoughts:

1. I like the find_in_parent_folders("otherfilename.tfvars") idea -- not just for secrets, but also to name the generic parent file in a way that makes it obvious that it's only applicable to Terragrunt and not to general Terraform. Having a terraform.tfvars file there which can't easily (without jumping through many extra hoops) be used for normal Terraform vars feels weird.

2. I definitely agree with the objections to magically inheriting parent .tfvars _in general_, but the terragrunt block is already a special case, so how about passing just the terragrunt block into Terraform (as a map variable) in a way that honors the include behavior? i.e.

   # "parent" terragrunt.tfvars
   terragrunt = {
     # lock stanza omitted
     remote_state = {
       backend = "s3"
       config {
         encrypt = "true"
         bucket = "foobarXXX"
         key = "Shared Networking/${path_relative_to_include()}/terraform.tfstate"
       }
     }
     set_terragrunt_variable = true
   }
   

   # child/terraform.tfvars
   terragrunt = { include { path = "${find_in_parent_folders("terragrunt.tfvars")}" } }
   

   # child/main.tf
   variable "terragrunt" { type = "map" }
   output "terragrunt" { value = "${var.terragrunt}" }
   

   would result in Terraform receiving a map that contains the actual rendered key value (because Terragrunt would pass it in using -var on the command line).

   Taking this one step further, suppose we now permit setting arbitrary user variables _inside_ the terragrunt block which could then be referenced both from elsewhere in the Terragrunt block and from within Terraform? Then the OP could write

   terragrunt = {
     user_variables = {
       environment_name = "dev"
     }
     remote_state = {
       backend = "consul"
       config {
         address = "my-consul.dev.local:8500"
         path = "terraform/state/${var.environment_name}"
         datacenter = "dus2-shared"
       }
     }
     set_terragrunt_variable = true
   }
   

   and have Terraform extract the same environment_name value (now set in exactly one place) from within the terragrunt map variable.

   Caveat: in practice this probably won't be very useful until Terraform can perform lookups on nested maps, but it appears somebody is working on that for Terraform 0.9: https://github.com/hashicorp/terraform/issues/11036, https://github.com/hashicorp/terraform/pull/11704

[dhoer]

I still think there is merit to using Convention over Configuration (think Java before Maven convention).

Possible convention rules:

1. pull in all terragrunt.tfvars files up and downstream when running terragrunt command (all other *.tfvars would still require the terragrunt dsl to get merged in)
2. terragrunt.tfvars can contain both terragrunt configurations and/or env vars
3. highest terragrunt.tfvars file found in path must contain the lock and remote state
4. a list of terragrunt.tfvars paths will be printed to stdout starting with the highest terragrunt.tvars file found in path
5. terragrunt.tfvars that contain env vars will have (contains environment variables) appended at the end of the path in list, e.g. /my/path/terragrunt.tfvars (contains environment variables)
6. terragrunt.tfvars with env vars will be merged together starting with first entry on the list
7. debug mode outputs the merged result for each terragrunt.tfvars file found, unless terragrunt sensitive flag is set

Concerns mentioned:

• It's error-prone: you may pick up random files you weren't expecting.
  Outputting a list of terragrunt.tfvars paths in order merged would help with this concern.

• It's hard to reason about: you can't just read the config files themselves to understand the behavior; you also have to scan the entire hard drive.
  I don't know enough about the use cases to comment on this. I don't think "scan the entire hard drive" is a true statement since you are only traversing up the path for terragrunt.tfvars files.

• It makes reproducible environments harder: you may have a file in a parent directory on your system that I don't have on mine.
  I think the convention addresses this.

• It encourages more magic. Once you're inheriting automatically from parents, you immediately get requests like "well, I want to inherit from parent files... except in these cases..." or "in dev, we also want to (magically) inherit from this special extra file". This exacerbates all the problems above.
  I think they just use the dsl to configure when not wanting to follow convention.

Shooting for 80/20 rule, 80% customers should be up running using the convention, the other 20% might require some form of configuration.

[brikis98]

OK folks, I'll try to reply to the flurry of questions one at a time.

First up, @notbrain.

This is definitely not the can of worms 🐛 I expected to find for wanting such a simple thing: that if a .tfvars variable assignment is made above a terragrunt {} block in the same file, to use whatever that value is inside the terragrunt {} block so you don't have N various places to change the value in that file. Wouldn't the rules of precedence "just work" in that case? No other variable value would override whatever is local to the file?

This is an unexpected side effect we got from merging .terragrunt configuration into terraform.tfvars. Suddenly, it's "obvious" that Terragrunt should be able to read Terraform variables, whereas before, it wasn't quite as clear :)

However, things aren't so simple.

First of all, Terraform doesn't support interpolations in variable definitions or .tfvars files. Try to set foo = "${var.bar}" and you'll see that no interpolation happens. Therefore, the interpolation syntax we support in the terragrunt = { ... } block is already a bit of an anomaly. If we were to add ${var.XXX} syntax for looking up variables, you'd only be able to use that with in the terragrunt = { ... } block and not the rest of the file, which will certainly confuse some users.

Second, while interpolating the value of ${var.XXX} will work fine if the value is set in the same file, it won't work if the value is set using other mechanisms, such as a default in the variable declaration. This will confuse even more users, as some of their var.XXX lookups will work and some will not.

In short, supporting ${var.XXX} lookups in the terragrunt = { ... } block feels like a very leaky abstraction.

I'm open to ideas here.

My two best thoughts so far are:

1. Show a clear error message for var.XXX lookups.
2. Add a new helper such as ${lookup_variable_in_current_file("XXX")}, which can return the value of a variable defined in the same .tfvars file, and makes it explicit that it does not offer the same semantics as ${var.XXX}.

On the other hand, Terraform 0.9 is looming around the corner, and that should add support for backends, which will handle the remote state configuration and locking for you. I assume those support variables already and would make any extra work in this area unnecessary. Terragrunt's main role at that point would be reduced to downloading source files and dealing with multiple modules, none of which should require much in the way of variable lookups.

Regarding -var-file technique described above: would this allow us to set up values inside a parent .tfvars file so that a terragrunt {} block like this would get the aws_region variable inserted in 2 places?

As mentioned above, we don't currently read any Terraform variables, so this won't really help.

But this was confusing, since I still needed to use get_env to get a TF_VAR_* that would otherwise be "automatically" picked up by terraform itself?

TF_VAR_XXX is useful if you're actually setting those env vars, but if you're defining those vars in .tfvars files, then you're probably not setting any env vars and this won't help you.

[brikis98]

Next up, @dhoer

Using find_in_parent_folders() does not work when you are also specifying a different repo as the terraform source. find_in_parent_folders() returns a relative path, but since terraform runs in a tmp directory, it needs the absolute path in order to properly find the file.

Ah, good point. That helper might have to execute before everything is copied to a tmp folder in order for this to work. A bit messy, but probably doable.

[brikis98]

@dmrzzz

how about passing just the terragrunt block into Terraform (as a map variable) in a way that honors the include behavior?

This is a very clever idea, but I don't think it'll work.

When setting a value for a Terraform variable (e.g. in a default or -var or a .tfvars file), you can't use interpolations, so if our terragrunt definition included a ${var.XXX} lookup, Terraform would return an error.

There is also a weird chicken-and-egg problem here: we need to run Terraform to get it to return the value of a variable, but Terragrunt doesn't necessarily know what Terraform code to run and with which settings until it has the value of those variables.

[brikis98]

@dhoer

pull in all terragrunt.tfvars files up and downstream when running terragrunt command

Up and downstream? So you'd pick up .tfvars files in child folders?

Also, you prefer every project to have two files, a terragrunt.tfvars and a terraform.tfvars?

Outputting a list of terragrunt.tfvars paths in order merged would help with this concern

It would help debugging the problem, but not avoiding it in the first place.

I don't know enough about the use cases to comment on this. I don't think "scan the entire hard drive" is a true statement since you are only traversing up the path for terragrunt.tfvars files.

My point here is that when you're explicit in your code, you can look at that code, and know exactly what it includes. You don't need to have any conventions memorized. If it includes some file in a parent folder, the code will say so explicitly, and you'll know to look in that parent folder.

If you have conventions, then you have to know to look in special places (e.g. all folders above and below), which is non-obvious, especially as teams grow larger and not everyone has spent hours reading the Terragrunt docs.

It makes reproducible environments harder: you may have a file in a parent directory on your system that I don't have on mine.
I think the convention addresses this.

Not at all. Imagine we have our Terraform code in a repo called terraform-code. You check that out on your local computer into /Users/dhoer/terraform-code and I check it out on my computer into /Users/brikis98/terraform-code. If I happen to have a terraform.tfvars (or terragrunt.tfvars) in /Users/brikis98, then when I run terragrunt apply, I'll get totally different behavior than you.

[notbrain]

@brikis98

Thanks for the lesson, and thanks for being so methodical about this--could get ugly quickly, especially with new terraform stuff on the way. I was unclear that .tfvars files are only for explicit assignment instead of an alternative to terraform variable declarations that would _also_ pick up interpolations. I think that explains most of my misunderstanding.

I am now running into the problem @dhoer mentioned when using another repo to hold the terraform modules; I can't seem to figure out a non-explicit absolute path since the execution is happening inside the temp folder inside /var/folders/6h/lflfn8v13j5_k2d91kc3gm9m0000gn/T/terragrunt-download/....

Given the things brought up in this discussion is there a current recommendation for how to proceed with this? Minimal duplication inside .tfvars files aside, how can I point terraform to my parent global.tfvars while plan/apply-ing inside sns-topics?

infra-live/
  global/
    global.tfvars
    sns-topics/
      sns-topics.tfvars

(Naming convention not set, might need to go back to terraform.tfvars -- but would be much nicer to be able to reduce the number of same-named files all over the hierarchy).

[dhoer]

@dhoer

pull in all terragrunt.tfvars files up and downstream when running terragrunt command
Up and downstream? So you'd pick up .tfvars files in child folders?

if you are using spin-up, yes. But only in that case, otherwise current directory and above

Also, you prefer every project to have two files, a terragrunt.tfvars and a terraform.tfvars?

No, just one, whatever the convention says it will be pulled in.

Outputting a list of terragrunt.tfvars paths in order merged would help with this concern
It would help debugging the problem, but not avoiding it in the first place.

I don't know enough about the use cases to comment on this. I don't think "scan the entire hard drive" is a true statement since you are only traversing up the path for terragrunt.tfvars files.
My point here is that when you're explicit in your code, you can look at that code, and know exactly what it includes. You don't need to have any conventions memorized. If it includes some file in a parent folder, the code will say so explicitly, and you'll know to look in that parent folder.

IMHO the convention would be easier than the steps I had to do to deal with env vars.

If you have conventions, then you have to know to look in special places (e.g. all folders above and below), which is non-obvious, especially as teams grow larger and not everyone has spent hours reading the Terragrunt docs.

You have to read the docs to do basic stuff today.

It makes reproducible environments harder: you may have a file in a parent directory on your system that I don't have on mine.
I think the convention addresses this.
Not at all. Imagine we have our Terraform code in a repo called terraform-code. You check that out on your local computer into /Users/dhoer/terraform-code and I check it out on my computer into /Users/brikis98/terraform-code. If I happen to have a terraform.tfvars (or terragrunt.tfvars) in /Users/brikis98, then when I run terragrunt apply, I'll get totally different behavior than you.

I think this is a good thing to allow flexibility. The above example would fail under suggested convention if it didn't have state and lock defined. I removed a convention item that mentioned only one state and lock can exist, otherwise, fail. That might be a way to manage that...

---

The bottom line here is that terragrunt currently makes keeping configurations DRY hard. It does an amazing job with state, locking, and cleaning up module clutter and that is why I choose this tool.

Use case; I need to tag each environment with customer and environment tag for billing purposes.

It is easy to do this if I repeat the env vars in each terraform.tfvars that sources a module, but I would like to have this defined once under the dev folder, e.g.:

my-infrastructure/env/dev/services/module/terraform.tfvars
my-infrastructure/env/dev/terraform.tfvars

Using relative paths works, but it can lead to dangerous cut and paste errors since the environment name is included:

arguments = ["-var-file=../../env/dev/services/environment.tfvars"]

The convention over configuration example was to an attempt to solve repeating product, customer, and environment variables to keep it DRY.

[dmrzzz]

On the other hand, Terraform 0.9 is looming around the corner, and that should add support for backends, which will handle the remote state configuration and locking for you.

Ooh, I hadn't seen that yet. Very exciting!

I assume those support variables already and would make any extra work in this area unnecessary.

Actually it looks like the backend configuration will not initially support variables (https://github.com/hashicorp/terraform/pull/12067) but no doubt they'll get there at some point, so I think your conclusion still stands.

[brikis98]

@notbrain I filed a separate issue for that: https://github.com/gruntwork-io/terragrunt/issues/143. I'm open to ideas (and PRs!) for how to fix it.

[brikis98]

@dhoer

The bottom line here is that terragrunt currently makes keeping configurations DRY hard.

I agree. I think the question is to figure out how to improve that in a way that doesn't introduce a bunch of new problems. With Terraform 0.9 right around the corner, I think I'd prefer to see what impact that has before making major changes to Terragrunt.

[brikis98]

BTW, I propose adding support for ${path.module} in Terragrunt configuration as a solution for the absolute/relative path issue with -var-file arguments: https://github.com/gruntwork-io/terragrunt/issues/143#issuecomment-281660992. I think it'll be a pretty easy change, so if anyone wants this fix soon, I'd be grateful for a PR!

[dhoer]

@brikis98 That sounds like a good approach. Thanks

[brikis98]

Here's a proposal for a way to keep your .tfvars files more DRY: https://github.com/gruntwork-io/terragrunt/issues/147#issuecomment-282276693.

Feedback (and PRs!) welcome.

[shaharmor]

Hey, I read the entire thread but still didn't figure out if its possible (And if not, how can we make it so), to use a variable inside the terragrunt config.
What I'm looking for is save the state/lock files in the corresponding AWS region to where that module is deployed, sort of like this:

terragrunt = {
  lock {
    backend = "dynamodb"

    config {
      state_file_id = "${path_relative_to_include()}/${var.aws_region}"
      region = "eu-west-1"
    }
  }

  remote_state {
    backend = "s3"

    config {
      encrypt = "true"
      bucket = "terraform-state"
      key = "${path_relative_to_include()}/${var.aws_region}/terraform.tfstate"
      region = "eu-west-1"
    }
  }
}

Not sure its the best practice. Maybe I should just copy the environment and create folders for env-eu-west-1 & env-eu-west-2 or something

[brikis98]

@shaharmor Not currently possible. That's why this issue is still open :)

I have a proposal for a solution here: https://github.com/gruntwork-io/terragrunt/issues/147#issuecomment-282276693. I may not be able to get to it for a week or two though. In the meantime, feedback and PRs very welcome.

[apottere]

It looks like the linked comment above was describing a solution to two different issues, and was abandoned in favor of a solution that only fixed the other issue and not this issue - is that a fair assessment?

We're running into problems organizing our terragrunt code where we would really like to use variables for the remote_state key and source, but the only real way to affect those properties is with ${path_relative_to_include(). Up until now they were the same value, so we were able to use a directory structure to work around not having variables, but now we'd like them to be different and I'm not sure how to accomplish that.

[brikis98]

@apottere Which comment are you referring to?

Also, what would your ideal solution to this problem look like?

[apottere]

I was referencing this comment: https://github.com/gruntwork-io/terragrunt/issues/147#issuecomment-282276693. Later on in that same thread (https://github.com/gruntwork-io/terragrunt/issues/147#issuecomment-308924425) is where the solution changed to no longer work for this issue.

I would be happy with something as simple as being able to specify variables for terragrunt only (that don't have to pass on to terraform), either in the terragrunt = {} block or in a seperate file associated with the terragrunt block by an "include" or by convention.

[brikis98]

@apottere To make sure I'm understanding, is your goal to be able to define some common variables in, say, a root .tfvars file as follows:

# infra/terraform.tfvars
terragrunt = {
  foo = "bar"
}

And to be able to reference those variables in some child .tfvars file?

# infra/stage/my-app/terraform.tfvars
terragrunt = {
  foo = "${variable_from_parent("foo")}"
}

[apottere]

I was thinking the other way around - for example if I wanted database and app tfstate files stored in different buckets (contrived):

# infra/terraform.tfvars
terragrunt = {
  remote_state {
    backend = "s3"
    config {
      lock_table = "terragrunt_locks"
      encrypt = true
      region = "us-east-1"
      bucket = "terraform-${get_aws_account_id()}-${variable_from_child("type")}-us-east-1"
      key = "${path_relative_to_include()}/terraform.tfstate"
    }
  }
}

# infra/my-app/mysql/terraform.tfvars
terragrunt = {
  type = "database"
  include {
    path = "${find_in_parent_folders()}"
  }
}

[brikis98]

Oh, I see, thanks for the clarification. Heh, that's yet another new use case :)

The variable_from_child approach is a bit awkward because the include actually copies the code from the parent into the child, so the method would be more accurately named variable_from_self. I suppose a read_var_from_file() helper could handle this use case (e.g. read_var_from_file("type", self) and others, but I can't help but think the programming model is getting a bit over-complicated. This may mean that instead of adding another potentially leaky abstraction, we need to step back and think about the best way to handle all these different use cases...

[apottere]

Would it be possible to merge variables like the extra_arguments are being merged from parent to child? It seems like that would simplify it a little: ${variable("type")}

[apottere]

Here's a more realistic example of what I'd love to be able to accomplish with terragrunt:

terragrunt = {
  remote_state {
    backend = "s3"
    config {
      lock_table = "terragrunt_locks"
      encrypt = true
      region = "${variable("region")}"
      bucket = "terraform-${variable("region")}-${get_aws_account_id()}"
      key = "${path_relative_to_include()}/terraform.tfstate"
    }
  }
}

Right now it doesn't seem like there's a good way in terragrunt or terraform to store the remote-state in the region you're configuring without duplicating configurations, which means that if the S3 region that has your remote state goes down it'll impact terraforming all other regions.

[tomdavidson]

Have not contrib to terraform code or terragrunt code - this could be utter rubbish:

I am digging the new locals {} and recently did a k8s plan that did not use tfvars at all. I used config.tf with locals that built the config from env vars and git refs. I used symlinks from common.tf to various plans. It was fairly slick an dry.

What if instead of using only tfvars, terragrunt could use locals. The terragrunt locals could reference other locals to compute values. Terraform could use the same locals to compute the same values. Terragrunt would need to be able to interpolate locals, presumably using the same libs that Terraform does, and would take a _first pass at computing its values before passing the computed values to terraform_ to avoid the chicken/egg situation.

#tg config local
locals {
  terragrunt = {
    remote_state {
      backend = "s3"
      config {
        bucket = "${local.backend_name}"
        dynamodb_table = "${local.backend_name}"
      }
    }

  backend_name="prak-${get_env("env", "exp")}-${local.region}-${get_aws_account_id()}"

}

#other locals
locals {
  region=${var.region}

}

# some plan
provider "aws" {
   region = "${local.region}"
}

[brikis98]

What if instead of using only tfvars, terragrunt could use locals. The terragrunt locals could reference other locals to compute values. Terraform could use the same locals to compute the same values. Terragrunt would need to be able to interpolate locals, presumably using the same libs that Terraform does, and would take a first pass at computing its values before passing the computed values to terraform to avoid the chicken/egg situation.

It's an interesting idea, but I suspect extracting or re-implementing how Terraform processes locals will be a very, very non-trivial task, and nearly impossible to keep up-to-date with the changes in Terraform itself.

[tomdavidson]

"very, very non-trivial" - what I nice way to say it :D

TF is pretty hot, I get the moving target issue. My terragrunt config is likely to be much more conservative than my tf plans and some lag will prob be ok. Import the tf config.go https://github.com/hashicorp/terraform/blob/master/config/config.go (not re-implement, but implement) ?

[brikis98]

TF is pretty hot, I get the moving target issue. My terragrunt config is likely to be much more conservative than my tf plans and some lag will prob be ok. Import the tf config.go https://github.com/hashicorp/terraform/blob/master/config/config.go (not re-implement, but implement) ?

Hehe, you're welcome to give it a shot, but from last I looked at it, Terraform builds a fairly complicated dependency graph and navigates it according to a very specific lifecycle to process variables. Replicating all of that in Terragrunt, even by using Terraform's code under the hood, is, uh, well, very, very non-trivial :)

[k-brooks]

+1

[bobchaos]

Would it be possible to reduce the scope of this issue? I'm trying to setup a version pinning system: I've made breaking changes to my modules and need to tell existing stacks to keep using the old code, but the only way I've found to pass a value is via ENV, which is less than ideal (can't commit the ENV vars to git :/ ).

If I could use arbitrary variables within the context of terragrunt blocks only I'd be happy, no need to share the module version with Terraform. I could do this:

terragrunt = {
  terraform {
    source = "git::[email protected]:myorg/terragrunt-modules.git?ref=${tg_var(TF_MODULES_VERSION)}//zemodule"
  }
# get the modules version from the the parent folder so it's shared with all the modules
  include = {
    path = "${find_in_parent_folders()}"
  }
}

and then in the parent directory...

terragrunt = {
  TF_MODULES_VERSION = "v0.1"
}

[mattford63]

I'd like to see something to allow version pinning also by specifying a top-level var.

This is a work around I'm using - would love to see a better way though

In env.tfvars file in stage dir I have

env_name = "stage"
env_version = "v0.0.8"

In the component dir say stage/vpc I have

terraform {
  source = "git::ssh://[email protected]/mf/vpct?ref=${run_cmd("get_env_ver.sh", "../env.tfvars")}"
}

Where get_env_ver.sh is on the path and is simply

#!/bin/bash
gawk '/env_version/ {gsub(/"/,"", $3) ; printf("%s",$3)}' $1 

Ideally this one liner script would be in the run cmd but I couldn't figure out the escaping.

Ugly but let's me specify the version of the modules dir in each env.

[flmmartins]

I think this issue is huge and should be divided in smaller scopes.

My use case is a bit different.
I have the following structure

repository - infra-repo

|prod - prod contains remote state + prod_variables which I provide the other folders bellow it
├── create_master_role
│   ├── terraform.tfvars
└── modules
├── role
│   └── main.tf
|── something_that_role_needs
└── main.tf
role.tfvars call modules/role.
Inside module/role there's a module call to terraform-modules which is a different repository.

Example:

create_master_role/tfvars contains:

terragrunt = {
  include {
    path = "${find_in_parent_folders()}"
  }
  terraform {
    source = "${path_relative_from_include()}/../../modules//role"
  }
}

module/role/main.tf

module "role" {
  source = "git::[email protected]/terraform-modules.git//role?ref=v1.0.0"
  module_version = "v1.0.0"

  name = "${var.account_name}.role"
  tags = "${merge(var.default_tags, local.local_tags)}"
  assume_role_policy = "${data.aws_iam_policy_document.role.json}"
  policies = ["${aws_iam_policy.policy.arn}"]
  tags2 = ${module.something_that_role_needs.arn}
}

I only use module/role to pass variables to the terraform-modules. Some of the variables use outputs from other modules... some use data.....

What I would love to have is:

terragrunt = {
  include {
    path = "${find_in_parent_folders()}"
  }
  terraform {
    source = "git::[email protected]/terraform-modules.git//role?ref=v1.0.0"
        name = "${var.account_name}.role"
        tags = "${merge(var.default_tags, local.local_tags)}"
       assume_role_policy = "${data.aws_iam_policy_document.role.json}"
       policies = ["${aws_iam_policy.policy.arn}"]
       tags2 = ${module.something_that_role_needs.arn}
  }
}

[brikis98]

This issue has indeed gotten huge and hard to follow. Terragrunt nowadays has support for:

1. All built-in Terraform functions
2. Local variables
3. read_terragrunt_config
4. generate block

Hopefully, between all of these, most of the use-cases mentioned in this issue are covered, so I'm going to close it. If there's anything missing, please file a new issue.