Terragrunt: can't get get_parent_tfvars_dir() to work

[ebarault]

Consider the following project:

/terragrunt live folder
├── terraform.tfvars <-- root
├── prd
│   ├── app1
│   |   └── terraform.tfvars <-- child

I'd like to access the terraform modules dir located one step above the folder where I have the root terraform.tfvars using get_parent_tfvars_dir() in the child terraform.tfvars as in:
source = "${get_parent_tfvars_dir()}/../modules//app"

but get_parent_tfvars_dir() keeps returning the absolute location of the child terraform.tfvars /path/to/live/prd/app1 instead of returning /path/to/.

Is there anything i'm doing wrong ?

[brikis98]

Do you have an include { ... } block in the child terraform.tfvars?

[ebarault]

Yes, I have the classic
```
terragrunt = {
include {
path = "${find_in_parent_folders()}"
}
#...
}

[brikis98]

You're right, this is a bug. I was just able to repro with your exact folder structure and the following contents in prd/app1/terraform.tfvars:

 terragrunt = {
   include {
     path = "${find_in_parent_folders()}"
   }

   terraform {
     extra_arguments "test" {
       commands = ["plan", "apply"]
       arguments = ["-var", "foo=${get_parent_tfvars_dir()}"]
     }
   }
 }

When I run terragrunt apply, I see:

terraform apply -var foo=/tmp/terratest/prd/app1

Not sure when this broke, as there are tests for it. Sorry about that!

[ebarault]

👍 Thanks for confirming. Unfortunately I don't know go... and therefore can't help you much here.

[brikis98]

Understood. I'm a bit buried at the moment, so I'm not sure when I'll be able to get to this one. @jocgir Any chance you have a spare few minutes to take a look, since I believe you were the one who originally added get_parent_tfvars_dir()?

[jocgir]

Hi, I'm a bit buried too, but I will take time to give it a quick look.

[brikis98]

Thanks!

[jocgir]

I found what is wrong. The get_parent_tfvars_dir() function has been designed to be used in the parent terraform.tfvars.

When the child terraform.tfvars configuration file is evaluated, it does not have the full context of its parent yet. So, these functions are only useful when used in the parent configuration file to retrieve the relative position of the child.

An example of utilization would be:

source = "${get_parent_tfvars_dir()}/../modules//${path_relative_to_include()"

defined in the parent configuration file. That requires the modules folder hierarchy to be exactly the same as the terragrunt folder.

That being said, this is still a bug and it is misleading for the user. I will check if it is possible to fix it.

[Indigenuity]

Are there any decent workarounds for this? The original usecase seems pretty essential for local development:

source = "${get_parent_tfvars_dir()}/../modules//app"

Currently the only solution I see is using very deep relative pathing like:

`source = "${get_tfvars_dir()}/../../../../modules//app"

But this requires careful refactoring every time directory structure changes even a little.

I don't quite follow the referenced PR, since I don't know the language. I don't think I could do it justice, but if there really isn't any other manpower to put on this I could take a stab at it.

[brikis98]

The original usecase seems pretty essential for local development:

For local dev, have you tried the --terragrunt-source cmd line param?

[Indigenuity]

That still carries the same problems, no? I either have to type the whole path manually for each apply at the CLI or use the extra_arguments in the the .tfvars files--which have the same parent/child problem discussed here.
If terragrunt directory mirrored exactly the modules directory, declaring source in root terraform.tfvars file would be no problem. Otherwise, only the leaf terraform.tfvars file knows what the source should be.

├── terraform
│   ├── dev
│   └── terraform.tfvars <-- root
│   │   ├─ customer_app
│   │   │   ├─ customer_1
│   │   │   └── terraform.tfvars <-- child
│   │   │   ├─ customer_2
│   │   │   └── terraform.tfvars <-- other child
│   ├── modules
│   │   ├── customer_app

In this structure, there's no way from dev/terraform.tfvars to know what module to use for a leafcustomer_1. So the source needs to be declared in the leaf:

// dev/customer_app/customer_1
terragrunt = {
  terraform {
    source = "${get_parent_tfvars_dir()}/../modules//customer_app"
  }
  include {
    path = "${find_in_parent_folders()}"
  }
}

Apologies if this makes no sense.

[Indigenuity]

I suppose this could be avoided by separating out the ideas of "where to get modules" and "what module to use". i.e. the // syntax in Terraform. But I imagine that's not parsed at the terragrunt level?

So rather than
source = "modules_location//myapp"
Instead it would be something like

modules_source = "modules_location"
module = "myapp"

That way you could define what module to use in a child but leave it to the parent to know where to fetch modules from.

[brikis98]

I either have to type the whole path manually for each apply at the CLI

Yup, that would be the idea for how you do manual testing when you want to override the source URL with a local file path. A bit tedious, but these are CLI args, so you can easily define vars and aliases to make it easy.

So the source needs to be declared in the leaf:

Yes, source URLs are always defined in the leaf nodes. Usually, they are set to versioned URLs, not local file paths, which gives you the ability to promote a single, immutable version of your Terraform code from environment to environment (e.g., dev -> qa -> prod). When you do manual testing, you can override the URL with a local file path using --terragrunt-source.

[brikis98]

I suppose this could be avoided by separating out the ideas of "where to get modules" and "what module to use". i.e. the // syntax in Terraform.

A neat idea, assuming that all the children of a parent come use the same modules_source.

Note that we will be working on a relatively large refactor of Terragrunt to support the next version of Terraform and HCL2. You can find the rough proposal here: https://github.com/gruntwork-io/terragrunt/issues/466#issuecomment-386258812. That includes a new get_input helper you can use to read variables from other Terragrunt configs, which offers another way to handle source URLs where you store the common source in a variable somewhere and reference it everywhere else:

source = "${get_input("../common.tfvars", "modules_source"}//foo/bar"

[Indigenuity]

That proposal looks mighty fine! I think a function like get_input would help here.

[jhoblitt]

I've run into same problem attempting to use get_parent_tfvars_dir() from a "child" config and getting the abs path of the child rather than the parent.

    extra_arguments "tls" {
      commands = ["${get_terraform_commands_that_need_vars()}"]

      env_vars = {
        TF_VAR_tls_crt_path = "${get_parent_tfvars_dir()}/..."
        TF_VAR_tls_key_path = "${get_parent_tfvars_dir()}/..."
      }
    }