Terragrunt: [Question] How to use wrapper data sources with terragrunt in live repos

[shinka81]

Hi! I am looking for documentation or an example of how to use data sources (remote state in particular) with Terragrunt config. For example I have the following Terragrunt config and tf data source:

# main.tf
data "aws_availability_zones" "available" {}

# terraform.tfvars
terragrunt = {
  terraform {
    source = "git::[email protected]:some_org/some_vpc_module.git?ref=0.1.0"
  }

  # Include all settings from the root terraform.tfvars file
  include = {
    path = "${find_in_parent_folders()}"
  }
}

cidr_blocks = {
  vpc       = "10.100.9.0/24"
  public_a  = "10.100.9.0/27"
  public_b  = "10.100.9.32/27"
  general_a = "10.100.9.64/27"
  general_b = "10.100.9.96/27"
}
availability_zones = ["${data.aws_availability_zones.available.names}"]

When I go to plan I get Invalid interpolation syntax. Expected syntax of the form '${function_name()}', but got '${data.aws_availability_zones.available.names}'. Seems like the only interpolation allowed is terragrunt specific functions? In past projects we would isolate live modules per env/folder but pass in inputs/outputs using remote state data sources (i.e. passing in a security group id to whitelist to another module, subnet id, an end point to use, etc)

My question is: is this possible? If not, how do you recommend passing in dependent information from other modules in a live repo with terragrunt?

Let me know if this needs some more clarification, thanks!

[josh-padnick]

The terraform.tfvars file is a Terraform feature that Terragrunt piggy backs on. It's not specific to Terragrunt. Unfortunately, Terraform does not support interpolation in terraform.tfvars files.

My question is: is this possible? If not, how do you recommend passing in dependent information from other modules in a live repo with terragrunt?

The short answer is that, as far as I know, you can't. However, see the Terragrunt core use case Keep Your Remote State Config Dry to see the technique we use at Gruntwork when setting up best-practices infrastructure for clients.

[brikis98]

To add to @josh-padnick's response, the way to handle this is to use the data source in your Terraform (.tf files) and not in the .tfvars files.

[shinka81]

Thanks for the info! Closing this issue.

[konstantin-recurly]

@brikis98 is there an example of the structure for this?
I have tried this structure, doesn't seem that it works in this way

us-central1
├── gke
│   ├── cluster
│   │   ├── data.tf
│   │   └── terragrunt.hcl
...

in data.tf I have

data "google_compute_zones" "available" {
}

in terragrunt.hcl I have

inputs = {
  project_id                 = local.common.devops_project_id
  name                       = "${local.service}-gke-cluster"
  region                     = local.regional["region"]
  zones                      = ["us-east4-a", "us-east4-b"]
}

So as you can see I have to manually specify the zones, what I would like to be able to do is to use
slice(data.google_compute_zones.available.names, 0, 2)
I have tried to create a local variable in terragrunt.hcl in locals like so

locals {
zones = slice(data.google_compute_zones.available.names, 0, 2)
}

here is the result of running terragrunt plan

[terragrunt] 2020/04/10 14:45:08 Not all locals could be evaluated:
[terragrunt] 2020/04/10 14:45:08    - zones
[terragrunt] 2020/04/10 14:45:08 Could not evaluate all locals in block.

[yorinasub17]

Terragrunt does not process terraform files inline during configuration parsing. What @brikis98 means here is to not attempt to depend on or pass in terraform data source from terragrunt as an input variable, and instead do that interpolation directly in your terraform module.

If you insist on this model where you want to dynamically look up values in your terragrunt configuration, then you will need to create a separate terraform + terragrunt module that outputs the data source information and pull that in using dependency blocks.

[konstantin-recurly]

well, in many cases we use community modules, that don't have that in place, which means we need to fork a community module to add one data source.

[yorinasub17]

I understand the challenge, but unfortunately adding terraform level processing even just for data sources goes beyond the scope of what terragrunt is intending to accomplish (it would be more than just a wrapper). The canonical way we recommend addressing this is to treat the community modules as library modules, and create wrapper modules as the top level module that you then call using terragrunt. This gives you the flexibility to use all the powers of terraform such as data source lookups, while keeping your terragrunt config simple.

Here are also a few other alternatives:

• Create a different terraform/terragrunt module to look up that info and use dependency blocks
• Use run_cmd to look them up using the awscli, or a custom script.

Unfortunately I don't have any examples for these and am a bit buried to generate them on the fly. The best we have is the walkthrough in our production deployment guide, although that is in the context of Gruntwork's private IaC library catalog.

[konstantin-recurly]

So, I found a different way or maybe what you ment.
What I have created a new data folder and now I have this structure

 tree
.
├── data
│   ├── data.tf
│   ├── providers.tf
│   └── terragrunt.hcl
├── kms
│   ├── providers.tf
│   └── terragrunt.hcl

in the data folder, I have
data.tf with following

data "google_project" "this" {
  project_id = var.project_id
}
#############
## Outputs ##
#############
output "project_number" {
  value = data.google_project.this.number
}
###############
## Variables ##
###############
variable "project_id" {
  type = string
}

terragrunt.hcl with following

terraform {
  source = "./"
}


locals {
  default_yaml_path = find_in_parent_folders("empty.yaml")
  common            = yamldecode(file("${get_terragrunt_dir()}/${find_in_parent_folders("common.yaml", local.default_yaml_path)}"))
  regional          = yamldecode(file("${get_terragrunt_dir()}/${find_in_parent_folders("regional.yaml", local.default_yaml_path)}"))

}


# Include all settings from the root terragrunt.hcl file
include {
  path = find_in_parent_folders()
}


inputs = {
  #This module uses the default common vars for this env/region
  #In the future we will reference states using dependencies
}

this is how I get all data source that I need and here is the example of how to use the data:
in the kms folder, I have
terragrunt.hcl with the following

terraform {
  source = "[email protected]:terraform-google-modules/terraform-google-kms.git?ref=v1.1.0"
}

dependency "data" {
  config_path = "../data"
}

dependencies {
  paths = ["../data"]
}

locals {
  default_yaml_path = find_in_parent_folders("empty.yaml")
  common            = yamldecode(file("${get_terragrunt_dir()}/${find_in_parent_folders("common.yaml", local.default_yaml_path)}"))
  regional          = yamldecode(file("${get_terragrunt_dir()}/${find_in_parent_folders("regional.yaml", local.default_yaml_path)}"))
...

}

# Include all settings from the root terragrunt.hcl file
include {
  path = find_in_parent_folders()
}

inputs = {
  #This module uses the default common vars for this env/region
  #In the future we will reference states using dependencies
  project_id = local.common.project_id
  location   = local.regional["region"]

  keyring            = local.kms_naming
  keys               = [local.kms_naming]
  set_encrypters_for = [local.kms_naming]
  set_decrypters_for = [local.kms_naming]
  encrypters         = ["serviceAccount:service-${dependency.data.outputs.project_number}@container-engine-robot.iam.gserviceaccount.com"]
  decrypters         = ["serviceAccount:service-${dependency.data.outputs.project_number}@container-engine-robot.iam.gserviceaccount.com"]
}

and it works as I expected
I hope this will help someone else.

[yorinasub17]

Yup that is exactly what I meant in one of my suggestions! Thanks for sharing the example 👍

[bluemalkin]

This saved me too, thank you !