Terragrunt: Would it be possible to add a get_output interpolation syntax function

[Roberttmoon]

I was wondering if adding the ability to pass an output from one Terragrunt managed module to another Terragrunt managed module using something like "${get_output("output_name", "path/to/.tfvars")}" would be helpful? This would help with passing outputs around a Terragrunt managed stack cleanly.

My thought is that since Terragrunt already knows where the .tfstate file is within the s3 bucket it could collect the state file, read the outputs block and return the "name: value" of the output from the .tfstate file.

[brikis98]

An interesting idea. Out of curiosity, what's the advantage of using this over doing the same thing in pure Terraform code with terraform_remote_state?

[Roberttmoon]

I don't know if I'm misunderstanding how to use terraform_remote_state: do you need to add a data resource for each output that you are importing to the module that you are using? (I don't know if you can add that to the terragrunt = {} section on the terrafrom.tfvars file.)

With using the get_output() interpolation, I would expect that you are just passing the value of the output which would lead to modules that could be more independent and would not need to know about each other structure (the output names for instance). Moreover all the referencing would happen in one place, Terragrunt rather than in the Terrafrom modules as well as Terragrunt. Again this could be moot if I am misunderstanding how to use remote state within the context of Terragrunt.

[brikis98]

I don't know if I'm misunderstanding how to use terraform_remote_state: do you need to add a data resource for each output that you are importing to the module that you are using?

terraform_remote_state is just a normal Terraform data source. It's designed to do more or less what you are requesting above: it reads in the state file of some other module and makes all the outputs in that state file available in your code. Example:

# Read the remote state from my VPC module
data "terraform_remote_state" "vpc" {
  backend = "s3" 
  config {
    bucket = "my-bucket"
    key = "vpc/terraform.tfstate"
    region = "us-east-1"
  }
}

# Use various outputs from the VPC module's state throughout my code
resource "aws_elb" "example" {
  name = "example"
  subnets = ["${data.terraform_remote_state.vpc.public_subnet_ids}"]
}

resource "aws_security_group" "example" {
  vpc_id = "${data.terraform_remote_state.vpc.vpc_id}"
}

With using the get_output() interpolation, I would expect that you are just passing the value of the output which would lead to modules that could be more independent and would not need to know about each other structure (the output names for instance). Moreover all the referencing would happen in one place, Terragrunt rather than in the Terrafrom modules as well as Terragrunt.

I agree with these advantages. The key question is whether they offer a solution that's so much better than terraform_remote_state to merit building into Terragrunt the ability to:

1. Read from a variety of remote state backends.
2. Parse Terraform's tfstate syntax, which is an internal API that is NOT guaranteed to be backwards compatible between releases.

That said, I can think of two potential shortcuts:

1. Reuse the Terraform code base to read the state files for us. Both Terragrunt and Terraform are written in Go, so assuming the actual APIs are exposed in a reasonable fashion, this should be doable.

2. Instead of reading the state file of other modules, perhaps we could just run terragrunt output on those modules? E.g., if in your .tfvars file you set vpc_id = "${get_output("../vpc", "vpc_id")}", Terragrunt would go to the ../vpc folder, run terragrunt output vpc_id, and fill the result in for you.

[Roberttmoon]

I agree that reading the .tfstate file is less than ideal and i feel that your second "shortcut" would actually be closer to the correct way to handle this by running output on the modules referenced in the location part of the get_output("location", "output"). I don't know if the output will change between versions like the state file which could lead to issues.

My confusion about terraform_remote_state (which does seem to be a really great tool) is where it gets called, if its called from a .tf file on the Terragrunt side, then sure this is completely viable, (if a bit confusing to a new comer to Terraform/Terragrunt like myself). If the terraform_remote_state needs to be on the module that Terragrunt calls, that means a lot of rigidity between modules. Its pretty easy with something like like a subnet where you have some needs that are pretty clear cut. I was thinking something more like IAM where you could build larger policies that may or may not take in multiple outputs from various sources, or linking together other, complex AWS services.

[brikis98]

If the terraform_remote_state needs to be on the module that Terragrunt calls, that means a lot of rigidity between modules.

This is actually the case. The terraform_remote_state data source has to live in .tf files, which means it's built into the modules you deploy with Terragrunt, and not anything that Terragrunt knows about itself. It's not quite as rigid as you may think, as you can use variables for all the params of the terraform_remote_state config:

data "terraform_remote_state" "vpc" {
  backend = "s3" 
  config {
    bucket = "${var.remote_state_bucket_name}"
    key = "${var.remote_state_key}"
    region = "${var.remote_state_region}"
  }
}

These variables can be set from the .tfvars files you manage with Terragrunt. That said, it's definitely more coupled than we'd want.

The even bigger downside to this coupling is that you now have to separately make Terragrunt aware of dependencies between your modules. This requires manually knowing which module depends on which (e.g., my Docker cluster module depends on my VPC module) and recording that in the .tfvars files (see the docs for more info). These dependencies essentially duplicate the terraform_remote_state data sources and are a massive PITA to maintain.

A huge advantage of your suggestion—having Terragrunt fill in the dependencies—is that Terragrunt would now implicitly know about all of these dependencies and could handle them for you automatically. This would dramatically improve multi-module interactions.

So, I'm a big +1 on a PR that would add this functionality, especially using my second suggestion from this comment, where Terragrunt calls terragrunt output on your other modules to fetch data from them. This seems very reasonable to implement and maintain, and now that I've thought through it, it has quite a few powerful advantages.

[Roberttmoon]

I have been trying to figure out how to add get_output() and am a little confused where Terragrunt handles the variables set in the root of the terraform.tfvars files.

[brikis98]

where Terragrunt handles the variables set in the root of the terraform.tfvars files.

Parsing of the terragrunt { ... } config in terraform.tfvars happens here: https://github.com/gruntwork-io/terragrunt/blob/master/config/config.go#L183

Parsing of anything outside the terragrunt { ... } config does not happen at all, currently. That's because terraform.tfvars isn't actually a Terragrunt concept, but Terraform's standard way of storing variables in a file. We tried to affect that as little as possible.

So to be able to use Terragrunt helpers, such as this new get_output(), outside of the terragrunt { ... } block would require changing how we parse the configuration a bit. This has often been requested for other reasons anyway—e.g., to be able to keep variable definitions DRY—so at this stage, it's probably worth doing.

[Roberttmoon]

Gotcha, I kinda had that thought a few hours ago but kept kicking stuff around wondering if I missed something. I was expecting to put get_output() within the root section of the terraform.tfvars though I can think of a few ways to add it in the terragrunt { ... } section. My immediate thought would be to add the filled value as a filled EnvVar TF_VAR_foo="foo value (I dont know that much about Go's session, so that could end up being messy) or add it to the command string ... -var foo="foo value". I'm not really sure if these would be solid choices to be honest.

(on proof-read I feel more attached to command line method but I'm still a bit fuzzy on the inner working on how Terragrunt and Terraform interact)

[brikis98]

I think to be useful, get_output() would have to work in the "normal" part of terraform.tfvars and not just the terragrunt { ... } block.

terragrunt = {
  # ...
}

vpc_id = "${get_output("../vpc", "vpc_id")}"

Having said that, I suspect we're going to have to start thinking about a real parser for interpolation syntax, rather than the sketchy regex thing we do now...

[Roberttmoon]

I totally agree that it should be on the variable assignment in the root of the .tfvars file. I think that would make it far more readable (all assignment happens in the same way) and it _seems more_ Terraform. Would a valid solution be to scan the .tfvars file and rebuild it to feed to the Terraform module that is currently being run? That would allow for backwards compatibility for the .tfvars file and would allow for Terraform and Terragrunt to stay pretty isolated.

Hum... other thought, if you feed the terraform.tfvars file in to Terraform, whats to keep someone from naming a variable in their module terragrunt and having a really confusing time?

[brikis98]

Would a valid solution be to scan the .tfvars file and rebuild it to feed to the Terraform module that is currently being run?

I suspect what we'll need to do is create a generated.tfvars with the fully resolved variables in it, and pass that as an extra parameter to Terraform using -var-file.

Hum... other thought, if you feed the terraform.tfvars file in to Terraform, whats to keep someone from naming a variable in their module terragrunt and having a really confusing time?

Nothing at all. Hopefully, terragrunt isn't a common enough variable name to worry about that too much :)

[brikis98]

Update: I'm now working on this.

[Roberttmoon]

Thats awesome! I am excited to see how you solve get_output(). I did try to spend some time learning Golang and figuring out a way to implement get_output() but the ways I saw to approach it were beyond my understanding of go and seemed to be very disruptive to the codebase (changing input parameters to collect the state and importing other modules). Moreover that would have been just for the method of adding get_output() to the args section of Terragrunt.

[ghost]

I like the sound of this. It certainly gets at half of what I was trying to suggest in #343 The other half would be some way to automatically add the outputs of a called module to outputs of the calling template or module, with optional prefix. That'd save a ton of typing AND accommodate changes in modules without requiring a cascade of updates through the call stack. That's probably more of a terraform thing than a terragrunt thing, but it sure would be nice to have.

[geekifier]

I came here by the way of @brikis98's comment in #466, where I noticed the mention of theget_output interpolation.

Will the proposed get_output interpolation somehow influence the way *-all commands function? The big issue being the fact that outputs of dependencies will not be available until those modules are applied at least once. It seems you'll need an ability to produce a "partial plan", apply it, then resume planning the rest of the modules, while maintaining the awareness of which step of the complete stack application is being processed.

This is currently a problem that needs to be solved in a CI workflow I am working on, and it seems that it would be non-trivial to solve it within Terragrunt itself.

[brikis98]

Will the proposed get_output interpolation somehow influence the way *-all commands function?

Yes. The general idea would be to use get_output instead of terraform_remote_state. That way, Terragrunt can see all the dependencies between modules and run apply-all in the right order.

The big issue being the fact that outputs of dependencies will not be available until those modules are applied at least once. It seems you'll need an ability to produce a "partial plan", apply it, then resume planning the rest of the modules, while maintaining the awareness of which step of the complete stack application is being processed.

That sounds too complicated and error prone. The general idea is:

1. If you run apply-all, Terragrunt will be able to use get_output to figure out the dependency graph and apply things in the right order.

2. If you run any other command, and Terragrunt finds a get_output, it'll try to call terraform output, but if that fails because the module you depend on hasn't been applied yet, you'll get an error, and know you need to apply (or apply-all) that other module.

Also, I got a bit buried, and had to temporary put my get_output work on hold. I will hopefully pick it up again as part of #466. My hope is that being able to use the HCL2 parser will make it quite a bit easier.

[acobaugh]

Could we also tie this in to terragrunt's dependencies setting with some sort of shortcut?

eg.

terragrunt = {
  include {
    path = "${find_in_parent_folders()}"
  }

  dependencies {
    paths = ["../vpc"]
  }
}

vpc_id = "${get_output("vpc", "vpc_id")}"

The first parm to get_output(), if it doesn't contain any path info, will refer to the dependency whose last path part matches. This might cause a problem if there were ever multiple deps with the same last path part, but I can't think of a reason this would ever happen in the real world unless someone was being intentionally confusing.

This way if I need to reference multiple outputs from the vpc module, I'm not having to encode the path to the vpc module in multiple places. Would also make it easier if the vpc module were to ever move, we would only have to update the path in one place.

I can foresee cases where we have dependencies that aren't just through output/variable dependencies where I still have a dependencies block, and it would be nice to have a single place to see what the deps are for a given module.

Definitely +1 this, though, even without my idea. This is the last piece of the puzzle for my intended use-case.

[brikis98]

get_output("foo", "bar"), if I can find some time to finish it, will automatically consider foo a dependency. You won't need to add it separately to the dependencies list.

[alwaysastudent]

@brikis98 - i am trying to interpolate the provider, credentials of the current module with an output from another module. is this possible ?

[brikis98]

@alwaysastudent Is that question related to this issue? If not, please open a separate one.

[ghost]

It seems relatively unlikely that there would be a dependency on a module without a corresponding terraform_remote_state in the template. Or, at least, it's incredibly unlikely that there would be a terraform_remote_state without a corresponding dependency to another template. It'd be really great if the existence of a terraform remote state and the path in the key to the state file could be used to track dependencies automatically. Now that my team is working directly in terraform on occasion, I'm finding a lot of missing dependencies which could have been implied by the addition of a t_r_s. Whether that gets renamed to something like get_output() versus just implying it via the existing syntax doesn't seem to matter much.

One thing I'd really hate to lose . I do, on occasion, dynamically compute the location of a t_r_s based on a flag or a particular environment's structure. In some cases, production environments use a 3rd party service, but developer environments use local services running in ECS or in an ASG cluster. I found it much easier to use completely different templates, but with the same outputs, rather than having a ton of conditionally declared resources within a single, shared template. I would definitely want get_output("template", "output") to be subject to some kind of path override.

In my case, I have a module which takes a bunch of flags that control behaviours, and returns path elements and name strings for use in t_r_s declarations.

# use a short module name here, to keep variable accesses short
module "locate_resources" {
  source = "../modules/locate_resources"

  # why oh why does terraform not implicitly use the value of a variable that 
  # exists in this scope if the name matches, unless overridden explicitly here?
  enable_outside_service1 = "${var.enable_outside_service1}"
  enable_outside_service2 = "${var.enable_outside_service2}"
  ...
}

data "terraform_remote_state" "service1" {
  backend = "s3"
  region = "${var.terraform_state_region}"
  key = "${var.aws_account_id}/${module.locate_resources.service1_region}/${module.locate_resources.service1_environment}/${module.locate_resources.service1_template}/terraform.tf_state"
  bucket = "${var.terraform_state_bucket}"
}

...

Then, in the dependencies section of the terragrunt block, I have to make sure to manually set the dependency path to point to the same template that will end up being resolved to a terraform_remote_state by the locate_resources module. In some cases, they point to a shared instance of a service running in my management vpc. In other cases, they point to a template in the current environment, which has outputs that point to a 3rd party service instead of a locally declared resource (sometimes, those resources are still managed via terraform, in other cases, service addresses and credentials are simply passed from vars and vaults to outputs). Setting the dependency path to match the template that will be used in terraform_remote_state is the source of a lot of user errors as my team starts to use terraform directly, since changing the value of a flag can also force a change in the path to a dependency. I'd REALLY love to have those dependency paths computed from the set of t_r_s data sources in the template.

Additionally, it is possible, in my system, that integration tests may cause certain services to be temporarily replaced by setting one of the flags to a different value for the duration of a test, letting me spoof certain services in order to generate test data.

This is all functionality I'd hate to lose in any update, though I'm not opposed to changing the means by which I access it.

A thought (added later):
For what it is worth, you can kind of implement the get_output() function proposed here via a module like locate_resources, as declared above. Exposing an output is kind of a verbose process, since you have to declare vars, then explicitly set them when calling the module, and then each t_r_s output must have its own output declaration, but a module with a bunch of terraform_remote_state resources in it and a bunch of output vars for each remote_state does get you what you are looking for, functionally - just move the t_r_s declaration from my example, above, into the module that is called from the example. Automatically picking up dependencies from any t_r_s resource in the graph fixes the dependency management part of the problem. Some kind of automatic mapping from t_r_s name and output to an output name would be the only missing piece. If there were a way to iterate over the set of outputs in a data source, it would be possible to populate a map of maps as an output - t_r_s name is the outer key, and the keys of the inner map match the keys of the named t_r_s. But just having terragrunt codegen a bunch of outputs with a naming convention would work just as well. Even better if it automatically codegens the calling module declaration with all of the implied variable copying that has to go on.

[ghost]

I guess what I like about the proposal for get_output, is it moves the dependency on the data from the template code into the template's tfvars, making it much easier to switch between explicitly setting a value and grabbing one from another template's outputs, since switching between a var and a t_r_s currently requires a conditional (which I have buried inside my locate_resources module, so I don't have to deal with dozens of such conditionals scattered throughout my templates). On the other hand, with the proposed solution, you do have to explicitly list each value that the template will be dependent upon, making the tfvars file much more verbose than it would ordinarily be - most of my variables are not passed in, but are acquired via t_r_s, so I'd have to add hundreds of get_output() calls to all of my templates just to handle the common default of getting values from the outputs of another template. Also, any time you add a new output to a template, you need to find all of its dependencies and add a get_output() call in the dependency. I'd much rather allow convention to keep the verbosity of my variable declarations to a minimum. Terraform already forces a ridiculous quantity of explicit variable setting. I'd hate to see that get even more verbose. It's great to allow overrides, but I'd really like variables that match output names to be passed through without having to explicitly declare them. If there's a an output with the same name in multiple dependencies, error out or allow the order of dependency declaration to imply precedence.

[ghost]

And for what it is worth, I have a wide variety of templates that have the same path ending.

My management VPC template is at <account>/<region>/_global/vpc and the main VPC for a given environment is at <account>/<region>/<environment>/vpc

If I have a 3rd party service1, A template which provides outputs with config values is at /<saas>/service1, while the shared instance running locally for developers may be at <account>/<region>/_global/service1. Granted, I wouldn't need a template which has outputs pointing to a 3rd party service that isn't configured via terraform if I could transparently switch between direct configuration in the tfvars and getting an output from another template. Instead, I wrote my code so it always gets values from terraform_remote_state, and then I just set up templates that do nothing but populate outputs with the values of vars for use in a t_r_s that gets dynamically configured.

[robh007]

The get_output() function would be helpful addition to this great project. Any idea @brikis98 when it maybe finished?

[brikis98]

@robh007 This is on hold until we do the HCL2 upgrade for Terraform 0.12. That will end up changing a lot of aspects of Terragrunt, including the config file. Hopefully, using HCL2 will also make it easier to parse and process interpolation functions such as get_output().

[robh007]

Thanks for the response @brikis98, yes HCL2 looks to bring a lot of nice changes to terraform.

[bradleyess]

@brikis98 - now we're at Terragrunt 0.19.6 and Terraform 0.12.3 I'm curious if we got any closer to get_output() yet.

[Tensho]

It would be nice to have the described feature in the Terragrunt core, meanwhile, you can try tfvars-annotations by @antonbabenko.

[msfuko]

I'm at v0.19.11 and Terraform v0.12.5..
get_output() would be extremely helpful so we don't need to put terraform_remote_state into every downstream module which has dependency with others.

[gevial]

@msfuko I've dived a bit into the code and it seems it's not so trivial to add get_output() as it introduces some kind of chicken and egg problem.

However you can work around with run_cmd() like that:

include {
  path = find_in_parent_folders()
}

terraform {
  source = "${get_parent_terragrunt_dir()}/../tf-modules//eks_cluster"
}

inputs = merge(
  yamldecode(file("${get_terragrunt_dir()}/${find_in_parent_folders("vars.yaml")}")),
  {
    kubernetes_version  = "1.13",
    instance_type       = "t3.small",
    vpc_id              = trimspace(run_cmd("terragrunt", "output", "vpc_id", "--terragrunt-working-dir", "../vpc"))
    vpc_private_subnets = run_cmd("terragrunt", "output", "private_subnets", "--terragrunt-working-dir", "../vpc")
  }
)

dependencies {
  paths = ["../vpc"]
}

Please note string values have to be trimspaced and list values are parsed just fine.

[gevial]

@brikis98 Following the conversation from https://github.com/gruntwork-io/terragrunt-infrastructure-live-example/issues/8

I've tried to re-arrange cli package but without success. Terragrunt has to read and parse the config in order to run anyway, so we have kind of chicken and egg problem.

I can't see how get_output() can be implemented without changing config parsing entirely. One option is to change config parsing in a way that Terragrunt doesn't need to know about i.e. get_output() until it actually sees it in the config.

Hope that makes sense.

[brikis98]

@gevial Thanks for trying! If a bigger refactor is what it takes, then that's what it'll take. We'll get to it when we can.

[Tensho]

Just wonder does run_cmd() block execution until external command completes?

[gevial]

@Tensho yes - as its output is interpolated into the config

[msfuko]

The only thing I noticed is if I used run_cmd in the input, terragrunt plan-all will fail since The state file either has no outputs defined, or all the defined outputs are empty.
I've set up the proper dependencies.paths and set --terragrunt-ignore-dependency-errors.. but just doesn't help. Any ideas as the workarounds?

[gevial]

@msfuko that's strange, I can successfully run plan-all in my repo which has the following structure:

.
├── common
│   ├── main.tf
│   ├── repositories.tf
│   └── terragrunt.hcl
├── dev
│   ├── eks
│   │   └── terragrunt.hcl
│   ├── rds
│   │   └── terragrunt.hcl
│   ├── vars.yaml
│   └── vpc
│       └── terragrunt.hcl
├── prod
│   └── terragrunt.hcl
└── terragrunt.hcl

Presumably you have some other errors in the configuration.

[gevial]

@KimSamba I've just run terragrunt plan --terragrunt-source /Users/gennadii_aleksandrov/r/tf-modules/aws_rds in dec/rds directory (see the structure above) and there are no issues.

run_cmd() just runs external command, I can't imagine it can affect Terragrunt behaviour itself.

[KimSamba]

@gevial Sorry i removed my post, yes it is working. It just did not work when I input ~/... instead of full path

[tomaszdudek7]

@msfuko Have you solved your issue? I just hit it.

[msfuko]

@gevial thanks for the response! If your environment is up. you won't get into the issue of The state file either has no outputs defined, or all the defined outputs are empty. during run_cmd, since it does output something. the issue we encountered only happen when provision a whole new environment.

@spaszek unfortunately I have not. the run_cmd output still got checked regardless of the dependency.

So the issue is - when I provision a new environment with the folders structure:

├── common
│   ├── main.tf
│   ├── repositories.tf
│   └── terragrunt.hcl
├── dev
│   ├── eks
│   │   └── terragrunt.hcl
│   ├── rds
│   │   └── terragrunt.hcl
│   ├── vars.yaml
│   └── vpc
│       └── terragrunt.hcl
├── prod
│   └── terragrunt.hcl
└── terragrunt.hcl

In eks I need the output from vpc by running run_cmd() (and I set the dependency). I will encounter the error when doing plan-all or apply-all. since the output of run_cmd() gets checked before the prerequisites are created.

[gevial]

@msfuko did you try explicitly inject a dependency with https://github.com/gruntwork-io/terragrunt#dependencies-between-modules ?

[gevial]

The workaround I'm thinking about is applying vpc as it should cost nothing, then plan-all will work.

[gevial]

or you can provide fake values instead of run_cmd() if you have nothing applied anyway

[KimSamba]

This run_cmd solution is nice but i have an issue with it: It cannot work with terragrunt -all functions since it will try to check outputs that haven't been created yet, even if you set terragrunt module dependencies.

[yorinasub17]

I have opened a PR with an implementation of get_output: https://github.com/gruntwork-io/terragrunt/pull/828. This also includes a fix for the issue mentioned above, where -all functions interpolate the inputs block during the dependency collection (and thus attempt to read the outputs before anything has been applied).

[yorinasub17]

Hi folks! https://github.com/gruntwork-io/terragrunt/pull/828 is now merged and released as https://github.com/gruntwork-io/terragrunt/releases/tag/v0.19.20. Be sure to read the details in the README. Please try it out to see if it addresses your needs!

Going to close this now. Please open a new issue if you find bugs in the implementation, or if it doesn't address your needs.

Thanks!

[Tensho]

Thank you very much for this enhancement 🙇 Just a thought – if we already have a dependencies block it's reasonable to take the outputs from there and don't duplicate module references in multiple dependency blocks. In 99% we depend on something we need to refer. Another one optional attribute, e.g. outputs = true, in the dependencies block could eliminate the constant terraform output call.

[yorinasub17]

Yup both are valid points, although I suspect we will deprecate the dependencies block and replace it with dependency block, given the flexibility of configuring those as well as the ease of maintainence of the underlying implementation.

We also had this idea of allowing dependency to read in the config so you can reference them. E.g dependency.vpc.inputs.name or dependency.vpc.locals.region.

Note that I already opened a PR for another enhancement, which is default_outputs: https://github.com/gruntwork-io/terragrunt/pull/840.

[kerr-bighealth]

Just wanted to drop a note here and say that the dependency block with outputs has solved my biggest issue with the new terragrunt.hcl implementation, which was upstream output dependencies being available as downstream inputs. I had gone so far as to declare remote state data blocks directly in downstream modules, and this did not feel right at all since I then had to pass environment details like tfstate bucket, key, etc. I've only got this feature working in a couple areas like iam > vpc > sg > ec2 dependency chains, but it's working well so far.