Terraform: Intermediate variables (OR: add interpolation support to input variables)

[mrwilby]

Not quite sure how to express this properly.

Input variables today cannot contain interpolations referring to other variables. I find myself bumping up against this now and then where I'd prefer to define a "default" value for an input variable based upon a composition of another input variable and some fixed string.

If we could interpolate values inside default properties of input variables OR, terraform supported some kind of internal transitive intermediate variable which only exists in order to act as a binding point between inputs and other interpolation expressions I could accomplish what I want without having a lot of redundancy in inputs.

Slightly related, but I also long for the ability to reference the input values of a module (not just the outputs) because this is often where I tend to create such bindings. Of course, I can propagate the input to the module all the way through the module and emit as an output but that gets quite repetitive and clunky after a while.

[apparentlymart]

Intermediate variables have been a desire of mine too.

In one particularly tricky spot (using the CIDR functions for network planning) I worked around this by creating a module whose entire purpose is to pass through a value verbatim:

variable "value" {}
output "value" {
    value = "${var.value}"
}

which I then instantiated for each "intermediate variable" I needed:

module "example" {
    source = "..."
    value = "${cidrsubnet(....blahblahblah)}"
}

/// ...

resource "aws_vpc" "example" {
    cidr_block = "${module.example.value}"
}

It's clunky, not least because terraform get then wants to create a separate copy (or symlink) of this "identity module" for each usage. But it got me out of a hairy spot.

[jonapich]

I have spent the better of the last 3 days trying out terraform. In practically all my attempts, I have bumped against this shortcoming; input variables are totally static and cannot be baked from other input variables.

Maintainability and readability is highly impacted by this limitation. Every time a calculation must be repeated across several resources, the DevOp is tempted to create a variable out of the calculation so that he can use just the variable and not having to "remember/copy" the calculation everywhere. As the repetitions increase, changing the calculation becomes more and more risky because you must catch it in all the files.

I have seen some of the propositions (such as the data-driven configuration) and while they all look great to me, they don't look like trivial changes and thus, I am afraid they will come in very late.

It looks to me that if the scope of this feature is limited to input variables, it would be a trivial and incredibly useful addition that would immediately raise user adoption and agility/flexibility. I am still trying to figure out how big of a deal breaker this is for our particular scenario.

Here's a very simple scenario:

variable "region" {
  description = "Please choose an AWS region: n - us-east-1, o - us-west-2"
}

variable "regions" {
  description = "internal use - AWS regions map"
  default = {
    n = "us-east-1"
    o = "us-west-2"
  }
}

variable "region_name" {
  description = "friendly name for the selected region"
  computed = "${lookup(var.regions, var.region)}"
}

Maybe it should be a whole new type of resource to make it really easy on the interpreter, and also very easy to deprecate once the data-driven configs are production-ready:

late_binding_variable "region_name" {
  value = "${lookup(var.regions, var.region)}"
}

Of course these should be included in "${var.*}" so maybe the variable needs a new parameter if it's hard to distinguish static values from interpolations when terraform reads the config:

variable "region_name" {
  value = "${lookup(var.regions, var.region)}"
}

In the above example, terraform knows that a variable without a default but that contains a "value" parameter must be expanded last through interpolation. Also, I guess if value is there, it should error out if a default is also included.

Of course, only variables would be allowed into this new interpolation stage. I don't expect them to work if interpolated from modules or resources.

[apparentlymart]

In the mean time since I wrote my earlier comment I found a new workaround, that became possible because of the null_resource changes in a recent release:

resource "null_resource" "intermediates" {
    triggers = {
        subnet_cidr = "${cidrsubnet(....blahblahblah)}"
    }
}

resource "aws_vpc" "example" {
    cidr_block = "${null_resource.intermediates.triggers.subnet_cidr}"
}

This hack has some different pros and cons than the module hack I was using before:

• You don't have to terraform get a bunch of silly do-nothing modules, which is nice.
• Since null_resource is a resource, it generates a bunch of extra noise in diffs whenever the values change, which can be off-putting to those who don't know this pattern.
• You can pack an arbitrary number of "intermediate variables" into a single resource, so it can serve to group together a bunch of intermediates that are related in some way and also reduce the amount of diff noise.

A long-term solution to this could potentially build on the architectural change described in #4149. An implication of that change is making the concept of "computed" more pervasive in Terraform, so that the various complex rules around what kinds of interpolations are allowed in different contexts can potentially be simplified into the three cases "allows all interpolations", "allows all interpolations but causes a deferral when computed" and "allows only non-computed interpolations".

[jonapich]

The null_ressource looks great! Definitely opens the door to better maintainability and readability. The doc is is sort of hidden down there in the providers section, which I haven't had a need for so far. Maybe it deserves a quick mention in the variable interpolation doc?

[igoratencompass]

@apparentlymart amazing, it works! Would have never thought of using null_resource this way, thank you!

[darrin-wortlehock]

@apparentlymart - having trouble grokking your workaround. In your example you reference ${null_resource.triggers.subnet_cidr}. Should that be ${null_resource.intermediates.triggers.subnet_cidr} ? I'm getting an error missing dependency: null_resource.triggers when I try and use your example, but an invalid syntax error when I include intermediates. Thanks.

[apparentlymart]

@darrin-wortlehock yes, sorry... you're right. I've corrected the example to use null_resource.intermediates.

[plombardi89]

I use the template_file resource to do this...

resource "template_file" "environment_fqn" {
  template = "${var.environment_type}_${var.environment_region}_${var.environment_label}"
  lifecycle { create_before_destroy = true }
}

Then you just use rendered to get the value later.

Is there anything wrong with doing it this way? I haven't run into an issue in my limited use of it.

[apparentlymart]

@plombardi89 that's an interesting hack... you're actually creating a template with _no_ interpolations in it, since the interpolations in your string are handled before the template is parsed, and then "rendering" that template.

This has a similar effect to my null_resource example above, but there is a caveat that if any of your variables expand to something that looks like Terraform interpolation syntax then the template_file resource will attempt to expand them, which is likely to produce unwanted results.

As long as it doesn't include any interpolation markers then it should work just fine.

[plombardi89]

@apparentlymart Good point. Not too worried about interpolation markers here, but I guess I could address it with an explicit vars blocks.

[joshrtay]

would really like to interpolate input variables.+1

[iamveen]

A welcome addition, indeed.

[andrey-iliyov]

+1

[geek876]

@apparentlymart : Referencing your null_resource example, what if there is more than 1 subnet_cidr and want to choose that via a variable ?

resource "null_resource" "intermediates" {
    triggers = {
        subnet_cidr_1 = "${cidrsubnet(....blahblahblah)}"
        subnet_cidr_2 = "${cidrsubnet2(.....)}"
        subnet_cidr_3 = "${cidrsubnet3(.....)}"
    }
}

resource "aws_vpc" "example" {
    cidr_block = "${null_resource.intermediates.triggers.<variable_to_choose_cidr>}"
}

[apparentlymart]

@geek876 that is a good question!

Something like this may work, but I've not tested it yet:

resource "aws_vpc" "example" {
    cidr_block = "${lookup(null_resource.intermediates.triggers, var.var_to_choose_cidr)}"
}

This presumes that the var_to_choose_cidr variable contains something from your trigger map, like "subnet_cidr_1".

[geek876]

@apparentlymart. Thanks. However, this doesn't work. cidr_block doesn't get evaluated if we do the lookup trick.

[apparentlymart]

@geek876 I'm sorry, you're right. I'd briefly forgotten the shenanigans that Terraform does to make lookup work.

[serialseb]

I've been abusing this a bit on some automation scripts. The problem is that you can't use the output of null_resource.triggers, or even length(null_resource.stuff.*.id) as it barks with a resource count can't reference resource variable: null_resource.environments.*.triggers.name.

I'd find it very useful to have a let, that can only interpolate, just like count, other variables, but be internal, in the sense that it cannot be set by the command line.

variable "environment_names" { default = ["dev","uat"] }
let "environment_count" { value="${length(var.environment_names)}" }
// or even a shorter version
let "environment_count = "${length(var.environment_names)}"

resource "null_resource" "env" {
  count="${let.environment_count}"
}

That would make, combined with modules taking in lists and maps, my modules much easier to read, without any of the text processing tricks i currently use.

[fwisehc]

I guess one could use a template resource here:

resource "template_file" "example" {
  template = "${hello} ${world}!"
  vars {
    hello = "goodnight"
    world = "moon"
  }
}

output "rendered" {
  value = "${template_file.example.rendered}"
}

[echohack]

+1 Terraform desperately needs the ability to interpolate variables in variables.

[drsolaris]

+1

[apparentlymart]

FYI to those who have been using some of the workarounds and hacks discussed above: in Terraform 0.7.0 it will be possible to replace uses of null_resource with null_data_source, and resource "template_file" with data "template_file" (see #6717) to make these workarounds behave a bit more smoothly.

[dennybaa]

The lack of intermediate variables or input vars interpolation can mitigated with use of the null_data_source in v0.7 the following way:

variable "project_name" {}

# Defaults for discovery
variable "discovery" {
    default = {
        backend = "consul"
        port = 8500
    }
}

# Data source is used to mitigate lack of intermediate variables and interpolation
data "null_data_source" "discovery" {
    inputs = {
        backend = "${var.discovery["backend"]}"
        port = "${var.discovery["port"]}"
        dns = "${lookup(var.discovery, "dns", "consul.${var.project_name}")}"
    }
}

output "discovery" {
    value = "${data.null_data_source.discovery.inputs}"
}

[adampats]

There are other ways to achieve this as of 0.7.0 - I'm not sure I understand the purpose of null_data_resource other than being an arbitrary location to perform non-DRY computations / map merges instead of an inline interpolation elsewhere. It does appear to be necessary for output, but I've found that you can actually modify the values of a map variable either inline or using a series of merge + map methods, at least within a module type resource:

Assuming this map variable:

variable "sns" {
  type = "map"
  default = {
    name = "sns_topic"
  }
}

I can add a key/value using inline addition / modification:

module "my_module" {
  source = "./my_module"
  mod_map = {
    name = "${var.sns["name"]}"
    display_name = "foo"
  }
}

Or alternatively I can add a key/value using merge/map methods:

module "my_module" {
  source = "./my_module"
  mod_map = "${merge( var.sns, map("display_name", "foo") )}"
}

However, when using map variables of 10+ key/values, this can get very ugly and will give your local DRY fanatic a twitch.

It would be nice if this new null_data_resource had some sort of cleaner hash merge type functionality to modify map variables:

data "null_data_source" "new_sns" {
    source_var = "${var.sns}"
    inputs = {
        display_name = "foo"
    }
    merge = "${var.other_map_var}"
}

But perhaps this is just me trying to make a map variable more like a mutable ruby hash like object, I don't know.

[apparentlymart]

I feel I ought to clarify my earlier comment: My use of null_data_source for is a hack, not a suggested, supported path, just like the earlier suggestions of using a module and null_resource.

null_data_source is primarily there as a cost-free way to test the data source mechanisms during core Terraform development, and pressing it into use as a stash of intermediate variables is a workaround at best, and won't be without its gotchas.

[mcqj]

+1 for interpolation of other variables when defining a new variable.
Not having this feature means terraform is not DRY

[garo]

Another use-case: I have a module which defines an aws_instance. I want to have a default tags for that instance like so:

variable "overwrite_tags" { # These are passed from parent when this module is instantiated
    type = "map"
}

variable "default_tags" {
    type = "map"
    default = {
        "Name" = "${var.hostname}-${var.settings["vpc_name"]}"
    }
}

Then I would want to set the tags field in the aws_instance like so: tags = "${merge(var.default_tags, var.tags)}"

Unfortunately this is not possible :(

[cchildress]

This would be a huge improvement for the Azure provider. The azurerm_virtual_machine resource needs a URI for where to store the hard drive image file and this is often region dependent.
I'd really like to be able to declare which region my virtual machine is going into when I implement a module and let a map lookup handle the rest.

In the main.tf for the module:

  storage_os_disk {
    name = "${var.node_name}_os_image"
    vhd_uri = "${lookup(var.node_os_storage_account_blob_endpoint, var.node_region)}${azurerm_storage_container.node_container_os.name}/foo/bar.vhd"
    caching = "ReadWrite"
    create_option = "FromImage"
  }

In the variables.tf:

variable "node_os_storage_account_blob_endpoint" {
  type = "map"
  default = {
    iowa = "${azurerm_storage_account.iowa_storage_os_standard.primary_blob_endpoint}"
    virginia = "${azurerm_storage_account.virginia_storage_os_standard.primary_blob_endpoint}"
  }
}

Then to implement it just add node_region = "iowa" in the module implementation.

[philidem]

The inability to create a variable whose value is computed from other variables was one of the first limitations that I encountered as a new user to Terraform. Just wanted to offer that perspective as someone just learning Terraform.

[tom-schultz]

This is also a blocker for me. I'm creating EC2 instances and want a default set of tags. These tags should have the owner, deploy type, and name which I want passed in as variables. Doh!

[tomstockton]

Another request for this. A new user to terraform, I really want to use it as a replacement for all my custom Python / Troposphere code.

My simple use case (similar to others above) is to define standard tags for resources (customer, project, environment) each of which will be defined as variables. I then want to create an additional 'name_prefix' variable which is a concatenation of these tags.

Is this feature on the development pipeline? Would be good to know how best to deal with this right now.

[rorychatterton]

Running into a similar issue when trying to interpolate an intermediate variable, except I'm using it to reference data in a remote state:

In remote state, I have the following data

data.terraform_remote_state.rs_core.subnet_mgmt_id
data.terraform_remote_state.rs_core.subnet_dmz_id
data.terraform_remote_state.rs_core.subnet_app_id
...
data.terraform_remote_state.rs_core.subnet_<subnet_type>_id

Inside a module, I want a user to be able to pass the variable "subnet_type", then use it to build a call to the remote state for the subnet_id.

I attempt to do this with the following syntax:

variable subnet_type {}
...

resource "azurerm_network_interface" "ni" {
...
  ip_configuration {
    ...
    subnet_id = "${format("%s_%s_%s", "$${data.terraform_remote_state.rs_core.subnet", "${var.subnet_type}", "id}")}"
    }
}

if I pass "subnet_type="mgmt", it outputs the following state with terraform plan.

+ module.vm_active_directory_2016.azurerm_network_interface.ni
    ... ...
    ip_configuration.2915114413.subnet_id:  "${data.terraform_remote_state.rs_core.subnet_mgmt_id}"
    ... ...

Instead of the data that is referenced by ${data.terraform_remote_state.rs_core.subnet_mgmt_id}

ip_configuration.2915114413.subnet_id:  /KeyURL/.../...

I figure trying to build the tag out of "format" is hacky, however I'm unsure if there is a supported method to substantiate the "format" function inside of the "data" function.

Has anybody got any advice how to approach this issue?

I've tried building it into a Null_reference and pass it, but haven't had much luck.

edit:

Solved for my use case, and documenting in case anybody is looking to do similar.

I've changed my output for my subnets remote state to output the following:

output "subnet_ids" {
  value = "${
    map(
      "dmz", "${module.network.subnet_dmz_id}",
      "mgmt", "${module.network.subnet_mgmt_id}",
      "app", "${module.network.subnet_app_id}",
    )
    }"
}

Which can then be referenced as a map at data.terraform_remote_state.rs_core.subnet_ids

I then reference them in the remote state using:

subnet_id = "${lookup("${data.terraform_remote_state.rs_core.subnet_ids}","${var.subnet_name}")}"

[MichaelDeCorte]

+1

[missingcharacter]

Seems like null_resource triggers no longer work like mentioned in https://github.com/hashicorp/terraform/issues/4084#issuecomment-176909372 on terraform 0.9.5

Same code that worked in 0.9.4 is now giving me the following error:

1 error(s) occurred:

* module.app.data.template_file.userdata: 1 error(s) occurred:

* module.app.data.template_file.userdata: At column 3, line 1: true and false expression types must match; have type unknown and type string in:

${var.default_puppet ? null_resource.puppet.triggers.default_role : var.puppet_roles}

Code:

resource "null_resource" "puppet" {
  triggers {
    default_role = "${format("%s::%s", "${var.tier}", "${var.app_name}")}"
  }
}

data "template_file" "userdata" {
  template   = "${file("${path.module}/userdata.tpl")}"

  vars {
    puppet_roles = "${var.default_puppet ? null_resource.puppet.triggers.default_role : var.puppet_roles}"
  }
}

Was this change intentional?

[apparentlymart]

Hi @missingcharacter! Sorry for that regression.

It looks like you're hitting the bug that was fixed by #14454. That'll be included in the next release, which will come out very soon.

In the mean time, there are some workarounds discussed in #14399, if staying on 0.9.4 for the moment isn't an option.

[cemo]

@apparentlymart What is the technical difficulty in this issue? It seems to me current architecture can handle this without so much work, can't it?

[apparentlymart]

Indeed I think the work here is not too hard conceptually:

• New configuration constructs to represent these intermediate values.
• New graph node types for them, probably similar to the existing implementations of module variables and outputs.
• New interpolation variable type to reference them.

In this particular case the main blocker is not on this architectural stuff but more on the fact that making this sort of multi-layer change is something we prefer to do carefully and deliberately, and that anything involving configuration warrants a careful design process to ensure that what we produce can be clearly explained and understood.

As I've been mentioning in other issues, we are currently in the early stages of some holistic discussion about next steps for the configuration language, and this is one of the use-cases being considered for it. We prefer to approach this holistically so that we can ensure that all of the configuration language features will meld well together to produce something that is, as a whole, as simple as it can be.

A current sketch I have for the syntax here is as follows, but this is very early and not a firm plan:

locals {
  # locals are visible only within the module where they are defined
  foo = "bar"
  baz = "${local.foo}-baz"
}

module "example" {
  source = "./example"

  # can pass locals to other modules via variables
  foo_baz = "${local.foo}-${local.baz}"
}

Not sure yet if "local" is the right term here. Another one floated was value with the interpolation being val.foo, but that was feeling a little too close to var.foo and thus probably confusing.

Also leaning towards this being a separate construct -- rather than just allowing interpolation in variable defaults -- because that aligns well with the concepts of other languages, where function arguments (populated by the caller) are a distinct idea from local variables (populated by the function code itself).

Again, this is a very early sketch, and not necessarily exactly what this will look like. I expect we'll have more to say on this subject once we get to a more firm place on this language design discussion.

[cemo]

@apparentlymart Once again throughout explanation. Thank you for it.

[charlessolar]

I used several bits of information from this thread to solidify an ansible inventory file output from terraform.
Its hacky, but doesn't require installing new packages or running any other program

Created a gist for anyone wanting to save some time:
https://gist.github.com/volak/515016139f0014cdfc029c7dd553d597

Looking forward to further updates on intermediate variables - the limitation on count not being able to be computed and not being able to create a map of lists were .. difficult to sort out

[practicalint]

Glad to see this getting addressed. It is obviously a missing feature users are asking for. I found this thread on yet another search of how to workaround this and stop repeating complicated expressions. I came up with the file_template work-around somewhat on my own with hints earlier on, but it is clunky. @apparentlymart you got through the noise to the real root of the problem with the locals solution, as it is not a need to alter "input variables", but rather a need to have scratch areas to do some manipulations within a module execution. Bummer it didn't make the 0.10 release, but I'll look for it following.

[bhechinger]

@apparentlymart Please update us ASAP regarding when you expect this to make it to store shelves. This has been driving me insane. :)

I guess I should throw my current use case out there. Who knows, maybe there is a better way to what I'm doing. :)

I've started creating environments with the region encoded into them:

us-east-1:Production
us-east-1:Stage
us-east-1:Test

It would be lovely to split those into variables that can be used later. Otherwise I end up having to scatter element(split(":", terraform.env), 0) and element(split(":", terraform.env), 1) all over the place which is extremely ugly.

[nbering]

@bhechinger I stumbled upon this a while ago by @apparentlymart. Didn't make the feature freeze for 0.10.0. #15449

[bhechinger]

@nbering oh, thanks for that link, I expect to see status there so I'll watch that instead!

[JonCubed]

@bhechinger until that hits this is how I get around this issue currently

data "null_data_source" "configuration" {
  inputs = {
    aws_region   = "${element(split("+",terraform.env), 0)}"
    environment  = "infra"
    cluster_name = "${element(split("+",terraform.env), 1)}"
  }
}

and you would use it like
region = "${data.null_data_source.configuration.inputs.aws_region}"

[ghost]

@JonCubed chapeau, very elegant -- one can argue that this is good enough

[apparentlymart]

Hi all!

Indeed #15449 is the thing to watch to see when this lands. As you can see over there, we weren't able to get it reviewed and merged in time for the 0.10.0 cutoff but now that 0.10.0 is out, once the dust has settled on any urgent fixes we need to deal with, we should be able to get that in.

[brikis98]

TIL about the null_data_source thanks to this thread and @dennybaa example code. Very handy!

[apparentlymart]

Hi everyone!

I'm happy to announce that #15449 has just been merged for inclusion in the next Terraform release. This introduces a new concept called _local values_, which are distinct from variables. Whereas _variables_ allow a parent module to pass values to a child, _local values_ allow different parts of the same module to share a value by name.

This new feature can be used both to avoid repetition of complex expressions and to factor out constants that will be used many times and that should _not_ be overridden by a calling module. For example:

# find the current AWS region
data "aws_region" "current" {
  current = true
}

locals {
  # constant lookup table for AMIs
  regional_amis = {
    us-east-1    = "ami-7b4d7900"
    eu-west-1    = "ami-1446b66d"
    ca-central-1 = "ami-c2c779a6"
  }

  # make the selected AMI available with a concise name elsewhere in this module
  ami = "${local.regional_amis[data.aws_region.current.name]}"
}

resource "aws_instance" "example" {
  instance_type = "t2.micro"
  ami           = "${local.ami}" # get the region-specific AMI from the local value above
}

Local values, unlike variables, may contain interpolation expressions that refer to variables, resource attributes, etc, and may even refer to _each other_ as long as there are no cyclic dependencies.

Thanks to everyone in this issue for sharing their use-cases and for your patience while we got this designed an implemented.

Given that this is a long-lived issue with many watchers, I'd like to request that if anyone finds bugs in this new feature once it's included in a release that they open a new top-level issue rather than leaving a comment here, since that way we can keep the notification spam to a minimum and also avoid "crossing the streams" of trying to debug potentially multiple issues in a single flat thread. Thanks!