Terraform: Add include functionality

[chrisferry]

Would like to follow DRY principles with my TF templates.
I currently have:

resource "terraform_remote_state" "vpc" {
    backend = "consul"
    config {
        address = "consul.domain.com"
        path    = "us-east-1/vpc"
        scheme  = "https"
    }
}

resource "terraform_remote_state" "inf-sg" {
    backend = "consul"
    config {
        address = "consul.domain.com"
        path    = "us-east-1/inf-security-groups"
        scheme  = "https"
    }
}

----- at the top of all my templates

would prefer the ability to use:

include "../../common/tf_remotes.tf"

[ocxo]

Are you looking for something like modules?

[chrisferry]

I use modules already for other functionality, but modules does not solve this use case.

[knuckolls]

I've wanted to be able to define my own resource templates that define defaults.

Something like...

resource_template "aws_instance" "my_aws_instance" {
  ... default values ...
}

resource "my_aws_instance" "foo" {
  ...specific stuff...
}

[ryanuber]

@knuckolls I did something exactly like that in the early TF days. The PR is probably totally not relevant now, but for reference here it is: https://github.com/hashicorp/terraform/pull/232

[knuckolls]

wow cool. modules certainly help dry things up but I feel like resource_templates would improve the readability of a lot of .tf files.

[tomzo]

I agree that modules are not the same.
An include that works more like c++ include would be useful.
It is to reduce text, not a concept in domain like module are.

[kaelumania]

I would also love to see this feature in the next tf release. I could also live with a simple C++ include, just copying the template into the tf file.

[kendawg2]

+1

[josephholsten]

@chrisferry what exactly is worse about modules than this straight string include? Would it be sufficient to have a shorthand for modules like

module "../../common/tf_remotes"

with the common code in common/tf_remotes/main.tf?

[tomzo]

what exactly is worse about modules than this straight string include?

I guess modules let you include whole sets of resources, you can use them at root level of terraform file.
include could be within resource so that you can have part of resource defined in separate file.

[kendawg2]

@josephholsten So my issue with modules (and perhaps I am not using them correctly) is needing to provide all of the extra logic for exporting and importing variables. If we look at the simplest terraform configuration, we would have a Variables.tf and a main.tf. The main would have all of the resources in it and Variables would be populated as needed from Variables.tf (or from within the main.tf file). All resources inherit those variables naturally.

To make it bit more readable, I have taken all of the servers and security groups etc, and made separate files of them. Because they are in the same directory, they get loaded automatically, just as though they were in the main.tf file, and thus naturally inherit all of the variables and pass information to each other without issue.

I have about a dozen environments where I have the exact same configuration, but some things change, like some are missing a few services, or the security groups are a bit different, but by and large the variables used are all the same, the values are the only difference.

What I would like to do, is have a tfvars file which is unique to each environment, and then create a main.tf which might have some tweaks or might be the same, but then include files from a master directory just as though they were in the current directory, but only the ones I want. I do that today by simply copying entire directories around and then making the tf files t_ files so they don't load if I don't want them to. I don't have to pass in variables, they just get loaded.

If I use modules, then I have to pass in and out variables, plus they get copied into a hidden directory anyway. With an include, they would just get loaded as needed. I know this could create some versioning issues, but that would be for the user to decide as they do now.

I believe this functionality existed in some form early on, but was removed. IMHO I think it's easy to do and a great option that would drastically reduce the number of times my files are copied around, even using modules. A module is like 5-10 lines of code in my main.tf while an include is one.

[stefanthorpe]

+1 for include
would make linking super simple. Modules have there part but are overkill in some cases

[danielcbright]

:+1: for include, this is a feature sorely needed

[MikeSchuette]

+1
At first I didn't agree with this, but I believe that was because the initial use case of passing around data actually threw me off a bit. The ability to keep your code DRY is far more commonly needed. As an example, I have my entire AWS VPC environment abstracted into one large module which takes a bunch of inputs. This module is then used by modules to represent a specific environment, like prod-us-east-1. Each one of them has to recreate the entire module call, when really they should be able to include a single call to the module and simply provide the variables.

As an addendum, I've seen instances where people are solving this now via symlinks and it's not pretty :)

[TheLinuxNinja]

+1

[sedan07]

+1. Would be very nice for variables and situations where exactly the same configuration is used across say multiple environments.

[graemedavidson]

+1

[kingpong]

:+1:

[akimyonoglu]

:+1:

[extraordinaire]

:+1:

[zoltanmaric]

:+1:

[Phylliade]

👍

[jennyfountain]

+1. We have three environments and modules just won't work in this situation.

[gpaiu]

:+1:

[peterticketfly]

+1

[ewold]

+1

[igormoochnick]

+1

[dennybaa]

+1

[defn]

+1 ... using symlinks to tf files works, too. Easy to grep *.tf to find where things are defined, too.

[berendt]

+1

[rorysavage77]

+1

[skandyla]

+1

[c4milo]

Please people, stop the +1 madness, just subscribe to the issue or click on the reaction emoji in the description of this issue if you really want to give your +1.

[bmurtagh]

+1

[jakubigla]

+1 comments should be banned. Any updates on that?

[apparentlymart]

It looks like this issue has already documented the main use-cases for such a feature, so I'm going to lock the conversation for now and we can keep this as a reminder of how include serves some different use-cases than modules, which might inform future core work in Terraform.

[apparentlymart]

Over in #18019, @m4rkw shared some examples of currently using symlinks to achieve include-like functionality to avoid duplication of identical calls to child modules.

As noted over there, once our current work for the next major release concludes, which includes a number of other improvements to the configuration language that should make modules more effective for reuse, we're planning to revisit this issue and see if some or all of the use-cases can be addressed with the new features being added, and thus whether a separate "include" feature still provides enough benefit to overcome the additional language complexity it implies.

More to come here once the dust has settled on the existing work.

[teamterraform]

Hi all! Sorry for the long silence here.

We looked at this again as part of planning the changes for Terraform 0.12, but ultimately decided not to move forward with it. Instead, we worked to improve the usability of Terraform modules so that module composition patterns would be more ergonomic. In particular, the _Data-only Modules_ section toward the end is describing a pattern meeting a similar use-case as the opening comment in this issue.

As others have pointed out in this discussion, modules do not cover _all_ of the same use-cases that a text pre-processor would cover. The Terraform language aims make it as straightforward as possible to read an unfamiliar configuration and understand what it does, and to understand the relationship between a set of planned changes and the configuration changes that produced it. Because Terraform modules are a structured decomposition construct, Terraform can understand the data flow between them and reflect that understanding into the UI, whereas simple text includes would require the user to manually review the code and understand how all the includes come together and what result they produce.

We understand that some users prefer brevity, and we'll concede that the Terraform language in general -- and modules in particular -- are not designed with brevity as a high-priority goal. Instead, the Terraform language is designed to be explicit and straightforward to read, which often leads to some verbosity and boilerplate.

As with all language design tradeoffs, there is no perfect answer and no design will please everybody. However, we hope that the new features of Terraform 0.12 will make modules more comfortable to use for some of these use-cases, and as usual we'd ask users with unusual needs to consider using Terraform in conjunction with other software to pre-process certain configuration files. The Terraform language's JSON variant exists primarily to ease the implementation of such preprocessing steps, but we know lots of users have successfully employed text-based templating of Terraform's native syntax too.

Thanks for the great discussion here, as usual. We understand that this outcome will be disappointing to some, but we hope that at least the above reasoning will help you to understand why we made this decision.