Terraform: Feature request: some kind of include directive

[m4rkw]

I like the simplicity of only processing .tf files in the current directory where Terraform is executed but this seems to make complicated environments even more complicated.

Example:

The logical hierarchy in our cloud environment is:

account => environment => region => product => service => customer

where account means an AWS or Azure account
environment is an environment within that account, eg dev/staging/prod

so this is how we organise the directories for our Terraform code. Underneath the customer directory are a bunch of workspaces. We have several products and services under any of the given accounts and have a bunch of config at several points in the hierarchy that need to be set for all of the services running underneath it.

to avoid repeating the account or env-level config we are currently using symlinks, so a workspace directory might look like this:

account.tf -> ../../../../../../account.tf env.tf -> ../../../../../env.tf main.tf

furthermore, the way modules are implemented means you typically will want the same module invocation block every time the module is called. to avoid having this invocation block repeated every time the module is called we abstract them, along with their input and output definitions, into files in a workspace-shared directory and symlink to them from the workspaces. we also typically need vars that are shared between all of the workspaces for a given service so those are in the parent directory with the workspaces. finally we also have a tags.tf template in order to standardise the tagging policy across our resources. so what we end up with is something like this:

account.tf -> ../../../../../../account.tf env.tf -> ../../../../../env.tf inputs.tf -> ../../../../../../../workspace-shared/core/some-service/security-groups/inputs.tf main.tf modules.tf -> ../../../../../../../workspace-shared/core/some-service/security-groups/modules.tf outputs.tf -> ../../../../../../../workspace-shared/core/some-service/security-groups/outputs.tf tags.tf -> ../../../../../../../templates/tags.tf vars.tf -> ../vars.tf

this is good from a DRY perspective but the heavy use of symlinks feels clunky and some of the developers are not very keen on it. we've discussed auto-generating these links at run-time rather than storing them in git, which wouldn't be very difficult to implement but would then make every deployment dependant on a link-generating algorithm and any changes to that algorithm would then potentially affect every deployment, which feels dangerous.

it would be really nice if there were some kind of include directive so we could load all of this stuff in from a static list within the workspace rather than having to litter our codebase with symlinks.

where this gets even more complicated is when we get to developer-owned infrastructure, where the bulk of the terraform code is in another repo but the workspace directory is in our [infrastructure] repo. i won't dump all the details of that here but safe to say it's even more symlinks at even more complicated paths, and there's very understandable and natural resistance to cross-repo symlinks being checked into git from some of our developers.

i realise terraform is quite a mature project and the fact that there is not currently an include directive suggests that either i'm missing something important or there is some kind of philosophical objection to having one, hopefully the former. i have tried experimenting with putting config into modules and referencing it via outputs but that just leads to other complicated problems and doesn't really do much to get rid of the symlink requirements.

in the absence of a better way, if i were to brush up on Go and roll my own include function is it something you would consider merging?

[mblakele]

related: https://github.com/hashicorp/terraform/issues/1478

[apparentlymart]

Hi @m4rkw! Thanks for sharing these use-cases.

As @mblakele pointed out, this seems to be the same request as #1478, so I'm going to close this one just to consolidate over there, but I'll link to this issue from that one to make sure we don't lose track of the use-cases you described here.

We're currently in the process of implementing a number of other improvements to the configuration language that'll be included in the next major release, so in terms of whether we'd accept an include implementation I think the answer is: not right now, since it is very likely to conflict with other work in progress, but our intent with #1478 was to revisit it once the dust has settled on the current work and see if other new features we're adding (in particular, being able to pass resource and module objects as a whole to other modules) can address some or all of these use-cases in a different way, avoiding having two different competing mechanisms for code reuse in Terraform.

[ebekker]

So since #1478 is locked down and won't accept comments, I'll put my thoughts here. The use of symlinks is nice if your configs are all local or you use some shared drive to store them all, but what about if you use Git? Won't all the symlinks get reduced to copies of the file?

[m4rkw]

@ebekker no

[ebekker]

Saw a few references, including this one that indicate Git will handle symlinks correctly -- but wasn't able to confirm they do so across platforms? So will they work correctly across Windows, Linux and MacOS?

[mblakele]

@ebekker linux and macos are ok. Historically symlinks on windows have been problematic

https://stackoverflow.com/questions/5917249/git-symlinks-in-windows

[korfuri]

Since #1478 is closed to discussion, I'll also add our use-cases for an include functionality here. Our use-case is very similar to the one described by @m4rkw. We mainly see two pain points where an include functionality would be necessary:

• The first use-case is setting environment-level, account-level and cluster-level info (mostly configuring providers, also setting a few local values e.g. locals { environment = "production" }). This is identical to @m4rkw's first example. These are generally files in high-level directories that get symlinked in each service's directory.
• The second and more complicated example is that our module-heavy setup has a lot of "pass-trhu" variables, that are required to support overrides, and that need to be defined for many services in a similar way. For instance, we'll have module modules/elb which is a basic module to provision an AWS load balancer. We'll then have modules/service/serviceA defining how to provision the resources for our service, "serviceA", and using modules/elb to create the service's ELB. The serviceA module is invoked in production/services/serviceA, staging/services/serviceA, etc. In most environments, serviceA uses ELBs with logging enabled, except in the test environment where we disable it, so the serviceA module has a variable "enable_elb_logging" { default = "true" }. The issue is that pretty much every service using modules/elb needs to define this same variable.

We already use symlinks for the first problem, and that works fine. We've had issues with people not realizing that the files are symlinks but we added linters who ensure that people don't accidentally copy the files instead of symlinking them.

For the second problem, we have a few options:

• Code repetition, which we use currently (i.e. serviceB, serviceC also have a variable "enable_elb_logging" { default = "true" } statement). This creates a lot of repetition and makes it hard to update the code but it has the advantage that if serviceX doesn't use ELB logging at all, it could simply not create the variable.
• Symlinks: create modules/elb/includes/pass-thru-vars.tf with variable "enable_elb_logging" { default = "true" }, then symlink it as modules/service/service[ABC]/vars-elb.tf -> modules/elb/includes/pass-thru-vars.tf. This is DRY but the symlinks get quite confusing, and devs get confused between when to use symlinks and when to use modules. We also see many cases where devs just cp a file instead of symlinking it.
• Homemade includes, i.e. use our own preprocessing on TF files - we've been trying hard to avoid adding a "build" step like that, and it wouldn't be compatible with Terraform Enterprise anyway.

These 3 solutions all feel hacky and/or brittle. I'd love to hear from someone who solved this problem elegantly.

[rosscdh]

+1 symlinking works but feels hacky

[gwvandesteeg]

I have a common set of map variables used for looking up data/settings that I'd like to reference in multiple modules to save on passing stupid amounts of settings through to the modules and repeating the variable definitions over and over (and passing maps through to modules is not fun/elegant).
A simple include directive would solve that problem nicely.

Hope some progres/announcement is coming soon.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.