terraform get: can't use variable in module source parameter?

[amaczuga]

I'm trying to avoid hard-coding module sources; the simplest approach would be:

variable "foo_module_source" {
  default = "github.com/thisisme/terraform-foo-module"
}

module "foo" {
  source = "${var.foo_module_source}"
}

The result I get while attempting to run terraform get -update is

Error loading Terraform: Error downloading modules: error downloading module 'file:///home/thisisme/terraform-env/${var.foo_module_source}': source path error: stat /home/thisisme/terraform-env/${var.foo_module_source}: no such file or directory

[radeksimko]

I'm trying to avoid hard-coding module sources

Is there any particular reason behind that? Do you expect some modules to have the same interface, so you can swap these?

[mitchellh]

This is as intended. We should add validation that this isn't allowed.

The reason is simply that it breaks our compile -> semantic check -> execute loop. i.e. imagine if your C code could arbitrarily download new C files during compile/execution. Would be weird.

[amaczuga]

Do you expect some modules to have the same interface

yes, that is exactly my point - for the flexible running plans against various versions/forks of identically interfaced modules, without refactoring base terraform code

[amaczuga]

This is as intended.

Er. Forgive me - I'm lost here, due to labels - that is - marked bug, yet your comment suggest a wontfix

[radeksimko]

marked bug, yet your comment suggest a wontfix

The fix is to add the validation so you get something a bit more clear rather than "error downloading module" I guess.

[clstokes]

FWIW, this is something I wanted to do as well and found wasn't supported. In my case, I wanted to avoid duplicating git::ssh://[email protected]/... across tens or hundreds of files and do something like source = "${var.module_path}//modules/common-vpc".

[clstokes]

Also to set the branch/tag via a variable would be helpful...

[radeksimko]

Also to set the branch/tag via a variable would be helpful...

See https://github.com/hashicorp/terraform/issues/1145

[clstokes]

@radeksimko I'm familiar with ref as added in a recent version, but I'm suggesting something like source = "github.com/clstokes/terraform-modules//modules/common-vpc?ref=${var.module_branch}".

Are variables allowed at all in modules sources?

[mitchellh]

@clstokes They're not yet

[ketzacoatl]

+1 on this. I want admins and automated-ci to be able to specify the local path, allow flexibility to pull from git or filesystem, etc, but this is not possible without allowing interpolation in the source param.

[mitchellh]

This is not a bad idea but it is very hard to do with the current architecture of how modules work with Terraform. It also shifts a lot of potential errors away from a compile-time error to a runtime error, which we've wanted to avoid. I'm going to keep this tagged with "thinking"

[ketzacoatl]

@mitchellh, how are _compile-tile_ and _runtime_ differentiated in Terraform? Are you referring to tf plan vs tf apply? If this is the case, I would like to share my experience as a user has never built confidence in tf apply succeeding if tf plan succeeds.

Said another way, TF as it is right now gives me a lot of compile time _and_ runtime errors. For example, you can easily tell TF to create an SSH key that seems fine with tf plan but errors out with tf apply. Either way, my vote for unblocking this capability (understanding it isn't simple, given current architecture) stems from wanting the ability (as a user) to choose whether or not a variable in the module source is a good decision for my code. Thanks for listening :)

[pikeas]

+1, I understand why this may be architecturally tricky to get right, but it would be great to have on the admin/DRY side of things.

[kaelumania]

+1 I also think that the gained flexibility would outweigh the disadvantages.

[kokovoj]

:+1:

Use-case for this would be allowing for the flexibility to store module source in a variable for :

a. module source pointing at a corporate source control behind a corporate VPN

variable "your_project_source" {
  default = "https://your_src_system/your_project//terraform"
}

OR
b. use a local path on the dev box (after that src was already checked out locally, so don't need to be on the corporate VPN)

variable "your_project_source" {
  default = "/Users/joeshmoe/projects/your_project/terraform"
}

(and overriding one or the other in terraform.tfvars) and then

module "your_project" {
  source = "${var.your_project_source}"
  ...
}

[apparentlymart]

One very specific complexity with this is that currently modules need to be pre-fetched using terraform get prior to terraform plan, and currently that command does not take any arguments that would allow you to set variables. By the time plan is running, Terraform is just thinking about the module name and paying no attention to the module source, since the module is assumed to already be retrieved into the .terraform subdirectory.

Perhaps in some cases this could be worked around by breaking a configuration into two separate runs, with an initial run creating a remote state that can be consumed by the second run. Since terraform_remote_state is just a regular resource its configuration arguments _can_ be interpolated, even by things that aren't known until apply time, as long as a dependency cycle doesn't result.

This is of course not as convenient as creating everything in one step using directly-referenced modules, but maybe it's a reasonable workaround for some situations in the mean time.

[apparentlymart]

@kokovoj 's use-case, of switching to a different version in a development environment, got me thinking about how that gets solved in other languages.

When I have a problem like that in e.g. Go, NodeJS or Python I don't use any runtime features to solve it, but rather I just ignore the location/version of the module given in the dependency list and just install whatever one I want, exploiting the fact that (just like in Terraform) the "get" step is separated from the "compile" and "run" steps, and so we can do manual steps in between to arrange for the versions we want.

Terraform obscures this ability a little by storing the local modules in a directory named after the MD5 hash of the module name under the .terraform directory, so it's harder to recognize which one is which by eye... but you can, if you locate the right one, install it from a different source or modify it in-place. (I've done this several times while debugging, in fact.)

So with all of this said, perhaps Terraform could just be a little more transparent about where it looks for modules and embrace the idea that terraform get just installs the _default_ module locations, but it's fine to manually install from other locations, or even to write your own separate tool to install from wherever you want. If we went this route, the only thing that would need to change in Terraform is to switch to a more user-friendly on-disk module representation and to commit not to change it in future versions of Terraform.

(It would also be nice to extend terraform get to be able to handle certain overrides itself, but that is made more complex by the fact that there can be nested modules that have their own dependencies, and so such syntax would probably end up quite complicated if it had to happen entirely on the command line.)

[blalor]

This is definitely something I'd like to see implemented. I'd be using it to solve the issue of where to pull a module's source when running in local dev or CI environments.

[orclev]

I'm also looking for a good solution to this. It seems like at the very least terraform get would need to be changed to support providing variables and to look in the conventional places for the variable settings.

Is the hash that modules get stored under a hash of the source attribute? If so it seems like so long as you hashed it after performing the interpolation everything would still be kosher. You would have had to run terraform get again if you modify any of the variables, so it isn't possible to self-modify included modules during a particular run of apply because the module won't exist yet. If you terraform apply with a different set of variables than you supplied for terraform get then it likewise would fail since the module wouldn't exist yet under the appropriate hash.

[mmell]

:+1: for @clstokes use case: https://github.com/hashicorp/terraform/issues/1439#issuecomment-93838774

[yokiworks]

:+1: for @kokovoj

[jonapich]

What threw me off here is the ${path.module} interpolation. I have a hierarchy like this:

|- ./dev  <--- this is my cwd
|- ./modules
|- ./modules/specific_module
|- ./modules/generic_module

I am using nested modules, so I thought I would use source = "${path.module}/../another_module" when loading the nested one and ran into the "source cannot use interpolation" error.

Turns out the interpolation is not necessary in this case! (wat?)

• From ./dev I can module "mod" { source = "../modules/specific_module" } as expected.
• From ./modules/specific_module/mod.tf, I can create a module "mod" { source = "../generic_module" } and it works.

It is very surprising given the following documentation in the "How to create a module" "Paths and Embedded Files" section:

(...) since paths in Terraform are generally relative to the working directory that Terraform was executed from

In the above, we use ${path.module} to get a module-relative path. This is usually what you'll want in any case.

The section that follows, "Nested Modules", doesn't mention that loading a module from a module uses the module's path as the cwd as an exception.

[franklinwise]

+1 - It would be very helpful to be able to use a variable for the source, particularly targeting different branches.

[johnrengelman]

Another user case is providing credentials for accessing a private repository over HTTPS. Each user could set their own username & password then.

[robcoward]

As @johnrengelman mentioned, the use of private sources requiring user credentials is just not realistic if we are forced to hard-code the credentials. We have to be allowed to pass in credentials to private github repos etc via some mechanism, variable interpolation or whatever, especially if you are trying to use Atlas with Terraform, pulling config from a GitHub repo in the first place.

[ketzacoatl]

Standard/best-practices for accessing private repos on git is (in my experience) done with SSH and pub/private keys, not HTTPS with username/passwords.

[cloudynetwork]

@ketzacoatl For other uses cases that is a reasonable way to access private repos. However we are having to work around this lack of ability to authenticate securely via HTTPS by having external scripting download the private repos for us and then reference the modules locally...all a bit mess. I don't quite understand the resistance to being able to specify the credentials as a variable... Clearly people are in need of this.

[robcoward]

And how would you propose using ssh and pub/private keys to configure terraform to access a modules in a private github repo when running under Atlas ? Seriously, the only rational thing to do if running terraform through atlas, is to either allow variables to be used in module sources, or provide an alternative format for authenticating against private repos. You can't expect people to hard-code user credentials into a file that is checked into any form of repository, public or private.

[ocxo]

We created an outside collaborator in github that we granted read only rights to the remote modules it needs. This seems to be the least horrible way we've found to do this which doesn't compromise security of our repos too much.

[ketzacoatl]

@robcoward, another option would have Atlas adding support for the use case as well.

[cloudynetwork]

@fromonesrc However this still doesn't get around the need to actually pass a credential in. Whatever way we look at this, passing a credential (without having to hardcode it into the file) is a very basic and rather essential function.

Imagine if terraform didn't allow the passing of provider credentials such as AWS secret keys? It's logical to pass those ...so why is it not just as logical to be able to pass in the HTTPS credentials for github?

This features needs to be added already, no reason not to.

[ocxo]

Yep. I'm providing a workaround, not defending the current design.

[brannn]

What's the current Hashicorp position on this one?

Being able to dynamically call modules would be a breath of fresh air.

Consider a nested Consul module that is used to manage keys in multiple environments. Passing a Jenkins parameter that determines the module source during a 'terraform apply' build step would be a common scenario.

Having the same kind of interpolation freedom we get with providers would be very powerful in our situation.

[hmcgonig]

@heavyliftco I have a similar scenario to the one you listed. Being able to pass the source in as a variable (ultimately coming from the jenkins parameter) would make this feature so much more robust.
+1 for this idea.

[tchemineau]

+1 for this idea

[proffalken]

My use-case is subtly different and yet I hope will lend some more support to this idea.

I am writing a project to be opensourced in the near future that will use Terraform to provision the infrastructure, however, I need to be able to allow the end-user to decide upon the cloud provider of their choice.

I've tried to do this:

variable "cloud_provider" {
  default = "aws"
}
module "cloud_provider" {
  source = "./${var.cloud_provider}"
}

However I get the error above.

The idea is that simply through running export TF_VAR_cloud_provider the appropriate module could then be included for AWS, Azure, DigitalOcean, GCE etc.

[bam0382]

In my case, looking to reference file <module>/policies/role.json from a git based module, but after doing a terraform get, because the module path under .terraform/ is a bunch of numbers, can't reference that in the config.

module "iam" {
    source                        = "git::https://github.com/<org>/<iam_module>"
    ecs_role_file                 = "${path.module}/policies/role.json"

I get the following error:

Errors:

  * file: open <terraform_root>/policies/role.json: no such file or directory in:

${file(var.ecs_role_file)}

I want to be able to reference <terraform_root>/.terraform/<module_hash>/policies/role.json. Is there a variable that will do this?

[johnrengelman]

@bam0382 that's not really related to this issue, but have your tried passing:

ecs_role_file = "policies/role.json"

?

[iroller]

We also need to access private modules but do not want to hardcode password for the repo.

[xuwang]

There are use cases for dynamic module source and different auth methods. Before we have an "all-in-terraform" solution, this is what we do in a production ci/cd for now:

1. pull terraform in build space
2. pull modules from module repos with proper get and auth methods to a local module stage space.
3. in terraform, modules' source always are "hard coded" to the predefined module stage space, for example source="../../modules/foo/bar"
4. terraform get and apply

Sure the modules are double stored during the build, but it gives us more control on how and where to get the modules based on the build jobs and leave terraform alone to do what it does the best.

[ketzacoatl]

I've been doing more or less the same (relative paths with ../../tf-modules) for the last year+

This allows CI and developers to auth with git in the ways that are easy for them, and de-coupled from Terraform, as well as manage revisions/branches, and to hack on the modules as needed.. without disrupting TF or requiring complicated work-arounds.

[benjah1]

+1 for dynamic modules load. This will bring a huge flexibility into terraform,

[maxenglander]

I am very interested in this feature as well.

The idea of parameterizing the terraform get stage, as articulated by @orclev, and requiring a re-run of terraform get whenever the module source field changes (i.e., the MD5 hash no longer matches the value of .terraform/modules/<md5>), seems to me like it would be an adequate for all of the use cases discussed so far.

@apparentlymart I think your comparison to how this is handled in Go, NodeJS, etc. is useful. In NodeJS for example, I can reference a package in package.json from origin, a remote clone, or a local clone. I might even reference an API-compatible package with a very implementation (e.g. React vs Preact). In this view, terraform get is analogous to npm install, both of which occur prior to runtime, the difference being that Terraform also does static analysis on the dependencies it downloads.

If Terraform were able to treat a change in the value of a module's source the same way it treats the presence of a newly defined module (i.e., requires user to re-run terraform get), then I believe the concern voiced by @mitchellh about shifting compile errors to the runtime would be neatly handled. If I'm understanding Mitchell's concern correctly (sorry if I'm not), this approach would treat source variables similar to C macros evaluated with #ifdef in which the interpolation and evaluation occurs prior to compilation during a pre-processing stage. Granted, in Terraform, there would have to be limits on what could be interpolated: probably only user defined variables could be allowed; resource attributes would have to be forbidden (I believe this is the same as the situation with the count parameter for resources).

I believe this solution could also be adequate for the credentials use case, articulated by @johnrengelman and others. I think the best way for Terraform to handle this would be to exclude the user's credentials from the module path, so, given a source = ${var.credentials}:${var.uri}, the credentials would be validated only during terraform get, and only the value of ${uri} would be evaluated when forming the MD5 hash of the module downloaded to .terraform/modules.

I believe this solution could also be adequate for the use case articulated by @kokovoj where a user wants to refer to a local module that may have drifted from origin because it is under development. I think in this case, requiring the user to re-run terraform get in order to fetch and store the local version isn't a huge hassle and, in theory, it only needs to be done once (unless the user changes the path of the local module, likely a rare situation). It might be a minor waste of disk space to have one locally downloaded module in .terraform/modules/<md5> that represents the origin module, and a nearly identical one representing the local dev version, but otherwise I don't see a drawback to allowing this duplication.

Finally, I think this solution would be adequate to the "driver" use case articulated by @proffalken. I don't think there's any difference from the viewpoint of Terraform between this use case and the local-vs-remote path use case.

Would be interested in hearing from others whether the approach of "parameterizing the terraform get stage" would address their use case, and from the HashiCorp team about whether this sounds like a feasible/reasonable approach. As a side note, I'd be very interested in taking a stab at this :).

[esword]

Adding a slightly different use case, similar to the credentials use case - we mirror our shared modules from a private GHE server into an external git repo. Initial testing and even plans are run from machines that have access to the internal GHE server, but the actual "apply" is run from systems that do not have access to our internal network and must access the modules from the external git repo.

[blaltarriba]

+1 for this feature.

I´m start using Terraform and when I will have dozens of projects using modules, if in some moment I decide to move a module to new source then I will have to change the source field in each project.

It would be great to be able to set the source field with interpolation sintax.

[igormoochnick]

+100

I'm at exactly this point in time that @blaltarriba mentioned - our terraform structure has reached massive size and I've started to move modules around into different locations and have to support versions for the nested modules

I need a way to split the module source path into 2 or 3 parts:

1. "modules root" location - this is optional but I wanted to use it to control the location of the local or git-based modules
2. "module id" - name of the module. This may be combined with #1
3. "module version" - this is the MUST. Nested modules versions have to be different for different terraform runs

[ketzacoatl]

FWIW, @igormoochnick, I recommend simplifying. As an example: I have multiple environments in multiple accounts, all of which progress at their own pace, but still source from the same code base. I've found it's often easier to maintain multiple checkouts of the repos on disk, instead of a) using the git syntax for sourcing modules, and b) having some sort of versioning or other use of variables.

[brikis98]

@xuwang's workaround is awesome. The fact that it works means there is nothing inherently wrong with dynamically specifying the path to a module, and since it can be quite useful for a variety of use cases (e.g. fast iteration while developing a module locally, controlling module version numbers in a single, central place, etc), there should be a way to do this within Terraform itself.

Perhaps Terraform should allow interpolation in the source parameter in "limited" way similar to the count parameter's limitations. Currently, you can use interpolation in the count parameter, but that interpolation is limited to data that can be resolved locally. For example, you could set count = "${var.some_variable}" but you could not set count = "${data.some_dynamic_data_source.value}". It seems like local variables could be processed during terraform get and could handle most of the use cases people are asking for in this issue.

[xsellier]

I'm working on a micro-services architecture.
Each service has its terraform recipe. Since I'm using the standard git-flow, I use the develop branch to deploy on develop, and master for production. Each service has its own git repository, and because its a micro-service architecture, I have another git repository to specify which service I have to deploy.
And this repository is _generic_. Meaning, it aggregates terraform recipes and deploy them. But it has to deal with branches (due to git flow). And because I don't want to hard-code everything, I need this feature !

module "gitlabrunner" {
  source = "git::ssh://[email protected]/service.git//deployment?ref=${var.branch}"

  environment = "${var.environment}"
  availability_zones = "${availability_zones}"
  cluster = "${var.cluster}"
}

I have a _workaround_:

# Terraform file
module "gitlabrunner" {
  source = "git::https://gitlab-ci-token:{CI_BUILD_TOKEN}@gitlab.hibernum.net/service.git//deployment?ref={BRANCH}"

  environment = "${var.environment}"
  availability_zones = "${availability_zones}"
  cluster = "${var.cluster}"
}

# Shell script
sed -e 's/{CI_BUILD_TOKEN}/'${CI_BUILD_TOKEN}'/gi' -e 's/{BRANCH}/'${BRANCH}'/gi' main.tmpl > main.tf

[dcharbonnier]

really annoying to not be able to set the authentication to a private git source in the tfvars, we end up with pre-terraformn scripts with sed like @xsellier but it's not what we expect from terraform.
We already have this as a workaround for the remote state... one more...

[mfischer-zd]

Has anyone considered using a $HOME/.netrc file for the GitHub HTTP authentication use case? (I'm showing my age here, but sometimes ancient solutions still have modern applicability.) The go-getter library used to fetch modules runs the git command, which uses libcurl under the hood, which in turn supports the .netrc file out of the box.

[Ginja]

We fetch our modules from an S3 bucket, and it works pretty well. It uses the exported IAM credentials in our CI pipeline.

[leigh507]

@mitchellh Any chance of getting an update on your thinking on this one? Really desperate for this feature, we want to create a proper CI pipeline for our terraform code. We would like to be able to release a single var file with the github branches that are good to move to prod.

[ghost]

Admittedly, we don't have a problem with the private module use case, since we haven't had issues using them via public key auth in CI pipelines:

module "private_repo" {
  source = "git::ssh://[email protected]/myproject/somerepo.git?ref=tags/v1.0
  username = "AzureDiamond"
  password = "hunter2"
}

However, parameterizing the tags is something we would love to do, but we end up having our wrapper scripts that install the SSH keys do a sed replacement, similar to what another comment mentioned.

This would be a nice thing to have, but I'd be interested in exactly how to do it. It would mean variable interpolation would need to happen before terraform get pulls the repo. Moreover, nested module retrieval would require interpolating, then pulling modules, and then interpolating those module variables before pulling the next module... and so forth recursively. Seems challenging. Given the graph changes that make #953 more doable, maybe this has hope.

[GarinKartes]

Parameterizing the source for modules seems like a must at this point for us. I am debating writing a script that will fill this value in and write a tf file for me since we can't have it interpolated.

module "workspace" {
  source = "git::ssh://...//v2/workspace/${var.app_name}?ref=${var.branch}"
}

this would allow the project/branch to be looked up dynamically from a shared file module

[hflamboauto1]

My use case:

my_lambda_0.0.1-beta.zip is lambda function and code is packed and published into a blob server (e.g artifactory, or any http(s) url) with a custom file deploy.tf that allows to plug this into a terraform module.

deploy.tf only contains a helper terraform output to help to find/reference the expanded directory on .terraform/modules

output "path" {
  value = "${path.module}"  
}

terraform deploy

# trigger the download / unzip of distribution as "terraform module":

module "lambda_blob_url" {
  source = "http://127.0.0.1/blobs/lambdas/my_lambda_0.0.1-beta.zip"
}

versus

variable "blob_server" {
     default = "http://127.0.0.1/"
}

variable "blob_context" {
     default = "blobs/lambdas/"
}

variable "lambda_name" {
     default = "my_lambda"
}

variable "artifact_version" {
     default = "0.0.1-beta"
}

# trigger the download of distribution as "terraform module":

module "lambda_blob_url" {
  source = "${blob_server}/${blob_context}/${lambda_name}_${var.version}.zip"
}



md5-b04dbb7857b60dacde0961571dbdaa70



data "archive_file" "lambda_deployment_zip" {
    type        = "zip"
    source_dir  = "${module.lambda_blob_url.path}"
    output_path = "${path.module}/${var.lambda_name}-${var.lambda_version}.zip"
}



md5-fcdcbc66c20c1d521ac91fa177d21905



# AWS Lambda function
resource "aws_lambda_function" "my_lambda" {
...
}

[ju2wheels]

+1 for module source parameterization. I really need a way to do this so I can customize the selected ref for dev, staging, prod etc. Not much of the fan of the other workarounds so im just going to create a common dot directory with all my shared module files and symlink all files into env specific folders with different tfvars for each environment.

Either way all of these workarounds are more work than is really necessary, in the long run, I dont particularly care if this becomes a runtime issue, the added effort on the end user doesnt justify keeping it the way it is IMO, and alternative solutions like Terragrunt arent really a solution if I need a third party addon that does nothing more than move these hard coded parts into tfvars.

[pll]

I too have the need to use variables in source = lines, which I've explained here: #14745

The short story: I want to use gitlab API tokens to access my repos via https like this:

module "sample_app" {
source = "https://gitlab-ci-token:${var.API_TOKEN}@gitlab.com:/sample-project/sample-module.git?ref=master//terraform_code"
...
}

[kevinpostlewaite]

+1 for module source parameterization, or otherwise specifying the location for a particular location for modules in only one single location. My use case isn't as complicated as the others here but it would definitely make it easier when I need to work on our in-house modules if I can specify the source location in a single location.

[jedi4ever]

+1 on this , my use case is switching between different branches for different environments and developing local

[WalkerGriggs]

+1 It has been said many times before, but source parameterization is crucial for keeping things DRY -- doubly so larger scale enterprise infrastructure.

[fmartingr]

+1 on this, my use case is to have a module with different variables per environment.

[cmacrae]

Checking in to express my interest, also.
For using private GitHub repos in the module source, it sucks to have to put the credentials in plain text. Looking forward to movement on this 🙂

[ketzacoatl]

I haven't commented on this in a while, but have been following along. I have two things to share:

1) For me personally, I have found a way to not care too much about this issue but using git sub-trees and otherwise including my upstream TF repos in a /vendor path within each project repo. Honestly, it works and scales pretty well, but I also work with a lot of different clients/projects/repos, so there's some overhead involved (not enough for me to care, as this use of git still lets different branches of the env to modify the upstream repos or point the subtrees at different branches of those upstreams).

2) This issue is less about "allowing variables and interpolation in a module's source parameter", and it is more about having better support for versioning/branches with modules. One sub-group/use-case there is developer friendliness, and the other is about CI pipelines. It seems TF could address the "allow interpolation in source" by giving modules some of these capabilities - for example, if source is a git URL, there could be an optional module parameter git_revision, or similar. Perhaps that is a better direction to take this problem?

[hflamboauto1]

I completely agree with @ketzacoatl and I had some similar thoughts as he explain on point 2
At least some "simple" interpolation features on module source level would help a bit at least.

[apparentlymart]

Hi all! Thanks for the great discussion here. I noticed there's been a lot of new discussion recently but it's been a while since we checked in.

As noted previously, there is a chicken-and-egg problem with allowing interpolations in module sources, since the interpolation language assumes a certain amount of work has already been done, which includes parsing and inspecting modules, etc. So with this in mind, I think the solution here must be something different than allowing interpolations into source, but I would like to find a good solution.

The two most common use-cases I see discussed in this issue are:

• Wanting to use different branches, tags, or other revision selectors for modules coming from a version control system without changing the source code of the calling module.

• Wanting to insert credentials into the URL to obtain modules from non-public sources.

To me these feel like distinct use-cases that are worth solving separately.

---

The first of these could be addressed by what I'd proposed earlier (note: my previous comments were from before I was a Hashicorp employee) of providing a mechanism to override module source, either via some extra arguments to init or via a separate command like terraform get module.foo git://...?ref=test that can be run _after_ init to override the default source for module.foo.

Overriding this via CLI actions rather than variables opens the door to third-party tools to help manage more complex/custom cases. As I'd noted before, I see this as similar to running go get github.com/hashicorp/terraform and then going into that package directory and switching branches using regular git commands. Similar approaches are used in other programming languages, either by exposing directly the git repository or providing a special command to install a named package/library from a non-default source.

---

For the second of these, we usually encourage passing credentials via non-configuration and non-variable means, such as environment variables, since that avoids them getting included in things like the plan files, console output, etc. Unfortunately for those not using git over protocols other than SSH, the story for authentication with environment variables is rather lacking. It _is_ possible to do something with the GIT_ASKPASS environment variable to temporarily set a Credentials Helper, but indeed that's rather more work than ideal.

It seems like some more thinking is required on _this_ problem, since we need to find a compromise that weighs keeping the credentials "out of band" while allowing different credentials per module, but in the mean time that non-ideal GIT_ASKPASS solution may pass as a workaround in simple cases where the same credentials are used for all modules hosted in private repositories.

[mtougeron]

The first use-case for different branches is the primary driver for my needs. Having an approach similar to the init and backend config would be good for us.

[MichaelDeCorte]

There's also the use case for local sources. Relative path's aren't the best in some cases. It would be better for absolute paths relative to TF-Home

[pll]

@apparentlymart,

Thanks for thinking about this a little more deeply here. I do not pretend to understand the internals of Terraform, or even Go for that matter, therefore, I don't understand the chicken-and-egg problem you're discussing (naively, it seems to me that tf should be able to eval it's variables files, recognize that there's a variable on the RHS of an equal sign, and appropriately fill in the blank :)

As for the credentials issue, this is a big on when using private Github and Gitlab installations and trying to checkout multiple repositories. As I explained above, in order for terraform get to be able to checkout my module when running in an automated pipeline container, the user must have credentials to access the git repo. The acceptable way to do this is to set an API_TOKEN environment variable which is inherited by the pipeline runner. As such, I need terrform to be able to evaluate something l like this:

module "sample_app" {
  source = "https://gitlab-ci-token:${var.API_TOKEN}@gitlab.com:/sample-project/sample-module.git?ref=master//terraform_code"
... 
}

such that API_TOKEN is pulled from the environment, or, even pulled from a variables file which is built on the fly by my automation code. Using something like GIT_ASKPASS is a non-started, since in an automated CI/CD pipeline scenario, there is no one to ask for said credentials. I suppose GIT_ASKPASS could be set to some external script to automatically pull the environment variable and pass it to git, but that seems rather hackish to me.

I guess I don't understand what's so special about the source = line. When I create a module, I'm passing all sorts of variable assignments to my module. Before tf can pass those variables to the module, it has to know the values on the RHS to assign. If it can evaluate variables on the RHS of parameters being passed into the module, why can't it evaluate the variables on the RHS of the source = line?

I am not asking this to be a jerk, I honestly don't know how TF does things behind the scenes, and I don't know Go either. It (again, naively) seems like it's a simple templating issue that I'm currently considering by using python and jinja2 to solve because I need this capability pretty badly.

Thanks.

Paul

[apparentlymart]

Indeed setting GIT_ASKPASS to a non-interactive script that reads credentials from some source was what I was proposing as a workaround. This is the solution Git provides for customizing its authentication mechanism, but I agree it's not very ergonomic for this use-case and would like to find something more user-friendly in the long run. Was just offering that as a path forward for those who need something _now_.

---

As for why interpolations can't work here, there are lots of different reasons why this is more complex than it first appears, but the primary one to consider is that variables aren't the _only_ kind of interpolation expression, and so the interpolator expects to have enough context to be able to resolve things like aws_instance.foo.id and module.baz.bar.

You're right that in principle there could be an entirely different interpolation mode that _only_ supports variables, but that's a pretty significant shift and lots of duplicated code, and doesn't really solve the problem anyway since the goal (as I've defined it, anyway) is for the credentials to _not_ be in the URL where they would then appear in the console output, disclosing the secret to anyone who can see the console output.

Believe me that if doing this with interpolations _were_ straightforward and _did_ address the problem completely then this is definitely the solution I'd favor!

[ju2wheels]

The first of these could be addressed by what I'd proposed earlier (note: my previous comments were from before I was a Hashicorp employee) of providing a mechanism to override module source, either via some extra arguments to init or via a separate command like terraform get module.foo git://...?ref=test that can be run after init to override the default source for module.foo.

@apparentlymart please no. My teammates already want to rip my head off for the number of commands they have to run per module (init, env select, plan, apply). I would not want this as a separate command to add to this chain. I would rather have something like terraform init -backend-config <file> -module-config <file> and be able to do it all via init (even if internally it does use the get mechanics). This would allow me to get the same effect of being able to specify a ref for each module as if I had done it via a variable.

[suneeta-mall]

+1. This is currently blocking us too and using sed to workaround this! Would be nice to have this feature.

[RobertFr0st]

• 1, would like to see!

[mukund1989]

pretty please!

[mcoulombe636]

Not being able to interpolate module's source location is a huge deal breaker for me. Any decent sized project that wants reusable parts would need something more flexible than hardcoding the files' locations.

[apparentlymart]

Hi everyone! Thanks again for the great discussion here, and thanks to everyone for confirming and clarifying their different use-cases.

I've captured the two big ones we discussed before in their own issues, so we can have a more focused discussion about each of them separately:

• #15613: Changing module version numbers without editing config
• #15614: Passing credentials for module sources without putting them in config

The first of these has some notes about a possible design building on some internal discussions we'd had on this subject. The second doesn't currently have a design proposal attached, but we can discuss some more the requirements and see if we can get to one.

---

Regarding @MichaelDeCorte's use-case of module sources that aren't relative to the source module, it is intentional that modules are specified relative to one another when they appear in the local filesystem, since we believe that reduces coupling by avoiding the modules needing to "know" where their root is. At this time I don't expect we will make changes to support absolute paths here, since that would complicate the model and add more different situations to document and test.

However, if you believe you have a compelling use-case that would warrant the additional complexity here, please feel free to open a fresh issue for this with some more details on the use-case. It's easier to have discussions about these things with concrete examples to work from, since that way we can potentially find alternative implementations that solve some or all of the same problems with fewer drawbacks.

---

Given that there are now some more specific issues covering the main use-cases here, and this is progressing from general "thinking" to more specific design work, I'm going to close this issue and encourage further discussion in those issues. If anyone has an additional use-case not covered already in this comment then please do open a new issue referencing this one -- ideally following a similar format to the ones I opened above, focusing on a problem statement rather than a candidate solution -- and we'll then be able to have a more detailed discussion about each separate use-case and find the best approach for each.

Thanks again for all the discussion here! People sharing their use-cases is always very helpful to feel out the shape of a problem and explore solutions for it.

[jjshoe]

@apparentlymart how about when I want to use one method to find a module, and if not found, fall back to another?

Think local development, before I push my changes up to git.

edit: https://github.com/hashicorp/terraform/issues/1439#issuecomment-101150068

[boostrack]

Private GitHub Repos If you need Terraform to be able to fetch modules from private GitHub repos on a remote machine (like Terraform Enterprise or a CI server), you'll need to provide Terraform with credentials that can be used to authenticate as a user with read access to the private repo. First, create a machine user on GitHub with read access to the private repo in question, then embed this user's credentials into the source parameter: module "private-infra" { source = "git::https://MACHINE-USER:[email protected]/org/privatemodules//modules/foo" }

Note: Terraform does not yet support interpolations in the source field, so the machine username and password will have to be embedded directly into the source string. You can track GH-1439 to learn when this limitation is addressed.

SOURCE GH-1439

https://www.terraform.io/docs/modules/sources.html

@apparentlymart maybe the docs should not reference this "closed" issue

[simplytunde]

Yes, the issue is already closed but the Doc says to keep tracking the same issue. Any suggestion on what to put instead?

[boostrack]

maybe just remove ?

Note: Terraform does not yet support interpolations in the source field, so the machine username and password will have to be embedded directly into the source string. You can track GH-1439 to learn when this limitation is addressed.

SOURCE GH-1439

The worst than not docs is no accurate docs. maybe just that interpolations in the source field are not supported

[apparentlymart]

Hmm thanks for pointing that out, @boostrack. We don't normally make direct references to github from the website so I didn't even think to check for that. For the moment I'm just going to remove that link altogether since we don't have any good means to ensure that this sort of problem won't happen again, and github issues are too transient to be referenced in the long-lived documentation.

[alevinetx]

The note for lack of source interpolation is buried under the Private GitHub Repos section. Because this section doesn't apply to me, I missed this note the many times I was looking at the page.

May I suggest you call this restriction out at the top of the page in the generic summary?

[TinajaLabs]

Perhaps this is the wrong place to bring this up, but how is one to build a system where one can create abstract modules that call to either an AWS deployment or an Openstack deployment.

I need to be able to deploy the same infrastructure into either environment and I'm not sure how to structure it. This is the beauty of "providers", right?

I should be able to build custom modules that have references to instance_image (ami for aws, image_id for openstack) and instance_type (instance_type for aws, flavor_id for openstack).

It seems one should be able to build a set of custom modules where the source is either a set of aws modules, or openstack modules. Hence the ability to turn this on a root level variable for source definition.

Is everyone else using a completely separate code base for each type of infrastructure deployment? How ultimately tedious and error prone that would be.

[ju2wheels]

@TinajaLabs Thats a discussion for the Gitter channel: gitter.im/hashicorp-terraform/Lobby

[dmikalova]

Can a warning be added that interpolations aren't allowed since there should be a validation?
Error loading modules: module name: invalid source string: ${path.root}/modules/name isn't very obvious at first.

[luisdavim]

I would like to at least be able to use variables to specify module versions

[nik-shornikov]

lack of interpolation is pretty stealthed-out in the doc and the error you get for trying is still opaque

[pixelicous]

Please implement this!! no more running away to use cases and finding workarounds, interpolation should have much greater support >.<

[ketzacoatl]

There are new issues to capture further discussion on more specific nuances of this topic.

[pll]

@ketzacoatl Can you please link to those new issues so we can follow those discussions?

Thanks.

[pixelicous]

@ketzacoatl i would love a reference to where this specific issue is solved, i need to interpolate the source of a module, is it possible to reference a different directory name based on a passed variable.. The variables should be graphed in a way that this would be attainable

[luisdavim]

We're maintaining a terraform wrapper to overcome some of these issues. We implement the Terrafile pattern. It would be a great improvement if terraform would support the Terrafile approach natively.

[coryjamesfisher]

I come from the software development side of the house and abstraction to an interface seems like a good move.

My ideal is to create an interface for a module representing one logical component of our infrastructure. Then each module worries about the provider specific details.

So for example I would create 2 modules:
module/aws/webserver
module/openstack/webserver
Finally a main.tf file with:
variable "cloud_provider" { default = "aws", description = "Pass aws or openstack" }
...
source = module/${vars.cloud_provider}/web

I'm looking for interpolation to work in the module source path.
If there is another way to do this I'd be happy to hear it. I have one idea of wrapping the Terraform script with a templating/replacement engine of my own driven by a shell script. But it'd be nice to have a solution in Terraform itself.

[pixelicous]

@coryjamesfisher Same issue as you here.. couldn't find a workaround besides using third party tools to handle this problem, which is really ugly.

I seriously do not understand the problem, there should be an option to init with the variable or based upon folder structure.. most of the workarounds and issues are caused due to this.. funny thing now when i started using workspaces, you cant even use the workspace name as interpolation for a default of a variable.. very ugly.. need to go through a local..

[eliblight]

+1 Here is my use case (would love to hear about a work around)

I have multiple projects/deployments, each needs to be deployed on multiple regions/zones.

following github.com/kung-foo/multiregion-terraform/blob/master/main.tf, I adopted the module per region approch.
For a single project this looks like this:

• main.tf

module "us-east4" {
  source = "per-region"
  region = "us-east4"
  zones = ["us-east4-a", "us-east4-b"]
}

module "us-west1" {
  source = "per-region"
  region = "us-west1"
  zones = ["us-west1-a", "us-west1-b", "us-west1-c"]
}
...

• per-region/main.tf

provider "google" {
  region      = "${var.region}"
  project     = "a-single-project"
}

resource "google_compute_instance" "gci" {
  count = "${length(var.zones)}"
  zone = "${var.zones[count.index]}"
  ...

• per-region/variables.tf

variable "region" {}

variable "zones" {
  type = "list"
}

BUT if I have multiple project, each and eveyone of them will need the main regions and zones file.

So I wanted to have the following workaround

• project-a/main.tf

module "deployment" {
  source = "../topology"
  dir = "../project-a/per-region"
}

• project-a/per-region/main.tf (same as before)
• project-a/per-region/variables.tf (same as before)
• topology/main.tf

module "us-east4" {
  source = "${var.dir}"
  region = "us-east4"
  zones = ["us-east4-a", "us-east4-b"]
}

module "us-west1" {
  source = "${var.dir}"
  region = "us-west1"
  zones = ["us-west1-a", "us-west1-b", "us-west1-c"]
}

• topology/variables.tf

variable "dir" {}

each project follows the same pattern

• project-X/main.tf - side stepping to topology and back
• project-X/per-region - specific deployment of each project per region

Alas this does not work.

Any workarounds / suggestions?

[Binternet]

+1 for that, I am organizing my AWS infrastructure to modules and trying to define each module path under /services folder. instead of hard-coding /services path in each module source parameter, it should be set as a variable.

[midacts]

Has this been implemented yet or on the books to be implemented?

[nbering]

To recap, here's the closing comment from @apparentlymart.

https://github.com/hashicorp/terraform/issues/1439#issuecomment-317085508

When closed, he split this issue into #15613 and #15614.

Also in https://github.com/hashicorp/terraform/issues/1439#issuecomment-91254101 @mitchellh, indicated that this is one of the places where variables cannot be used (by design) because of the execution order of various steps. I can't find anything to indicate that this was considered for change.

[lorengordon]

Personally I'd love to see interpolation for the entire source parameter. The chosen direction to implement support for just the version is very limiting. I'd rather like to pull all my source definitions to the top of a configuration, in a locals definition, so I don't have to go hunting through every file to find/update the string. I expect it would make modules much more maintainable overall.

[pixelicous]

@lorengordon I agree.. this is nonsense.. that and the fact that everytime you pull a whole repository instead of a leaf

[luisdavim]

We use this http://bensnape.com/2016/01/14/terraform-design-patterns-the-terrafile/ I think it would be reasonable to have something like that natively.

[jjshoe]

@mitchellh - It would be great if hashicorp could re-look at this. Though it's been closed, and split into two cases, which don't address all the reasons for this, it's more commented then any current open issue.

[akvadrako]

The best workaround I have found is by using putting something like this in override.tf

module "core" {
    source = "/Users/dev/terraform-modules/core"
}
...

Not ideal, but seems to work.

[MichaelDeCorte]

@mitchellh agreement with @jjshoe the original issue of allowing interpolation for the source parameter has not been addressed. This issue should be opened, or a new one forked off. But it should not be closed.

[MichaelDeCorte]

@akvadrako I'm not following your workaround. Can you elaborate?

[akvadrako]

@MichaelDeCorte It's just that it's possible to override the module source parameters with an external file. My use case is module development, where I want to replace several references to git repos with local checkouts.

[MichaelDeCorte]

@mitchellh elaborating an example to allow the for absolute paths relative to TF-Home. Assume the below directory / file structure. This is a common pattern where repo1 is a shared repository that is downloaded locally via a script as a workaround for the source interpolation issue.

TF-Home /

• app1.tf
• appSubDir1
  - app2.tf
  
  
  
  • repo1 /
    
    
    
    
    
    • foo1.tf
    
    
    
    • foo2.tf
    
    
    
  
  

Assume that app1, app2 and foo1.tf all depend on foo2. The source parameter would be:
app1: repo1/foo2.tf
app2: ../repo1/foo2.tf
foo1: foo2.tf

I hope it's clear that its not great.

[pixelicous]

@akvadrako
Guys the best method to get around it is to wrap your terraform in a script.

Our powershell wrapper does so many things to over come terraform restrictions, we cant use terraform without, basically we did something like the guys in terragrunt did, plus many more addons on it, i cant understand how somebody can even use terraform as is out of the box without some interpolation in those missing places..

anyhow, i really hope hashicorp will decide to change some parts of the product, because it is really constricting, some of those things should have been thought of much before

[luisdavim]

Yeah, we've been using the Terrafile approach (see my comment above) it works pretty well but it forces us to use a wrapper script, I think that the Terrafile pattern should be supported by Terraform.
This could easily be added to the get phase.

[nikolay]

The rationale to disallow this so that intelligent people can't download random modules is the same as not having a division operator as somebody may decide to divide by zero one day.

[tuusberg]

+1. Changing module versions manually is error prone.

[midacts]

How else can you do it. :o

On Sat, Oct 20, 2018, 10:17 AM Matthew Tuusberg [e-mail]
wrote:

+1. Changing module versions manually is error prone.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/hashicorp/terraform/issues/1439#issuecomment-431585637,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADxtkMTqJSkZ98V__pZRc_eVZVqyMbZfks5umzBjgaJpZM4D9Dyw
.

[richardgavel]

There is a similar issue in not being able to use interpolation syntax when providing configuration for back ends (say S3 bucket/region). You get around that by using terraform init -backend-config so that value is known at the beginning of the lifecycle. Couldn't something be done similarly (provide the value as some kind of command line param)?

[discreet]

This is something I've been wanting for a while and have been thinking a lot about. The use case I have is I wrote a bunch of terraform code to deploy a kubernetes cluster. In my code I have a variables module which lives in a git repo and contains all my input variables based on region and environment. This allows me to use the same exact code to deploy my kubernetes cluster to multiple AWS account and into multiple regions and environments with only changing two inputs to terraform apply. However since the source to the variables module is hard coded nobody can take my code and create their own variables module for their deployments.

[josephcaxton]

I need to be able to pass variable. Is Hashcorp looking to resolve this issue?

[GarinKartes]

With workarounds being provided and they intentionally made it this way, not likely we will see parameters in the source line.

I recommend using different folder paths and wiring up all relative pathing in your TF files.

From: josephcaxton [e-mail]
Reply-To: hashicorp/terraform [e-mail]
Date: Wednesday, December 5, 2018 at 6:30 AM
To: hashicorp/terraform [e-mail]
Cc: Garin Kartes [e-mail], Comment [e-mail]
Subject: Re: [hashicorp/terraform] terraform get: can't use variable in module source parameter? (#1439)

I need to be able to pass variable. Is Hashcorp looking to resolve this issue?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fhashicorp%2Fterraform%2Fissues%2F1439%23issuecomment-444504173&data=02%7C01%7Cgarin.kartes%40alaskaair.com%7C1692108d43a74281574e08d65abe4217%7C0f44c5d442b045c2bf55d0fea8430d33%7C1%7C0%7C636796170540379315&sdata=44aW3hZTTeccEDntjYPI03TeU11tqXtlJSKfJThwknk%3D&reserved=0, or mute the threadhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FARwnyDDvgV-3yvBNCAQes2gsVqzbYiZNks5u19iXgaJpZM4D9Dyw&data=02%7C01%7Cgarin.kartes%40alaskaair.com%7C1692108d43a74281574e08d65abe4217%7C0f44c5d442b045c2bf55d0fea8430d33%7C1%7C0%7C636796170540389334&sdata=99pGIuhS1Td8MJQahoDjOJnsCWJGguO6x9amTi4BZco%3D&reserved=0.

[major0]

While I can understand the reasons for not supporting general var/local inclusion .. I feel that many (all?) of the above use cases could be resolved by adding${path.root} to the list of allowed local module source prefixes.

[damon-atkins]

Or even something like source yaml_lookup://../lookupfile.yaml which contains module name and source pairs. Or some sort of cli option --source_overrides=something.yaml The value is saved in the state, and warns if anything is different to the last run.

[jmturwy]

Another example as to why this is beneficial:

`####################### Global value #######################
locals {
orgname = "acmeCorp"
}

################

module "iam" {
source = "./iam/customer/${local.orgname}"
org-name = "${local.orgname}"
providers = {
aws = "aws.customer-${local.orgname}"
}

}

################

module "s3-bucket" {
source = "./s3/customer/${local.orgname}"
org-name = "${local.orgname}"
env = "production"
providers = {
aws = "customer-${local.orgname}"
}
}

#

module "vpc" {
source = "./vpc/customer/${local.orgname}"
org-name = "${local.orgname}"
env = "production"
region = "us-westt-1"
cidr-octs = "10.7"
peer-account = "xxxxxxxxxxxxxx"
peer-vpc = "vpc-xxxxxxxxxxxxxxxxx"
peer-cidr = "192.10.0.0/16"
}`

this would be called acmecorp.tf, we would just copy this module and renamed it to loonytoons.tf and change the local var to loonytoons thus saving a lot of copy pasta

[non7top]

Adding to a comment by richardgavel from Nov 14, 2018

Backend configuration is stored in .terraform/terraform.tfstate, so store module sources in there and require re-init if those change, i.e something like module.cluster1.app -> source="github.com/example/example"

Having such feature is particularly useful if you want to test new module version which is located in some feature branch in another (shared) repo, you then have to edit all paths to module manually and re-init anyways.

[BattleBrisket]

+1 for this. I have a git-based module to configure team permissions, and I have ~80 teams. That's a lot of wet, brittle code that won't stand up to any significant change in the repository structure.

If we cannot have the source set as a variable, could we specify some module-specific config values that would load at runtime? If I could store the git URL and a ref tag somewhere in tfvars, for example, that would meet my needs.

[anderson-dan-w]

A use I see easily popping up (in that literally my first project that I'm working on terraform with), I want to have multiple modules that I pull from, but I will always want those to use same branch, within a project:

prod_git_tag = "v.0.0.1"
staging_git_tag = "v.0.5.7"

and then

module "vpc" {
  source = "${git_repo_path}?ref=${git_tag}"
}

module "kubernetes-cluster" {
  source = "${git_repo_path}?ref=${git_tag}"
}
 # etc...

which seems pretty reasonable to me - when I pass in git_tag=prod_git_tag, now they all reference the same git_tag and can be updated with one line, rather than in all the various places.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.