Terraform: external module source subdirectory downloads the whole repo rather than just the subdirectory

[moss04]

Terraform Version

# terraform -v
Terraform v0.7.0

Affected Resource(s)

core.
This affects the 'terraform get' behavior and is not limited to a specific resource.

Terraform Configuration Files

module "mod0" {
  source = "git::https://moss04:[email protected]/moss04/modtest//modules/mod0"
  vsphere_password = "${var.vsphere_password}"
  rootssh_password = "${var.rootssh_password}"
}
module "mod1" {
  source = "git::https://moss04:[email protected]/moss04/modtest//modules/mod1"
  vsphere_password = "${var.vsphere_password}"
  rootssh_password = "${var.rootssh_password}"
}
variable "vsphere_password" {}
variable "rootssh_password" {}

Expected Behavior

When running 'terraform get' it should download the contents of the modules/mod0 subdirectory from the moss04/modtest project for the first module, and the contents of the modules/mod1 subdirectory for the second module.

Expected result of 'terraform get':

# terraform get
Get: git::https://moss04:[email protected]/moss04/modtest/modules/mod0
Get: git::https://moss04:[email protected]/moss04/modtest/modules/mod1
# ls .terraform/modules/15b0b1bbc68da21c8e9afd1bf5f2c9df/
module1.tf
# ls .terraform/modules/f55ce9db6ad3ce0e3f80e3b0bf2666af/
module0.tf

Actual Behavior

When running 'terraform get' it downloads the entire project directory tree for _each_ module.

Result of 'terraform get':

# terraform get
Get: git::https://moss04:[email protected]/moss04/modtest
Get: git::https://moss04:[email protected]/moss04/modtest
# ls .terraform/modules/15b0b1bbc68da21c8e9afd1bf5f2c9df/
modules  README.md
# ls .terraform/modules/f55ce9db6ad3ce0e3f80e3b0bf2666af/
modules  README.md

Steps to Reproduce

included in Result from above, along with the provided config and an appropriate github tree.

[apparentlymart]

Hi @moss04! Sorry Terraform isn't behaving as you expected here.

At present the layout of the .terraform/modules directory is essentially an implementation detail of Terraform and subject to change, so if you'd be willing I'd love to hear a little more about what higher-level problem you're trying to solve that makes you care about how the modules are laid out on disk, since I expect there's a potential feature request hiding in there.

We could certainly lay out the files on disk as you describe, but it would have some other implications. The first one that springs to my mind is if the module directory contains symlinks to files in the parent directory... currently that works as long as all of the files are in the same git repository, though I could certainly understand the argument that Terraform never actually guaranteed that it would be true and it might just be an accidental artifact of the current implementation.

[moss04]

Thanks for the reply @apparentlymart.

After reading your response and further testing I realize that the end behavior is in fact what I want, but just wasn't implemented the way I expected from the documentation.

I had thought that the subdir portion of the module source value was implemented by terraform's 'get' function, but it appears that this is rather utilized by functions further along in the process like 'apply'/etc. I have confirmed this by running an apply and recieving the correct module configurations for each of my VMs.

However the fact that the subdir is not applied to the source download but to rather to further terraform actions is not obvious from from the 'source' documentation. I would suggest clarifying this there, but this issue can be closed as it was a misunderstanding on my part. I don't need tight control of the actual on-disk layout of the modules directory, and if there access issues with other/unused modules those can be placed in another git project.

[mitchellh]

There are a number of improvements we could make for disk space and network optimizations when grabbing and storing modules but at this point in time we're more focused on their functionality and improving their edge cases versus stuff like this.

I don't think this specifically requires any changes since the .terraform/modules directory is indeed an implementation detail as @apparentlymart said, so we don't expect folks to get super meaningful information out of there by spelunking around. We welcome you to do it, however! We're not trying to hide anything, just haven't optimized that disk layout for humans (as clear by the SHA directory names).

[constructdian]

Hi Gents

I'm having an issue due to the above behaviour, the issue is slightly different but the relevant behaviour is the same so I hope reopening this is OK.

We use a central infrastructure repo that contains a lot of shared code and reference subdirs inside that repo for our modules. The issue I'm having is that because TF grabs the entire repo for each module that is externally sourced, I end up with n complete copies of the repo being downloaded, and as this central repo is growing terraform get is getting slower and slower.

I don't think there is a way to prevent TF from downloading the superfluous parts of the repo, and I know there isn't a way to tell it to download the repo only once and reference it for multiple modules, is there any way I can work around this that I'm missing or is there a potential change request here?

Cheers

[jch254]

In the same boat as @constructdian, any movement on this? Cheers!

[chamindg]

Same boat here. My config uses more than 10 modules. +1 for downloading once and reusing for all subdirs which share same repo path.

[yuliyantsvetkov]

Same here. Only for my EC2 instances the .terraform directory is like 500 MB. For 10 modules it downloads the repository 10 times :/ :-1:

[rverma-nikiai]

Hi, how is this resolved. Also wanted to know if terraform allow sparse checkout module

[richguernsey]

https://www.terraform.io/docs/modules/sources.html#modules-in-package-sub-directories

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.