Terraform: In 0.12, modules can no longer be installed from local git repositories at relative paths

[robrankin]

Terraform Version

Have reproduced this with beta1, beta2, and a dev compiled version from master today.

Terraform v0.12.0-beta1
Terraform v0.12.0-beta2
Terraform v0.12.0-dev

Terraform Configuration Files

module "modx" {
  source                      = "git::../../../repo//path/to/module?ref=621"

Debug Output

Doesn't appear to be extremely useful, except perhaps this line:

2019/04/25 10:33:15 [TRACE] go-getter detectors rewrote "git::../../../repo?ref=621" to "git::file:///repo?ref=621"

Expected Behavior

terraform init should succeed

Actual Behavior

terraform init fails as follows:

0.12.0-beta1

Error: Failed to download module

Error attempting to download module "modx" (mod_x.tf:6)
source code from "git::../../../repo?ref=621": error
downloading 'file:///repo?ref=621': /usr/bin/git exited with
128: Cloning into '.terraform/modules/modx'...
fatal: '/repo' does not appear to be a git repository
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

0.12.0-beta2 and 0.12.0-dev

Error: Failed to download module

Error attempting to download module "modx" (mod_x.tf:6)
source code from "git::./../../../repo?ref=621": <nil>

Steps to Reproduce

1. Use a local filesystem Git repo and reference it as specified above
2. terraform init

Additional Context

These local filesystem git references work in 0.11.x

[apparentlymart]

Thanks for reporting this, @robrankin!

I don't think this was _intentionally_ supported in v0.11 and prior and was probably just a side-effect of an implementation detail that has changed now, but we'll see what we can do to restore the behavior nonetheless, since I can see that it would be useful.

[salimane]

I am not sure if it is related but I am getting the same error with tf 0.12.0-rc1

module "my_module" {
  source  = "git::ssh://[email protected]/private/repo-modules.git?ref=tag_v0.0.1//folder/sub_folder"
}

Error: Failed to download module

Error attempting to download module "my_module"
(file.tf:2) source code from
"git::ssh://[email protected]/private/repo-modules.git?ref=tag_v0.0.1//folder/sub_folder":
<nil>

am I missing something @apparentlymart ?

[kipkoan]

@apparentlymart - We just ran into this same issue while working on upgrading our terraform repos. We use pinning of local repo hashes to support environment promotion. Do you think a fix for this might be coming soon? Thanks!

[kipkoan]

$ terraform init -upgrade
Upgrading modules...
Downloading git::../..?ref=5067df412b6 for testing...

Error: Failed to download module

Could not download module "testing" (main.tf:22) source code from
"git::../..?ref=5067df412b6": error downloading
'file:///.terraform/modules/..?ref=5067df412b6': /usr/bin/git exited with 128:
Cloning into '.terraform/modules/testing'...
fatal: '/.terraform/modules/..' does not appear to be a git repository

[mnikhil-git]

I noticed similar issue. I have been using the module sources from git paths but when I switch the module source to local filesystem path, same module does not seem to work because of use of path.module functions.

Also, when module source is git and terraform12 is initialized, modules get checked out and initialized in .terraform/modules//

whereas for the localfilesystem, modules are simply symlinked under .terraform/modules/ --> module_directory_local_filesystem.

Is this intentional? why is not there a copy of the module in the initialized directory instead?

[salimane]

Error: Failed to download module

Error attempting to download module "my_module"
(file.tf:2) source code from
"git::ssh://[email protected]/private/repo-modules.git?ref=tag_v0.0.1//folder/sub_folder":

I have found the solution to the above issue. here is the correct way of sourcing private repositories in terraform assuming you have the right permissions to clone the repository

module "my_module" {
  source  = "git::ssh://[email protected]:private/repo-modules.git//folder/sub_folder?ref=tag_v0.0.1"
}

[hunkjun]

source = "git::ssh://[email protected]:sre/vvv/legacy.git//aws_userdata?ref=master"

[maxmanders]

For what it's worth, I've had success with

module "my_module" {
  source  = "git::ssh://[email protected]/organisation/repo.git//path/to/module?ref=tag_v0.0.1"
}

Note the / rather than : after [email protected]. I saw errors when using the previous suggestion but that may be specific to my set up.

[kipkoan]

@salimane , @hunkjun , @maxmanders -- you guys are misunderstanding the problem that @robrankin posted, and I confirmed. Installing modules from remote git repos has always worked. Installing modules from local git repos, using relative paths, _used_ to work prior to 0.12.

This worked in 0.11:

source = "git::../../../repo//path/to/module?ref=621"

But no longer does in 0.12.

It's super useful for local development without having to push commits to the remote before running init/plan/apply.

[maxmanders]

Copy that, apologies for fuelling the confusion!

[Erokos]

Any updates regarding this issue? It's fairly tedious having to commit and push commits that don't work so I can find out that they don't work :)

[salewski]

I just left a comment over in issue #25488 that I implemented a (forthcoming) fix for supporting git:: forced local paths (both relative and absolute). After rummaging around in the code, though, I'm a little surprised that previously worked (though I did not look back in time very far); if I find time, I might try a little archaeology on it. In any event, I'll post a PR with my proposed changes in the next day or so to gather feedback.

[salewski]

...I'll post a PR with my proposed changes in the next day or so to gather feedback.

There's an issue write-up in hashicorp/go-getter#268, as that's the library that provides the Git repo cloning support for Terraform. The string value of a Terraform module call's source parameter is parsed and handled by the go-getter lib.

The draft PR for it is hashicorp/go-getter#269

Assuming that PR (or something like it) gets merged, then getting the support for this feature in Terraform will require updating the dependency on go-getter to a version that includes the change.

[apparentlymart]

Thanks for capturing that context over in the go-getter repository, @salewski!

I want to add a little more context here that is Terraform-specific:

Terraform handles relative paths to modules already on disk (without the git:: prefix, etc) directly itself, rather than with go-getter, because the rule is that a relative path is interpreted against the directory of the module that called it, rather than against the current working directory.

Implementing the handling of relative paths for git inside go-getter would, unfortunately, make this behavior inconsistent: go-getter has no awareness of the idea of Terraform modules, and so the relative path would end up being interpreted relative to the current working directory rather than to the calling module. As well as just being inconsistent, that would also make it impossible to reliably use git:: relative paths in any shared module, because it would not be able to predict its own path relative to the current working directory for similar reasons that today modules can't predict their own _absolute_ paths to find their neighbor local git modules.

I'm not sure at this time what the best way forward is on this. In order to properly meet the use-cases described in that go-getter issue I think we would need to somehow make a git:: relative path be interpreted relative to the current module directory, but it's not clear to me how to do that within the constraints of the existing go-getter API. The GitDetector type could potentially have an optional field to allow overriding its sense of "current working directory" for the purposes of resolving relative paths, but that would require Terraform to construct those instances dynamically on a per-request basis (not super difficult, but quite different than the current design) and it doesn't really fit in with the current proposed PR in hashicorp/go-getter#269 where the lookup is handled directly inside the Detect function, rather than inside GitDetector.

With all of that said, it seems like this might require some more thought to figure out what the appropriate separation of concerns is. Because go-getter is a shared library, we must also consider what impacts this might potentially have on other go-getter callers which may have other ideas about what "current working directory" means, or whether reading objects from the local filesystem is even desirable.

[salewski]

[...]
I want to add a little more context here that is Terraform-specific:
[...]
...because the rule is that a relative path is interpreted against the directory of the module that called it, rather than against the current working directory.

@apparentlymart: Thanks for that additional context; that's quite helpful. I see what you mean about the module's context vs. $PWD -- my local testing setup was too simplistic to have that be an issue.

In the work thus far on hashicorp/go-getter#269, there was definitely some friction against the current Detector API. In fact, my comment that starts on line 55 of detect_git.go is something of an apology for introducing a function outside of the Detector interface. I don't love the leaked abstraction into the calling Detect(...) dispatch function, but I held my nose in order to avoid breaking the API compatibility. The tasklist item "document limitations and/or possible future work" in the PR was there to shine a light on that wart and open a discussion on how to maybe clean it up. But it won't fly, anyway, without a better approach to the $PWD thing, like you said.

I def. agree that more thought is required.

Thanks for the great feedback!

[salewski]

[...]
...or whether reading objects from the local filesystem is even desirable.

@apparentlymart, Could I ask you to elaborate on that notion a little?

The current Detect(...) dispatcher function does give the caller some control over which specific Detector implementations are used, so conceivably could be used to avoid applying particular detectors that have unwanted behaviors for a given scenario. But right now the implementations are "all or nothing" by technology (Git, File) or service (GCS, S3, GitHub, BitBucket).

[apparentlymart]

Hi @salewski,

Regarding the "is local filesystem access desirable?" question: I don't know if this actually matters in practice, but previously it seems like a non-Terraform go-getter caller could specifically have omitted detectors/getters that access the local filesystem and keep only the ones that interact with network services. For example, HashiCorp Nomad also uses go-getter in the context of a clustered network service, where accessing the local disk of a particular node _might_ be conceptually weird -- though I will qualify that I'm only talking hypothetically here; I don't have any specific Nomad knowledge to confirm that.

If this did turn out to be true then one way we could potentially address it is to make the local file support opt-in, by whether this hypothetical new "base directory" field we've been talking about is set at all. Of course, whether that makes sense would depend on how the rest of that plays out, given the still-open question of whether this functionality can be the responsibility of the GitDetector rather than just `Detector.

(On re-read I also see that you've suggested here that _absolute_ local filesystem paths to git repositories _have_ been working, in which case this would be less of a concern because it doesn't introduce any new access that wasn't present before. I didn't quite catch that on first read, so I was assuming that local directories were not possible _at all_ before.)

[salewski]

...previously it seems like a non-Terraform go-getter caller could specifically have omitted detectors/getters that access the local filesystem and keep only the ones that interact with network services.
[...]
If this did turn out to be true then one way we could potentially address it is to make the local file support opt-in

Thanks @apparentlymart. Make sense; I'll keep it in mind.

(On re-read I also see that you've suggested here that _absolute_ local filesystem paths to git repositories _have_ been working, in which case this would be less of a concern because it doesn't introduce any new access that wasn't present before. I didn't quite catch that on first read, so I was assuming that local directories were not possible _at all_ before.)

Oh, in my comment above, I did not mean to imply that I knew absolute paths had been working previously.

I did not find this GH issue until after I had done most of the work behind the current state of hashicorp/go-getter#269. My experience was that neither absolute nor relative filepaths worked with git:: forced strings. And after having been in the code to make it work[0], I was surprised to read the reports from others who have commented here that git:: with local file paths of any sort ever worked.

[0] only partially, as it turns out, at this point