Terraform: path.module fails in sub-module on Docker for Windows mount

[kcd83]

I have found a quirky issue where the .terraform/module/<uuid> symlinks trip up Terraform and we get a file not found error when one module calls another module with a ${path.module} reference to a file.

This _only_ affects Docker for Windows users when it the underlying file system is actually a mount. I.e. Docker mount to MobyLinux to the Windows shared drive.

This _only_ trips up if it is a second module calling the first module with path.module, directly calling the first module is fine.

I'm guessing something particular to that setup in this scenario makes GO not realise it is a symlink or something in that ilk. Sorry it is gross, so I have put extra effort in packaging up how to reproduce the issue. Please don't hate on the Windows devs ;)

See https://github.com/kcd83/terraform_submodule_symlink_breaker

Terraform Version

Terraform v0.11.7

Terraform Configuration Files

A module with a file lookup using ${path.module}

data "local_file" "hello" {
  filename = "${path.module}/hello.txt"
}
# or
data "null_data_source" "something" {
  inputs = {
    something = "${file("${path.module}/hello.txt")}"
  }
}

Debug Output

See https://gist.github.com/kcd83/30dd742c46d007c40ba953729525c249#file-docker_for_windows_terrform_apply_trace-log-L917

2018/08/15 02:09:11 [TRACE] vertex 'root.two_hello.sub_module_one.module.two_hello.module.sub_module_one.data.local_file.hello': evaluating
2018/08/15 02:09:11 [TRACE] [walkRefresh] Entering eval tree: module.two_hello.module.sub_module_one.data.local_file.hello
...
data.local_file.hello: Refreshing state...
data.local_file.hello: Refreshing state...
...
2018/08/15 02:09:11 [ERROR] root.two_hello.sub_module_one: eval: *terraform.EvalReadDataApply, err: data.local_file.hello: open /mount/workspace3/.terraform/modules/8d8a0a393113b8c77ea1e849b0532b3c/hello.txt: no such file or directory
2018/08/15 02:09:11 [ERROR] root.two_hello.sub_module_one: eval: *terraform.EvalSequence, err: data.local_file.hello: open /mount/workspace3/.terraform/modules/8d8a0a393113b8c77ea1e849b0532b3c/hello.txt: no such file or directory
2018/08/15 02:09:11 [TRACE] [walkRefresh] Exiting eval tree: module.two_hello.module.sub_module_one.data.local_file.hello
2018/08/15 02:09:11 [TRACE] root.one_hello: eval: *terraform.EvalWriteState
2018/08/15 02:09:11 [TRACE] root.one_hello: eval: *terraform.EvalUpdateStateHook
2018/08/15 02:09:11 [TRACE] [walkRefresh] Exiting eval tree: module.one_hello.data.local_file.hello
2018/08/15 02:09:11 [TRACE] dag/walk: upstream errored, not walking "provider.local (close)"

Expected Behavior

In docker on windows, but not a mount

# terraform apply
data.local_file.hello: Refreshing state...
data.local_file.hello: Refreshing state...

Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Actual Behavior

In Docker on Windows, in a mounted directory

/mount/workspace # terraform apply
data.local_file.hello: Refreshing state...
data.local_file.hello: Refreshing state...

Error: Error refreshing state: 1 error(s) occurred:

* module.two_hello.module.sub_module_one.data.local_file.hello: 1 error(s) occurred:

* module.two_hello.module.sub_module_one.data.local_file.hello: data.local_file.hello: open /mount/workspace/.terraform/modules/8d8a0a393113b8c77ea1e849b0532b3c/hello.txt: no such file or directory

Steps to Reproduce

In Docker on Windows:

git clone https://github.com/kcd83/terraform_submodule_symlink_breaker.git
cd terraform_submodule_symlink_breaker
docker run -i -t -v ${pwd}:/mount -w /mount/workspace --entrypoint sh hashicorp/terraform:light
terraform init
terraform apply

Additional Context

Yes, running Terraform inside Docker, on Windows sounds like an evil use case. Actually we are using it like the "portable development environment" use case for Vagrant. I'm 90% sure this would affect Windows Vagrant users if they used the docker provisioner.

So the rational is the we have many Windows users, WSL is only sometimes available and Docker allows us to package a number of tools such as Ansible. Some of these tools require pinning of particular python libraries. Now we want people to adopt Terraform too.

I am hoping the symlink handling in this scenario can simply be copied from the symlink handing in other scenarios to resolve this

[apparentlymart]

Hi @kcd83! Sorry this isn't behaving well.

Unfortunately Terraform's Linux builds tend to assume that the filesystem will behave in a Unix-ish way, including that symlinks will be supported. I agree with your theory that Docker for Windows is probably breaking this assumption somehow, such as those symlinks not traversing the Docker for Windows VM's virtual filesystem bridge properly.

Some good news is that as of the next major release (0.12) we won't be creating symlinks in that directory anymore _anyway_, and instead Terraform will just refer directly to the location where the local module is installed.

In the mean time I have a few workaround ideas, based on what you've mentioned above:

• Although by default the .terraform directory is within the root module, it's possible for it to be in another location if you run the Terraform commands with an additional argument. In principle then, you could create a new directory that is _not_ visible from Windows, cd into that directory, and then run terraform init /mount/workspace3 to force it to use the configuration from that directory, and similarly terraform apply /mount/workspace3.

• Reference the module source addreses in a non-local way, such as via an absolute git:: address, to force Terraform to clone another copy rather than trying to use a symlink.

• Manually "flatten" those symlinks by deleting them and replacing them with copies of the directories they referred to.

We may be able to find some other way around this prior to 0.12, but given that we are currently very focused on getting the 0.12 release ready unfortunately it may end up being that we end up just holding on for the change to stop using symlinks altogether.

Thank for the detailed reproduction case. If possible we'll use it to produce an interim fix, but if not we will at least use it to very the behavior of 0.12 to verify that it's no longer tripped up by this or other symlink-related behavior.

[kcd83]

Great response thanks.

Yep this works

docker run --rm -i -t -v ${pwd}:/mount  --entrypoint sh hashicorp/terraform:light
terraform init /mount/workspace
terraform apply /mount/workspace

Also so does a docker volume mount for .terraform

docker run --rm -i -t -v ${pwd}:/mount -w /mount/workspace -v /mount/workspace/.terraform --entrypoint sh hashicorp/terraform:light
terraform init
terraform apply

I built master recently, do you know if the removal of symlinks has happened yet or do you have a rough ETA on 0.12? I think we can wait until the release but it would be nice to verify this problem will go away.

[apparentlymart]

The work I was referring to is not yet on the master branch since it's quite a disruptive refactor and so we've been doing it on a development branch in order that master can be used for ongoing 0.11 releases. 0.12 is not expected for at least a couple more months, since once it is feature complete we will need to get through the alpha and beta testing cycles and respond to any feedback from those.

[kcd83]

Yeah understood, I might poke around those branches if I have time

[apparentlymart]

Hi again, @kcd83!

The changes I was referring to have now been merged to the master branch and are included in the v0.12.0-alpha2 prerelease builds.

I don't have a Docker for Windows environment ready here to test with, but I did confirm that indeed the symlinks that were causing this problem are no longer used. If you'd be willing to give your examples a try with v0.12.0-alpha2 I'd appreciate any feedback you can give me to verify this. If you don't have time to check this or if you no longer have the docker for windows environment to test with then no worries: I can see about setting up a test environment for this myself, if needed... I'm just hoping you have a test environment more immediately to hand to test this more quickly than I could! :grinning:

[kcd83]

Yep that works, instead of symlinks we now get

root@ab6db49e64cf:/mount/workspace# cat .terraform/modules/modules.json | jq .
{
  "Modules": [
    {
      "Key": "two_hello.sub_module_one",
      "Source": "../one",
      "Dir": "../modules/one"
    },
    {
      "Key": "",
      "Source": "",
      "Dir": "."
    },
    {
      "Key": "one_hello",
      "Source": "../modules/one",
      "Dir": "../modules/one"
    },
    {
      "Key": "two_hello",
      "Source": "../modules/two",
      "Dir": "../modules/two"
    }
  ]
}

[apparentlymart]

Great! Thanks for verifying that, @kcd83.

In that case, I'm going to close this out. This change will be included in the forthcoming v0.12.0 final release.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.