Terraform: Terraform eating TF module files

[matthewhartstonge]

I've just upgraded to 0.8.6 today on Windows 10 x64 and have been able to reproduce this about 5 times on my machine via CMD (even after restart). Trying to gain a greater understanding as to why I have deleted the .terraform folder and relevant state files to try and fix after replacing files.

Terraform Version

0.8.6

Affected Resource(s)

AWS Resources

Debug Output

C:\ ... \amazon>terraform get
Get: file://C:/ ... /amazon/aws-vpc
Error loading Terraform: Error downloading modules: module production-vpc: No Terraform configuration files found in directory: .terraform\modules\6c358148a16fc2f22bf420d4d37dc14a

Panic Output

None.

Expected Behavior

Should have loaded modules, or complained.

Actual Behavior

Terraform ate all the module .tf files.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. terraform plan
2. terraform get
3. terraform plan

Important Factoids

• Normal use (AWS VPC with 3 subnets)
• Also noted that it started playing up with asking what the default AWS region is, when already defined in vars/passed through from the top level.

[matthewhartstonge]

Not sure if if this helps, but at least shows the difference in the files it's been eating:

Wish I could give you more, but there are no errors output, no error log file e.t.c.

[jbardin]

Hi @MatthewHartstonge,

Are these files stored on a shared filesystem?

Can you show the output of fsutil behavior query SymlinkEvaluation?
(I don't have a windows system handy to check the command/path, but some documentation is here: https://technet.microsoft.com/en-us/windows-server-docs/management/windows-commands/fsutil-behavior)

[matthewhartstonge]

Hi @jbardin thanks for having a look at this.

No, all files are local.

Output from fsutil behavior query SymlinkEvaluation:

C:\Users\ ... \amazon>fsutil behavior query SymlinkEvaluation
Local to local symbolic links are enabled.
Local to remote symbolic links are enabled.
Remote to local symbolic links are disabled.
Remote to remote symbolic links are disabled.

[jbardin]

That looks normal, so my first hunch wasn't right. If you could get a standalone reproduction, I'll get a windows system up to test it out. Also, if you could post the debug logs from the command that is removing the files, that may help track down what's going on here.

Thanks!

[matthewhartstonge]

I re-upgraded to 0.8.6 late yesterday haven't been able to reproduce the issue again. If I can narrow it down some more or experience it again I'll re-open the issue.

Thank you for your help.

[matthewhartstonge]

After playing around today, I realized that the vendored upstream go-getter uses windows hardlinks. Therefore, if anything breaks using terraform get, terraform get -update or simply deleting the .terraform folder all module files are deleted due to using NTFS Junctions (hard links) over a soft link (symlink).

See upstream PR hashicorp/go-getter#51 for fix.

[matthewhartstonge]

@jbardin are you able to test my upstream go-getter changes with TF to see if it generates the expected behaviour? (Symlinking, not hardlinking) The CI errors are appearing from master by the looks of it.
I'm happy to give it a shot - just quite new to Golang.

[jbardin]

Oh yes, I missed the call out to cmd to create a junction point. We may need to keep that behavior which I noted in the linked PR. I'm not too familiar with the details of windows behavior here, but I have a new hunch.

Terraform has go-getter pull the files into a temp dir which is later removed. I think it could be that a junction point to the directory is being created in that temp dir, and the RemoveAll is walking back into the original directory when cleaning up the temp files. (there's a reason unix doesn't allow hard links to directories)

[jbardin]

@MatthewHartstonge:

I'm still not able to reproduce this myself, though it may be my unfamiliarity with windows. It appears that a junction is removed like a normal file and not recursed into under normal circumstances.

Is it possible for you to create a small standalone reproduction I could clone and run with the steps to reproduce?

[matthewhartstonge]

@jbardin all good, hopefully this helps. I was clearing the .terraform folder to fix a corrupt state I had got into after having to clear out AWS and it was easier doing that than manually rm'ing all resources from teh state file. Steps to reproduce:

1. Open up a terraform project with local modules
2. Run terraform get
3. Delete .terraform folder that appears
4. Local module files should now have disappeared

[jbardin]

Yes, that's basically what I've done, with various combination of nested modules and some go programs to add/remove the directories. This was on a standalone Windows Server 2016, not joined to a domain.

[matthewhartstonge]

Okay, I'll see if I can set up something generic next week - pushing hard for the end of our sprint this week ;)

[matthewhartstonge]

Closing due to upstream hashicorp/go-getter#51

[Defozo]

I'm having the same problem. What should I do to make terraform working again?

[Defozo]

Oh, in my situation, I didn't have any .tf file in my module root directory (they were in 1 level deeper subdirectory). Everything's fine now.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.