Terraform: backend "remote" is copying files outside of PWD, including untracked invisibles

[u2mejc]

Terraform Version

Terraform version: 0.11.14
tfe.v2.1

Terraform Configuration Files

terraform {
  required_version = "= 0.11.14"

  backend "remote" {
    organization = "example"

    workspaces {
      name = "foobar"
    }
  }
}

Debug Output

$ time terraform plan
2019/06/19 23:20:10 [INFO] Terraform version: 0.11.14
2019/06/19 23:20:10 [INFO] Go runtime version: go1.12.4
2019/06/19 23:20:10 [INFO] CLI args: []string{"/tmp/terraform", "plan"}
2019/06/19 23:20:10 [DEBUG] Attempting to open CLI config file: /home/ubuntu/.terraformrc
2019/06/19 23:20:10 Loading CLI configuration from /home/ubuntu/.terraformrc
2019/06/19 23:20:10 [INFO] CLI command args: []string{"plan"}
2019/06/19 23:20:10 [DEBUG] Service discovery for app.terraform.io at https://app.terraform.io/.well-known/terraform.json
2019/06/19 23:20:11 [DEBUG] Retrieve version constraints for service tfe.v2.1 and product terraform
2019/06/19 23:20:12 [INFO] command: backend initialized: *remote.Remote
2019/06/19 23:20:12 [DEBUG] checking for provider in "."
2019/06/19 23:20:12 [DEBUG] checking for provider in "/tmp"
2019/06/19 23:20:12 [DEBUG] checking for provider in ".terraform/plugins/linux_amd64"
2019/06/19 23:20:12 [DEBUG] found provider "terraform-provider-aws_v2.15.0_x4"
2019/06/19 23:20:12 [DEBUG] found provider "terraform-provider-local_v1.2.2_x4"
2019/06/19 23:20:12 [DEBUG] found provider "terraform-provider-template_v2.1.2_x4"
2019/06/19 23:20:12 [DEBUG] found valid plugin: "template", "2.1.2", "/home/ubuntu/terraform/providers/aws/prod/us-east-1/.terraform/plugins/linux_amd64/terraform-provider-template_v2.1.2_x4"
2019/06/19 23:20:12 [DEBUG] found valid plugin: "aws", "2.15.0", "/home/ubuntu/terraform/providers/aws/prod/us-east-1/.terraform/plugins/linux_amd64/terraform-provider-aws_v2.15.0_x4"
2019/06/19 23:20:12 [DEBUG] found valid plugin: "local", "1.2.2", "/home/ubuntu/terraform/providers/aws/prod/us-east-1/.terraform/plugins/linux_amd64/terraform-provider-local_v1.2.2_x4"
2019/06/19 23:20:12 [DEBUG] checking for provisioner in "."
2019/06/19 23:20:12 [DEBUG] checking for provisioner in "/tmp"
2019/06/19 23:20:12 [DEBUG] checking for provisioner in ".terraform/plugins/linux_amd64"
2019/06/19 23:20:12 [INFO] backend/remote: starting Plan operation
Running plan in the remote backend. Output will stream here. Pressing Ctrl-C
will stop streaming the logs, but will not stop the plan running remotely.

Preparing the remote plan...


Error: error uploading configuration files: Failed copying file "/home/ubuntu/terraform/test.dump" to archive: archive/tar: write too long

The configured "remote" backend encountered an unexpected error. Sometimes
this is caused by network connection problems, in which case you could retry
the command. If the issue persists please open a support ticket to get help
resolving the problem.

2019/06/19 23:20:22 [DEBUG] plugin: waiting for all plugin processes to complete...

real    0m11.923s
user    0m8.988s
sys     0m0.220s

Crash Output

n/a

Expected Behavior

Terraform should not traverse out of it's PWD. It should not upload unrelated files / folders (especially those outside git control).

Actual Behavior

Terraform appears to hang indefinitely as it attempts to upload hundreds of untracked environmental file, or errors depending on file size.

Appears to be looking for the git root. But instead of transferring up a shallow copy of visible files, starting at the PWD, it attempts to upload everything from the parents, including untracked and invisible files.

Steps to Reproduce

cd <root of repo>

ln -s /dev/random testfile

cd <child dir with terraform code>

terraform init

terrform plan

Additional Context

References

[mildwonkey]

Hi @u2mejc !

This behavior is as designed and we are updating the documentation and adding a warning in terraform to help clarify the behavior. The (undocumented, but we're fixing that!) assumption is that, if you've configured a working directory, the top-level configuration directory is _required_, so terraform grabs everything.

Instead of closing this issue, I am going to re-label this as a feature request so that we can re-evaluate the _correctness_ of this behavior, and consider alternate approaches.

[u2mejc]

Hi @mildwonkey, thank you for your work on this and #22096 ! 🚀

Sounds like there is still some discussions being done how to refine this, like not breaking local module paths and avoiding terraform hanging for mins as is compiles virtual env contents into slug.

I really appreciate it!

[jonathanhle]

Hi @mildwonkey, I don't 100% understand why the assumption is made that the top-level configuration directory is required. They way we structure our projects and now that we're on TFE, doing a local TF plan is REALLY SLOW. Something like 20 minutes before a plan will start. I figured out today it was uploading like a 1 gig of .terraform cache directories across various folders each representing a different workspaces. Really plainful to now have local plans be an option for us because of the slowness with so much to upload.

I thought that working in "MainApp-nonprod-us-west-2", would only have the TF executable deal with the stuff in MainApp-nonprod-us-west-2 + any thing that was referenced in the TF files in that folder. Uploading stuff that isn't specifically referenced seems weird to me.

.
├── README.md
├── modules
│   ├── module-a
│   └── module-b
└── projects
    ├── account-base
    ├── AppA-nonprod-us-east-1
    ├── AppA-nonprod-us-west-2
    ├── AppB-prod-us-east-1
    ├── AppB-prod-us-west-2
    ├── MainApp-nonprod-us-east-1
    └── MainApp-nonprod-us-west-2
        └── << most of the stuff in here are symlinks to main.tf, outputs.tf, common.tfvars in account-base folder >>
        └── custom-main.tf for things that are specific to that workspace.

[jonathanhle]

^^ I was able to workaround my issue with TF uploading lots of stuff I didn't expect it to by wiping out the all TF cache directories across the different projects folders within my repo. Then I reinit the folder I'm working in and do the plan. Hope this might be helpful to someone else running into this issue.

function_clear_tf_cache () {
  find ~/dev -type d -name ".terraform" -prune -exec rm -rf {} \;
}

alias tfplan='function_clear_tf_cache && terraform init && terraform plan'

from one of my bosses:
"something similar to the .dockerignore would be great"
"like i understand the situation where it has to send context over to TFE but being able to specify what paths/files to ignore from the payload would be ideal, especially if i know that i have sensitive stuff in some of these directories"

[brianpham]

I am encountering a similar same issue when doing remote plan as well. It takes about ~30min for remote plan because it is copying about ~4gb of .terraform. Is there an option we could set for it to ignore the .terraform folder completely? I haven't tried the suggestion that @jonathanhle posted above.

[pselle]

Hello! The origin of this issue reports for 0.11, but I have news for those on this thread: the 0.12.11 release added support for a .terraformignore file, detailed here: https://www.terraform.io/docs/backends/types/remote.html#excluding-files-from-upload-with-terraformignore.

In addition, starting from this release, all .terraform and .git directories are ignored (previously, just the directories at the configuration root dir were ignored) so you should see improvements from upgrading, regardless of using .terraformignore or not.

I'm going to close this issue to reflect the release of this feature, because w.r.t. 0.11, no further work is being done towards 0.11 releases/we recommend the 0.12 upgrade path.

[jonathanhle]

thank you for then note about the fix in .12.

[brianpham]

Thanks for the fix in .12! Greatly appreciated!

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.