Terraform: Terraform should not upload .terraform directories to TFE

[lizthegrey]

Terraform Version

$ terraform -v
Terraform v0.12.5

Terraform Configuration Files

(with remote execution enabled)

terraform {
  backend "remote" {
    hostname     = "app.terraform.io"
    organization = "[...]"

    workspaces {
      name = "[...]"
    }
  }
}

provider "aws" {
  region  = "us-east-1"
  version = "~> 2.19.0"
}

[...]

Debug Output

https://gist.github.com/lizthegrey/864fef40012c522fff41191bf0868dad for full output.

Preparing the remote plan...

The remote workspace is configured to work with configuration at
terraform/env-production relative to the target repository.

Therefore Terraform will upload the full contents of the following directory
to capture the filesystem context the remote workspace expects:
/home/elizabeththegrey/infra

Crash Output

n/a

Expected Behavior

Only 540k of data should be uploaded to TFE

Actual Behavior

240MB+ are uploaded to TFE containing the AWS provider binaries.

Steps to Reproduce

1. terraform init
2. terraform plan

Additional Context

Yes, we have symlinks.

References

n/a

[koblas]

Just to add how big an issue this is.

In our case, we've got a bunch of distinct sub-folders that are each their own component. So the upload is 3.5GB of .terraform directories.

Note; this will also upload the .git directories or any other files/directories that shouldn't be shared in some cases.

At a minimum there needs to be a .terraignore or similar to prevent uploads of some files.

[brianpham]

I am hitting the same issue here. A local terraform plan takes only a few seconds before. Now if I am running terraform plan on the cli, it will take on average takes about 30min to plan for 6 resources to be added because it is uploading about ~4gb of data of .terraform data.

Note: This is only happening when running using the cli workflow. If I am starting a queue from Terraform Cloud or starting a speculative plan from a GitHub PR. It runs fairly quickly. Can we have some kind of workaround to ignore the .terraform files?

Output:

Preparing the remote plan...

The remote workspace is configured to work with configuration at
terraform2/environments/dev relative to the target repository.

Therefore Terraform will upload the full contents of the following directory
to capture the filesystem context the remote workspace expects:
    /Users/bpham/Dev/infra

Terraform v0.12.9
Configuring remote state backend...
Initializing Terraform configuration...
2019/09/23 17:59:49 [DEBUG] Using modified User-Agent: Terraform/0.12.9 TFC/f9d44e3688
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.


------------------------------------------------------------------------

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

[ilyasotkov]

@brianpham A workaround would be to copy all plugin (provider) binaries to ~/.terraform.d/plugins/, that way they won't be fetched to the local ./.terraform directory when you run terraform init.

E.g. if you run Terraform from Docker, you can do that when building the Docker image like so: https://github.com/exekube/exekube/blob/master/dockerfiles/google/Dockerfile#L96-L104

[brianpham]

Thanks @ilyasotkov That seems to work or the suggestion that @koblas suggested above works too.

[vkruoso]

Also looking forward to something in the direction of better control on the upload process. We have a quite large repo and lots of stuff that are non-related to TF execution: only one directory is relevant for terraform. This makes impossible to iterate in a local cli environment.

[yardensachs]

Yea this is not a small issue, and putting it in the root is good if you have a single repo?

[yardensachs]

Any update on this? This is a huge issue.
We now have to create documentation for our teams to workaround this issue, we might have to stop using Terraform cloud until this is fixed.

[pselle]

@yardensachs Hi! There's work in progress to allow you to customize your uploads (specifically in the CLI flow) to Terraform Cloud. You will be able to define a file at your configuration root (a .terraformignore file) to customize your build.

w.r.t. the original title of this issue, .terraform directories at the root configuration directory are excluded from the archive uploaded to Terraform Cloud, but I believe what we're talking about here is the larger issue if being unable to customize what you send to TFC. In addition, the work in progress also ignores all .terraform directories, previously only .terraform directories at the root were ignored.

If you're interested in testing this feature out, we'd appreciate the help. @pkolyvas can assist if you want to get in touch!

[lizthegrey]

@pkolyvas, yes, Honeycomb is interested in testing .terraformignore.

[pkolyvas]

Terraform Ignore is a work in progress, and may have issues. You can use this build of Terraform via this feature branch: https://github.com/hashicorp/terraform/tree/f-tfignore.

A plan or apply with this build will be very noisy at present, as it shares all the files that are ignored, but that's in there currently to help with debugging while this is in progress.

How to test:

Checkout and build Terraform from the f-tfignore branch (i.e. run "go install").

To use/test this feature:

• Have a Terraform setup that leverages the remote backend
• Add a .terraformignore file to your configuration root
• Run terraform plan or terraform apply and see which files are excluded/you can add patterns to the .terraformignore file to exclude files/directories from your slug

About the .terraformignore file:

• Blank lines or lines starting with # are ignored.
• Each line defines a pattern
• You can end patterns with a forward slash / to specify a directory.
• You can negate a pattern by starting it with an exclamation point !.
• Standard glob patterns work.

If you'd like to report an issue and/or provide some additional feedback we've got a form here for this feature: https://forms.gle/czsWEYqag8drdwDj6 .

Notes:
If building terraform from source is new-to-you (it is to me), you'll want to:

• Download Go 1.12.x (https://golang.org/dl/) and set it up (Add go to your PATH)
• Use go install to build the terraform binary

[pkolyvas]

https://www.terraform.io/docs/backends/types/remote.html#excluding-files-from-upload-with-terraformignore

.terraform and .git ignored by default.

.terraformignore been released as part of 0.12.11! Thanks to @pselle for all the hard work.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.