Terraform: Treat .terraform/providers as a cache to avoid re-downloading on subsequent "terraform init" in the same directory

Created on 5 Dec 2020  路  7Comments  路  Source: hashicorp/terraform

This new issue re-raises an issue that I believe was incorrectly closed. Issue #27099 was closed with the reason "duplicate issue of #27092", however the issue used as a reason to close #27099 is not the same use case.

Problem: running terraform init multiple times should re-use previously downloaded provider versions. It seems like terraform providers are always downloaded, regardless of any additional settings.

Due to this issue, the added time to re-download providers with every run causes terraform workflows to take longer

Terraform Version

Terraform v0.14.0

Expected Behavior

Running terraform init 2 or more times, with no provider version changes

Initializing modules...

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/local from the dependency lock file
- Reusing previous version of hashicorp/null from the dependency lock file
- Reusing previous version of hashicorp/template from the dependency lock file
- Reusing previous version of hashicorp/archive from the dependency lock file
- Reusing previous version of hashicorp/aws from the dependency lock file

Terraform has been successfully initialized!

Actual Behavior

Terraform Init First Run

Initializing modules...

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/local from the dependency lock file
- Reusing previous version of hashicorp/null from the dependency lock file
- Reusing previous version of hashicorp/template from the dependency lock file
- Reusing previous version of hashicorp/archive from the dependency lock file
- Reusing previous version of hashicorp/aws from the dependency lock file
- Installing hashicorp/archive v2.0.0...
- Installed hashicorp/archive v2.0.0 (signed by HashiCorp)
- Installing hashicorp/aws v3.20.0...
- Installed hashicorp/aws v3.20.0 (signed by HashiCorp)
- Installing hashicorp/local v2.0.0...
- Installed hashicorp/local v2.0.0 (signed by HashiCorp)
- Installing hashicorp/null v3.0.0...
- Installed hashicorp/null v3.0.0 (signed by HashiCorp)
- Installing hashicorp/template v2.2.0...
- Installed hashicorp/template v2.2.0 (signed by HashiCorp)

Terraform has been successfully initialized!

Terraform init Second run

Initializing modules...

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/local from the dependency lock file
- Reusing previous version of hashicorp/null from the dependency lock file
- Reusing previous version of hashicorp/template from the dependency lock file
- Reusing previous version of hashicorp/archive from the dependency lock file
- Reusing previous version of hashicorp/aws from the dependency lock file
- Installing hashicorp/archive v2.0.0...
- Installed hashicorp/archive v2.0.0 (signed by HashiCorp)
- Installing hashicorp/aws v3.20.0...
- Installed hashicorp/aws v3.20.0 (signed by HashiCorp)
- Installing hashicorp/local v2.0.0...
- Installed hashicorp/local v2.0.0 (signed by HashiCorp)
- Installing hashicorp/null v3.0.0...
- Installed hashicorp/null v3.0.0 (signed by HashiCorp)
- Installing hashicorp/template v2.2.0...
- Installed hashicorp/template v2.2.0 (signed by HashiCorp)

Terraform has been successfully initialized!

Relevant Configuration

terraform {
  required_version = ">= 0.14"
  required_providers {
    archive = {
      source  = "hashicorp/archive"
      version = "~> 2.0.0"
    }
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.0"
    }
    local = {
      source  = "hashicorp/local"
      version = "~> 2.0.0"
    }
    null = {
      source  = "hashicorp/null"
      version = "~> 3.0.0"
    }
    template = {
      source  = "hashicorp/template"
      version = "~> 2.2.0"
    }
  }
}

Steps to Reproduce

Please list the full steps required to reproduce the issue, for example:

  1. terraform init (multiple times for the same configuration)

References

Are there any other GitHub issues (open or closed) or Pull Requests that should be linked here? For example:

  • #27092 (closed, associated with this problem, but not the same use case)
cli enhancement
Source
picture joncolby
👍2

Most helpful comment

Hi all,

What's happened here is that previously Terraform was using the contents of the .terraform directory as a sort of _implied_ dependency lock -- the presence of a provider in there caused Terraform to re-select that version unless you use -upgrade -- but now the .terraform.lock.hcl file has replaced it, so Terraform _only_ considers the lock file when deciding what to install, and it does not consider what's already installed in .terraform because the lock file is always authoritative.

Of course, a side-effect of that new design is that the lock file can _only_ specify a version, and can't also serve as a sort of implied "cache" of the packages. Therefore Terraform always installs what the lock file says because the lock file doesn't contain any packages.

It would be possible in principle for Terraform to treat the .terraform directory _just_ as a cache, with .terraform.lock.hcl serving as the authority for which packages are needed. That's some new behavior that we'll need to design, so I'm going to relabel this as an enhancement just to send this into our "design a new behavior" process, rather than our "make the implementation match the intended design" (bug fixing) process.

picture apparentlymart on 8 Dec 2020
👍2

All 7 comments

This behavior seems to be by design.

to solve the caching issue you mentioned you need to tell terraform to cache the plugins. Add this line in $HOME/.terraformrc 

  plugin_cache_dir   = "$HOME/.terraform.d/plugin-cache"

then it should work as expected. Not sure why this is not the default behavior.

see https://www.terraform.io/docs/commands/cli-config.html

adicostil picture adicostil on 6 Dec 2020
👍2

@adicostil - thanks for this! This helps to "workaround" the problem seen with terraform 0.14.1

I tested this with terraform 0.13.5 and the providers download behaviour is different (it is the expected behaviour)

This is the Terraform 0.13.5 output after running terraform init a second time:

Initializing modules...

Initializing the backend...

Initializing provider plugins...
- Using previously-installed hashicorp/aws v3.20.0
- terraform.io/builtin/terraform is built in to Terraform
- Using previously-installed hashicorp/archive v2.0.0
- Using previously-installed hashicorp/local v2.0.0
- Using previously-installed hashicorp/null v3.0.0
- Using previously-installed hashicorp/external v2.0.0
- Using previously-installed hashicorp/template v2.2.0
- Using previously-installed hashicorp/kubernetes v1.13.3
- Using previously-installed terraform-providers/statuscake v1.0.0
- Using previously-installed carlpett/sops v0.5.3
- Using previously-installed hashicorp/random v3.0.0

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
joncolby picture joncolby on 6 Dec 2020

Hi all,

What's happened here is that previously Terraform was using the contents of the .terraform directory as a sort of _implied_ dependency lock -- the presence of a provider in there caused Terraform to re-select that version unless you use -upgrade -- but now the .terraform.lock.hcl file has replaced it, so Terraform _only_ considers the lock file when deciding what to install, and it does not consider what's already installed in .terraform because the lock file is always authoritative.

Of course, a side-effect of that new design is that the lock file can _only_ specify a version, and can't also serve as a sort of implied "cache" of the packages. Therefore Terraform always installs what the lock file says because the lock file doesn't contain any packages.

It would be possible in principle for Terraform to treat the .terraform directory _just_ as a cache, with .terraform.lock.hcl serving as the authority for which packages are needed. That's some new behavior that we'll need to design, so I'm going to relabel this as an enhancement just to send this into our "design a new behavior" process, rather than our "make the implementation match the intended design" (bug fixing) process.

apparentlymart picture apparentlymart on 8 Dec 2020
👍2

@apparentlymart - Thanks for the explanation. Now I understand how this behaviour relates to the new lock mechanism.

For now, I will use the plugins-cache-dir as a workaround to achieve the desired cache behaviour.

keep up the good work!

joncolby picture joncolby on 8 Dec 2020

@apparentlymart - Thanks for the explanation. Now I understand how this behaviour relates to the new lock mechanism.

For now, I will use the plugins-cache-dir as a workaround to achieve the desired cache behaviour.

keep up the good work!

Only that the offered work around doesn't work too.

erezool picture erezool on 8 Dec 2020

@erezool - The workaround works for me. I am using a client configuration file .terraformrc containing the following. what did you try exactly?

plugin_cache_dir = "mycachedir"
joncolby picture joncolby on 10 Dec 2020

@joncolby Thanks for replying. It does work now and I didn't change a thing. Pretty strange I must say

erezool picture erezool on 10 Dec 2020
👍1
Was this page helpful?
0 / 5 - 0 ratings

Related issues

pawelsawicz picture pawelsawicz  路  3Comments
larstobi picture larstobi  路  3Comments
jrnt30 picture jrnt30  路  3Comments
ronnix picture ronnix  路  3Comments
franklinwise picture franklinwise  路  3Comments