Terraform: Terraform init is extremely slow

[AlmogCohen]

Terraform init takes forever and it seems like it is due to provider download. This extremely slow behaviour is consistent for the last 2 days. Working like that is hellish as I'm using terragrunt which runs the terraform init command often in my case.

Drilling down on the trace logs, I've seen that the download of https://releases.hashicorp.com/terraform-provider-aws/3.22.0/terraform-provider-aws_3.22.0_darwin_amd64.zip takes usually 1-5 minutes minimum. When I simply download this url in my browser, it never takes more than 15 seconds....usually less than 10. When I run the init command with trace logs, it is almost always takes many many minutes to finish.

I have no internet issues at ALL

Terraform Version

Terraform v0.13.5
+ provider registry.terraform.io/hashicorp/aws v3.22.0

Terraform Configuration Files

I have created only one file in my example project, backend.tf and this is the its content
# Generated by Terragrunt. Sig: nIlQXj57tbuaRZEa terraform { backend "s3" { encrypt = true key = "staging/elastic_beanstalk/terraform.tfstate" profile = "some-profile" region = "us-east-1" bucket = "company-terraform-state" dynamodb_table = "terraform-lock-table" } }

Debug Output

https://gist.github.com/AlmogCohen/b8af6e33c170c393a63e126226ad7108

Crash Output

Not relevant

Expected Behavior

Terraform provider download should take as fast as it takes in the browser, and terraform init should not take 5 minutes for a very very simple terraform proejct

Actual Behavior

Terraform provider/plugin downloads takes forever

Steps to Reproduce

1. terraform init

Additional Context

I'm using terragrunt, so the init command runs often on one of my dependency projects. It doesn't really matter for this bug report because I've run the terraform command directly in a folder which has only tf files in.

References

Not relevant

[AlmogCohen]

Here is another trace log example from a project that is includes a little more pieces https://gist.github.com/AlmogCohen/19495e839bc4894d8084e9ab191e23f6

A billion of things are being downloaded, but mostly when it is requested from the hashicorp releases host, the download takes forever.

The download of https://releases.hashicorp.com/terraform-provider-aws/2.70.0/terraform-provider-aws_2.70.0_darwin_amd64.zip took almost a minute. Download of https://releases.hashicorp.com/terraform-provider-template/2.2.0/terraform-provider-template_2.2.0_darwin_amd64.zip took almost 40 seconds via the terraform init command, while via browser it took literally 2 seconds.

[danieldreier]

Thanks for reporting this! You're right that it shouldn't be so slow. I'm running on a mac and it's not that slow for me. Can you try and do an install of terraform via homebrew? Homebrew's Terraform has cgo enabled, and I'm wondering if you're stalling on DNS resolution.

[AlmogCohen]

Thanks! I appreciate your quick (and affirming) response!
I'm pretty sure I'm using the homebrew version as I've upgraded from Terraform v0.12 a few days ago...the last homebrew command I have in history is brew reinstall [email protected].

If I had DNS issues...won't it be affecting the browser download speed as well? For all modules I tried, the browser speed was like 20 times faster than the terraform init for that specific module.

EDIT: I've just verified locally, the terraform init is still extremely slow

[zzn01]

same issue here, terrafrom version is 0.13.5, installed via homebrew
hang on aws provider for very long time(10 minutes+)

- Installing -/aws v3.22.0...

[danieldreier]

@AlmogCohen the reason I wanted to look into DNS issues is that golang uses its own DNS resolver rather than the native one, and this has caused issues in Terraform (and many other projects) that can currently only be worked around with a cgo-enabled build. However, if you're not seeing it on a homebrew build then that's probably not the issue.

Most of the Terraform Core engineering team is out until next week, so I'm just doing basic triage of urgent issues. As other folks run into this - please upvote with a 👍 rather than a comment, to help us prioritize this. I don't think this will get looked at until the team is back but if it turns out there's a more widespread issue I can call folks in.

[danieldreier]

I went and tried to reproduce this again on my mac with the following trivial config:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.0"
    }
  }
}

# Configure the AWS Provider
provider "aws" {
  region = "us-east-1"
}

My init took about 7 seconds:

time terraform init

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 3.0"...
- Installing hashicorp/aws v3.22.0...
- Installed hashicorp/aws v3.22.0 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
terraform init  1.77s user 0.56s system 33% cpu 6.883 total

I do believe you that you're running into this, just wanted to clarify that I'm not able to reproduce a more widespread problem.

Can you share a traceroute to releases.hashicorp.com? Okay to obscure the first few hops for privacy, I'm just trying to get clues on what's different between our environments. Also, can you share what OS version you're running, if it's X86 vs ARM, anything interesting about your network environment like an HTTP proxy, etc?

[AlmogCohen]

I copied your trivial config into main.tf and run time terraform init and this is the result:

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 3.0"...
- Installing hashicorp/aws v3.22.0...
- Installed hashicorp/aws v3.22.0 (signed by HashiCorp)

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
terraform init  2.73s user 1.57s system 7% cpu 1:00.33 total

Just to prove the point again regarding no internet issues, I've run time wget https://releases.hashicorp.com/terraform-provider-aws/3.22.0/terraform-provider-aws_3.22.0_darwin_amd64.zip and got this:

--2020-12-31 08:19:33--  https://releases.hashicorp.com/terraform-provider-aws/3.22.0/terraform-provider-aws_3.22.0_darwin_amd64.zip
Resolving releases.hashicorp.com (releases.hashicorp.com)... 151.101.1.183, 151.101.129.183, 151.101.65.183, ...
Connecting to releases.hashicorp.com (releases.hashicorp.com)|151.101.1.183|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 42005468 (40M) [application/zip]
Saving to: ‘terraform-provider-aws_3.22.0_darwin_amd64.zip’

terraform-provider-aws_3.22.0_darwin_amd64.zip     100%[================================================================================================================>]  40.06M  10.1MB/s    in 4.7s

2020-12-31 08:19:38 (8.61 MB/s) - ‘terraform-provider-aws_3.22.0_darwin_amd64.zip’ saved [42005468/42005468]

wget   0.10s user 0.25s system 6% cpu 5.137 total

Mac specifications:

For the traceroute, I think some Firewall drops it midway because nothing can be seen after the first few hops:

traceroute -q 1 -w 1 -m 100 releases.hashicorp.com
traceroute: Warning: releases.hashicorp.com has multiple addresses; using 151.101.1.183
traceroute to dualstack.s.shared.global.fastly.net (151.101.1.183), 100 hops max, 52 byte packets
 1   #### Removed #####
 2   #### Removed #####
 3  *
 4   #### Removed #####
 5  #### Removed #####
 6  *
 7  bb302-pt-e49-100g-hot.israelinternet.co.il (185.149.252.109)  11.057 ms
 8  10.141.226.60 (10.141.226.60)  12.209 ms
 9  10.185.180.51 (10.185.180.51)  15.525 ms
10  10.185.180.11 (10.185.180.11)  12.616 ms
11  be6037.203.rcr22.fra06.atlas.cogentco.com (149.14.210.161)  69.101 ms
12  *
13 *
.......(nothing answered here....)
97  *
98  *
99  *
100  *

[pselle]

@AlmogCohen Thanks for running the timings and providing additional data.terraform init will take longer than the GET request alone.

One suggestion I have if you're running init often (not a workflow that is currently optimized), do make sure that if you can you're using provider cache settings. This should help with regard to your workflow.

On the final comments with timing, I see @danieldreier's report with ~7 seconds, and yours with ~5 seconds ... neither of these are close to the original 1-5 minutes in the original report. @danieldreier was also correct to point to CGO builds, as there are known DNS issues/traps that occur with other versions.

Would enabling the cache help your workflow?

[AlmogCohen]

@pselle Thanks for your comment. I'll look into enabling the cache later this week and respond here.

Regardless, it is important to note that the original problem still stands in place. The timing was 1 minute (and not 5 seconds like you said). If you'll take a closer look, you can see that the 5s timing is of wget. Terraform init is still 12 times slower, and from previous logs you could see that 95% of the time spent on terraform init is due to the resource download of the aws provider (which was timed as 5s via wget).

[pselle]

@AlmogCohen Thanks, you're correct, I was looking at the wrong timing! It was the 1 minute.

[future-media]

I have the same problem on Windows. Checking with Wireshark, I can see that the releases.hashicorp.com server starts off by accepting a TLSv1.2 connection but then gets stuck returning messages saying "ignored unknown record". It took so long that I killed it.

I was just doing the learn-terraform-aws-instance tutorial. I was able to create infrastructure once, but never again after that.

My provider has a short MTU of 1472 bytes. I think this requires servers to accept ICMP messages saying that packets are too long and must be resent with the shorter packet size. If routers are dropping ICMP packets it would be a problem. Just a thought.

[future-media]

Although learn-terraform-aws-instance tutorial fails completely on my office connection as described above, it works just fine when executed from Windows Server hosted by a different provider.

Other Windows PCs in my office also fail, but someone was able to get it to work on Linux. So it's the combination of my office connection + Windows that fails.

[lmserrano]

Something that also aggravates this issue is the fact that apparently, if you run multiple consecutive terraform init commands without any changes in the used providers nor their versions, Terraform will still proceed to install them each time:

$ terraform init
Initializing modules...

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/random from the dependency lock file
- Reusing previous version of hashicorp/local from the dependency lock file
- Reusing previous version of hashicorp/null from the dependency lock file
- Reusing previous version of hashicorp/template from the dependency lock file
...
- Installing hashicorp/template v2.2.0...
- Installed hashicorp/template v2.2.0 (signed by HashiCorp)
- Installing hashicorp/random v3.0.0...
- Installed hashicorp/random v3.0.0 (signed by HashiCorp)
- Installing hashicorp/local v2.0.0...
- Installed hashicorp/local v2.0.0 (signed by HashiCorp)
- Installing hashicorp/null v3.0.0...
- Installed hashicorp/null v3.0.0 (signed by HashiCorp)
...

Terraform has been successfully initialized!

Enabling cache may improve the speed of this operation, but it will still take a while because in Terraform 0.14.4 it is doing all this each time, while in previous versions < 0.13.x (if I'm not mistaken) it would only download what was new or that it wouldn't have, or providers which had new versions in case their version numbers were referenced with other symbols like >.