Terraform: Error: Failed to query available provider packages: Client.Timeout exceeded while awaiting headers

[snemetz]

Terraform Version

Terraform v0.13.4
Also tried 0.13.3 got same issue

Terraform Configuration Files

terraform {
  required_version = ">= 0.13"
  required_providers {
   aws = {
     source  = "hashicorp/aws"
     version = "~> 3.0"
   }
 }
}

Expected Behavior

Expected init to work

Actual Behavior

Initializing provider plugins...

• Finding hashicorp/aws versions matching "~> 3.0"...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/aws:
could not query provider registry for registry.terraform.io/hashicorp/aws: the
request failed after 2 attempts, please try again later: Get
"https://registry.terraform.io/v1/providers/hashicorp/aws/versions": net/http:
request canceled while waiting for connection (Client.Timeout exceeded while
awaiting headers)

Steps to Reproduce

terraform init

Additional Context

I can use curl and access the url that terraform says it can't access with no issues
I've tried multiple times over 2 days and issue doesn't change. Terraform always fails, curl always succeeds
Had no issues with 0.12, but 0.13 has not worked yet
Terraform section was the only change made going to 0.13
This is on Mac
I also tried with only the posted terraform config and still get same issue

[crosbymichael1]

Im having the same issue trying to use the helm provider:

terraform {
  required_providers {
    helm = {
      source = "hashicorp/helm"
      version = "1.3.2"
    }
  }
}

provider "helm" {

  kubernetes {
    host                   = local.kube_host
    cluster_ca_certificate = local.kube_cluster_ca_certificate_decoded
    token                  = local.kube_token
    load_config_file       = false
  }
}

---

Error Message:

`Initializing the backend...

Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.

Initializing provider plugins...

• Finding latest version of hashicorp/aws...
• Finding hashicorp/helm versions matching "1.3.2"...
• Using hashicorp/aws v3.9.0 from the shared cache directory

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/helm:
no available releases match the given constraints 1.3.2
`

[pselle]

Thank you for the report! Short of more information to help replicate this issue, I've tagged this as a registry issue as perhaps the concern may lie there.

@crosbymichael1 -- it looks like your message cut of at the error message, so I'm not sure you have the same error here (the timeout).

[crosbymichael1]

@pselle I figured out what was going on

so I did a tg apply in one of the helm folders and got this error, which I thought was pretty weird because there is 100% a

---

helm provider version > 1.3
Initializing provider plugins...

• Finding hashicorp/helm versions matching "~> 1.3"...
• Finding latest version of hashicorp/aws...
• Using hashicorp/aws v3.9.0 from the shared cache directory
  Error: Failed to query available provider packages
  Could not retrieve the list of available versions for provider hashicorp/helm:

no available releases match the given constraints ~> 1.3

After changing the version a bunch of times I found out that since I already had a different version of the binary in my tf plugins folder, that instead of downloading the new version, terraform just checked locally to see what I had installed, and since only 10.5 is there, well that is the only version I'd be allowed to use
/Users/michael.crosby/.terraform.d/plugins/registry.terraform.io/hashicorp/helm/10.5
So I just deleted my local helm provider folder and then did a tg apply and it downloaded the version I specified and it worked fine then

[pselle]

Thanks for the additional information @crosbymichael1! I've re-added the new label here because that gives more information that might help someone create a reproduction case for this particular angle of the issue (which could be separate from the original post, unsure, but what you've described does seem problematic, and tricky to detangle!)

[crosbymichael1]

@snemetz Try to delete your local aws file in the terraform plugins folder and then rerun terraform init and see if it resolves your issue:

rm -rf ~/.terraform.d/plugins/registry.terraform.io/hashicorp/aws

If your changing aws provider version I have a feeling if terraform see you already have a provider installed locally then its only checking for versions within your local folder and not out in the terraform registry.

[snemetz]

That dir was empty. plugin-cache had a lot of versions. Remove all
` ls -lR
total 8
-rw-r--r-- 1 stevennemetz staff 394 Nov 16 2018 checkpoint_signature
drwxr-xr-x 3 stevennemetz staff 96 Jul 1 2019 plugin-cache
drwxr-xr-x 3 stevennemetz staff 96 Jun 21 2019 plugins

./plugin-cache:
total 0
drwxr-xr-x 2 stevennemetz staff 64 Oct 14 19:37 darwin_amd64

./plugin-cache/darwin_amd64:

./plugins:
total 0
drwxr-xr-x 2 stevennemetz staff 64 Nov 5 2019 darwin_amd64

./plugins/darwin_amd64:`

Made no change. Same error.
The error claims it is about downloading the versions file. Which should be before comparing with local files.

[snemetz]

Debug log
`
2020/10/14 19:49:59 [INFO] Terraform version: 0.13.4
2020/10/14 19:49:59 [INFO] Go runtime version: go1.14.7
2020/10/14 19:49:59 [INFO] CLI args: []string{"/usr/local/bin/terraform13.4", "init"}
2020/10/14 19:49:59 [DEBUG] Attempting to open CLI config file: /Users/stevennemetz/.terraformrc
2020/10/14 19:49:59 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2020/10/14 19:49:59 [DEBUG] checking for credentials in "/Users/stevennemetz/.terraform.d/plugins"
2020/10/14 19:49:59 [DEBUG] checking for credentials in "/Users/stevennemetz/.terraform.d/plugins/darwin_amd64"
2020/10/14 19:49:59 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2020/10/14 19:49:59 [DEBUG] will search for provider plugins in /Users/stevennemetz/.terraform.d/plugins
2020/10/14 19:49:59 [TRACE] getproviders.SearchLocalDirectory: /Users/stevennemetz/.terraform.d/plugins is a symlink to /Users/stevennemetz/.terraform.d/plugins
2020/10/14 19:49:59 [DEBUG] ignoring non-existing provider search directory /Users/stevennemetz/Library/Application Support/io.terraform/plugins
2020/10/14 19:49:59 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins
2020/10/14 19:49:59 [INFO] CLI command args: []string{"init"}
2020/10/14 19:49:59 [TRACE] Meta.Backend: no config given or present on disk, so returning nil config
2020/10/14 19:49:59 [TRACE] Meta.Backend: backend has not previously been initialized in this working directory
2020/10/14 19:49:59 [DEBUG] New state was assigned lineage "d727669d-a920-07ec-4a64-bc51e80e8402"
2020/10/14 19:49:59 [TRACE] Meta.Backend: using default local state only (no backend configuration, and no existing initialized backend)
2020/10/14 19:49:59 [TRACE] Meta.Backend: instantiated backend of type
2020/10/14 19:49:59 [DEBUG] checking for provisioner in "."
2020/10/14 19:49:59 [DEBUG] checking for provisioner in "/usr/local/bin"
2020/10/14 19:49:59 [DEBUG] checking for provisioner in "/Users/stevennemetz/.terraform.d/plugins"
2020/10/14 19:49:59 [DEBUG] checking for provisioner in "/Users/stevennemetz/.terraform.d/plugins/darwin_amd64"
2020/10/14 19:49:59 [INFO] Failed to read plugin lock file .terraform/plugins/darwin_amd64/lock.json: open .terraform/plugins/darwin_amd64/lock.json: no such file or directory
2020/10/14 19:49:59 [TRACE] Meta.Backend: backend does not support operations, so wrapping it in a local backend
2020/10/14 19:49:59 [TRACE] backend/local: state manager for workspace "default" will:

• read initial snapshot from terraform.tfstate
• write new snapshots to terraform.tfstate
• create any backup at terraform.tfstate.backup
  2020/10/14 19:49:59 [TRACE] statemgr.Filesystem: reading initial snapshot from terraform.tfstate
  2020/10/14 19:49:59 [TRACE] statemgr.Filesystem: snapshot file has nil snapshot, but that's okay
  2020/10/14 19:49:59 [TRACE] statemgr.Filesystem: read nil snapshot
  2020/10/14 19:49:59 [TRACE] providercache.fillMetaCache: scanning directory .terraform/plugins
  2020/10/14 19:49:59 [TRACE] getproviders.SearchLocalDirectory: .terraform/plugins is a symlink to .terraform/plugins
  2020/10/14 19:49:59 [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
  2020/10/14 19:49:59 [TRACE] HTTP client GET request to https://registry.terraform.io/.well-known/terraform.json
  2020/10/14 19:50:02 [ERR] Checkpoint error: Get "https://checkpoint-api.hashicorp.com/v1/check/terraform?arch=amd64&os=darwin&signature=069ea326-f2fa-49a8-980f-5a8c5830d0cb&version=0.13.4": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  2020/10/14 19:50:10 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions
  2020/10/14 19:50:10 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/aws/versions
  2020/10/14 19:50:20 [ERR] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions request failed: Get "https://registry.terraform.io/v1/providers/hashicorp/aws/versions": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  2020/10/14 19:50:20 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions: retrying in 1s (1 left)
  2020/10/14 19:50:21 [INFO] Previous request to the remote registry failed, attempting retry.
  2020/10/14 19:50:21 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/aws/versions
  2020/10/14 19:50:31 [ERR] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions request failed: Get "https://registry.terraform.io/v1/providers/hashicorp/aws/versions": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
  `

[techdragon]

I have this issue as well, its been happening since I switched to using any 0.13.x version. To me it looks like the new provider resolution functionality doesn't function correctly with having set a plugin_cache_dir setting.

[techdragon]

This may actually be a case of https://github.com/hashicorp/terraform/issues/26223

Edit: Yep, I just fixed my case of this issue with mv ~/.terraform.d/plugins ~/.terraform.d/plugin-cache and updating my .terraformrc file from plugin_cache_dir = "$HOME/.terraform.d/plugins" to plugin_cache_dir = "$HOME/.terraform.d/plugin-cache" problem solved instantly with all existing cached providers still recognised.

@snemetz Does this work for you?

[C-Howard]

I can't speak for snemetz but I appear to be seeing the same issue and I have no files in ~/.terraform.d/ besides a checkpoint_signature file

[mildwonkey]

It looks like there are two different problems being reported in this issue: a registry timeout error for the AWS provider, and some provider installer-specific questions and issues. I'll see how many I can address!

---

Error: Failed to query available provider packages w/net/http: request canceled while waiting for connection

Unfortunately I am not sure what caused this - I can't reproduce the issue now, and it's suspicious that curl was working for you while init was failing. I see that the same error messages were reported on two different days; do any of you still have this still problem today? I will check with our internal registry team and see if they have any ideas or information.

---

Terraform not updating provider version

Terraform v0.13 will not query the registry for any provider it finds locally installed under ${CONFIG_DIR}.terraform/plugins. Run terraform init -upgrade to tell terraform to query the registry for provider versions and install any updates meeting your version constraints.

This mechanism is changing in v0.14 so there is a separate installation directory for locally installed (non-registry) providers; I'm afraid I will mangle the details but you can take a look at the (draft!!) of the v0.14 upgrade guide for a nice explanation of the new options: https://github.com/hashicorp/terraform/blob/master/website/upgrade-guides/0-14.html.markdown#the-local-provider-cache-directory

The TL;DR is that we recommend you use the filesystem_mirror configuration for local providers, but in v0.14 you will also be able to drop them in the following directory (relative to the terraform configuration files): terraform.d/plugins/ . It's important that we can designate directories that _only_ terraform is allowed to write to, and will make sure the documentation is clear and add warnings where appropriate so it's clear to y'all.

---

Issues when setting plugin_cache_dir to implicit local mirror directory #26223

Terraform expects to _write_ to the plugin_cache_dir and so it is invalid as a local mirror. We'll take a look at improving the documentation and printing a warning or error message if the provider installer finds problematic configuration like this. I'd like to keep further conversation about this specifically in #26223 , please.

[mildwonkey]

Quick update: the registry team dug into their logs based on the timestamp on the original reported issue, and did not see anything suspicious, nor were there any reported outages with our various service providers. At this point I can't offer any explanation for the timeout beyond "vague unspecified network hiccups", but we'll definitely keep our eyes open and dig in if this happens again.

[C-Howard]

I am currently verifying and seeing the issue consistently.

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 2.70.0"...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/aws:
could not query provider registry for registry.terraform.io/hashicorp/aws: the
request failed after 2 attempts, please try again later: Get
"https://registry.terraform.io/v1/providers/hashicorp/aws/versions": net/http:
request canceled while waiting for connection (Client.Timeout exceeded while
awaiting headers)

My versions.tf file for the module seeing the error is the following

terraform {
  required_providers {
    aws = {
     source  = "hashicorp/aws"
     version = "~> 2.70.0"
    }
  }
  required_version = ">= 0.13"
}

This looks to be the same scenario as the original reported issue. We use no in-house providers, only the hashicorp AWS provider.

[mildwonkey]

@C-Howard thanks! Is that happening _right now_ ? I'll report this to the registry folks; it will help if you can share your TRACE logs

[C-Howard]

Yes it happens every time I run terraform init.

[terragrunt] [[redacted]/account] 2020/10/19 12:59:11 Running command: terraform --version
[terragrunt] 2020/10/19 12:59:15 Terraform version: 0.13.4
[terragrunt] 2020/10/19 12:59:15 Reading Terragrunt config file at [redacted]/account/terragrunt.hcl
[terragrunt] 2020/10/19 12:59:23 Running command: terraform init -backend-config=bucket=[redacted] -backend-config=dynamodb_table=[redacted] -backend-config=encrypt=true -backend-config=key=[redacted] -backend-config=region=us-east-1 -upgrade
2020/10/19 12:59:23 [INFO] Terraform version: 0.13.4  
2020/10/19 12:59:23 [INFO] Go runtime version: go1.14.7
2020/10/19 12:59:23 [INFO] CLI args: []string{"/usr/local/bin/terraform", "init", "-backend-config=bucket=[redacted]", "-backend-config=dynamodb_table=[redacted]", "-backend-config=encrypt=true", "-backend-config=key=[redacted]/terraform.tfstate", "-backend-config=region=[redacted]", "-upgrade"}
2020/10/19 12:59:23 [DEBUG] Attempting to open CLI config file: /Users/[redacted]/.terraformrc
2020/10/19 12:59:23 [DEBUG] File doesn't exist, but doesn't need to. Ignoring.
2020/10/19 12:59:23 [DEBUG] ignoring non-existing provider search directory terraform.d/plugins
2020/10/19 12:59:23 [DEBUG] ignoring non-existing provider search directory /Users/[redacted]/.terraform.d/plugins
2020/10/19 12:59:23 [DEBUG] ignoring non-existing provider search directory /Users/[redacted]/Library/Application Support/io.terraform/plugins
2020/10/19 12:59:23 [DEBUG] ignoring non-existing provider search directory /Library/Application Support/io.terraform/plugins

Initializing provider plugins...
2020/10/19 13:00:16 [TRACE] providercache.fillMetaCache: scanning directory .terraform/plugins
2020/10/19 13:00:16 [TRACE] getproviders.SearchLocalDirectory: .terraform/plugins is a symlink to .terraform/plugins
- Finding hashicorp/aws versions matching "~> 2.70.0"...
2020/10/19 13:00:16 [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2020/10/19 13:00:16 [TRACE] HTTP client GET request to https://registry.terraform.io/.well-known/terraform.json
2020/10/19 13:00:26 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions
2020/10/19 13:00:26 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/aws/versions
2020/10/19 13:00:36 [ERR] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions request failed: Get "https://registry.terraform.io/v1/providers/hashicorp/aws/versions": context deadline exceeded
2020/10/19 13:00:36 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions: retrying in 1s (1 left)
2020/10/19 13:00:37 [INFO] Previous request to the remote registry failed, attempting retry.
2020/10/19 13:00:37 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/aws/versions
2020/10/19 13:00:48 [ERR] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions request failed: Get "https://registry.terraform.io/v1/providers/hashicorp/aws/versions": context deadline exceeded (Client.Timeout exceeded while awaiting headers)

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/aws:
could not query provider registry for registry.terraform.io/hashicorp/aws: the
request failed after 2 attempts, please try again later: Get
"https://registry.terraform.io/v1/providers/hashicorp/aws/versions": context
deadline exceeded (Client.Timeout exceeded while awaiting headers)

[terragrunt] 2020/10/19 13:00:48 Hit multiple errors:
exit status 1

[mildwonkey]

Thanks, I appreciate the quick response - we'll dig into this on our end.

For anyone else reading this issue: Any _new_ information is appreciated; anyone else experiencing this same issue please reply to the original comment with a 👍 (instead of adding comments that don't add information).

[snemetz]

This does not seem to be a registry issue
I can run curl with the same url successfully before and after the failed terraform run.
I've done this on a number of days, always with the same results

Unless this can be figured out, 0.13 is a non-starter at my company

[snemetz]

Not sure if this will help. But I removed ~/.terraform and the error changed.
Now it displays

`
Initializing provider plugins...

• Finding hashicorp/aws versions matching "~> 3.0"...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/aws:
could not connect to registry.terraform.io: Failed to request discovery
document: Get "https://registry.terraform.io/.well-known/terraform.json":
context deadline exceeded (Client.Timeout exceeded while awaiting headers)
`

And via curl
curl https://registry.terraform.io/.well-known/terraform.json {"modules.v1":"/v1/modules/","providers.v1":"/v1/providers/"}

[justincampbell]

@snemetz Thanks for the additional info! We're still not able to find any indication of failing requests at our application or CDN layer. Could you try a more detailed cURL request? I don't have the exact headers Terraform sends off-hand, but something like:

curl https://registry.terraform.io/v1/providers/hashicorp/aws/versions \
  --user-agent "Terraform/0.13.4" \
  --header "X-Terraform-Version: 0.13.4" \
  --compressed

I'll try to recreate the exact headers and options now and update you when I have a better arg set.

Also, where are you running these commands from? Are you on a home or office network, VPN, or if you're running this within a cloud provider, which provider and region?

[justincampbell]

I did verify that the cURL above is the request that Terraform is making:

Accept-Encoding: gzip
Connection: keep-alive
Host: registry.terraform.io
User-Agent: Terraform/0.13.4
X-Terraform-Version: 0.13.4

[snemetz]

curl https://registry.terraform.io/v1/providers/hashicorp/aws/versions \
  --user-agent "Terraform/0.13.4" \
  --header "X-Terraform-Version: 0.13.4" \
  --compressed

Works but terraform fails right afterward. In same shell

$ terraform13.4 init

Initializing the backend...

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 3.0"...

Error: Failed to query available provider packages

Could not retrieve the list of available versions for provider hashicorp/aws:
could not query provider registry for registry.terraform.io/hashicorp/aws: the
request failed after 2 attempts, please try again later: Get
"https://registry.terraform.io/v1/providers/hashicorp/aws/versions": net/http:
request canceled while waiting for connection (Client.Timeout exceeded while
awaiting headers)

I'm running from home on a Mac.

[mildwonkey]

Well this is Officially Weird!

We do update the go version and various dependencies with pretty much every release, and it is always possible that _something_ in there is interacting with your local setup.

Is there anything customized or atypical about your home network? Do you have a proxy/VPN/firewall/custom dns settings? What version mac os are you on?

[C-Howard]

As before I am not the original poster but seeing the same issue and it looks to be a definite issue with delay due to DNS. Typically due to internal server connectivity we connect to DNS servers running out of the office and eventually fall back to a major provider like 1.1.1.1. I have not been using those dns servers outside of the office but using the fallback dns server. This results in roughly two dns server timeouts before a request succeeds.

So you can reproduce this issue on mac by having dns servers similar to the following

Unreachable server 1
Unreachable server 2
1.1.1.1

This will reliably cause the observed failure, and if you rearrange servers to something like

1.1.1.1
Unreachable server 1
Unreachable server 2

You will find your requests resolving again, which is probably why you are not seeing anything on the registry side. This suggests that any noticeable delay in dns resolution will cause the timeout to be hit.

[mildwonkey]

@snemetz : @C-Howard 's last comment touches on what I was about to suggest - the old "It's always DNS" adage. Can you check your DNS configuration, or do you have a proxy?

I'm not as skilled at pouring through the changelogs, so I couldn't say just yet if this has to do with upgrading our golang version or some other dependency. Here are some maybe-related issues in the golang issue repository (emphasis on maybe).
https://github.com/golang/go/issues/30702
https://github.com/golang/go/issues/23008
https://github.com/golang/go/issues/33006

[jacobbednarz]

I was also experiencing a similar issue with the cloudflare provider however we have managed to find a workaround that is showing promise(🤞). In our case, we upgraded the provider block to explicit 0.13 syntax and we hadn't yet ran an apply against all configurations. Some of the confgurations would be pulling /-/cloudflare as the /cloudflare/cloudflare version and while the plan would succeed without any changes, it wasn't until I forced a noop apply against all configurations that this went away for us.

It's worth noting, I wasn't able to replicate the issue consistently though. We would see it periodically within our automated CI tests every few hours.

[kzap]

I am experiencing the same thing with Terraform 0.13.4 and 0.13.5

Initializing provider plugins...
- Finding hashicorp/aws versions matching "~> 3.12"...
2020/10/29 17:21:31 [DEBUG] Service discovery for registry.terraform.io at https://registry.terraform.io/.well-known/terraform.json
2020/10/29 17:21:31 [TRACE] HTTP client GET request to https://registry.terraform.io/.well-known/terraform.json
2020/10/29 17:21:41 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions
2020/10/29 17:21:41 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/aws/versions
2020/10/29 17:21:51 [ERR] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions request failed: Get "https://registry.terraform.io/v1/providers/hashicorp/aws/versions": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2020/10/29 17:21:51 [DEBUG] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions: retrying in 1s (1 left)
2020/10/29 17:21:52 [INFO] Previous request to the remote registry failed, attempting retry.
2020/10/29 17:21:52 [TRACE] HTTP client GET request to https://registry.terraform.io/v1/providers/hashicorp/aws/versions
2020/10/29 17:22:02 [ERR] GET https://registry.terraform.io/v1/providers/hashicorp/aws/versions request failed: Get "https://registry.terraform.io/v1/providers/hashicorp/aws/versions": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

My openssl is

$ openssl version
LibreSSL 2.6.5

Same result using openssl from brew

$ openssl version
OpenSSL 1.1.1h  22 Sep 2020

This does not happen when I am on a VPN located in AWS US-East-1, but not on my home ISP

[paultyng]

You can override the default registry timeout (10 seconds) by setting TF_REGISTRY_CLIENT_TIMEOUT to a number of seconds. You can also override the number of retries (default 1) by setting TF_REGISTRY_DISCOVERY_RETRY. I'm not sure what the issue is here but I wonder if possibly bumping the timeout or something could help you mitigate this, or at least give us another data point.

[paultyng]

Also if its possible, you may want to test the 0.14.0-beta2 release. Its built with a different Go version, which has a slightly different http.Client implementation, so it may work differently.

[pradeepbhadani]

I am also experiencing the same issue with Terraform 0.13 and Terraform 0.14- Issue #26716.
I tried TF_REGISTRY_CLIENT_TIMEOUT and TF_REGISTRY_DISCOVERY_RETRY option but did not help.

[mildwonkey]

I was finally able to reproduce the timeout issue and confirm the necessary fix: https://github.com/hashicorp/terraform/issues/26716#issuecomment-720549842

The TL;DR is that it is indeed the linked, upstream golang issue: binaries build with cgo don't honor certain DNS settings on mac osx (more info in the linked comment above). Anyone with this problem and a golang development setup can test this out by building terraform with CGO_ENABLED=1 and trying again.

We are reaching out to our release team to modify terraform's build, and we'll also need to coordinate with our in-house provider development teams and communicate to third-party provider developers.

[mildwonkey]

update: I closed the other issue, so to save y'all a click here's the same comment that I linked to above:

This is how I reproduced this on my local mac:

1. modified my DNS settings to use entirely non-existent DNS resolvers (3.3.3.3, 2.2.2.2)
2. manually edited /etc/resolv.conf to use actual dns resolvers
   
   
   
   • curl, dig, etc registry.terraform.io: works just fine
   
   
   • terraform init: fails w/ timeout
   
   
3. re-build terraform locally with CGO_ENABLED=1
   
   
   
   • now it works
   
   

So, as we've suspected for awhile, the issues lies with golang/go#12524. We have reached out to our build team here at hashicorp about re-working our build to not use CGO.

One caveat: when we make this change, terraform will no longer have this issue, but it's entirely possible that folks will still see this same problem again because individual providers might be built with CGO (terraform init would succeed but the actual plan/apply could fail with similar looking errors).

We will try and coordinate this build change with the provider developers we have direct relationships with, and communicate the necessity with all provider developers, but since we don't control their build processes we can't guarantee that all providers will make this same change. (this is not immediately relevant to the issue at hand, but I wanted to get the information out there in for future reference).

[mmrobins]

FWIW set homebrew to do CGO_ENABLED=1 https://github.com/Homebrew/homebrew-core/pull/65148

[fawaf]

still occurs on Terraform v0.13.5

[0dragosh]

Still having this issue on v0.14.4.

[paultyng]

@0dragosh that was most likely the Fastly outage earlier today, see if your timing of it lines up with the incident on our status page or Fastly's.

[0dragosh]

@0dragosh that was most likely the Fastly outage earlier today, see if your timing of it lines up with the incident on our status page or Fastly's.

Timing coincides, thanks a lot for letting me know!

[LBoraz]

i have the issue in 0.14.4 with the kubectl provider. For some reason it tries to download it from hashicorp/kubectl instead of gavinbunney/kubectl

Could not retrieve the list of available versions for provider
hashicorp/kubectl: provider registry registry.terraform.io does not have a
provider named registry.terraform.io/hashicorp/kubectl

[pselle]

@LBoraz Thank you for trying to connect to a relevant issue, but I think you've actually run into something else! We also have discussion forums that might help in the future.

What I suspect is happening for you is that you have a module that doesn't have the required_providers block declared. Required providers/provider source is a per-module concept, so if you have a module that doesn't have it declared, Terraform is trying to find it in the default namespace (hashicorp) and failing, thus your error.