Terraform: Terraform provider downloads fail with TLS handshake timeout

[xgerman]

Terraform Version

0.10.0

Expected Behavior

We are running terraform as part of a Jenkins CI system. Basically every job builds a docker container containing terraform and then executes a tectonic installer in said container.

Terraform should download the providers and their checksums

Actual Behavior

`
Error installing provider "aws": error fetching checksums: Get https://releases.hashicorp.com/terraform-provider-aws/0.1.4/terraform-provider-aws_0.1.4_SHA256SUMS: net/http: TLS handshake timeout.

Terraform analyses the configuration and state and automatically downloads
plugins for the providers used. However, when attempting to download this
plugin an unexpected error occured.

This may be caused if for some reason Terraform is unable to reach the
plugin repository. The repository may be unreachable if access is blocked
by a firewall.
`

This happens with several other providers as well.

Steps to Reproduce

Please list the steps required to reproduce the issue, for example:

1. Run terraform apply inside a container which runs a container executing terraform apply

Important Factoids

I attached to the container running terraform and executed curl:
``root@jenkins-slave-4s7xc-r828q:~# docker exec -it 3e85ee5a4d9b curl -v https://releases.hashicorp.com/terraform-provider-tls/0.1.0/terraform-provider-tls_0.1.0_SHA256SUMS

• Hostname was NOT found in DNS cache
• Trying 151.101.1.183...
• Connected to releases.hashicorp.com (151.101.1.183) port 443 (#0)
• successfully set certificate verify locations:
• CAfile: none
  CApath: /etc/ssl/certs
• SSLv3, TLS handshake, Client hello (1):
• SSLv3, TLS handshake, Server hello (2):
• SSLv3, TLS handshake, CERT (11):
• SSLv3, TLS handshake, Server key exchange (12):
• SSLv3, TLS handshake, Server finished (14):
• SSLv3, TLS handshake, Client key exchange (16):
• SSLv3, TLS change cipher, Client hello (1):
• SSLv3, TLS handshake, Finished (20):
• SSLv3, TLS change cipher, Client hello (1):
• SSLv3, TLS handshake, Finished (20):
• SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
• Server certificate:
• subject: C=US; ST=California; L=San Francisco; O=Fastly, Inc; CN=s.ssl.fastly.net
• start date: 2017-06-19 20:08:43 GMT
• expire date: 2018-06-20 20:08:43 GMT
• subjectAltName: releases.hashicorp.com matched
• issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign CloudSSL CA - SHA256 - G3
• SSL certificate verify ok.

GET /terraform-provider-tls/0.1.0/terraform-provider-tls_0.1.0_SHA256SUMS HTTP/1.1
User-Agent: curl/7.38.0
Host: releases.hashicorp.com
Accept: /

< HTTP/1.1 200 OK
<``

So there must be a difference in how curl handles TLS and how terraform does it.

[otarabai]

I have the same issue since upgrading to 0.10, downgrading back to 0.9.11 fixes it.
Nope, just sporadic behaviour, issue exists with 0.9.11 as well.

[cagriy]

We started having this issue as well. On 0.10.8 for a long time but the issue started recently, a week or so ago. Terraform run is performed on a Jenkins instance similar to the original report but not in a container. TLS error occurs sporadically and breaks the pipeline, rerunning it usually solves the problem.

[fundead]

We're also seeing the same behavior described by @cagriy and @xgerman - our setup also runs Terraform from Jenkins, and we've seen the issue both when it's run from a container context and on host instances. Seen from versions including 0.11.2 and 0.11.7.

This results in sporadic occurrences of errors such as:

Initializing the backend...



Error configuring the backend "s3": RequestError: send request failed

caused by: Post https://sts.amazonaws.com/: net/http: TLS handshake timeout

This is using S3-backed Remote State, but failures have been seen from all TF commands. I have observed this across more than one AWS account from multiple Jenkins servers/agent clusters, all using a similar VPC/network setup. Our workaround is also to re-run builds, obviously not ideal.

[apparentlymart]

Hi @xgerman, and every one else! Sorry for the weird behavior here.

In Terraform versions prior to v0.11.2, when running on a host that has both IPv4 and IPv6 connectivity Terraform will prefer to use IPv6 for reaching releases.hashicorp.com. If Terraform is running on a system that appears to have IPv6 connectivity but that connectivity is _poor_ in relation to the IPv4 connection, this can result in timeouts.

From Terraform v0.11.2 onwards, Terraform implements RFC 6555, which means Terraform will attempt both an IPv4 and an IPv6 connection simultaneously and prefer IPv4 if it returns faster.

If you aren't able to upgrade to Terraform v0.11.2 or later, you may find that disabling IPv6 connectivity on your Jenkins agent host will help make things work more smoothly.

[thejmazz]

Just wanted to leave a note here for anyone else with the problem. Somewhat duplicate of #16448.

In my case, our OpenStack environment set the MTU on ens3 on the host to 1450, and a container running via drone was attached to a docker network with MTU 1500. Setting the MTU to 1300 (< 1450) on the docker0 interface ({ "mtu": 1300 } in /etc/docker/daemon.json) and with further networks using docker network create -d bridge -o com.docker.network.driver.mtu=1300 mynet is our immediate solution. This also then requires having a drone admin enable "trusted" on each repository so that they can set network_mode: mynet, or bridge. Importantly, the mtu that shows up in ip addr for a custom network will only actually change to 1300 when there is a container in it.

[woohgit]

I had the same issue and the disabling IPV6 solved the issue.

I had the issue on my local computer. I tried everything but nothing worked well. Except disabling the IPV6.

[apparentlymart]

Hi all,

Further to my previous comment, I just wanted to sum up a few different causes we've seen for this kind of issue for future reference:

• On a host with both IPv4 and IPv6 connectivity, Terraform versions prior to v0.11.2 will prefer IPv6. This can be problematic on systems where the IPv6 connection is slower or is actually inoperable in practice. From v0.11.2 onwards, Terraform implements RFC 6555 to mitigate this problem.
• Some environments have either explicit or transparent HTTP proxies that are required for outbound access. Occasionally we've seen reports that poorly-performing or misconfigured proxies have led to timeout and TLS-related issues. In this case, there is no known Terraform-specific workaround and so working with the administrator of that proxy is the primary path to resolution.
• Some users run Terraform on WiFi networks with "captive portal" intercepts which can cause confusion. There are several different approaches to intercepting outgoing traffic to redirect to a captive portal, including DNS intercepts and HTTP-level intercepts, and some of these can lead to Terraform appearing to timeout or have TLS handshake issues due to the interference of that system.

Since the IPv6 connectivity issue was addressed in v0.11.2 and the other causes here are local configuration problems, I'm going to close this out. If you find this comment in future and are seeing a problem with similar symptoms but none of these causes seem to apply, please open a new top-level issue to discuss that, being sure to include any relevant information about your network configuration.

[holyketzer]

I have the same issue with terraform version 0.11.10.

On my MacOS laptop terraform wasn't able to get modules

Error installing provider "kubernetes": Get https://releases.hashicorp.com/terraform-provider-kubernetes/: net/http: TLS handshake timeout.

Terraform analyses the configuration and state and automatically downloads
plugins for the providers used. However, when attempting to download this
plugin an unexpected error occured.

This may be caused if for some reason Terraform is unable to reach the
plugin repository. The repository may be unreachable if access is blocked
by a firewall.

If automatic installation is not possible or desirable in your environment,
you may alternatively manually install plugins by downloading a suitable
distribution package and placing the plugin's executable file in the
following directory:
    terraform.d/plugins/darwin_amd64

On my Linux desktop connected to the same router it works.
Issue isn't in my internet (it works from desktop) or router (I don't use captive portal etc). And this issue happens only with Terraform, all another apps and tools works without any connection issues.

[alfemy]

The same issue, at MacOS I get the error, at Linux PC located in the same network all is OK.
Funny fact, _sudo terraform init_ works as well at Mac, and downloads modules

terraform init

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
Error installing provider "aws": Get https://releases.hashicorp.com/terraform-provider-aws/: net/http: TLS handshake timeout.

This may be caused if for some reason Terraform is unable to reach the
plugin repository. The repository may be unreachable if access is blocked
by a firewall.

➜   sudo terraform init
Password:

Initializing provider plugins...
- Checking for available provider plugins on https://releases.hashicorp.com...
- Downloading plugin for provider "aws" (1.58.0)...

[Penryn]

+1 for sudo terraform init, Thanks, @alfemy.

Makes no sense to me why you'd need admin permissions to secure a TLS handshake, but it worked for us when disabling IPv6 did not.
(macOS Mojave)

[vesper8]

Tried to run commands as sudo, it isn't helping me. I already disabled ipv6

I downloaded the digitalocean provider manually and it's reporting it correctly. Have the latest version of both the DO provider and Terraform

I keep getting this error just trying to apply the droplet creation example from the official digitalocean provider repository

Error: Error applying plan:

1 error(s) occurred:

* digitalocean_droplet.mywebserver: 1 error(s) occurred:

* digitalocean_droplet.mywebserver: Error creating droplet: Post https://api.digitalocean.com/v2/droplets: net/http: TLS handshake timeout

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

On OSX Mojave.

If anyone has any idea please.. I'd love to solve this

[katdev]

I'm having the same issue after updating to Mojave 10.14.4 using the New Relic provider. sudoing works for me, so will use that as workaround for now.

[dgmorales]

I'm on Mojave 10.14.5, and I'm getting the same "net/http: TLS handshake timeout" message on init when it tries to access my remote state backend on azurerm. Puzzling as it is, sudoing works, specially because I'm using sudo -E to preserve the environment (to access the ARM_ACCESS_KEY env var I've set on my regular user session).

terraform init
Initializing modules...

Initializing the backend...

Error: Failed to get existing workspaces: Get https://xxxxxxxx.blob.core.windows.net/tfstates?comp=list&prefix=xxxxxx.tfstateenv%3A&restype=container: net/http: TLS handshake timeout

Additionaly, terraform plan works fine without sudo, despite that it also has to access the state.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.