Terraform-provider-aws: terraform apply hanging for long time

[hashibot[bot]]

_This issue was originally opened by @sureshoao as hashicorp/terraform#18580. It was migrated here as a result of the provider split. The original body of the issue is below._

---

main.tf 
provider "aws" {
  region = "us-east-1"
}
resource "aws_instance" "web" {
  ami           = "ami-b70554c8"
  instance_type = "t2.micro"

  tags {
    Name = "HelloWorld"
  }
}

terraform apply

2018-08-01T10:58:40.066-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: 2018/08/01 10:58:40 [DEBUG] [aws-sdk-go] DEBUG: Send Request sts/GetCallerIdentity failed, will retry, error RequestError: send request failed
2018-08-01T10:58:40.066-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: caused by: Post https://sts.amazonaws.com/: EOF
2018-08-01T10:58:40.066-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: 2018/08/01 10:58:40 [DEBUG] [aws-sdk-go] DEBUG: Retrying Request sts/GetCallerIdentity, attempt 12
2018-08-01T10:58:40.066-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: 2018/08/01 10:58:40 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:
2018-08-01T10:58:40.066-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: POST / HTTP/1.1
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: Host: sts.amazonaws.com
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: User-Agent: aws-sdk-go/1.14.31 (go1.9.2; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.8-dev
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: Content-Length: 43
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: Authorization: AWS4-HMAC-SHA256 Credential=ASIAXDWZ4B6U4W3J2KH3/20180801/us-east-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date;x-amz-security-token, Signature=aea1a110d837b40ac3d992a22e244df33400578900fd646078b16a96e843a787
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: Content-Type: application/x-www-form-urlencoded; charset=utf-8
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: X-Amz-Date: 20180801T145840Z
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: X-Amz-Security-Token: FQoGZXIvYXdzECAaDFO+v5nVdl07xgcyLyKlArCjiiepIMm2I0DFZ5XCyTX0ahmssS8JLCoQThdf6oFkHev+6xhT+Ioh/4eUEG4kpXuCHyk6hA/cR7RKW2nmhyVaHMygcBRlXr0voihN1y4dV728ETJuiI8MaMIhxmsom9Y6OgiMJ5MUcZMLhx9FVv0kwmv/ufqosTtJ/kyWPweRAEBSCpvgFuL18nvZ00Qclwza6NTMFTmdzXNIpYrobsFffaG27ED6ARxPYnIGof+fW+bvOGQgDBva1dVD6mMw5D3xcUGq3+4wkQAExNAQaBa7RibaO7jW/al3VK3bQ9Y3U9FjU5a4o05BumgmOIHktf1A/WEgaIDgxxq7ELTZw/rG/TABul9BCadJRTLqqhCuf31V6GVdw+yucP4bqGj4D8tBE32hKOKMh9sF
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: Accept-Encoding: gzip
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: 
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: Action=GetCallerIdentity&Version=2011-06-15
2018-08-01T10:58:40.067-0400 [DEBUG] plugin.terraform-provider-aws_v1.29.0_x4: -----------------------------------------------------
2018/08/01 10:58:43 [TRACE] dag/walk: vertex "root", waiting for: "meta.count-boundary (count boundary fixup)"
2018/08/01 10:58:43 [TRACE] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "aws_instance.web"
2018/08/01 10:58:43 [TRACE] dag/walk: vertex "provider.aws (close)", waiting for: "aws_instance.web"
2018/08/01 10:58:43 [TRACE] dag/walk: vertex "aws_instance.web", waiting for: "provider.aws"
2018/08/01 10:58:48 [TRACE] dag/walk: vertex "root", waiting for: "meta.count-boundary (count boundary fixup)"
2018/08/01 10:58:48 [TRACE] dag/walk: vertex "provider.aws (close)", waiting for: "aws_instance.web"
2018/08/01 10:58:48 [TRACE] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "aws_instance.web"
2018/08/01 10:58:48 [TRACE] dag/walk: vertex "aws_instance.web", waiting for: "provider.aws"
2018/08/01 10:58:53 [TRACE] dag/walk: vertex "root", waiting for: "meta.count-boundary (count boundary fixup)"
2018/08/01 10:58:53 [TRACE] dag/walk: vertex "provider.aws (close)", waiting for: "aws_instance.web"
2018/08/01 10:58:53 [TRACE] dag/walk: vertex "meta.count-boundary (count boundary fixup)", waiting for: "aws_instance.web"
2018/08/01 10:58:53 [TRACE] dag/walk: vertex "aws_instance.web", waiting for: "provider.aws"

[sureshoao]

please resolve this issue

[jjkirby]

I am having a similar problem with v1.30.0_x4 on a Mac 10.13.6 with go1.10.3 darwin/amd64. It appears to be a DNS issue because I can do a nslookup/dig on the sts endpoint. Is this possibly a go issue?

data.http.workstation-external-ip - *terraform.NodeRefreshableDataResourceInstance
2018-08-08T09:56:36.460-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: 2018/08/08 09:56:36 [INFO] Building AWS region structure
2018-08-08T09:56:36.460-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: 2018/08/08 09:56:36 [INFO] Building AWS auth structure
2018-08-08T09:56:36.460-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: 2018/08/08 09:56:36 [INFO] Setting AWS metadata API timeout to 100ms
data.http.workstation-external-ip: Refreshing state...
2018-08-08T09:56:36.745-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: 2018/08/08 09:56:36 [INFO] Ignoring AWS metadata API endpoint at default location as it doesn't return any instance-id
2018-08-08T09:56:36.751-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: 2018/08/08 09:56:36 [INFO] AWS Auth provider used: "SharedCredentialsProvider"
2018-08-08T09:56:36.751-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: 2018/08/08 09:56:36 [INFO] Initializing DeviceFarm SDK connection
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: 2018/08/08 09:56:36 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: POST / HTTP/1.1
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: Host: sts.us-east-1.amazonaws.com
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: User-Agent: aws-sdk-go/1.14.33 (go1.9.2; darwin; amd64) APN/1.0 HashiCorp/1.0 Terraform/0.11.8-dev
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: Content-Length: 43
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: Authorization: AWS4-HMAC-SHA256 Credential=AKIAJ7TUHS4VCO5WT7PQ/20180808/us-east-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=22bda1b7dc458acc64d5cc7e389178ab4bfc2661200c272edac9839a2111bfc9
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: Content-Type: application/x-www-form-urlencoded; charset=utf-8
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: X-Amz-Date: 20180808T155636Z
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: Accept-Encoding: gzip
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: 
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: Action=GetCallerIdentity&Version=2011-06-15
2018-08-08T09:56:36.752-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: -----------------------------------------------------
2018-08-08T09:56:36.805-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: 2018/08/08 09:56:36 [DEBUG] [aws-sdk-go] DEBUG: Send Request sts/GetCallerIdentity failed, will retry, error RequestError: send request failed
2018-08-08T09:56:36.805-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: caused by: Post https://sts.us-east-1.amazonaws.com/: dial tcp: lookup sts.us-east-1.amazonaws.com on 10.20.199.19:53: write udp 192.168.234.35:59290->10.20.199.19:53: write: host is down
2018-08-08T09:56:36.805-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: 2018/08/08 09:56:36 [DEBUG] [aws-sdk-go] DEBUG: Retrying Request sts/GetCallerIdentity, attempt 1
2018-08-08T09:56:36.806-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: 2018/08/08 09:56:36 [DEBUG] [aws-sdk-go] DEBUG: Request sts/GetCallerIdentity Details:
2018-08-08T09:56:36.806-0600 [DEBUG] plugin.terraform-provider-aws_v1.30.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------

(Maintainer edit note: edited to use triple backticks for formatting instead of single backticks)

[bflad]

Hi @sureshoao 👋 Sorry you're running into trouble here. Can you confirm a few things about your setup?

• Is STS enabled in all regions for this account?
• Where is Terraform running?
• Are there any HTTP proxy servers or captive portals that might be between where Terraform is running and Amazon STS?
• Is where Terraform running able to re-negotiate MTU if necessary? (e.g. if on an EC2 instance and using network ACLs to block all traffic, that Destination Unreachable ICMP Packets (type 3) are allowed)

@jjkirby I suspect you're having a separate issue.

The original report seems to point at something closing the connection with EOF (end of file):

...: 2018/08/01 10:58:40 [DEBUG] [aws-sdk-go] DEBUG: Send Request sts/GetCallerIdentity failed, will retry, error RequestError: send request failed
...: caused by: Post https://sts.amazonaws.com/: EOF

While your report seems to point at the DNS server not responding:

...: 2018/08/08 09:56:36 [DEBUG] [aws-sdk-go] DEBUG: Send Request sts/GetCallerIdentity failed, will retry, error RequestError: send request failed
...: caused by: Post https://sts.us-east-1.amazonaws.com/: dial tcp: lookup sts.us-east-1.amazonaws.com on 10.20.199.19:53: write udp 192.168.234.35:59290->10.20.199.19:53: write: host is down

You'll want to triple check via dig @10.20.199.19 sts.us-east-1.amazonaws.com from where Terraform is running in your case. It might have also been a transient issue with your DNS setup. We might be able to reduce the retry threshold similar to how we did for non-existent service endpoints to also include this type of error message for the DNS server not being available so it errors quicker in that case.

[jjkirby]

bflad -
Thanks for getting back. dig @10.20.199.19 sts.us-east-1.amazonaws.com resolves perfectly as well as dig @10.20.199.19 sts.amazonaws.com. And yes sts is turned on. I have tried to execute both at my home and work networks. Same result. What did you mean _by a transient issue with your DNS setup_?

I also found these interesting: https://github.com/golang/dep/issues/1838 and https://github.com/golang/dep/pull/1839. If it is a go DNS issue I would imagine someone on Mac OS X would have a similar problem?

[jjkirby]

@bflad -

I wrote a little go program thinking if it was a go issue:
```package main

import (
"fmt"
"net"
"os"
)

func main() {
ips, err := net.LookupIP("sts.amazonaws.com")
if err != nil {
fmt.Println(err)
os.Exit(1)
}
fmt.Println(ips)
}
```
Returned:
[54.239.29.25]

So in this case DNS works. Hmmmm

[jjkirby]

Case solved! Little Snitch was blocking because it didn't have a code signature. I turned off network filtering completely even though thought I had a rule to allow. Sorry for false alarm

[bflad]

Closing due to lack of response from original author.

[jkodroff]

Just a note for anyone else experiencing this:

I seem to have this issue when working with Terraform on Xfinity (Comcast) Wi-Fi. (Comcast offers customers the ability to use a public-ish wi-fi network.) Switching to my proper home network seems to have fixed the issue.

[Gowiem]

Another note for folks experiencing this issue -- If you have recently set ENV vars for AWS_SESSION_TOKEN or AWS_ACCESS_KEY_ID and similar, but you're using the credentials file to pull your terraform user creds then you'll hit this when that ENV variable session token expires. Use unset to remove those ENV vars and your terraform config will go back to normal.

[hashibot[bot]]

I'm going to lock this issue because it has been closed for _30 days_ ⏳. This helps our maintainers find and focus on the active issues.

If you feel this issue should be reopened, we encourage creating a new issue linking back to this one for added context. Thanks!