Eksctl: Error on MacOS - checking AWS STS access – cannot get role ARN for current session: NoCredentialProviders: no valid providers in chain

[nimbusscale]

What happened?
When running eksctl get cluster on my MacBook pro I get the following error message:

2018-11-20T10:53:43-05:00 [✖]  checking AWS STS access – cannot get role ARN for current session: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials: dial tcp 169.254.169.254:80: connect: host is down

What you expected to happen?
A list of the one cluster created by eksctl previously running in us-west-2.

How to reproduce it?
eksctl create cluster or eksctl get cluster on MacOS.

Anything else we need to know?
aws creds file (cross account role is being used):

$ cat ~/.aws/credentials
[bct-master-long-term]
aws_access_key_id = SNIP
aws_secret_access_key = <SNIP>
region = us-west-2

[default]
role_arn = arn:aws:iam::899826514230:role/SandboxAdmin
source_profile = bct-master-long-term
region = us-west-2

aws config file:

$ cat ~/.aws/config 
[default]
region = us-west-2

The following AWS CLI commands work just fine:

$ aws eks list-clusters
{
    "clusters": [
        "adorable-party-1541899386"
    ]
}

$ aws sts get-caller-identity 
{
    "Account": "899826514230", 
    "UserId": "AROAIK3I7IPINEIATJ2DK:botocore-session-1542727810", 
    "Arn": "arn:aws:sts::899826514230:assumed-role/SandboxAdmin/botocore-session-1542727810"
}

$ aws sts assume-role --role-arn "arn:aws:iam::899826514230:role/SandboxAdmin" --role-session-name "RoleSessionTest"
{
    "AssumedRoleUser": {
        "AssumedRoleId": "AROAIK3I7IPINEIATJ2DK:RoleSessionTest", 
        "Arn": "arn:aws:sts::899826514230:assumed-role/SandboxAdmin/RoleSessionTest"
    }, 
    "Credentials": {
        "SecretAccessKey": <SNIP>, 
        "SessionToken": <SNIP>,
        "Expiration": "2018-11-20T16:55:30Z", 
        "AccessKeyId": <SNIP>
    }
}

eksctl works just fine on Linux for me (using same creds and config file from above). From what I can see on the Internet this could be related to the Go SDK. I tried various troubleshooting items to see if I can get it work, but no luck. All produce the same error.

$ export AWS_PROFILE=default
$ eksctl get cluster

$ export AWS_REGION=us-west-2
$ eksctl get cluster

$ eksctl get cluster --region=us-west-2



md5-aae14a711880ae93cca7e47750f6e25f



$ export AWS_SDK_LOAD_CONFIG=1
$ eksctl get cluster
$ eksctl get cluster --profile=us-west-2



md5-80abc918e671f3828b560bed686e20dd



$ eksctl create cluster
2018-11-20T11:44:34-05:00 [ℹ]  using region us-west-2
2018-11-20T11:44:45-05:00 [✖]  checking AWS STS access – cannot get role ARN for current session: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials: dial tcp 169.254.169.254:80: connect: host is down



md5-f2793555f252a076482fc28cee02a314



$ aws --version
aws-cli/1.16.58 Python/2.7.10 Darwin/16.7.0 botocore/1.12.48
$ uname -a
Darwin Joes-MBP.home 16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64 x86_64
$ eksctl version
2018-11-20T11:17:49-05:00 [ℹ]  versionInfo = map[string]string{"builtAt":"2018-11-09T16:15:40Z", "gitCommit":"191474b2b0a6e6856b5f9c652c38b5f2f01bf7c9", "gitTag":"0.1.11"}



md5-80e2fb6bcaaecae8dafec06ca86292d9



https://amazon-eks.s3-us-west-2.amazonaws.com/1.10.3/2018-07-26/bin/darwin/amd64/aws-iam-authenticator`



md5-9d8fd7cffbe0468c30f23de6e58282b5



$ eksctl get cluster -v4
2018-11-20T11:39:21-05:00 [✖]  checking AWS STS access – cannot get role ARN for current session: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials: dial tcp 169.254.169.254:80: connect: host is down

[errordeveloper]

Thanks for detailed report! I am on holiday at the moment, but @christopherhein maybe able to help here, or perhaps @richardcase.

[christopherhein]

@nimbusscale I would try following the thread https://github.com/weaveworks/eksctl/issues/57#issuecomment-396160263 it might help. Seems like it's thinking there should be a default profile?

I typically have AWS_DEFAULT_PROFILE set as well if I'm not using the default profile, this might help too.

[nimbusscale]

I am using the default profile, but I was able to figure it out.

For some reason on Mac I needed to set the AWS_SHARED_CREDENTIALS_FILE Env Var, even though I'm using the default path.

$ eksctl get clusters
2018-12-04T10:45:14-08:00 [✖]  checking AWS STS access – cannot get role ARN for current session: NoCredentialProviders: no valid providers in chain
caused by: EnvAccessKeyNotFound: failed to find credentials in the environment.
SharedCredsLoad: failed to load profile, .
EC2RoleRequestError: no EC2 instance role found
caused by: RequestError: send request failed
caused by: Get http://169.254.169.254/latest/meta-data/iam/security-credentials: dial tcp 169.254.169.254:80: connect: host is down
$ export AWS_SHARED_CREDENTIALS_FILE=~/.aws/credentials
$ eksctl get clusters
NAME                REGION
attractive-sheepdog-1543796753  us-west-2

I don't need to set this on Linux. I assume this is a bug with the Go SDK and not an Eksctl bug.

[nimbusscale]

Closing this issue with the assumption this is not an eksctl bug.

[TreverW]

I still see this issue on Mac OSX 10.14.4 but it is intermittent. Sometimes it works, sometimes it doesn't.

I am using:
eksctl version 0.1.29,
aws cli version aws-cli/1.16.140 Python/3.7.3 Darwin/18.5.0 botocore/1.12.130

I have tried setting the AWS_SHARED_CREDENTIALS_FILE environment variable as suggested by @nimbusscale.

Here is an example session where it worked then didn't.

# aws cli commands work fine (consistently)
$ aws eks list-clusters
{
    "clusters": [
        "kubernetes-cd5"
    ]
}

# Intermittently, eksctl commands fail.
$ eksctl create nodegroup --config-file=cluster-cd5.yml --only nodes-test
[ℹ]  using region us-east-1
[✖]  checking AWS STS access – cannot get role ARN for current session: RequestError: send request failed
caused by: Post https://sts.amazonaws.com/: net/http: TLS handshake timeout

$ export AWS_SHARED_CREDENTIALS_FILE=/Users/twilhelm/.aws/credentials

# Setting the env var seemed to have helped, at least for one command...
$ eksctl create nodegroup --config-file=cluster-cd5.yml --only nodes-test
[ℹ]  using region us-east-1
[ℹ]  nodegroup "nodes-test" will use "ami-0bd7eb0889ba1a301" [AmazonLinux2/1.11]
[!]  retryable error (Throttling: Rate exceeded
    status code: 400, request id: 44311633-606d-11e9-b4e7-236a52c9e7fc) from cloudformation/DescribeStacks - will retry after delay of 731ms
[!]  retryable error (Throttling: Rate exceeded
    status code: 400, request id: 44ccb9c0-606d-11e9-b69a-1967de8b00be) from cloudformation/DescribeStacks - will retry after delay of 1.594s
[ℹ]  3 nodegroup(s) were filtered out: --only="nodes-test" was given, 5 nodegroup(s) (cluster-management1, ignite, nodes-default2) already exist
[ℹ]  will create a CloudFormation stack for each of 1 nodegroups in cluster "kubernetes-cd5"
[ℹ]  1 task: { create nodegroup "nodes-test" }
[ℹ]  buildings nodegroup stack "eksctl-kubernetes-cd5-nodegroup-nodes-test"
[ℹ]  --nodes-min=6 was set automatically for nodegroup nodes-test
[ℹ]  --nodes-max=6 was set automatically for nodegroup nodes-test
[ℹ]  deploying stack "eksctl-kubernetes-cd5-nodegroup-nodes-test"
[ℹ]  adding role "nodes-kubernetes-cd5" to auth ConfigMap
[ℹ]  nodegroup "nodes-test" has 0 node(s)
[ℹ]  waiting for at least 6 node(s) to become ready in "nodes-test"
[ℹ]  nodegroup "nodes-test" has 6 node(s)
[ℹ]  node "ip-172-22-104-124.ec2.internal" is ready
[ℹ]  node "ip-172-22-112-83.ec2.internal" is ready
[ℹ]  node "ip-172-22-134-152.ec2.internal" is ready
[ℹ]  node "ip-172-22-156-21.ec2.internal" is ready
[ℹ]  node "ip-172-22-166-71.ec2.internal" is ready
[ℹ]  node "ip-172-22-179-29.ec2.internal" is ready
[✔]  created 1 nodegroup(s) in cluster "kubernetes-cd5"
[ℹ]  checking security group configuration for all nodegroups
[ℹ]  all nodegroups have up-to-date configuration

# next command failed
$ eksctl get ng --cluster=kubernetes-cd5 --output=json
[✖]  checking AWS STS access – cannot get role ARN for current session: RequestError: send request failed
caused by: Post https://sts.amazonaws.com/: net/http: TLS handshake timeout

Adding the response on a failing call with -v5

eksctl get ng --cluster=kubernetes-cd5 --output=json -v5
2019-04-16T10:42:40-07:00 [▶]  DEBUG: Request sts/GetCallerIdentity Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: sts.amazonaws.com
User-Agent: eksctl/{"BuiltAt":"","GitTag":"0.1.29"} aws-sdk-go/1.17.7 (go1.12.3; darwin; amd64)
Content-Length: 43
Authorization: AWS4-HMAC-SHA256 Credential=<redacted>/20190416/us-east-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=<redacted>
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20190416T174240Z
Accept-Encoding: gzip

Action=GetCallerIdentity&Version=2011-06-15
-----------------------------------------------------
2019-04-16T10:42:50-07:00 [▶]  DEBUG: Send Request sts/GetCallerIdentity failed, will retry, error RequestError: send request failed
caused by: Post https://sts.amazonaws.com/: net/http: TLS handshake timeout
2019-04-16T10:42:50-07:00 [▶]  DEBUG: Retrying Request sts/GetCallerIdentity, attempt 1
2019-04-16T10:42:50-07:00 [▶]  DEBUG: Request sts/GetCallerIdentity Details:
---[ REQUEST POST-SIGN ]-----------------------------
POST / HTTP/1.1
Host: sts.amazonaws.com
User-Agent: eksctl/{"BuiltAt":"","GitTag":"0.1.29"} aws-sdk-go/1.17.7 (go1.12.3; darwin; amd64)
Content-Length: 43
Authorization: AWS4-HMAC-SHA256 Credential=<redacte4d>/20190416/us-east-1/sts/aws4_request, SignedHeaders=content-length;content-type;host;x-amz-date, Signature=<redacted>
Content-Type: application/x-www-form-urlencoded; charset=utf-8
X-Amz-Date: 20190416T174250Z
Accept-Encoding: gzip

Action=GetCallerIdentity&Version=2011-06-15
-----------------------------------------------------
2019-04-16T10:43:00-07:00 [▶]  DEBUG: Send Request sts/GetCallerIdentity failed, not retrying, error RequestError: send request failed
caused by: Post https://sts.amazonaws.com/: net/http: TLS handshake timeout
2019-04-16T10:43:00-07:00 [✖]  checking AWS STS access – cannot get role ARN for current session: RequestError: send request failed
caused by: Post https://sts.amazonaws.com/: net/http: TLS handshake timeout

[TreverW]

Is it possible to reopen this issue? I can try to help provide any details necessary to reproduce and fix it.

Thanks.

[Davidrjx]

same error on debian distribution. executing "eksctl create ..." outputs error resembling

checking AWS STS access – cannot get role ARN for current session: InvalidClientTokenId: The security token included in the request is invalid.
        status code: 403, request id: 9a89b74d-807b-11e9-9198-3b3470540988

[MitchyBAwesome]

Also seeing this issue on macOS 10.13.6 with eksctl version 0.1.32. As above, all other aws cli commands are working as expected.

Checking AWS STS access – cannot get role ARN for current session: RequestError: send request failed
caused by: Post https://sts.amazonaws.com/: net/http: TLS handshake timeout

[benhg]

I'm still seeing this issue as well, on MacOS 10.14

[errordeveloper]

@benhg from our slack conversation, I was under impression you resolved this on your end and it was a local configuration issue, correct?

[benhg]

Yes, sorry to bother you guys.

[nimbusscale]

I just reopened as I saw there were a few others who had the issue. I've since moved on to work on other things. Please feel free to reclose if you don't think this is an issue that needs to be looked at.

I won't reopen again.

[errordeveloper]

Thanks everyone!

[imago-storm]

I see this as well, on MacOS 10.14, constantly. I've tried setting AWS_SHARED_CREDENTIALS_FILE variable. My eksctl version is [ℹ] version.Info{BuiltAt:"", GitCommit:"", GitTag:"0.3.0"}.

[MLDMoritz]

Experiencing this on Ubuntu.

[mirr254]

I am also experiencing this on Ubuntu

[nrdlngr]

@benhg, @errordeveloper, what was the configuration issue? It seems like others are having the same problem.

[nrdlngr]

Does anyone know what causes the "Post https://sts.amazonaws.com/: net/http: TLS handshake timeout" error?

[benhg]

I believe it was a problem with my EC2 credentials

[qmilangowin]

I'm still getting this, macOS Catalina, latest version of Docker Desktop with Kubernetes installed.
It's very frustrating. Had to create my own docker image to be able to use eksctl with our clusters at work.
Cannot use it natively from my mac.

[michaelbeaumont]

@qmilangowin Are you able to use AWS cli otherwise?

[qmilangowin]

@michaelbeaumont Yes, I can use it no problem otherwise. Just with eksctl I have this issue.