Aws-iam-authenticator: error: the server doesn't have a resource type "svc"

[laszlocsontos]

Steps taken

1) Created an role as per https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html#eks-prereqs
2) Created a K8S cluster as per https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html#eks-create-cluster
3) Created an IAM user with programmatic access with the following permissions

• AmazonEKSClusterPolicy
• AmazonEKSWorkerNodePolicy
• AmazonEKSServicePolicy
• AmazonEKS_CNI_Policy
• AmazonEC2ReadOnlyAccess

Note: the original IAM user we had in .aws/credentials used AmazonEC2ReadOnlyAccess.
I added that in order not to break existing scripts.

4) Upgraded AWS CLI

% aws --version
aws-cli/1.16.30 Python/2.7.14 Linux/4.14.72-68.55.amzn1.x86_64 botocore/1.12.20

5) Installed kubectl as per https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html#eks-configure-kubectl

% kubectl version --short --client
Client Version: v1.10.3

6) Installed aws-iam-authenticator

7) Generated kubeconfig

% aws sts get-caller-identity
XXXXXXXXX   arn:aws:iam::XXXXXXXXX:user/bastion XXXXXXXXXXXXXXXXXXX

% aws eks update-kubeconfig --name test
Updated context arn:aws:eks:us-east-1:XXXXXXXXX:cluster/test in /home/ec2-user/.kube/config

% kubectl get svc
error: the server doesn't have a resource type "svc"

Debugging

1) Fetch token

% aws-iam-authenticator token -i wizenoze-test 
{
   "kind":"ExecCredential",
   "apiVersion":"client.authentication.k8s.io/v1alpha1",
   "spec":{

   },
   "status":{
      "token":"k8s-aws-v1.XXX"
   }
}

2) Verify token

% aws-iam-authenticator verify -i test -t k8s-aws-v1.XXX

&{ARN:arn:aws:iam::XXXXXXXXX:user/bastion CanonicalARN:arn:aws:iam::XXXXXXXXX:user/bastion AccountID:XXXXXXXXX UserID:XXXXXXXXXXXXXXXXXXX SessionName:}

Could you please help me in finding out what's wrong?

Thanks,
László

[weotch]

I was experiencing this until I opened a new tab in Terminal

[Samze]

We are also seeing this exact problem.

This is the output we get:

kubectl get svc --v=10
I1016 16:12:26.421644   49924 loader.go:359] Config loaded from file /Users/pivotal/.kube/config
I1016 16:12:26.421874   49924 round_trippers.go:386] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.11.1 (da
rwin/amd64) kubernetes/b1b2997" 'https://XXXXX.sk1.eu-west-1.eks.amazonaws.com/api?timeout=32s'
could not get token: NoCredentialProviders: no valid providers in chain. Deprecated.
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors
I1016 16:12:31.826447   49924 round_trippers.go:405] GET https://XXXXX.sk1.eu-west-1.eks.amazonaws.com/api?timeout
=32s  in 5404 milliseconds
I1016 16:12:31.826470   49924 round_trippers.go:411] Response Headers:
I1016 16:12:31.826519   49924 cached_discovery.go:111] skipped caching discovery info due to Get https://XXXXX.sk1
.eu-west-1.eks.amazonaws.com/api?timeout=32s: getting credentials: exec: exit status 1
I1016 16:12:31.826652   49924 round_trippers.go:386] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.11.1 (da
rwin/amd64) kubernetes/b1b2997" 'https://XXXXX.sk1.eu-west-1.eks.amazonaws.com/api?timeout=32s'
could not get token: NoCredentialProviders: no valid providers in chain. Deprecated.
        For verbose messaging see aws.Config.CredentialsChainVerboseErrors

We additionally assumed the role we created for our EKS cluster using

aws eks update-kubeconfig --name eksgism --role-arn arn:aws:iam::XXXXX:role/eksgism-role

We too can seemingly successfully use the aws-iam-authenticator

aws-iam-authenticator token -i eksgism -r arn:aws:iam::XXXXX:role/eksgism-role
{"kind":"ExecCredential","apiVersion":"client.authentication.k8s.io/v1alpha1","spec":{},"status":{"token":"k8s-aws-v1.XXXXX}

aws-iam-authenticator verify -i eksgism -t k8s-aws-v1.XXXXX
&{ARN:arn:aws:sts::XXXXX:assumed-role/eksgism-role/XXXXX CanonicalARN:arn:aws:iam::XXXXX:role/eksgism-role AccountID:XXXXX UserID:XXXXX SessionName:XXXXX}

...And opening a new tab did not work :(

Cheers,
Sam & @teddyking

[nckturner]

@laszlocsontos Whichever user/role's credentials you were using when you created the EKS cluster are the credentials you should be using when you try kubectl get svc. (In your case, user/bastion). Was this the same user that you used to create the cluster? The EKS service role has nothing to do with it.

[nckturner]

@Samze it appears that your AWS credentials are not accessible to the authenticator binary. What happens when you run aws sts get-caller-identity?

[KailasJanardhanan]

I am facing the exact same issue
aws sts get-caller-identity gives the following
{
"Account": "XXXXXXXXXXXX",
"UserId": "YYYYYYYYYYYYYYYYYYY:i-XXXXXXXXXXXXX",
"Arn": "arn:aws:sts::XXXXXXXXXXXX:assumed-role//i-"
}

[bestbear]

I'm facing the exact same problem as well. Interestingly enough, the token generation works but the verify fails: aws-iam-authenticator verify -i test -t "k8s-aws-v1.aMGI3YjY0M2M0YjE1Njk5"
could not verify token: sts getCallerIdentity failed: error from AWS (expected 200, got 403) However, when I execute "kubectl" with debugging enabled it returns an "unauthorized".

kubectl get svc --v=10

I1111 18:27:30.124663 58459 round_trippers.go:386] curl -k -v -XGET -H "Accept: application/json, /" -H "User-Agent: kubectl/v1.11.3 (darwin/amd64) kubernetes/a452946" 'https://63817EFA98124CC7D863C5AA77B202C1.sk1.us-west-2.eks.amazonaws.com/api?timeout=32s'
I1111 18:27:30.961638 58459 round_trippers.go:405] GET https://63817EFA98124CC7D863C5AA77B202C1.sk1.us-west-2.eks.amazonaws.com/api?timeout=32s 401 Unauthorized in 836 milliseconds

Then I try to run the kubectl curl directly that just failed making no changes and voila it works:

curl -k -v -XGET -H "Accept: application/json, /" -H "User-Agent: kubectl/v1.11.3 (darwin/amd64) kubernetes/a452946" 'https://63817EFA98124CC7D863C5AA77B202C1.sk1.us-west-2.eks.amazonaws.com/api?timeout=32s'
Note: Unnecessary use of -X or --request, GET is already inferred.

• Trying 52.40.85.170...
• TCP_NODELAY set
• Connected to 63817EFA98124CC7D863C5AA77B202C1.sk1.us-west-2.eks.amazonaws.com (52.40.85.170) port 443 (#0)
• ALPN, offering http/1.1
• Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
• successfully set certificate verify locations:
• CAfile: /opt/local/share/curl/curl-ca-bundle.crt
  CApath: none
• TLSv1.2 (OUT), TLS header, Certificate Status (22):
• TLSv1.2 (OUT), TLS handshake, Client hello (1):
• TLSv1.2 (IN), TLS handshake, Server hello (2):
• TLSv1.2 (IN), TLS handshake, Certificate (11):
• TLSv1.2 (IN), TLS handshake, Server key exchange (12):
• TLSv1.2 (IN), TLS handshake, Request CERT (13):
• TLSv1.2 (IN), TLS handshake, Server finished (14):
• TLSv1.2 (OUT), TLS handshake, Certificate (11):
• TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
• TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
• TLSv1.2 (OUT), TLS handshake, Finished (20):
• TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
• TLSv1.2 (IN), TLS handshake, Finished (20):
• SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
• ALPN, server accepted to use http/1.1
• Server certificate:
• subject: CN=kube-apiserver
• start date: Nov 5 20:31:28 2018 GMT
• expire date: Nov 5 20:33:37 2019 GMT
• issuer: CN=kubernetes
• SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.

GET /api?timeout=32s HTTP/1.1
Host: 63817EFA98124CC7D863C5AA77B202C1.sk1.us-west-2.eks.amazonaws.com
Accept: application/json, /
User-Agent: kubectl/v1.11.3 (darwin/amd64) kubernetes/a452946

< HTTP/1.1 200 OK
< Audit-Id: de7d427b-4c5f-4e91-bebc-e200a40b4b83
< Content-Type: application/json
< Date: Mon, 12 Nov 2018 01:30:10 GMT
< Content-Length: 164
<
{"kind":"APIVersions","versions":["v1"],"serverAddressByClientCIDRs":[{"clientCIDR":"0.0.0.0/0","serverAddress":"ip-172-16-56-17.us-west-2.compute.internal:443"}]}

• Connection #0 to host 63817EFA98124CC7D863C5AA77B202C1.sk1.us-west-2.eks.amazonaws.com left intact

[narup]

I was having the same issue. Doc is bit misleading, make sure your aws cli is configured with the key and secret from the user which created EKS cluster

[bestbear]

narup, yes, you might be on to something. I created the cluster from the GUI using SSO credentials which would have used temporary access/secret keys. All the kubectl work was done using my permanent access/secret key pair. I will have to go back and validate this again using only the CLI with my access/secret.

[pastephens]

Also seeing this. I created two clusters, one in the console and one from the CLI. The result is the same error. I can fetch and verify my token is my SAML user, which I used to create both clusters. I'm using 'assume-role' command to align the caller with the MFA token.

~> aws --version
aws-cli/1.16.57 Python/2.7.5 Linux/3.10.0-862.14.4.el7.x86_64 botocore/1.12.47

~> kubectl version --short --client
Client Version: v1.12.2

[luthes]

Might you guys be using Terraform at all? I know it's still just an API call for both. I'm seeing this when using Terraform that assumes a role, and regardless of if I assume that role or not, I can't access the control plane.

[bestbear]

Negative, I'm not using Terraform for this only the Console and the CLI.

[Fatxx]

My problem was related to not having my IAM user keys set as default at ~/.aws/credentials, I was using the --profile flag to run AWS CLI commands as I normally do but it doesn't seems to work for this case

[luispabon]

I'm having the exact same issue. Kubectl logs say

I1123 13:09:45.789218   28300 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

I created my cluster via the AWS console using one user, and I have another user (programmatic only) for the command line. The programmatic user has full EKS permissions on that cluster.

aws sts get-caller-identity

This outputs my programmatic user correctly. However:

~ aws-iam-authenticator token -i XXXX 
[copy token]

~ aws-iam-authenticator verify -i test -t TOKEN
could not verify token: sts getCallerIdentity failed: error from AWS (expected 200, got 403)

[stormguardian]

I had the same issue.
After looking around a bit, i've found that running this command:
aws eks update-kubeconfig --name xxxxxxxx --role-arn arn:aws:iam::xxxxxxxxx:role/EKS-Manager
kubectl get svc
Response:
could not get token: AccessDenied: Access denied
status code: 403, request id: c86f28cf-f16c-11e8-a02d-bf81fbdd8d60
Unable to connect to the server: getting credentials: exec: exit status 1

Then I've found that you can connect to the kubectl cluster with the same user that created it from the console.

So after creating the kube cluster with the same user that is in my aws creds, running
aws eks update-kubeconfig --name xxxxxxxx
kubectl get svc

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
kubernetes ClusterIP 10.100.0.1 443/TCP 21m

[jaydipdave]

EKS is a suicide mission by AWS. This Elastic Kube Shit is not working.

1) Created a role
2) Set my user as trusted entities.
3) Assigned a policy to my user so it can assume the role.
4) Created EKS Cluster
5) Installed latest aws*, kubectl
5) aws eks update-kubeconfig --name xxxxxxxx --role-arn arn:aws:iam::xxxxxxxxx:role/EKS-Manager
6) Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

All so tried using a readily available EKS-cli docker images, but same error.

Tried curl as well:

# TOKEN=$(aws-iam-authenticator token -i shit | jq -r .status.token)
# curl $APISERVER/version --header "Authorization: Bearer $TOKEN" --insecure
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

Try 2:

# TOKEN=$(aws-iam-authenticator token -i shit -r arn:aws:iam::xxxxxxxxxxxxx:role/EKS-manager | jq -r .status.token)
# curl $APISERVER/version --header "Authorization: Bearer $TOKEN" --insecure
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

Tried old version:

# TOKEN=$(heptio-authenticator-aws token -i shit | jq .status.token)

curl $APISERVER/version --header "Authorization: Bearer $TOKEN" --insecure
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {

  },
  "status": "Failure",
  "message": "Unauthorized",
  "reason": "Unauthorized",
  "code": 401
}

[mrichman]

@jaydipdave Who are you at the CLI when getting your token (i.e. aws sts get-caller-identity)? Is this the same user that created the cluster? I'm at re:Invent right now. Hoping to find some EKS folks to talk to about the user experience here.

[stormguardian]

From AWS Docs: https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html

When an Amazon EKS cluster is created, the IAM entity (user or role) that creates the cluster is added to the Kubernetes RBAC authorization table as the administrator (with system:master permissions. Initially, only that IAM user can make calls to the Kubernetes API server using kubectl. For more information, see Managing Users or IAM Roles for your Cluster. Also, the AWS IAM Authenticator for Kubernetes uses the AWS SDK for Go to authenticate against your Amazon EKS cluster. If you use the console to create the cluster, you must ensure that the same IAM user credentials are in the AWS SDK credential chain when you are running kubectl commands on your cluster.

[jaydipdave]

@jaydipdave Who are you at the CLI when getting your token (i.e. aws sts get-caller-identity)? Is this the same user that created the cluster? I'm at re:Invent right now. Hoping to find some EKS folks to talk to about the user experience here.

Hi @mrichman ,

Thanks for you quick response. I am using a different user at AWS CLI. i.e. "tools".
But EKS-user has a policy to assume the role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": "arn:aws:iam::xxxxxxxxxxxxx:role/*"
        }
    ]
}

And in the role, who created the EKS cluster, I set the Trusted Entities.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "eks.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    },
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::xxxxxxxxxxx:user/tools"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

I am assuming the role using "-r arn:aws:iam::xxxxxxxxxxxxx:role/EKS-manager ".

Should this be sufficient?

[luthes]

This really should be sufficient, but I don't think it is, and I think it's the problem that we're running into. Using an assumed role hasn't worked for me at all when creating a cluster, I've tried this a bunch of different ways. The only way I can do it is by using a logged in user (which uses an assumed role through SAML auth) Maybe we're all doing something wrong, but I haven't been able to figure it out.

And a little off topic, adding other users/roles at creation is a huge oversight by AWS, if one is added at creation, I can't see why they woudn't have thought to give us the ability to add more.

From AWS Docs: https://docs.aws.amazon.com/eks/latest/userguide/getting-started.html

When an Amazon EKS cluster is created, the IAM entity (user or role) that creates the cluster is added to the Kubernetes RBAC authorization table as the administrator (with system:master permissions. Initially, only that IAM user can make calls to the Kubernetes API server using kubectl. For more information, see Managing Users or IAM Roles for your Cluster. Also, the AWS IAM Authenticator for Kubernetes uses the AWS SDK for Go to authenticate against your Amazon EKS cluster. If you use the console to create the cluster, you must ensure that the same IAM user credentials are in the AWS SDK credential chain when you are running kubectl commands on your cluster.

Maybe the issue is coming from multiple assumed roles. Using SAML, I'm authenticating using 2FA, which assumes a role. That role that I assume then assumes another role, my Terraform role. So I'm a couple IAM roles deep at this point, and maybe something wierd is happening.

[mrichman]

I’m here at reinvent trying to get an audience with anyone on the EKS team who can help.

[bestbear]

I think the nut of the problem has do to with the initial EKS cluster creation and then the following commands via kubectl. My hypothesis is that the initial EKS cluster setup is done via the console and then the subsequent provisioning via the kubeclt on the cli breaks the aws-iam-authenticator model because the initial cluster creation stores data in the master node that is hidden. I will attempt to prove this theory as soon as I get a moment. At the Reinvent conference in Vegas.

Regards

Mark Cwetna
Slalom

From: Mark Richman [e-mail]
Reply-To: kubernetes-sigs/aws-iam-authenticator [e-mail]
Date: Tuesday, November 27, 2018 at 11:07 PM
To: kubernetes-sigs/aws-iam-authenticator [e-mail]
Cc: Mark Cwetna [e-mail], Comment [e-mail]
Subject: Re: [kubernetes-sigs/aws-iam-authenticator] error: the server doesn't have a resource type "svc" (#157)

I’m here at reinvent trying to get an audience with anyone on the EKS team who can help.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_kubernetes-2Dsigs_aws-2Diam-2Dauthenticator_issues_157-23issuecomment-2D442331310&d=DwMFaQ&c=fa_WZs7nNMvOIDyLmzi2sMVHyyC4hN9WQl29lWJQ5Y4&r=bizeWICKQs8YNpGSnQCdAWt6454ZIJWOmCQaNqXvmH4&m=l1zjz-TjXuvbsQ5L_znEchS-Hvu1Avt3_q9pWEluPcA&s=RbaBT5AH7kY7BiYlIaOSkVV0VZzXQhsIAc56vnJ2pEY&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AE5bgnAYay3H7PLHaRuN6rhMd9VqRDP-5Fks5uzig6gaJpZM4XUt7I&d=DwMFaQ&c=fa_WZs7nNMvOIDyLmzi2sMVHyyC4hN9WQl29lWJQ5Y4&r=bizeWICKQs8YNpGSnQCdAWt6454ZIJWOmCQaNqXvmH4&m=l1zjz-TjXuvbsQ5L_znEchS-Hvu1Avt3_q9pWEluPcA&s=j-dzY-VHOfS6d6-7P6YdwNx_Ezk-_YaJm4bLOPX5pIw&e=.

[yazfaz]

Possible solution if you created the cluster in the UI

If you created the cluster in the UI, it's possible the AWS root user created the cluster. According to the docs, "When an Amazon EKS cluster is created, the IAM entity (user or role) that creates the cluster is added to the Kubernetes RBAC authorization table as the administrator (with system:master) permissions. Initially, only that IAM user can make calls to the Kubernetes API server using kubectl. "

You'll need to first login to the AWS CLI as the root user or the user who created the cluster in order to update the permissions of the IAM user you want to have access to the cluster.

1. You'll need to get an access key for the root user or the user you were logged in as when you created the cluster and put this info in .aws/credentials under the default user. You can do this using the command aws configure

   Now kubectl get svc works, since you're logged in as the root user that initially created the cluster.

2. Apply the aws-auth ConfigMap to the cluster. Follow step 2 from these docs, using the NodeInstanceRole value you got as the Output from Step 3: Launch and Configure Amazon EKS Worker Nodes

3. To add a non-root IAM user or role to an Amazon EKS cluster, follow step 3 from these docs.
   Edit the configmap/aws-auth and add other users that need kubectl access in the mapUsers section.

4. Run aws configure again and add the access key info from your non-root user.

Now you can access your cluster from the AWS CLI and using kubectl!

[jaydipdave]

Getting the root user credentials is not possible for everyone. In my case, I am using centrally managed role by the organization. I have admin access but don't have root credentials.

[yazfaz]

Are you able to get the Access Key ID and Secret Access Key for the role you were logged in as when you created the cluster? (Or have an admin role get this for you, if you don't the permissions to do this). And then put that info in .aws/credentials and follow the steps above?

[bestbear]

Root credentials are taboo.

This tool does not account for the use-case of SAML providers. The STS AssumeRole always uses temporary keys. If the master is looking at the temporary keys that is broken. The keys expire in 60 minutes.

Mark Cwetna
•••sent from my iPhone•••

---

From: Yasmin [e-mail]
Sent: Thursday, November 29, 2018 2:10 PM
To: kubernetes-sigs/aws-iam-authenticator
Cc: Mark Cwetna; Comment
Subject: Re: [kubernetes-sigs/aws-iam-authenticator] error: the server doesn't have a resource type "svc" (#157)

Are you able to get the Access Key ID and Secret Access Keyhttps://urldefense.proofpoint.com/v2/url?u=https-3A__docs.aws.amazon.com_IAM_latest_UserGuide_id-5Fcredentials-5Faccess-2Dkeys.html&d=DwMFaQ&c=fa_WZs7nNMvOIDyLmzi2sMVHyyC4hN9WQl29lWJQ5Y4&r=bizeWICKQs8YNpGSnQCdAWt6454ZIJWOmCQaNqXvmH4&m=1qvTmmy2jCwQTaseeb1l6AfLs-yGiOC07A8e3NrwC1I&s=ZH28w6e3zX9Go8YIZ63XzLn3Qh2Iron8Jm9G7Sn7C3s&e= for the role you were logged in as when you created the cluster? (Or have an admin role get this for you, if you don't the permissions to do this). And then put that info in .aws/credentials and follow the steps above?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_kubernetes-2Dsigs_aws-2Diam-2Dauthenticator_issues_157-23issuecomment-2D443012625&d=DwMFaQ&c=fa_WZs7nNMvOIDyLmzi2sMVHyyC4hN9WQl29lWJQ5Y4&r=bizeWICKQs8YNpGSnQCdAWt6454ZIJWOmCQaNqXvmH4&m=1qvTmmy2jCwQTaseeb1l6AfLs-yGiOC07A8e3NrwC1I&s=nUlEMjW1pCzIZ7VJv3VKsv8X6i_hundPxmmtm1t5O7M&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AE5bgtZcLvg9BqDD91KbstgJYkQSjqoeks5u0FttgaJpZM4XUt7I&d=DwMFaQ&c=fa_WZs7nNMvOIDyLmzi2sMVHyyC4hN9WQl29lWJQ5Y4&r=bizeWICKQs8YNpGSnQCdAWt6454ZIJWOmCQaNqXvmH4&m=1qvTmmy2jCwQTaseeb1l6AfLs-yGiOC07A8e3NrwC1I&s=o_1P9xlhRIg0gUTJI6I8pI84KB9y-LL-wIMqNxRsuW4&e=.

[GeoffMillerAZ]

I usually use inter-account role assumption and terraform for aws. After reading above, I decided to do a terraform destroy on my eks master. Then I used the CLI to create it manually and I planned on just doing an import into terraform later. Before I used the CLI to create the new master, I created an IAM user just for the purpose of setting up this master and managing user access. I planned on deleting this IAM user later. But even with this IAM user, I am getting the exact same problem.

[staging_ekscreator]
aws_access_key_id=xxx
aws_secret_access_key=xyz

aws --profile=staging_ekscreator eks create-cluster --cli-input-json file://eks.json
aws --profile=staging_ekscreator eks update-kubeconfig --name my-eks
kubectl get svc --v=10
I1130 15:23:45.150251   17868 loader.go:359] Config loaded from file /Users/gmiller/.kube/config
I1130 15:23:45.151062   17868 loader.go:359] Config loaded from file /Users/gmiller/.kube/config
I1130 15:23:45.151634   17868 round_trippers.go:386] curl -k -v -XGET  -H "User-Agent: kubectl/v1.12.3 (darwin/amd64) kubernetes/xxx" -H "Accept: application/json, */*" 'https://xxx.sk1.us-west-2.eks.amazonaws.com/api?timeout=32s'
I1130 15:23:45.929480   17868 round_trippers.go:405] GET https://xxx.sk1.us-west-2.eks.amazonaws.com/api?timeout=32s 401 Unauthorized in 777 milliseconds
I1130 15:23:45.929518   17868 round_trippers.go:411] Response Headers:
I1130 15:23:45.929525   17868 round_trippers.go:414]     Audit-Id: xxx
I1130 15:23:45.929530   17868 round_trippers.go:414]     Content-Type: application/json
I1130 15:23:45.929535   17868 round_trippers.go:414]     Content-Length: 129
I1130 15:23:45.929542   17868 round_trippers.go:414]     Date: Fri, 30 Nov 2018 22:23:45 GMT
I1130 15:23:45.931723   17868 request.go:942] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
I1130 15:23:45.933017   17868 cached_discovery.go:111] skipped caching discovery info due to Unauthorized
F1130 15:23:46.625767   17868 helpers.go:119] error: the server doesn't have a resource type "svc"

[GeoffMillerAZ]

update:

https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html

apiVersion: v1
clusters:
- cluster:
    server: <endpoint-url>
    certificate-authority-data: <base64-encoded-ca-cert>
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: aws
  name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: aws-iam-authenticator
      args:
        - "token"
        - "-i"
        - "<cluster-name>"
        # - "-r"
        # - "<role-arn>"
      env:
        - name: AWS_PROFILE
          value: "staging_ekscreator"

adding this:

      env:
        - name: AWS_PROFILE
          value: "staging_ekscreator"

fixed my issue. So maybe on cluster creation you can just use a manually created IAM and then import it into terraform after you use this user to setup your user permissions. Then you can just delete this IAM user to keep your cloud tidy and secure. I'm trying that next.

[orangewise]

It works for me now. I had to set the aws credentials in another session (tab) than the one that is using kubectl. Thanks @weotch.

[pkelleratwork]

im running into this exact same thing trying to deploy from circleci to an eks cluster. horrible experience.

has anyone found a solution? Im stuck trying to get a prod cluster up. Its beyond frustrating...

i can access if from my laptop kubectl works and i can see everything.
i copied the same aws keys and kubeconfig from my laptop to ciriclei
try to connect in a build get

kubectl get svc
error: the server doesn't have a resource type "svc"
Exited with code 1

logging in as root is a no-go for me. not going to happen.

[jpatters]

I've tried about 15 times in the last 24 hours to create a cluster that I can actually access.
I've tried from the console, from the cli, with the terraform module.
I've double and triple checked that the IAM credentials I created it with are the same as the ones I am trying to connect with.
I've tried using a role, using root credentials, using an IAM user. I've tried specifying a profile or even the exact keys when using aws-iam-authenticator.
I have followed the official amazon getting started guide, the terraform getting started guide, and another random one I found.
Nothing I have done has resulted in a cluster that I can actually authenticate with or that nodes can even connect to. The authenticator can create a token but cannot verify it.

$ aws-iam-authenticator verify -i <clusterid> -t <token>
could not verify token: sts getCallerIdentity failed: error from AWS (expected 200, got 403)

Is EKS just broken?

[mrichman]

@jpatters This has happened to me before too. You're not using Cloud9 by chance are you?

Also, what's in your ~/.aws/credentials (don't post it here) just curious if you have multiple profiles, or anything in there other than just aws_access_key_id and aws_secret_access_key. I once had aws_session_token= (no value) which screwed me up like you're seeing.

What's the output from aws sts get-caller-identity? Does it match the ID in your credentials?

[jpatters]

@mrichman I am not using cloud9. However I did just get this to work by explicitly exporting aws key and secret

export AWS_ACCESS_KEY_ID=key
export AWS_SECRET_ACCESS_KEY=secret

(I know I said I tried this in my comment above but I must not have)

I do have multiple profiles in ~/.aws/credentials but I have verified that aws sts get-caller-identity returns the correct one.

At the moment, it seems to me like aws-iam-authenticator is not properly reading ~/.aws/credentials or maybe ~/.aws/config

It was working on a test cluster just now. I am now launching my intended cluster via terraform and I will report back what I find.

[jpatters]

The only way it is working for me is if I explicitly export the credentials. It seems to be caused by the authenticator not reading ~/.aws/credentials properly (or not it at all)

[reachhsubbu]

Yes. second it. It works only if I explicitly export the credentials.

[GeoffMillerAZ]

I spoke to a gentleman at the AWS EKS booth at Kubecon. I explained this problem and how I typically use credentials that are temporary from STS due to MFA and role assumption. He said they are getting a lot of complaints on this problem and that they are trying to come up with a good solution. I shared that what I had done was to make an IAM user in that specific account for the purpose of account creation and that I intended to delete the user after I setup permissions the way I needed on RBAC. He said that is a good solution except to not delete the account but to keep it as a break-glass account in case problems arose.

He also mentioned they _may_ be working on a solution to expose master logs to you so you can see and troubleshoot without as much guessing.

[nakulpathak3]

Are there any other workarounds/fixes for this? I tried explicitly exporting AWS_ACCESS_KEY_ID and AWS_ACCESS_SECRET_KEY and that did not work. Also tried adding

env:
        - name: AWS_PROFILE
          value: "staging_ekscreator"

and that failed on cluster create too. The error I'm getting is ERROR:root:exec: process returned 1. could not get token: AccessDenied: Access denied status code: 403, request id: e63b2a9f-03e2-11e9-9cea-e510a9909b67 when trying to get_nodes and I'm using the Kubernetes Python Client not kubectl. For some reason, disconnecting and connecting to the EKS cluster solves the issue but it continues to show up randomly.

This only happens when using the role I created as part of the EKS Getting Started guide, any commands made without the role (by my own user account which is same as the account that created the cluster) work correctly with no problems.

[falfaro]

Is this a problem with aws-iam-authenticator or not? Exporting the credentials is a valid workaround but very ugly. Perhaps this is because I am using multiple profiles?

[schumacher-m]

I followed https://docs.aws.amazon.com/eks/latest/userguide/add-user-role.html
to add ARN of my colleagues who wanted to manage the cluster:

kubectl edit -n kube-system configmap/aws-auth

[joshuabaird]

Like many others here, we use STS and assumed roles with profiles to access our AWS account(s). We encountered the same problem(s) that others here have. Thanks to @GeoffMillerAZ, I was able to get this working by doing the following:

• Provision a new EKS cluster using aws eks create-cluster and specifying the correct profile using --profile blah
• Run aws eks update-kubeconfig --name clustername --profile blah to create a shell ~/.kube/config
• Modify ~/.kube/config and add:

      env:
      - name: AWS_PROFILE
        value: "blah"

After this, I was able to run `kubectl get svc':

➜  kubectl get svc
Assume Role MFA token code: 123456
NAME         TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)   AGE
kubernetes   ClusterIP   172.20.0.1   <none>        443/TCP   13m

This is mostly documented here: https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html.

I would have to agree with others that the documentation (especially the "Getting Started" documentation) is not clear. Using STS with assumed roles is a very common pattern for many customers, and this should be addressed more eloquently.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[falfaro]

Thanks for the investigation. Do I understand that this is a problem with EKS and not with BKPR?

[falfaro]

Oh, and BTW, another alternative that I believe can work is exporting the environment variable AWS_PROFILE instead of using the --profile flags and modifying the Kube configuration.

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[lliknart]

We had the same issue with EKS cluster.

We use Jenkins to deploy. After many errors and tries, we verify the token with the following command
aws-iam-authenticator verify -i cluster_name -t token_id
it appears that the token has been generated for an IAM user than the expected role

We check the env variables. It appears that the following environment variables are set:
AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
Those are injected by Jenkins (with the aws-credentials plugin).

So, the AWS profile was good, but it used env variables to generate the token.

By unsetting this env variables the token is well generated and we can now request the EKS cluster

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[SarasaGunawardhana]

I could not solve this issue by creating a cluster on AWS EKS GUI. I tried eksctl to create cluster and it worked.