Aws-iam-authenticator: How do you troubleshoot error: You must be logged in to the server (Unauthorized)?

[holyjak]

Hello, I would very much appreciate tips on how to troubleshoot

How do you troubleshoot error: You must be logged in to the server (Unauthorized)

Facts

1. kubectl get svc fails with error: You must be logged in to the server (Unauthorized)
2. aws-iam-authenticator token -i mycluster returns a token so the communication with IAM seems to be OK; the error is somewhere later in the process
3. Other users, with other IAM roles, can access the cluster without a problem so it is something with this particular (assumed) role. The role is "readonly" in AWS but I guess that should be good enough for connecting to the cluster.
4. I have added the role to kubernetes' aws-auth and applied it (see below); though it would be awesome if the tool provided a more helpful error message if I fail to do so (or at least a way to check that it works as intended)

# aws-auth.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::123456:role/admin
      username: system:node:{{SessionName}}
      groups:
      - system:masters
    - rolearn: arn:aws:iam::123456:role/readonly
      username: system:node:{{SessionName}}
      groups:
      - db-user
      - system:masters

  mapUsers: |

  mapAccounts: |
      - "555555"

Applied with kubectl apply -f aws-auth.yaml.

NOTE: We use a "jump account" for all users and only after authenticating with it, we assume a role in the target account containing kubernetes. This works just fine for the "admin" role but, as described, fails for the "readonly" role.

kubeconfig:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: redacted...
    server: https://228DD8BE5749AB4552832A0869A1921B.yl4.eu-west-1.eks.amazonaws.com
  name: mycluster
contexts:
- context:
    cluster: mycluster
    namespace: common
    user: mycluster
  name: mycluster
current-context: mycluster
kind: Config
preferences: {}
users:
- name: mycluster
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - token
      - -i
      - mycluster
      command: aws-iam-authenticator
      env: null

User info:

 aws sts get-caller-identity
{
    "UserId": "AAAAA:john.doe@777777",
    "Account": "123456",
    "Arn": "arn:aws:sts::123456:assumed-role/readonly/john.doe@7777773"
}

Thank you!

[holyjak]

My error solved, question remains: how to troubleshoot?

Writing this description helped me realize what the problem was: my aws-auth.yaml was wrong, the account numbers there were incorrect (i.e. they _should_ be 123456 as shown above but were different - a copy+paste error).

However the question remains - how to troubleshoot such problems? If the login/auth fails because of missing role, why doesn't an attempt to use kubectl tell me so? Or at least aws-iam-authenticator should have the functionality to verify this, so that I could run something like

$ aws-iam-authenticator check-login -i mycluster
FAILED: Your role is "arn:aws:sts::123456:assumed-role/readonly/john.doe@7777773" but t
he aws-auth for the cluster mycluster does not map this role to any groups. You won't be 
able to use kubectl until the role is mapped there.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[saniic]

INFO! Testing kubectl connection....
error: You must be logged in to the server (Unauthorized)
ERROR! Test Failed, AWS role might not be recongized by cluster