Aws-iam-authenticator: EKS Connections using role switching with MFA ask everytime for MFA token

[daraacca]

I wish to connect to an EKS Instance using kubectl with the authenticator. To connect to this instance I want to assume a role in the necessary AWS Account where EKS Lives. The config map of the EKS Instance has also been updated to specify the same role -

I have the following kube config

apiVersion: v1
clusters:
- cluster:
    server: https://xxxxxxx.sk1.us-east-1.eks.amazonaws.com
    certificate-authority-data: xxxxxxxxxxxxxxx
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: aws
  name: aws
current-context: aws
kind: Config
preferences: {}
users:
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: aws-iam-authenticator
      env:
      - name: "AWS_PROFILE"
        value: "playground-power-user"
      args:
        - "token"
        - "-i"
        - "test-cluster"

my AWS config file is as follows

[profile playground-power-user]
region = us-east-1
source_profile = main
role_arn = arn:aws:iam::12345678:role/power-user
mfa_serial = arn:aws:iam::112233445566:mfa/[email protected]

my aws-auth-cm.yaml config map is as follows

apiVersion: v1
kind: ConfigMap
metadata:
  name: aws-auth
  namespace: kube-system
data:
  mapRoles: |
    - rolearn: arn:aws:iam::12345678:role/eks-cluster-role
      username: system:node:{{EC2PrivateDNSName}}
      groups:
        - system:bootstrappers
        - system:nodes
    - rolearn: arn:aws:iam::12345678:role/power-user
      username: admin:{{AccountID}}:{{SessionName}}
      groups:
        - system:masters

This is all good. When i connect to EKS, I am asked for my MFA token, which is accepted and command is executed

kubectl get clusterrolebindings
Assume Role MFA token code: 275230
NAME AGE
aws-node 7d
cluster-admin 7d
eks:kube-proxy 7d
eks:node-bootstrapper 7d

HOWEVER, for every kubectl command, I have to give the MFA token again. This is obviously unworkable. Any ideas to avoid this???

[jukkafor]

I'm facing this issue as well and it is hugely annoying. While waiting for caching to be implemented into aws-iam-authenticator I made a small Bash wrapper script for caching the output of this command.

[daraacca]

Hi Jukka, that sounds really interesting. I found an example wrapper in one of the other issues here. That should keep us going until the Pull Request for Caching is merged. Cheers!

[3r1co]

Hi @jukkafor , do you mind sharing this bash wrapper? I did not find the wrapper mentioned by @daraacca here :( This issue is really annoying for us

[daraacca]

Hi @3r1co - i found the script in a separate Issue here. I repost it below 👍

``#!/usr/bin/env bash CACHE_FILE=${HOME}/.kube/heptio-authenticator-${AWS_PROFILE}.cache MAXTIME=800 if [ -f $CACHE_FILE ]; then TS_DB=$(stat -t "%s" ${CACHE_FILE} | cut -d' ' -f10 | tr -d '"') AGE=$((date +%s` - $TS_DB ))
if [[ $AGE -le $MAXTIME ]]; then
cat ${CACHE_FILE}
else
aws-iam-authenticator "$@" | tee $CACHE_FILE
fi
else
aws-iam-authenticator "$@" | tee $CACHE_FILE
fi

[Martin-Andersen]

@daraacca how do you use the script?

[daraacca]

Hi @Martin-Andersen - Just name the script something like "aws-iam-authenticator-wrapper", and then reference this instead of "aws-iam-authenticator" in the kube config file.

[bgora]

@daraacca

when I use the wrapper script in my kubeconfig I get

No resources found.
Unable to connect to the server: getting credentials: exec: fork/exec /Users/bgirsch/bin/aws-iam-authenticator-wrapper: exec format error

kubectl 1.11.6

[mazzy89]

I can say that the wrapper for a while it worked for me. then it stopped to work. Maybe because i've updated the kubectl via Homebrew

[bgora]

my problem was solved by setting
$AWS_ACCESS_KEY_ID $AWS_SECRET_ACCESS_KEY $AWS_SESSION_TOKEN
if you execute kubectl in the same bash session, it will not ask constantly for MFA token

[tbekas]

@bgora @mazzy89 add this to the beginning of the wrapper script. Works for me on kubectl v1.13.1

#!/bin/bash

[jtstrohl]

Workaround suggested above is helpful, but still a pain, especially if you have users with workstations on multiple platforms (windows, mac, linux, etc.). We are left having to access our clusters using individual IAM user credentials instead of roles, which means we have to manage individual user access to each cluster. This is also quickly becoming unsustainable.

This is THE tool for authenticating to EKS via IAM permissions and MFA is AWS best practices, so if we want to use EKS the right way, we are forced into this issue.

Is there an ETA on a fix for this issue?

[daraacca]

Hi all - i found a better way than using the wrapper workaround.. I have been using AWS-VAULT ( which is available for Linux, Max & Windows ) as a credentials store.. This stores the temp credentials which can then be re-used by the authenticator.. You can find it here https://github.com/99designs/aws-vault

[jtstrohl]

Thanks @daraacca for posting an update with your improved workaround! I appreciate it. We will take a look.

[mazzy89]

For me it does not work that workaround with aws-vault

- name: mycluster--aws-auth
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
        - token
        - -i
        - mycluster
      command: aws-vault exec kubernetes-admin -- aws-iam-authenticator

when I try to run a command like this:

Unable to connect to the server: getting token: exec: exec: "aws-vault exec kubernetes-admin -- aws-iam-authenticator": executable file not found in $PATH

[daraacca]

1. make sure your .aws/config & credentials files are properly configured.
2. add your default profile to aws-vault( where you have your aws access keys )
   -> aws-vault add "default-profile"
3. Execute a connection to the role switching profle
   -> aws-vault exec "switching-profile" --
   This should initiate the connection and save the temporary credentials ( check with aws-vault list )
4. Execute kubectl command ( no need to use aws-vault directly with exec as the correct environment variables are already set ).
   -> kubectl get nodes

My kube config file looks like this

users:
- name: aws
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      command: aws-iam-authenticator
      args:
        - token
        - -i
        - staging-cluster

[mazzy89]

@daraacca In that case, make sense and it works. Thank you

[robatwave]

I've been using https://github.com/broamski/aws-mfa to work around this limitation.

[genebean]

Looks like #193 may take care of this soon :)

[christopherhein]

@genebean should have taken care of this, please give the 0.4.0 release a try and use --cache to cache the token. Closing for now, reopen if this is still an issue.

/close

[k8s-ci-robot]

@christopherhein: Closing this issue.

In response to this:

@genebean should have taken care of this, please give the 0.4.0 release a try and use --cache to cache the token. Closing for now, reopen if this is still an issue.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.