K9s: Unable to switch context

[raoulus]

Describe the bug
I've access to different EKS instances and want to switch the context in k9s but there are authentication problems according to the logs. AWS Okta is used for authentication.

To Reproduce
Steps to reproduce the behavior:

1. Open k9s
2. Go to :ctx
3. Select a different EKS instance
4. See error, k9s exits after 15 tries

Expected behavior
Context is switched and I can continue using k9s.

Screenshots

Versions (please complete the following information):

• OS: OSX 10.15.4
• K9s 0.19.6 [53542]
• K8s v1.15.11-eks-af3caf

Additional context
Happy to provide more details.

Logs

5:50PM INF 🐶 K9s starting up...
5:50PM INF ✅ Kubernetes connectivity
5:50PM INF No context specific skin file found -- /Users/me/.k9s/eks-eu-central-1-xxx-dev_skin.yml
5:50PM INF No skin file found -- /Users/me/.k9s/skin.yml. Loading stock skins.
5:50PM INF No namespace specified using cluster default namespace
5:50PM ERR List metrics failed error="nodes.metrics.k8s.io is forbidden: User \"XXX\" cannot list resource \"nodes\" in API group \"metrics.k8s.io\" at the cluster scope"
5:50PM WRN Fail CRDs load error="`list access denied for user on \"\":apiextensions.k8s.io/v1beta1/customresourcedefinitions"
5:50PM ERR Custom view load failed /Users/me/.k9s/views.yml error="open /Users/me/.k9s/views.yml: no such file or directory"
5:50PM ERR CustomView watcher failed error="lstat /Users/me/.k9s/views.yml: no such file or directory"
5:50PM WRN Fail CRDs load error="[list watch] access denied on resource \"\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""
5:50PM WRN   Dial Failed! error=Unauthorized
5:50PM WRN Fail CRDs load error=Unauthorized
5:50PM INF No context specific skin file found -- /Users/me/.k9s/eks-eu-central-1-xxx-int_skin.yml
5:50PM INF No skin file found -- /Users/me/.k9s/skin.yml. Loading stock skins.
5:50PM ERR Huh? `po` command not found
5:50PM WRN   Dial Failed! error=Unauthorized
5:50PM ERR Custom view load failed /Users/me/.k9s/views.yml error="open /Users/me/.k9s/views.yml: no such file or directory"
5:50PM ERR CustomView watcher failed error="lstat /Users/me/.k9s/views.yml: no such file or directory"
5:50PM ERR K9s can't connect to cluster error="the server has asked for the client to provide credentials"
5:50PM WRN Conn check failed (1/15)
5:50PM ERR Goto resource for "pod" failed error="Huh? `pod` command not found"
5:50PM ERR Huh? `pod` command not found
5:50PM ERR K9s can't connect to cluster error="the server has asked for the client to provide credentials"
5:50PM WRN Conn check failed (2/15)

[derailed]

@raoulus It looks like you creds are indeed incorrect for this cluster. Might be a kubeconfig issue with certs either being incorrect or not loading correctly. Could you try kubectl get po --context my_eks_context does this work? If so please send the debug k9s logs using this command k9s --context my_eks_context -l debug. Also are your kubeconfig for the clusters in a single kube config file or multiple files?

[radeklat]

I'm getting the same error message. In my case, the issue appears only on start of k9s if I try to switch to one of two contexts (only that one, not the other). And it does not appear if I wait for couple of seconds until information about K8s Rev: appears in the header.

To reproduce

1. Start of k9s, K8s Rev: n/a

3:13PM INF 🐶 K9s starting up...
3:13PM ERR failed to connect to cluster error="No connection to cached dial"
3:13PM INF No context specific skin file found -- /home/***/.k9s/***-production_skin.yml
3:13PM INF No skin file found -- /home/***/.k9s/skin.yml. Loading stock skins.
3:13PM INF No namespace specified using cluster default namespace
3:13PM ERR no API server connection
3:13PM WRN Fail CRDs load error="no API server connection"
3:13PM ERR Custom view load failed /home/***/.k9s/views.yml error="open /home/***/.k9s/views.yml: no such file or directory"
3:13PM ERR CustomView watcher failed error="no such file or directory"

1. Switch context immediately:

3:13PM WRN No namespace specified in context. Using K9s config
3:13PM ERR no API server connection
3:13PM WRN Fail CRDs load error="no API server connection"
3:13PM INF No context specific skin file found -- /home/rlat/.k9s/cookpad-global-1-production_skin.yml
3:13PM INF No skin file found -- /home/rlat/.k9s/skin.yml. Loading stock skins.
3:13PM ERR Custom view load failed /home/rlat/.k9s/views.yml error="open /home/rlat/.k9s/views.yml: no such file or directory"
3:13PM ERR CustomView watcher failed error="no such file or directory"

2.2 Wait for K8s Rev: to get populated, then switch context:

3:18PM WRN No namespace specified in context. Using K9s config
3:18PM INF No context specific skin file found -- /home/***/.k9s/***-production_skin.yml
3:18PM INF No skin file found -- /home/***/.k9s/skin.yml. Loading stock skins.
3:18PM ERR Custom view load failed /home/***/.k9s/views.yml error="open /home/***/.k9s/views.yml: no such file or directory"
3:18PM ERR CustomView watcher failed error="no such file or directory"

Versions (please complete the following information):
OS: Linux Mint 19.3 Tricia
K9s 0.20.1 [21576]
K8s v1.15.11-eks-af3caf

[derailed]

@raoulus Fixed v0.20.4

[raoulus]

Hi @derailed,

thanks for taking care! I updated to K9s Rev: 0.20.4 [71564] but when I now switch contexts the terminal freezes and I can't do anything anymore, not even ctrl + c.

These are the logs

9:38AM INF 🐶 K9s starting up...
9:38AM INF ✅ Kubernetes connectivity
9:38AM INF No context specific skin file found -- /Users/xxx/.k9s/eks-xxx_skin.yml
9:38AM INF No skin file found -- /Users/xxx/.k9s/skin.yml. Loading stock skins.
9:38AM INF No namespace specified using cluster default namespace
9:38AM ERR List metrics failed error="nodes.metrics.k8s.io is forbidden: User \"OktaXXX\" cannot list resource \"nodes\" in API group \"metrics.k8s.io\" at the cluster scope"
9:38AM WRN Fail CRDs load error="`list access denied for user on \"\":apiextensions.k8s.io/v1beta1/customresourcedefinitions"
9:38AM ERR Custom view load failed /Users/xxx/.k9s/views.yml error="open /Users/xxx/.k9s/views.yml: no such file or directory"
9:38AM ERR CustomView watcher failed error="lstat /Users/xxx/.k9s/views.yml: no such file or directory"
9:38AM WRN Fail CRDs load error="[list watch] access denied on resource \"-\":\"apiextensions.k8s.io/v1beta1/customresourcedefinitions\""

[sadedil]

I've faced same issue with multiple kubeconfig then understand underlying issue. My main problem was, same user name was exists in both config files, but one token of them was stale.