K9s: Pods not visible

[awwitecki]

Hello,
this problem is there for couple of months so sorry for a delayed call-out. I'm able to see all the k8s resources with k9s except pods for which the view is always empty albeit I've got a lot of pods there (visible with kubectl ofc).
Logs suggest the lack of access. For sure it's worth mentioning that I'm using multi-tenant cluster with the access limited to my namespace only (I'm able to list all of ns on k9s btw).

I'm open to provide any other non-confidential details if required.

Below I attach sample logs:

4:54PM INF 🐶 K9s starting up...
4:54PM INF ✅ Kubernetes connectivity
4:54PM INF No skin file found. Loading stock skins.
4:54PM INF No benchmark config file found, using defaults. error="open ...: no such file or directory"
Log file created at: 2019/07/21 16:54:08
Running on machine: ...
Binary: Built with gc go1.12.6 for darwin/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
Log file created at: 2019/07/21 16:54:08
Running on machine: ...
Binary: Built with gc go1.12.6 for darwin/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
Log file created at: 2019/07/21 16:54:08
Running on machine: ...
Binary: Built with gc go1.12.6 for darwin/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0721 16:54:08.643961   78121 reflector.go:134] pkg/mod/k8s.io/[email protected]+incompatible/tools/cache/reflector.go:95: Failed to list *v1.Pod: pods is forbidden: User "..." cannot list pods at the cluster scope
E0721 16:54:08.643961   78121 reflector.go:134] pkg/mod/k8s.io/[email protected]+incompatible/tools/cache/reflector.go:95: Failed to list *v1beta1.PodMetrics: pods.metrics.k8s.io is forbidden: User "..." cannot list pods.metrics.k8s.io at the cluster scope
E0721 16:54:08.643984   78121 reflector.go:134] pkg/mod/k8s.io/[email protected]+incompatible/tools/cache/reflector.go:95: Failed to list *v1beta1.NodeMetrics: nodes.metrics.k8s.io is forbidden: User "..." cannot list nodes.metrics.k8s.io at the cluster scope
E0721 16:54:08.644068   78121 reflector.go:134] pkg/mod/k8s.io/[email protected]+incompatible/tools/cache/reflector.go:95: Failed to list *v1.Node: nodes is forbidden: User "..." cannot list nodes at the cluster scope
E0721 16:54:09.796635   78121 reflector.go:134] pkg/mod/k8s.io/[email protected]+incompatible/tools/cache/reflector.go:95: Failed to list *v1beta1.NodeMetrics: nodes.metrics.k8s.io is forbidden: User "..." cannot list nodes.metrics.k8s.io at the cluster scope
E0721 16:54:09.796659   78121 reflector.go:134] pkg/mod/k8s.io/[email protected]+incompatible/tools/cache/reflector.go:95: Failed to list *v1beta1.PodMetrics: pods.metrics.k8s.io is forbidden: User "..." cannot list pods.metrics.k8s.io at the cluster scope

[derailed]

@awwitecki Thank you so much for the report! It looks like your user can list namespaces hence you get to see the namespaces in K9s. If you ran the following:

 k9s -n my_ns 

Where my_ns is your confined namespace, do you now see your pods?

[awwitecki]

This is actually the way I run k9s, i.e.

k9s --context ... --namespace ...

[derailed]

@awwitecki Thank you for your reply! So when you start K9s unrestricted, it will attempt to list pods in all namespaces. If the kubeconfig specifies a namespace that namespace will be used. What would you expect the behavior to be when launching K9s on RBAC enabled clusters?

[awwitecki]

Sorry for late response. I'm not completely sure what did you mean by that. Basically I always start k9s with context and namespace specified. I have all the rights to list, describe, etc. my pods there but despite this fact my pods are not visible with k9s while simple kubectl --context ... -n ... get pods shows them all. Or maybe I've configured my k9s wrong? Once again, all the other resources are visible with k9s without any issues.

[derailed]

@awwitecki Thank you for the follow up! Sorry I think I have misunderstood your question. I thought you could see the pods when using --context and -n. So I think this is indeed a bug as your user can actually list namespaces at the cluster level but can't list pods at the cluster level. I will fix this on the next drop. In the short term, I think if you further confine your user to only get but not list all namespaces then you should see your pods given these cli options.

[derailed]

@awwitecki Are you still having this issue or are things better in newer K9s revisions?

[awwitecki]

Unfortunately not :(
My revision:

Version:   0.8.1
Commit:    f7bd5301a8fff52d21ada6a99e7e1cd51ec25384
Date:      2019-08-13T04:55:09Z

Logs:

8:51AM INF 🐶 K9s starting up...
8:51AM INF ✅ Kubernetes connectivity
8:51AM INF No skin file found. Loading stock skins.
8:51AM INF No benchmark config file found, using defaults. error="open (...)/.k9s/bench-eu-west-1a-nonprod.yml: no such file or directory"
Log file created at: 2019/08/13 08:51:20
Running on machine: (...)
Binary: Built with gc go1.12.6 for darwin/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
E0813 08:51:20.755651   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Pod: pods is forbidden: User "(...)" cannot list pods at the cluster scope
E0813 08:51:20.755793   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1beta1.NodeMetrics: nodes.metrics.k8s.io is forbidden: User "(...)" cannot list nodes.metrics.k8s.io at the cluster scope
E0813 08:51:20.756043   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1beta1.PodMetrics: pods.metrics.k8s.io is forbidden: User "(...)" cannot list pods.metrics.k8s.io at the cluster scope
E0813 08:51:20.756296   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Node: nodes is forbidden: User "(...)" cannot list nodes at the cluster scope
E0813 08:51:21.846025   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1beta1.PodMetrics: pods.metrics.k8s.io is forbidden: User "(...)" cannot list pods.metrics.k8s.io at the cluster scope
E0813 08:51:21.846058   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Pod: pods is forbidden: User "(...)" cannot list pods at the cluster scope
E0813 08:51:21.846718   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Node: nodes is forbidden: User "(...)" cannot list nodes at the cluster scope
E0813 08:51:21.846864   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1beta1.NodeMetrics: nodes.metrics.k8s.io is forbidden: User "(...)" cannot list nodes.metrics.k8s.io at the cluster scope
E0813 08:51:22.909916   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1beta1.PodMetrics: pods.metrics.k8s.io is forbidden: User "(...)" cannot list pods.metrics.k8s.io at the cluster scope
E0813 08:51:22.967925   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1beta1.NodeMetrics: nodes.metrics.k8s.io is forbidden: User "(...)" cannot list nodes.metrics.k8s.io at the cluster scope
E0813 08:51:22.967951   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Node: nodes is forbidden: User "(...)" cannot list nodes at the cluster scope
E0813 08:51:22.968025   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Pod: pods is forbidden: User "(...)" cannot list pods at the cluster scope
E0813 08:51:23.975392   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1beta1.PodMetrics: pods.metrics.k8s.io is forbidden: User "(...)" cannot list pods.metrics.k8s.io at the cluster scope
E0813 08:51:24.029423   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Pod: pods is forbidden: User "(...)" cannot list pods at the cluster scope
E0813 08:51:24.029437   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1beta1.NodeMetrics: nodes.metrics.k8s.io is forbidden: User "(...)" cannot list nodes.metrics.k8s.io at the cluster scope
E0813 08:51:24.029814   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Node: nodes is forbidden: User "(...)" cannot list nodes at the cluster scope
E0813 08:51:25.035543   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1beta1.PodMetrics: pods.metrics.k8s.io is forbidden: User "(...)" cannot list pods.metrics.k8s.io at the cluster scope
E0813 08:51:25.090388   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1beta1.NodeMetrics: nodes.metrics.k8s.io is forbidden: User "(...)" cannot list nodes.metrics.k8s.io at the cluster scope
E0813 08:51:25.090395   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Node: nodes is forbidden: User "(...)" cannot list nodes at the cluster scope
E0813 08:51:25.091543   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Pod: pods is forbidden: User "(...)" cannot list pods at the cluster scope
E0813 08:51:26.146233   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1beta1.PodMetrics: pods.metrics.k8s.io is forbidden: User "(...)" cannot list pods.metrics.k8s.io at the cluster scope
E0813 08:51:26.202052   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Pod: pods is forbidden: User "(...)" cannot list pods at the cluster scope
E0813 08:51:26.202165   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Node: nodes is forbidden: User "(...)" cannot list nodes at the cluster scope
E0813 08:51:26.202238   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1beta1.NodeMetrics: nodes.metrics.k8s.io is forbidden: User "(...)" cannot list nodes.metrics.k8s.io at the cluster scope
E0813 08:51:27.219135   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1beta1.PodMetrics: pods.metrics.k8s.io is forbidden: User "(...)" cannot list pods.metrics.k8s.io at the cluster scope
E0813 08:51:27.278842   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Pod: pods is forbidden: User "(...)" cannot list pods at the cluster scope
E0813 08:51:27.278883   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1beta1.NodeMetrics: nodes.metrics.k8s.io is forbidden: User "(...)" cannot list nodes.metrics.k8s.io at the cluster scope
E0813 08:51:27.278944   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Node: nodes is forbidden: User "(...)" cannot list nodes at the cluster scope
E0813 08:51:28.279559   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1beta1.PodMetrics: pods.metrics.k8s.io is forbidden: User "(...)" cannot list pods.metrics.k8s.io at the cluster scope
E0813 08:51:28.390039   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Node: nodes is forbidden: User "(...)" cannot list nodes at the cluster scope
E0813 08:51:28.390007   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1beta1.NodeMetrics: nodes.metrics.k8s.io is forbidden: User "(...)" cannot list nodes.metrics.k8s.io at the cluster scope
E0813 08:51:28.390100   47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Pod: pods is forbidden: User "(...)" cannot list pods at the cluster scope

Keen to provide any additional information if required

[derailed]

@awwitecki Thank you for the reply!! Something is off here... You said you are starting K9s as follows correct?

k9s --context fred -n blee

I find it odd that we see this message in the logs ie
"E0813 08:51:21.846058 47872 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Pod: pods is forbidden: User "(...)" cannot list pods at the cluster scope
"
As this would indicate that the namespace is currently not set and K9s is trying to find pods in all namespaces and not blee per the command above.
Could you try moving you ~/.k9s/config.yml to ~/.k9s/config1.yml.

Also could you perhaps email me your rbac rules for this user his: clusterrole, clusterrolebinding,role and rolebinding?

Thank you!!

[antoniaklja]

hey, @derailed @awwitecki I'm wondering why default namespace is used for Namespace RBAC scope - what if someone doesn't have access to the default ns? will cluster-wide approach will be used as a fallback?

I think when k9s is namespace-scoped e.g. k9s --context fred -n blee it should be possible to use roles from that namespace. Does it make sense for you?

[awwitecki]

Hi @derailed ,
I always run k9s with context and namespace explicitly specified (like in you sample).
Tried with deleted config many times but with no luck.
WRT RBAC rules, due to the nature of cluster we use I don't have proper rights to attain them on my own. Will try to do this though.

[derailed]

@antoniaklja Thank you! I should probably beef the docs some. This is a sample rbac rule for K9s for an example namespace aka default. You can use any namespace you want there assuming K9s will be started with that namespace. Does this make sense?

@awwitecki Thank you! I'll keep digging, I've tried to repro this and so far everything is working as I would expect but perhaps I've missed something. It would help so see how your user RBAC rules look like so I could repro it on a cluster. My guess right now is you don't have rbac watch access on the namespace...

[awwitecki]

My Role and RoleBinding looks like this:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: k9s
  namespace: (...)
rules:
  - apiGroups:
      - ""
      - "metrics.k8s.io"
    resources:
      - pods
    verbs:
      - get
      - list
      - watch

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: k9s
  namespace: (...)
subjects:
  - kind: User
    name: (...)
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: k9s
  apiGroup: rbac.authorization.k8s.io

I'm not allowed to embrace more apiGroups and resources with this rule in the cluster but it seems to be sufficient.
Sadly, I'm also unable to obtain ClusterRule nor ClusterRuleBinding for you due to security measures.

On the other hand, looking on logs once again it seems weird that k9s tries to invoke cluster scope operation whilst in my view should do this namespace scope.

[derailed]

@awwitecki Thank you for sending these in! I agree it's strange. I have used your role and rb and can't seem to repro the logs you've send to me notably the issue about listing pods at the cluster level. Given this rbac rule you should be able to see pods. You will have no listing for nodes and node metrics unless your cluster role allows it. Here is what I did to try to repro. Let's see if you can see a delta from your current setup??

NOTE: I will add some more debug info in the next drop so we can better see what's happening in your cluster.

Thank you for you patience and helping me figure out what's happening with K9s!!

RBAC

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: k9s
  namespace: fred
rules:
  - apiGroups:
      - ""
      - "metrics.k8s.io"
    resources:
      - pods
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: k9s
  namespace: fred
subjects:
  - kind: User
    name: fred
    apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: Role
  name: k9s
  apiGroup: rbac.authorization.k8s.io

Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  namespace: fred
spec:
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
        - name: nginx
          image: k8s.gcr.io/nginx-slim:0.8
          ports:
            - containerPort: 80
          resources:
            limits:
              cpu: 200m
              memory: 20Mi
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
  namespace: fred
spec:
  type: ClusterIP
  selector:
    app: nginx
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 80

Commands

kubectl create ns fred
kubectl apply -f rbac.yml -f nginx.yml
# Launch K9s 0.8.2 in namespace fred as user fred -> Can see the pods and pod metrics.
k9s -n fred --as fred 

[pmaccacaro]

Hello,

We have the same issue. K9s trying to list/get cluster's wide pods also when running it with

k9s -n namespace --context context

Everything is visible (deployments, secrets, ingresses, etc) but pods

here our namespaced role's rules

rules:
  - apiGroups:
      - "*"
    resources:
      - "*"
    verbs:
      - get
      - list
      - watch

[derailed]

@pmaccacaro Thank you for the extra info. I'll take another pass and see what's up with that. If convenient could you send a snippet of the K9s logs when viewing pods in your cluster? You can email me directly if you don't want to post them here. Thank you!

[derailed]

@pmaccacaro @awwitecki I think v0.9.0 should fix this. If not please open with a more detail report so we can track this down. Thank you!!

[awwitecki]

It worked! Thank you so much for help here :)