Dashboard: Secrets visible in UI when referenced via environment variable, v2

[zerkms]

This is a followup for the https://github.com/kubernetes/dashboard/issues/2018 (and https://github.com/kubernetes/dashboard/issues/1993 ?) which either regressed or never worked as intended.

Environment

Installation method: kubeadm
Kubernetes version: 1.15.1
Dashboard version: v1.10.1
Operating system: ubuntu
Node.js version ('node --version' output): N/A
Go version ('go version' output): N/A

Steps to reproduce

When a pod created with

          envFrom:
            - secretRef:
                name: secretName

the corresponding environment variables with secrets are still visible on the pod's details page.

Observed result

It still exposes secrets even though the account does not have access to secrets.

Expected result

Those should not be exposed if the account does not have access to the secrets.

Comments

[floreks]

The real issue is that API server allows access to this. It should be handled by API, not by UI.

[zerkms]

So, should this then be closed as "works as designed" then?

[floreks]

@zerkms Unfortunately this works as intended. kubectl works exactly the same and allows to see the value of variable taken from secret. If you feel that this is a security issue then you should report it to the core team. If API will block access to it then it will not be displayed.

See my listing with kubectl configured as pod-reader only.

10:21 $ kubectl get pod
NAME            READY   STATUS    RESTARTS   AGE
env-variables   1/1     Running   0          16m
10:24 $ kubectl get secret
Error from server (Forbidden): secrets is forbidden: User "system:serviceaccount:kube-system:pod-reader" cannot list resource "secrets" in API group "" in the namespace "default"
10:24 $ kubectl describe pod
Name:         env-variables
Namespace:    default
Priority:     0
Node:         kind-control-plane/172.17.0.2
Start Time:   Tue, 30 Jul 2019 10:07:20 +0200
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"v1","kind":"Pod","metadata":{"annotations":{},"name":"env-variables","namespace":"default"},"spec":{"containers":[{"args":[...
Status:       Running
IP:           10.244.0.4
Containers:
  test-container:
    Container ID:  containerd://2d7a29241f043b2e7491704bf5cb0323dec2015992696b8c51f3d2d49e6879fd
    Image:         k8s.gcr.io/busybox
    Image ID:      sha256:36a4dca0fe6fb2a5133dc11a6c8907a97aea122613fa3e98be033959a0821a1f
    Port:          <none>
    Host Port:     <none>
    Command:
      sh
      -c
    Args:
      while true; do echo -en '\n'; printenv MY_POD_NAME; printenv MY_MEM_REQUEST MY_MEM_LIMIT; printenv MY_CONST_VALUE; printenv MY_SECRET_VALUE; sleep 10; done;
    State:          Running
      Started:      Tue, 30 Jul 2019 10:07:23 +0200
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     250m
      memory:  64Mi
    Requests:
      cpu:     125m
      memory:  32Mi
    Environment:
      MY_POD_NAME:      env-variables (v1:metadata.name)
      MY_MEM_REQUEST:   33554432 (requests.memory)
      MY_MEM_LIMIT:     67108864 (limits.memory)
      MY_CONST_VALUE:   test-const-value
      MY_SECRET_VALUE:  <set to the key 'username' in secret 'secret-env'>     Optional: false
      MY_CM_VALUE:      <set to the key 'test' of config map 'configmap-env'>  Optional: false
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-468fw (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             True 
  ContainersReady   True 
  PodScheduled      True 
Volumes:
  default-token-468fw:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-468fw
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:          <none>

[zerkms]

That's perfectly understandable indeed. Thanks a lot for your time!

[zerkms]

@floreks what API method is used to extract those secret in the dashboard? (I tried to find it but I'm not well familiar with the dashboard codebase).

I need that to explain why what I requested from the kubernetes API team makes sense (see the referred issue link above)

[zerkms]

UPD: I found it pod, err := client.CoreV1().Pods(namespace).Get(name, metaV1.GetOptions{})

If I understand it correctly - it's this place, but I'm not sure if it's possible to obtain the same amount of information (including plain text secrets) using kubectl.

If you have a minute, could you please add notes to the kubernetes api ticket?

[zerkms]

https://github.com/kubernetes/kubernetes/issues/80789#issuecomment-517060306:

The dashboard also has its own API credentials, though I didn't think those were used by the dashboard to fetch data exposed to users. If they are in this case, that is also an issue for the dashboard component

Let's move this issue to that component and get more details about the dashboard configuration and permissions.

@floreks sorry for bothering, but I think we need a more detailed input to know where exactly secrets leak from.

[liggitt]

Knowing the following would help:

• how is the dashboard configured? (do you have the deployment manifest you are using)?
• what version of the dashboard are you are using?
• do you have the apiserver audit log available?

[zerkms]

how is the dashboard configured? (do you have the deployment manifest you are using)?
the default unmodified one

https://raw.githubusercontent.com/kubernetes/dashboard/v1.10.1/src/deploy/recommended/kubernetes-dashboard.yaml

what version of the dashboard are you are using?

the latest available stable: v1.10.1

do you have the apiserver audit log available?

not at the moment, but if it would be absolutely necessary I may enable it (I'm sure though the problem can be reproduced in any cluster as mine is not unique in any way). Please give me some time to obtain those as at this very moment I'm in rush to finish something else, sorry! :-)

[liggitt]

For the dashboard team:

• what API request (get or list) is used to fetch secret data for display on that page
• is the logged in user or the dashboard client used to fetch that info?

cc @jeefy

[jeefy]

Heya!

When viewing a pod, the Dashboard backend gathers all details of a pod. The relevant line we're looking for is:

https://github.com/kubernetes/dashboard/blob/master/src/app/backend/resource/pod/detail.go#L100

That invokes this: https://github.com/kubernetes/dashboard/blob/6d184730c68d52b74c402f6c2324c4219c5a146e/src/app/backend/resource/common/resourcechannels.go#L659

Several lines down you can see it calling list, err := client.CoreV1().Secrets(nsQuery.ToRequestParam()).List(api.ListEverything) and that then goes down a rabbit hole of collating and presenting container-level data.

With Dashboard v1.10.1 (and earlier AFAIK), the RBAC for the deployment should be heavily restricted (by default) and any API calls are done as the current-logged-in-user to the Dashboard.

[zerkms]

After some more research I found that secrets exposed via environment variables started being shown as long as I allow list for secret via this RBAC:

  - apiGroups: [""]
    resources:
      - secrets
    verbs:
      - list

To summarise: as long as I bind just a built-in view role to a service account: in the dashboard I can see the pod's details but not secrets (expose via env)

But as long as I additionally attach that secrets list access policy: those start being visible.

[liggitt]

list allows fetching the content of all instances of a resource. it exposes more information than get

[zerkms]

O M G

That is totally not obvious from the https://kubernetes.io/docs/reference/access-authn-authz/authorization/#determine-the-request-verb

In that case - my apologies for taking all your time.

Does it mean there is no way to only allow an account to list secret names, but not expose their contents?

[liggitt]

That is totally not obvious from the https://kubernetes.io/docs/reference/access-authn-authz/authorization/#determine-the-request-verb

apologies, will take a look at making that clearer

Does it mean there is no way to only allow an account to list secret names, but not expose their contents?

Correct

[liggitt]

That is totally not obvious from the https://kubernetes.io/docs/reference/access-authn-authz/authorization/#determine-the-request-verb

apologies, will take a look at making that clearer

doc PR open at https://github.com/kubernetes/website/pull/15623