Linkerd2: Use ServiceAccountToken volumes

[grampelberg]

What problem are you trying to solve?

Currently, service account tokens are used to validate identity for pods. These are passed as part of the CSR to identity. In situations where the token is not mounted automatically, a pod's identity cannot be verified and identity must be disabled. This token also is needlessly over permissive and shouldn't be shipped around.

How should the problem be solved?

A new type of volume - ServiceAccountToken reached beta in k8s 1.12. This is mounted on a per-container basis, can have a specific expiration and allows restriction by audience. Instead of relying on the default service account token to be auto-mounted, injection should add the volume to the proxy's pod and restrict to exactly the audience required.

This should not be a configuration option and instead be the only way to view service account tokens moving forward.

Concerns

While this feature reached beta in k8s 1.12, it is unclear that it is available in most cloud providers and local solutions (docker desktop, minikube). Before implementing, support for at least GKE, AKS, EKS, minikube and docker desktop should be validated.

[grampelberg]

Related to #2843 and #3183.

[alpeb]

As for provider support, this is supported wherever we can specify the --service-account-issuer, service-account-signing-key-file and --service-account-api-audiences flags to the kube-apiserver:

kind

Supported, although not enabled by default. One needs to pass a config file such as:

kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
kubeadmConfigPatches:
  - |
    apiVersion: kubeadm.k8s.io/v1beta2
    kind: ClusterConfiguration
    metadata:
      name: config
    apiServer:
      extraArgs:
        "service-account-issuer": "kubernetes.default.svc"
        "service-account-signing-key-file": "/etc/kubernetes/pki/sa.key"

Minikube

Supported, although not enabled by default. Minikube needs to be started with those flags enabled:

minikube start --extra-config=apiserver.service-account-signing-key-file=/var/lib/minikube/certs/sa.key --extra-config=apiserver.service-account-issuer=kubernetes/serviceaccount --extra-config=apiserver.service-account-api-audiences=api

Microk8s

Supported, although not enabled by default. The flags need to be set in /var/snap/microk8s/current/args/kube-apiserver:

...
--service-account-issuer=api
--service-account-signing-key-file=${SNAP_DATA}/certs/server.key
--service-account-api-audiences=api

Docker Desktop

Supported, although not enabled by default. To enable first open a session against Docker's tty:

screen ~/Library/Containers/com.docker.docker/Data/vms/0/tty

Then edit the kube-apiserver config:

vi /etc/kubernetes/manifests/kube-apiserver.yaml

and append the flags to the spec.containers.command section:

    - --service-account-issuer=api
    - --service-account-signing-key-file=/run/config/pki/apiserver.key
    - --service-account-api-audiences=api

Close the session with ctrl-a :quit and restart Docker.

GKE

Supported, and enabled by default!

AKS

Not supported and there's no way to set it up.

EKS

Not supported and there's no way to set it up. Not through the console nor through eksctl.

---

Edit: Completed Docker Desktop section
Edit: Updated command for Minikube, according to this comment

[grampelberg]

Whelp, this sucks. Should we add a config option just for GKE or shelve this for now?

[grampelberg]

I'm moving out of the release for now. Once this is more widely adopted it'd be great to do.

[reegnz]

AWS EKS now supports this by default since early september with kubernetes 1.14 and they encourage using projected service account tokens for IAM Role assumption.
https://aws.amazon.com/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/

As for AKS, let's see how this issue gets resolved: https://github.com/Azure/AKS/issues/1288 https://github.com/Azure/AKS/issues/1208

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[reegnz]

AKS ETA is early February https://github.com/Azure/AKS/issues/1208#issuecomment-574303709 So is support in GKE+EKS+AKS enough to have a go at this feature?

[grampelberg]

@reegnz AKS has support now?

Given the lack of support in minikube/microk8s, we'll need to degrade gracefully no matter what. Interested in taking up the contribution?

[reegnz]

@grampelberg I'm not that proficient in go yet, so I'll have someone else pick this up. :)
I am very enthusiastic about having this as a feature though!

[irizzant]

Minikube won't start with:
minikube start --extra-config=apiserver.service-account-issuer=api --extra-config=apiserver.service-account-signing-key-file=/var/lib/minikube/certs/apiserver.key --extra-config=apiserver.service-account-api-audiences=api

at least not version 1.8.2
You need to use:
minikube start --extra-config=apiserver.service-account-signing-key-file=/var/lib/minikube/certs/sa.key --extra-config=apiserver.service-account-issuer=kubernetes/serviceaccount --extra-config=apiserver.service-account-api-audiences=api

[alpeb]

Thanks @irizzant :+1:
I've updated the comment above.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 14 days if no further activity occurs. Thank you for your contributions.

[Olshansk]

I recently came by the following solution for minikube:

export GOOGLE_APPLICATION_CREDENTIALS=<creds-path>.json 
minikube addons enable gcp-auth

I'm not sure it completely address the problems here but figured it's worth mentioning.

Reference: https://minikube.sigs.k8s.io/docs/handbook/addons/gcp-auth/

[taman9333]

Hello @alpeb there's a type in Docker Desktop which will make cluster stuck starting...
this

- --service-account-signing-key-file=/var/run/config/pki/apiserver.key

should be

- --service-account-signing-key-file=/run/config/pki/apiserver.key

[alpeb]

Thanks @taman9333! I've updated the comment above :+1: