Kind: TokenRequest feature not enabled by default

[daixiang0]

What happened:

TokenRequest feature does not work.

root@kind-control-plane:/# grep Token /var/lib/kubelet/config.yaml
  TokenRequest: true
  TokenRequestProjection: true

What you expected to happen:

TokenRequest feature works.

How to reproduce it (as minimally and precisely as possible):

• enable TokenRequest feature with following config:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  # any feature gate can be enabled here with "Name": true
  # or disabled here with "Name": false
  # not all feature gates are tested, however
  TokenRequest: true
  TokenRequestProjection: true

• refer to https://github.com/kubernetes-sigs/apiserver-network-proxy/blob/master/examples/kubernetes/README.md

• the agent is failed:

  Warning  FailedMount  14s (x12 over 8m28s)  kubelet, kind-control-plane  MountVolume.SetUp failed for volume "konnectivity-agent-token" : failed to fetch token: the API server does not have TokenRequest endpoints enabled

Anything else we need to know?:

Environment:

• kind version: (use kind version):

kind v0.9.0 go1.14.4 linux/amd64

• Kubernetes version: (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.2", GitCommit:"52c56ce7a8272c798dbc29846288d7cd9fbae032", GitTreeState:"clean", BuildDate:"2020-04-16T11:56:40Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.1", GitCommit:"206bcadf021e76c27513500ca24182692aabd17e", GitTreeState:"clean", BuildDate:"2020-09-14T07:30:52Z", GoVersion:"go1.15", Compiler:"gc", Platform:"linux/amd64"}

• Docker version: (use docker info): 19.03.6
• OS (e.g. from /etc/os-release): Ubuntu 18.04

[daixiang0]

Easily use this to test:

apiVersion: v1
kind: Pod
metadata:
  name: sa-token-test
spec:
  containers:
  - name: container-test
    image: busybox
    volumeMounts:
    - name: token-vol
      mountPath: "/service-account"
      readOnly: true
  volumes:
  - name: token-vol
    projected:
      sources:
      - serviceAccountToken:
          audience: api
          expirationSeconds: 3600
          path: token

[BenTheElder]

This is due to kubeadm defaults IIRC. We could enable it but we'd be deviating from the defaults which we usually don't do.

https://github.com/istio/istio/blob/e02690fbfb8bda564582b27d22d9e8e6e00422a5/prow/config/trustworthy-jwt.yaml#L1-L13

[daixiang0]

Are there any docs about those configs at Kubeadm or Kubernetes side?

[BenTheElder]

https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection (upstream docs mention you must set these flags)

regarding kubeadm see e.g.:
https://jpweber.io/blog/a-look-at-tokenrequest-api/
https://github.com/linkerd/linkerd2/issues/3260

[BenTheElder]

I had a brief discussion with @howardjohn about enabling this by default previously, but didn't get to following up with kubeadm upstream about if it makes sense to do there. cc @neolit123

[howardjohn]

I think I saw a pr about enabling this by default in kubeadm. Perhaps it's not merged or only in 1.20 though

[neolit123]

hello,

normally features will get enabled in kubeadm by default when they graduate to Beta (feature gates' default values become "true" at that point). enabling Alpha features in kind by default before that, may work, but i would not recommend it.

https://github.com/kubernetes/enhancements/issues/542

in the case of:

TokenRequest: true
TokenRequestProjection: true

these features are already Beta so the above two lines are redundant.

for:

projected:
sources:

projected volumes require enabling BoundServiceAccountTokenVolume: true which is still Alpha and is planned to graduate to Beta in 1.21. (EDIT: see further comments bellow for corrections)

as noted, if you have questions about the required flags:
https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection
https://jpweber.io/blog/a-look-at-tokenrequest-api/
seem to have good information.

[howardjohn]

I thought the BoundServiceAccountTokenVolume was different and not relevant here? "This ServiceAccountTokenVolumeProjection is beta in 1.12 and enabled by passing all of the following flags to the API server:"

[neolit123]

I thought the BoundServiceAccountTokenVolume was different and not relevant here? "

ok, that is true. i confused bound vs non-bound tokens.
(the overall state of TokenRequest* features is confusing).

the flags mentioned here are required:
https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection

kubeadm started adding these flags in 1.20, but we have no plans to backport this change to older releases after discussions with SIG Auth.

[BenTheElder]

Do you have a link to the discussion with sig-auth?

Even if kubernetes won't cherry pick it, I'd be interested in understanding
exactly why, so we can consider "backporting" some consistent flags in kind
across versions. This one has come up pretty frequently with users.

On Thu, Dec 3, 2020 at 7:38 AM Lubomir I. Ivanov [e-mail]
wrote:

I thought the BoundServiceAccountTokenVolume was different and not
relevant here? "

ok, that is true. i confused bound vs non-bound tokens.
(the overall state of TokenRequest* features is confusing).

the flags mentioned here are required:

https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection

kubeadm started adding these flags in 1.20, but we have no plans to
backport this change to older releases after discussions with SIG Auth.

—
You are receiving this because you were assigned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes-sigs/kind/issues/1954#issuecomment-738085000,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAHADK3TNXFBQVOTEWIXB6DSS6WFZANCNFSM4ULUZDGQ
.

[neolit123]

kubeadm started adding these flags in 1.20, but we have no plans to backport this change to older releases after discussions with SIG Auth.

Even if kubernetes won't cherry pick it, I'd be interested in understanding
exactly why

the change to add the flags to kubeadm was made in here for 1.20:
https://github.com/kubernetes/kubernetes/pull/93258
the same PR moved TokenRequest and TokenRequestProjection to GA.

this PR (planned for .21, moves BoundServiceAccountTokenVolume to Beta) also has related discussion around usage and upgrade:
https://github.com/kubernetes/kubernetes/pull/95667

while TokenRequestProjection and TokenRequest were enabled since 1.12, due to communication reasons kubeadm missed the opportunity to enable them early on. similar goes to Azure AKS apparently where the relevant flags were optional.

a couple of blockers for the kubeadm backport from my POV are the the following:

• we generally do not backport features inc. flag additions to older releases, unless they solve a bug / security problem.
• it's not clear to me if users will be content with the following default issuer binding:
  https://github.com/kubernetes/kubernetes/pull/93258/files#diff-d870edeb820f49abf2f52cd357a8a5396c7cd1a36e024d339146e512186c7297R141
  (probably yes)

to the general question of whether we could backport the flags for ServiceAccountTokenVolumeProjection to older kubeadm versions, my guess would be - it's safe, but would defer to @liggitt @zshihang for an expert opinion.

so we can consider "backporting" some consistent flags in kind
across versions. This one has come up pretty frequently with users.

i somehow question if the demand has enough volume for this particular use case on the side of kind, but it seems doable.

[BenTheElder]

kind only supports kubernetes 1.13+* at the moment, so we could set the flags in kind config consistently across all versions. Even if it is a "breaking change" it would be one when upgrading kind rather than when choosing kubernetes version, so that's fine given we're still alpha at the moment.

I've gotten more pings about this feature than most, they're just scattered. Interest is increasing as more managed clusters enable.

*ish, more like 1.14+

[howardjohn]

FWIW Istio will either break or give a lot of warning messages if its not enabled, which may account for some demand

[liggitt]

whether we could backport the flags for ServiceAccountTokenVolumeProjection to older kubeadm versions, my guess would be - it's safe, but would defer to @liggitt @zshihang for an expert opinion.

Other than the general policy not to backport features (which is a good policy), my main hesitation is that prior to 1.20, this was a beta feature. Would backporting the flags cause problems for someone setting up a GA-only clusters (like we do for the ga-only conformance job at https://github.com/kubernetes-sigs/kind/blob/master/hack/ci/e2e-k8s.sh#L120-L121)

[BenTheElder]

Other than the general policy not to backport features (which is a good policy), my main hesitation is that prior to 1.20, this was a beta feature. Would backporting the flags cause problems for someone setting up a GA-only clusters (like we do for the ga-only conformance job at https://github.com/kubernetes-sigs/kind/blob/master/hack/ci/e2e-k8s.sh#L120-L121)

Good point.

Worst case though we'd set the flag again to empty/unset via a patch in the GA-only mode to clear it (IIRC kubeadm has these as a map, so last patch will win and be the only flag definition).

OTOH, 1.20 is almost here, we've been discussing shipping a kind release immediately following so the default cluster at least would pick this up anyhow.

[BenTheElder]

@daixiang0 @howardjohn 1.20 comes out today (I just built the debs/RPMs actually...), is updating kind to 1.20 sufficient for your needs?

[daixiang0]

@BenTheElder 1.20 is too new for me, i do not have a plan to upgrade it now, so I want docs with more details about those things.

[fentas]

I would also rather stay close to our systems, which run 1.18

[howardjohn]

I am not sure what the problem is. If you want to stay close to your systems - this feature is not enabled by default in these versions either. If you are enabling this flag in prod why is it an issue to configure it in kind?

[BenTheElder]

@daixiang0 you can enable it with the config here: https://github.com/kubernetes-sigs/kind/issues/1954#issuecomment-737775492

If that's not sufficient kind would be willing to host a docs page about this feature. I can't speak for kubernetes.io and this is not their issue tracker 😅

John makes a good point. It is rather simple to enable the flags ...

[fentas]

Ah, I understood this issue that enabling this feature gates within kind wouldn't work.
I also tried

featureGates:
  # any feature gate can be enabled here with "Name": true
  # or disabled here with "Name": false
  # not all feature gates are tested, however
  TokenRequest: true
  TokenRequestProjection: true

and it didn't work.
But if it should work I'll dig deeper. : )

[fentas]

Interestingly enough api-server received the feature gate parameter

    - --feature-gates=TTLAfterFinished=true,TokenRequest=true,TokenRequestProjection=true

but I am still seeing

MountVolume.SetUp failed for volume "istio-token" : failed to fetch token: the API server does not have TokenRequest endpoints enabled

[fentas]

Ok did some digging around.
Found this: https://github.com/kubeflow/manifests/issues/959#issuecomment-593289634

    - --service-account-signing-key-file=/etc/kubernetes/pki/sa.key
    - --service-account-issuer=kubernetes.default.svc

is missing for this to work / tested manually and it worked. : )
Is there an easy way to say kind to set these for me? :+1:

edit dangs overread comment above https://github.com/kubernetes-sigs/kind/issues/1954#issuecomment-737775492
Sorry all good :100:

[BenTheElder]

Yeah, the featuregate is not actually disabled for most versions (been in beta a while), but the feature requires setting these flags to actually function. The featuregate functionality in kind should work but isn't necessary here.

kubeadm only defaults to setting those flags in 1.20+, https://github.com/kubernetes-sigs/kind/issues/1954#issuecomment-737775492 has the kind config you need to pass the flags.