Kops: Why BoundServiceAccountTokenVolume featureGate is not working?

[a8j8i8t8]

1. What kops version are you running? The command kops version, will display
this information.
1.15.1
2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.
1.15.9
3. What cloud provider are you using?
aws
4. What commands did you run? What is the simplest way to reproduce this issue?
Created cluster with below feature gate enabled for api-server.

kubeAPIServer:
    featureGates:
        BoundServiceAccountTokenVolume: true

5. What happened after the commands executed?
kube-api-server pod didn't start on master nodes.
6. Anything else do we need to know?
Is any other configuration needed for this feature gate to work properly?

[robinpercy]

Thanks @a8j8i8t8 . I was able to reproduce the issue based on your description. Need to investigate further. Will let you know if I find additional required configs.

[robinpercy]

@a8j8i8t8 looks like you also need to specify the ServiceAccountIssuer and ServiceAccountSigningKeyFile

Here's the error reported by the api-server
service-account-issuer is a required flag when BoundServiceAccountTokenVolume is enabled, --service-account-signing-key-file and --service-account-issuer are required flags

[rvirdiz]

@robinpercy were you able to solve this issue. It tried by specifying the required values, but no luck.
--service-account-issuer
--service-account-signing-key-file
--service-account-key-file
--service-account-api-audiences (--api-audiences now)

I am using kops 1.15.2 on K8s cluster 1.15.10 on AWS, to install istio-1.3.2, which requires the above values to be added to kube-apiserver, I am not sure what values needs to be setup for kops managed cluster, please let me know.
https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection

[mathieu-gilloots]

Same problem here, with istio sds, we need to add the extra flags like here :
https://github.com/istio/istio/blob/master/prow/config/trustworthy-jwt.yaml
How to process to do that with kops ?
Thanks

[rvirdiz]

Same problem here, with istio sds, we need to add the extra flags like here :
https://github.com/istio/istio/blob/master/prow/config/trustworthy-jwt.yaml
How to process to do that with kops ?
Thanks

I tried to made changes in K8s 1.15 API server manifest, and restarted kubelet service, it worked for me.

/etc/kubernetes/manifests/kube-apiserver.manifest, added following parameters to this file.
- --feature-gates=TokenRequest=true
- --service-account-issuer=kubernetes.default.svc
- --service-account-signing-key-file=/srv/kubernetes/server.key
- --service-account-key-file=/srv/kubernetes/server.key
- --api-audiences=api,istio-ca

[rvirdiz]

For kops, add following parameters to your cluster spec yaml file to deploy istio-1.3+ SDS by enabling ServiceAccountTokenVolumeProjection feature.
https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection

spec:
  kubeAPIServer:
    apiAudiences:
    - api
    - istio-ca
    featureGates:
      TokenRequest: "true"
    serviceAccountIssuer: kubernetes.default.svc
    serviceAccountKeyFile:
    - /srv/kubernetes/server.key
    serviceAccountSigningKeyFile: /srv/kubernetes/server.key

[jmymy]

working on updating docs https://github.com/istio/istio.io/pull/6744 and https://github.com/kubernetes/kops/pull/8680

For kops, add following parameters to your cluster spec yaml file to deploy istio-1.3+ SDS by enabling ServiceAccountTokenVolumeProjection feature.
https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/#service-account-token-volume-projection

spec:
  kubeAPIServer:
    apiAudiences:
    - api
    - istio-ca
    featureGates:
      TokenRequest: "true"
    serviceAccountIssuer: kubernetes.default.svc
    serviceAccountKeyFile:
    - /srv/kubernetes/server.key
    serviceAccountSigningKeyFile: /srv/kubernetes/server.key

note, featureGates for TokenRequest is not needed in k8s 1.12+, as its enabled by default

[olemarkus]

This one can be closed now?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[olemarkus]

/close

[k8s-ci-robot]

@olemarkus: Closing this issue.

In response to this:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.