Teleport: Make teleport easier to run and configure on Kubernetes

[klizhentas]

Feature Request

Customers notice that Teleport is quite hard to run on kubernetes:

• Static file configuration require teleport to restart
• Teleport resource configuration are not integrating well with Kubernetes - users have to setup a parallel RBAC system.
• Teleport should play better with modern proxies like Envoy
• Application access can discover microservices automatically

Solution

Kubernetes CRD are not a suitable backend for teleport. They miss many features like notification watches for example.
However it should be possible to implement limited integration:

• [ ] Storing local x509 process credentials as a secret will allow teleport-kube-agent to become stateless and use dynamic tokens.
• [ ] Adding internal service that watches a set of CRDs - Roles, Users uses Teleport API to sync values will make it easier to configure Teleport.

[stevenGravy]

Feedback I've seen is that they often don't have an existing user/groups RBAC across many k8s clusters. If Teleport could apply those to clusters (groups I assume is the most straightforward) that would make it a lot easier.

[awly]

@klizhentas

Static file configuration require teleport to restart

So IIUC, users have a config file that changes on disk, and teleport should pick up those changes without a restart?
Two implementation details here:

• if config file is malformed, teleport should keep running with the last good config and not crash
• we may need to worry about gradual config rollouts, by e.g. adding some randomized wait time before applying an updated config

Teleport should play better with modern proxies like Envoy

Can you elaborate on this?

[awly]

One other feature request: use k8s API (a Secret or ConfigMap) to store teleport instance UUID and credentials.

[awly]

Another suggestion - use k8s API as a backend (via CRDs) (https://github.com/gravitational/teleport/issues/3177)

[wreed4]

Specifically with regards to RBAC configuration, this would be massively improved if these were CRDs that teleport read. Or short of that, if these resources can be put into a configmap that the teleport pods will read on start up and update if they change, that would be the next best thing.