teleport-kube-agent doesn't re-register after proxy is recreated

[fraenkel]

Description

What happened:

We have an automated deploy of the teleport server behind an ASG. When we make some configuration change, a new instance is spun up.
We noticed that when we do a tsh kube ls we do not see any previously deployed teleport-kube-agent. Only newly deployed kube-agents appear as their pod is started.

What you expected to happen:
I would expect that eventually all running kube-agents register themselves.

Reproduction Steps

As minimally and precisely as possible, describe step-by-step how to reproduce the problem.

1. Deploy a standalone teleport server with the kubernetes proxy enabled
2. Deploy an EKS cluster using teleport-kube-agent pointing to the proxy
3. Create a new standalone teleport server with the same DNS name
4. tsh kube ls does not show the cluster
5. kubectl -n TELEPORT_NS delete po -l app.kubernetes.io/name=teleport-kube-agent
6. tsh kube ls shows the cluster

Server Details

• Teleport version (run teleport version): Teleport v5.1.2 git:v5.1.2-0-g822d10b44 go1.15.5
• Server OS (e.g. from /etc/os-release): Ubuntu 20.04.2 LTS
• Where are you running Teleport? (e.g. AWS, GCP, Dedicated Hardware): AWS
• Additional details:

Client Details

• Tsh version (tsh version): Teleport v5.1.2 git: go1.15.7
• Computer OS (e.g. Linux, macOS, Windows): macOS
• Browser version (for UI-related issues):
• Installed via (e.g. apt, yum, brew, website download): website
• Additional details:

Debug Logs

Please include or attach debug logs, when appropriate. Obfuscate sensitive information!

• Start Teleport with --debug flag (teleport --debug)
• Run tsh with --debug flag (tsh --debug)

teleport-kube-agent logs after the server was recreated

[KUBERNETES]   Kubernetes service 5.1.2:v5.1.2-0-g822d10b44 is starting via proxy reverse tunnel.
E0223 23:00:09.984802       6 v2.go:105] EOF
E0223 23:00:22.676411       6 v2.go:105] EOF
E0223 23:00:26.243752       6 v2.go:105] EOF
E0223 23:00:36.236028       6 v2.go:105] EOF
E0223 23:00:39.879072       6 v2.go:105] EOF
E0223 23:00:40.020085       6 v2.go:105] EOF
ERRO [KUBERNETE] Error copying upstream response Body: context canceled forward/fwd.go:203
2021/02/23 23:12:29 http: superfluous response.WriteHeader call from github.com/gravitational/teleport/lib/kube/proxy.(*responseStatusRecorder).WriteHeader (forwarder.go:1643)
E0223 23:13:49.662650       6 v2.go:105] EOF
E0223 23:13:58.250475       6 v2.go:105] EOF
E0223 23:14:01.766591       6 v2.go:105] EOF
E0223 23:14:10.603254       6 v2.go:105] EOF
E0223 23:14:12.390978       6 v2.go:105] EOF
E0223 23:14:12.411914       6 v2.go:105] EOF
ERRO [KUBERNETE] Error copying upstream response Body: context canceled forward/fwd.go:203
2021/02/23 23:20:05 http: superfluous response.WriteHeader call from github.com/gravitational/teleport/lib/kube/proxy.(*responseStatusRecorder).WriteHeader (forwarder.go:1643)
ERRO [KUBERNETE] Error copying upstream response Body: context canceled forward/fwd.go:203
2021/02/23 23:20:27 http: superfluous response.WriteHeader call from github.com/gravitational/teleport/lib/kube/proxy.(*responseStatusRecorder).WriteHeader (forwarder.go:1643)
ERRO [KUBERNETE] Error copying upstream response Body: context canceled forward/fwd.go:203
2021/02/23 23:22:39 http: superfluous response.WriteHeader call from github.com/gravitational/teleport/lib/kube/proxy.(*responseStatusRecorder).WriteHeader (forwarder.go:1643)

teleport-kube-agent logs after the pod was recreated

[KUBERNETES]   Kubernetes service 5.1.2:v5.1.2-0-g822d10b44 is starting via proxy reverse tunnel.

Which platform(s)

• [x] Teleport Open Source
• [ ] Teleport Cloud
• [ ] Teleport Enterprise
• [ ] Other

Which component(s)

• [ ] Server Access
• [x] Kubernetes Access
• [ ] Application Access
• [ ] Database Access
• [ ] CLI Clients
• [ ] Web Clients
• [ ] Other

[webvictim]

Is your proxy connected to the same Teleport auth server as before? If so, the kube agents should re-register themselves and appear once they realise that their connection to the proxy has dropped.

If the auth server changes, the kube agents will still be trying to connect with cached certificates for the old auth server - this requires manual intervention to fix (i.e. deleting /var/lib/teleport inside the pod, or restarting it)

[fraenkel]

Its an all in one so its a new auth server as well.
I do have a static kube token that I use as the authToken for the teleport-kube-agent.
Are you saying I need more static tokens? I see references in the docs, but none of them are specified in the Teleport Kubernetes Access.

The docs mention tokens for nodes and proxy, and then a separate one for auth. But there is no clear explanation of how they are used on the agent side.

[webvictim]

When you use a static token to join an agent to a Teleport cluster, the cluster's certificate authority generates a key and signs a certificate which the agent then caches on disk and uses for any future authentication with the cluster (via mutual TLS).

Without persistent storage, when you start a new Teleport auth server it generates a fresh certificate authority, so all old certificates are invalidated. Therefore, even if you define the same static token string in the cluster's config, a running agent which has previously joined a different Teleport cluster will not be able to authenticate with the new one. This is somewhat by design, to prevent a scenario where a Teleport cluster could be taken over by an attacker, who would then gain control of all the existing nodes when they rejoined.

The only way to force a running agent with cached certificates to rejoin a new cluster is to delete cached certificates on the agent (the easiest way is to remove the contents of /var/lib/teleport) and then restart the Teleport process. Killing the pod and letting it get restarted by Kubernetes also works, because the pod's storage is ephemeral so the certificates are deleted when the pod terminates.

One way to avoid this would be to make sure that your Teleport auth server has some kind of persistent storage configured, then kill -HUP $(pidof teleport) on the auth server when the config changes. This will fork a new Teleport process using the changed configuration, and shut the old process down when there are no active user connections to the old process. As this would keep the same certificate authority, your agents would then rejoin without a problem.

(related to #2838)

[fraenkel]

Its a bit of a chicken and egg problem. We need teleport to access the k8s cluster. But this says I have to have inbound access to all my networks and furthermore have a kubeconfig so I can bounce the pods.

[webvictim]

I can see what you're saying. The recommendation here would definitely be that your central Teleport cluster (which the agents connect to) should be a lot more static/persistent/highly available than the agents themselves so that you can maintain access.

[fraenkel]

I see I either use dynamodb, etcd or magically make ebs work with ASGs to enable a stateless auth instance.