Teleport: Teleport-proxy does not serve updated https certificates without restart

[dampersand]

Description

We use short-lived SSL certificates from a private CA in our environment, and we have provisioned our private teleport-proxies to serve these short-lived https certificates. (teleport.yaml directive: proxy_service.https_cert_file)

It appears that when these short-lived certificates are rotated and placed on the server, the teleport service continues to pass the old certificate out. Obvs this is no good - we COULD code 'systemctl restart teleport' into the cert rotation, but much better for teleport to handle automated cert changes (so we don't have to force-disconnect users)!

What happened:
contents of proxy_service.https_cert_file changed, but teleport service did not reflect the change until after a restart

What you expected to happen:
the teleport service to accurately reflect the cert file after any possible updates without requiring a restart

How to reproduce it (as minimally and precisely as possible):
Helps to have your own CA, but you can spoof this, too!

1. have a testing teleport proxy up-and-running. Be sure you can reach it from a browser.
2. generate a keypair on teleport-proxy and self-sign the public key into a certificate with some unique metadata (for instance, expiry date of tomorrow)
3. point your teleport proxy's 'proxy_service.https_cert_file' and 'proxy_service.https_key_file' to the generated key/cert
4. start the teleport service on the teleport-proxy
5. Point a browser to the teleport-proxy node and view the certificate that is returned (paying close attention to the unique metadata you've set in step 2)
6. on the teleport-proxy, regenerate/re-sign the certificate, but vary the unique metadata from step 2 (for instance, change the expiry date to next week)
7. Confirm that teleport-proxy's teleport.yaml is still pointing to the correct key/cert, but DO NOT restart the teleport service
8. Point a browser to the teleport proxy node and view the certificate that is returned. Notice that the unique metadata from steps 2/6 have NOT been changed from the previous cert you received!

Environment

• Teleport version (use teleport version): Teleport Enterprise v4.2.10git:v4.2.10-0-g7fd6c7348 go1.13.2
• Tsh version (use tsh version): Teleport v4.2.10 git:v4.2.10-0-g7fd6c7348 go1.13.2

• OS (e.g. from /etc/os-release): CentOS Linux release 7.7.1908 (Core)

• Where are you running Teleport? (e.g. AWS, GCP, Dedicated Hardware): AWS EC2s, but treat it like baremetal

Browser environment

• Browser Version (for UI-related issues): Google Chrome, Version 83.0.4103.97 (Official Build) (64-bit)
• Install tools: NA
• Others: NA

Relevant Debug Logs If Applicable

NA

[webvictim]

This is the same issue as #3198. There was a fix in #3272 but it wasn't resilient/tested enough.

[dampersand]

My bad. I actually saw that one and then skipped it when I read 'cert-manager', thinking it was a different situation.

[webvictim]

That's no problem - this explanation is more thorough anyway. I definitely want to see a fix for this issue and will try to see if we can get it worked on.

[webvictim]

I got hit by this one myself this morning.

error: Get https://teleport.example.com:3080/v1/webapi/ping: x509: certificate has expired or is not yet valid
kex_exchange_identification: Connection closed by remote host

At first I thought I'd failed to set up certbot renew to run properly on my Teleport auth/proxy server, but it actually turned out that the certificates had been renewed just fine - they just hadn't been reloaded due to this exact issue.

My current workaround is to add a post-renew hook to certbot which should tell Teleport to reload the certs after they've been renewed:

# cat /etc/letsencrypt/renewal-hooks/post# cat 001-restart-teleport 
#!/bin/bash
kill -HUP $(pidof /usr/local/bin/teleport)

[russjones]

@fspmarshall This is related to cluster upgrades. State machine that can cover CA rotation, but also other upgrades, like nodes, as well as proxy certificates. Watch file and trigger self-reload.

[pierrebeaucamp]

We're running into this as well with Cloud. Fixing it in Teleport would be preferred as it would remove the need to do any sidecar shenanigans as a workaround.

[alex-kovoy]

@russjones

It seems that you can update the keys without restarting the service and do it in place where you can use TLSConfig.GetCertificate method to pass your own callback.

(without knowing all details) we can add a flag to teleport config to autoupdate the keys. When enabled, we would listen to these files and once updated the TLSConfig.GetCertificate callback will make sure that the new keys will be used for the new connections.

https://github.com/robustirc/robustirc/commit/3b83e51eeed62e3e5421f03023503bd6e8c34193
https://stackoverflow.com/questions/37473201/is-there-a-way-to-update-the-tls-certificates-in-a-net-http-server-without-any-d