Teleport: Remove the need to `rm /var/lib/teleport`

[kontsevoy]

Problem

When users are learning Teleport, it's common for them to dramatically re-configure the auth server (change the cluster name, etc). When they do this, they frequently get stuck because the existing nodes fail to re-join the cluster (or they lose trusted clusters).

This happens because nodes (and trusted clusters) have their _old cluster credentials_ stored in /var/lib/teleport and those credentials are no longer valid. The error message "access denied" in the logs aren't very helpful, even though it's truthful.

Proposal

When a node re-connects to the cluster, it should so something like this:

1. If there are no node credentials in /var/lib/teleport -> use the join token
2. If there are existing credentials in /var/lib/teleport -> use the credentials
3. If "access denied" error happens because the credentials are not valid, try to use token.
4. If token was accepted and a node received new credentials, _overwrite the existing data in /var/lib/teleport with the new creds.

The same logic should apply to trusted clusters. If they suddenly start getting "access denied", they should try to use the join token.

Re-join intervals

Customers are also complaining that it takes up to 15 minutes for a trusted cluster to re-appear if there's a brief network disconnect. Let's do the following:

• The logic for node (or a trusted cluster, or an IoT node) for re-connecting should be the same. (back-off intervals, etc).
• Once a connection is restored, let's make sure that it takes _no more than a minute_ for nodes/clusters to reconnect after that point. This is especially important for IoT applications, where waiting for 15 minutes at scale causes issues.

Question: can we safely be even more aggressive? 30 seconds? 15 seconds?

Warnings

• Watch out for endless loops! i.e. use token -> get creds -> access denied -> use token, etc.
• Make sure NOT to blow up valid credentials.

CC @klizhentas to review.

┆Issue is synchronized with this Asana task by Unito

[klizhentas]

the design looks good to me, very helpful feature

[kontsevoy]

On cluster/node disconnects. I suspect we don't have a code in place which detects a dead idle socket on the node side, i.e. when a node/cluster holds a live connection to a cluster, the node (probably) only detects failure when it tries to ping it, NOT when the network cable is unplugged, that's why our reconnect logic is so relaxed (because it's probably tied to the ping frequency, which must NOT be increased because it will lead to severe network over-utilization).

[klizhentas]

I've investigated this problem a bit more today and came to conclusion that this fix is harder to accomplish than I thought for a couple of reasons:

• It is hard to reliably detect out of sync errors, as there are quite many of them and not always manifest themselves in the same way
• The system that fails to connect needs a full reboot - from proxies to nodes, and this will complicate the existing state machine even more - after trying several recovery approaches I ended up in a dead end state with completely broken cluster.

So I'm cautiously removing this from this milestone until I get some better ideas on how to fix this.

[webvictim]

This (or at least a very similar) issue caught up a user on evaluation recently and also stumped me for quite a while trying to figure out why they were having issues trying to join a node to a cluster.

The issue here went as follows:

• The user ran tctl nodes add on their auth server and tried to run the resulting command (to join a node with a token) on a separate node machine

teleport start \
   --roles=node \
   --token=dd739637c606a4cf1434031ff0ecfebb \
   --ca-pin=sha256:2e491d37c8ce1d3509fe31bfc46018728a70725cbc09697fbf71c4fb776d0657 \
   --auth-server=172.17.0.3:3025

• Because they were using an IP address and not a hostname (for evaluation), they got an error: ERRO [PROC:1] Node failed to establish connection to cluster: Get https://172.17.0.3:3025/v1/webapi/find: x509: cannot validate certificate for 172.17.0.3 because it doesn't contain any IP SANs. service/connect.go:65
• To fix this, I advised them to set public_addr under auth_service in their config to 172.17.0.3:3025 and restart Teleport, so the auth server would regenerate the certificate to include the IP SAN
• They did this, but rather than fixing the error, it changed to ERRO [PROC:1] Node failed to establish connection to cluster: Get https://172.17.0.3:3025/v1/webapi/find: x509: certificate signed by unknown authority. service/connect.go:65
• On my advice they tried removing /var/lib/teleport on the auth server and restarting, but still got the same error on the node

Eventually we figured out that the node had cached the old cert from the auth service and as such the join was failing. After removing /var/lib/teleport on the node and running the join command again, it worked first time.

I think this may be a subtly different bug to the original issue and if so, it would be nice to fix it.

I would suggest one of two fixes:
1) warn the user that they might try deleting /var/lib/teleport to fix their issue
2) (preferably) automatically delete the old auth server cert from the database, fetch a new cert and try joining again in the event of that error (x509: certificate signed by unknown authority)

Given that we're providing the CA pin hash on the command line it seems like it'd be fairly simple to detect that the existing cert we're caching doesn't match that CA pin, so delete the cache and refetch a fresh version.

[one000mph]

I encountered a similar error message due to cached cert mismatch and was able to resolve it by removing the offending node tctl rm nodes/<node-uuid> and reregistering it. I agree that it would be better to avoid the problem in the first place and/or document a better fix (working on this!).

[webvictim]

Another customer got hit with a very similar issue to this recently. It wasn't at all clear to them that they needed to delete /var/lib/teleport, but after they did everything just worked as expected.

This is a big cause of failures and frustration that we should improve on.

[zmb3]

+1 - Would love to see an easier workflow here. As we get our first teleport cluster up and running we're constantly making changes that require us to reconfigure or redeploy the auth server, and getting each node to rejoin the cluster is tedious, even for a small amount of nodes.

[Planktonette]

Definitely needs a less manual process to rejoin nodes to a cluster. We're targeting automated infrastructure, so in a worst case scenario where we lose teleport, it would be much friendlier if we didn't need to go through every node that was registered to trigger a re-registration.

[stevenGravy]

In cases where a node attempts to connect to a cluster who's been reset with a different cluster name the log can get overwhelmed with these messages. In some cases you do get the node address attempting the access even with the right token. In others not which makes it harder to troubleshoot. The fix is to reset the node's /var/lib/teleport dir and reconnect.

ERRO [AUTH:1]    "Failed to retrieve client pool. Client cluster <oldname>, target cluster <newname>, error:  \nERROR REPORT:\nOriginal Error: *trace.NotFoundError \"/authorities/host/<old name>\" is not found\nStack Trace:\n\t/gopath/src/github.com/gravitational/teleport/lib/backend/dynamo/dynamodbbk.go:816 github.com/gravitational/teleport/lib/backend/dynamo.(*DynamoDBBackend).getKey\n\t/gopath/src/github.com/gravitational/teleport/lib/backend/dynamo/dynamodbbk.go:395 github.com/gravitational/teleport/lib/backend/dynamo.(*DynamoDBBackend).Get\n\t/gopath/src/github.com/gravitational/teleport/lib/backend/sanitize.go:103 github.com/gravitational/teleport/lib/backend.(*Sanitizer).Get\n\t/gopath/src/github.com/gravitational/teleport/lib/backend/report.go:130 github.com/gravitational/teleport/lib/backend.(*Reporter).Get\n\t/gopath/src/github.com/gravitational/teleport/lib/services/local/trust.go:207 github.com/gravitational/teleport/lib/services/local.(*CA).GetCertAuthority\n\t/gopath/src/github.com/gravitational/teleport/lib/cache/cache.go:540 github.com/gravitational/teleport/lib/cache.(*Cache).GetCertAuthority\n\t/gopath/src/github.com/gravitational/teleport/lib/auth/middleware.go:336 github.com/gravitational/teleport/lib/auth.ClientCertPool\n\t/gopath/src/github.com/gravitational/teleport/lib/auth/middleware.go:152 github.com/gravitational/teleport/lib/auth.(*TLSServer).GetConfigForClient\n\t/opt/go/src/crypto/tls/handshake_server.go:147 crypto/tls.(*Conn).readClientHello\n\t/opt/go/src/crypto/tls/handshake_server.go:43 crypto/tls.(*Conn).serverHandshake\n\t/opt/go/src/crypto/tls/conn.go:1364 crypto/tls.(*Conn).Handshake\n\t/opt/go/src/net/http/server.go:1783 net/http.(*conn).serve\n\t/opt/go/src/runtime/asm_amd64.s:1358 runtime.goexit\nUser Message: \"/authorities/host/<old name>\" is not found\n." auth/middleware.go:160 

[alanlubold]

I have an issue that starts out the same way as what is discussed earlier in this thread. I certainly fall into the category of someone who is learning Teleport.

I try to make a node join a freshly installed single node cluster, using the IP, and get the error :

ERRO [PROC:1] Node failed to establish connection to cluster: Get https://172.17.0.3:3025/v1/webapi/find: x509: cannot validate certificate for 172.17.0.3 because it doesn't contain any IP SANs. service/connect.go:65

Then I try adding public_addr under auth_service in my config to 172.17.0.3:3025 (up until now I did not have a teleport.yaml file) and restarting, etc., and now I get

ERRO [PROC:1] Node failed to establish connection to cluster: Get https://172.17.0.3:3025/v1/webapi/find: x509: certificate signed by unknown authority. service/connect.go:65

At this point I am advised to delete /var/lib/teleport to remove old cert, but upon restarting the teleport service and trying to generate a new cert with tctl nodes add I get an error that I don't even have a auth server:

error: tctl must be either used on the auth server or provided with the identity file via --identity flag

And I find upon looking at /var/lib/teleport that it is indeed almost empty.

Is there something about how I am cleaning out the var/lib/teleport folder that I am missing? I am just deleting it entirely, like so: sudo rm -r /var/lib/teleport

Thanks

[russjones]

Related https://github.com/gravitational/teleport/issues/2122.

[webvictim]

5355 is a prime example of the need for this.