Teleport: Terminated node is still present in cluster

[igtsekov]

Description

What happened:
I created a teleport cluster using this how-to.
When I terminated one node it was still present in the output of tctl nodes ls command and shown in the UI.

What you expected to happen:
I expected the node to be removed from the output of tctl nodes ls command after it shutdowns.

How to reproduce it (as minimally and precisely as possible):
Create a cluster using this how-to. Verify that the cluster is up and running. Log in to the auth server and execute tctl nodes ls, after that terminate the node. Execute tctl nodes ls again or visit the UI.

Environment

• Teleport version (use teleport version):
  ```teleport version
  Teleport v4.3.5 git:v4.3.5-0-g122349e78 go1.13.2

- Tsh version (use `tsh version`):

tsh version
Teleport v4.3.5 git:v4.3.5-0-g122349e78 go1.13.2

- OS (e.g. from `/etc/os-release`):

cat /etc/os-release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
```

• Where are you running Teleport? (e.g. AWS, GCP, Dedicated Hardware): AWS

Browser environment

• Browser Version (for UI-related issues):
• Install tools:
• Others:

Relevant Debug Logs If Applicable

• tsh --debug
• teleport --debug

[webvictim]

Teleport nodes check in with a heartbeat every ten minutes or so. Idle/dead nodes will be removed from the tctl nodes ls listing after a missed heartbeat.

Did this happen for you?

[igtsekov]

Yes, it did. Is there a config option for this timeout?

[luckispac]

We have a similar aws setup with two load-balanced auth servers running in version Teleport Enterprise v4.2.10, in our case the dead node remained for a day or so until we noticed.
Not sure if this fits 100% here since in our case both auth-server had different nodes (UUID) with the same name (short lived vm) and a restart on both auth servers "flushed" it.

[russjones]

You can do tctl rm nodes/uuid to remove a node once you've shut it down (tsh ls -v will give you UUIDs)

Also, can you tell me a little bit more about your use case - why is it you want a node to TTL out faster?

[igtsekov]

In our setup, we will have around 500 nodes that are changing constantly. We want to keep the list of nodes up to date because the nodes are accessed very often and it is very annoying to see a list that is not accurate and you might try to access a node that is terminated. The tctl rm node / uuid command crossed my mind, but I am not sure if it can be automated.

[webvictim]

We have an issue for a better display of offline nodes (https://github.com/gravitational/teleport/issues/3657) but it isn't quite what you're asking for.

It isn't currently possible to alter the timeout, or the heartbeat interval. The reason for this decision is that on very big Teleport clusters, more regular heartbeats would generate a lot more traffic, which is undesirable for nodes which connect over the public internet. It could also give rise to a thundering herd problem when clusters are restarted.

It looks like for now there isn't an easy way to achieve what you want, I'm afraid. You could try running tctl get nodes every 30-60 seconds and processing the metadata.expires data for each node:

kind: node
metadata:
  expires: "2020-10-08T14:47:29.352820138Z"
  id: 1602167849354187953
  labels:
    environment: demo
    role: node
  name: e97c8c9d-5ddf-4270-92a0-8e578dc85a01
spec:
  addr: 10.4.0.217:3022
  hostname: gusdemo-root-node-1
  rotation:
    current_id: ""
    last_rotated: "0001-01-01T00:00:00Z"
    schedule:
      standby: "0001-01-01T00:00:00Z"
      update_clients: "0001-01-01T00:00:00Z"
      update_servers: "0001-01-01T00:00:00Z"
    started: "0001-01-01T00:00:00Z"
  version: 4.3.0
version: v2

The metadata.expires timestamp should usually be about 9-10 minutes in the future, as the timeout is 10 minutes and the heartbeat occurs every minute to update the timestamp. If you see that this timestamp drops to 7-8 minutes in the future, you could reasonably assume that the node has gone offline and run tctl nodes rm/id (where id is the value of metadata.name). If the node does check in again, it will be readded (but nobody would be able to connect to it until this happens, which is a risk).

This would probably get you to a point where offline nodes could be removed after being offline for 1-2 minutes.

[igtsekov]

Yes, it isn't what I am asking for but is a good idea and will be very helpful. Thank you for the response. I will test your workaround.