Teleport: Can not add node to cluster ?

[huifanglu2018]

while I run a standalone cluster ,it works fine, but it often get errors while I add another node to this cluster.
one node run as auth,node,proxy:

/usr/local/bin/teleport start  --config=/etc/teleport.yaml  --pid-file=/run/teleport.pid --insecure
add another node as below:
/usr/local/bin/teleport start  --roles=node --config=/etc/teleport.yaml  --pid-file=/run/teleport.pid --insecure

it always shows:

DEBU [HTTP:PROX] No valid environment variables found. proxy/proxy.go:222
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:137
ERRO [PROC:1]    Node failed to establish connection to cluster: ssh: handshake failed: no matching keys found. time/sleep.go:148

can anybody help me get node to the cluster?

[webvictim]

Can you share the content of the /etc/teleport.yaml files for both nodes?

[huifanglu2018]

teleport:
    data_dir: /var/lib/teleport
    auth_token: f7adb7ccdf04037bcd2b52ec6010fd6f0caec94ba190b765
    auth_servers:
      - 10.10.10.136:3025
    connection_limits:
        max_connections: 1000
        max_users: 250
    log:
        output: stderr
        severity: DEBUG
auth_service:
    enabled: true
    #    session_recording: "proxy"
    proxy_checks_host_keys: no
    cluster_name: "teleportcluster"
    listen_addr: 0.0.0.0:3025
    tokens:
    - proxy,node:f7adb7ccdf04037bcd2b52ec6010fd6f0caec94ba190b765
    authentication:
      type: local
      second_factor: off
ssh_service:
    enabled: true
    labels:
        env: staging
proxy_service:
    enabled: true
    listen_addr: 0.0.0.0:3023
    web_listen_addr: 0.0.0.0:3080
    tunnel_listen_addr: 0.0.0.0:3024
    public_addr: 10.10.10.136:3080

---

teleport:
    data_dir: /var/lib/teleport
    auth_token: f7adb7ccdf04037bcd2b52ec6010fd6f0caec94ba190b765
    auth_servers:
            - 10.10.10.136:3080
    connection_limits:
        max_connections: 1000
        max_users: 250
    log:
        output: stderr
        severity: DEBUG
auth_service:
    enabled: false
ssh_service:
    enabled: true
    labels:
        env: staging
proxy_service:
    enabled: false

@webvictim Thanks for your help.

[webvictim]

@huifanglu2018 I think the reason for the issue is because the token you've set is valid for both proxy,node roles, but your node is only trying to join with the node role. With Teleport, tokens must use the full set of roles.

Change - proxy,node:f7adb7ccdf04037bcd2b52ec6010fd6f0caec94ba190b765 to - node:f7adb7ccdf04037bcd2b52ec6010fd6f0caec94ba190b765, then restart your Teleport auth server and try joining the node again - it should work.

[huifanglu2018]

/home/ubuntu/teleport# /usr/local/bin/teleport start  --roles=node --config=/home/ubuntu/teleport/teleport.yaml --auth-server=https://10.254.9.135:**3080**  --pid-file=/run/teleport.pid  --insecure
DEBU [SQLITE]    Connected to: file:/var/lib/teleport/proc/sqlite.db?_busy_timeout=10000&_sync=OFF, poll stream period: 1s lite/lite.go:173
DEBU [SQLITE]    Synchronous: 0, busy timeout: 10000 lite/lite.go:218
DEBU [KEYGEN]    SSH cert authority started with no keys pre-compute. native/native.go:107
DEBU [PROC]      Adding service to supervisor. service:register.node service/supervisor.go:181
DEBU [PROC]      Adding service to supervisor. service:ssh.node service/supervisor.go:181
DEBU [PROC]      Adding service to supervisor. service:ssh.shutdown service/supervisor.go:181
DEBU [PROC]      Adding service to supervisor. service:common.rotate service/supervisor.go:181
DEBU [PROC:1]    Service has started. service:register.node service/supervisor.go:242
DEBU [PROC:1]    Service has started. service:ssh.shutdown service/supervisor.go:242
DEBU [PROC:1]    No signal pipe to import, must be first Teleport process. service/service.go:761
DEBU [PROC:1]    Service has started. service:ssh.node service/supervisor.go:242
DEBU [PROC:1]    Service has started. service:common.rotate service/supervisor.go:242
DEBU [PROC:1]    Connected state: never updated. service/connect.go:99
INFO [PROC]      Connecting to the cluster worker7 with TLS client certificate. service/connect.go:127
DEBU [PROC]      Attempting to connect to Auth Server directly. service/connect.go:793
DEBU [PROC]      Attempting to connect to Auth Server through tunnel. service/connect.go:801
DEBU [CLIENT]    HTTPS client init(proxyAddr=10.254.9.135:3080, insecure=true) client/weblogin.go:307
WARNING: You are using insecure connection to SSH proxy https://10.254.9.135:3080
DEBU [PROC]      Discovered address for reverse tunnel server: 10.254.9.135:3024. service/connect.go:881
DEBU [HTTP:PROX] No valid environment variables found. proxy/proxy.go:222
DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:137
ERRO [PROC:1]    Node failed to establish connection to cluster: ssh: handshake failed: no matching keys found. time/sleep.go:148

@webvictim Thanks so much for your reply, but I was so sad it still so many issues.
It still same error after I change it to node:f7adb7ccdf04037bcd2b52ec6010fd6f0caec94ba190b765 . and I find if I change the auth server to 10.254.9.135:3025 , it shows log as below:
node log:

/home/ubuntu/teleport# /usr/local/bin/teleport start  --roles=node --config=/home/ubuntu/teleport/teleport.yaml --auth-server=https://10.254.9.135:3025  --pid-file=/run/teleport.pid  --insecure --insecure-no-tls
DEBU [SQLITE]    Connected to: file:/var/lib/teleport/proc/sqlite.db?_busy_timeout=10000&_sync=OFF, poll stream period: 1s lite/lite.go:173
DEBU [SQLITE]    Synchronous: 0, busy timeout: 10000 lite/lite.go:218
DEBU [KEYGEN]    SSH cert authority started with no keys pre-compute. native/native.go:107
DEBU [PROC]      Adding service to supervisor. service:register.node service/supervisor.go:181
DEBU [PROC]      Adding service to supervisor. service:ssh.node service/supervisor.go:181
DEBU [PROC]      Adding service to supervisor. service:ssh.shutdown service/supervisor.go:181
DEBU [PROC]      Adding service to supervisor. service:common.rotate service/supervisor.go:181
DEBU [PROC:1]    Service has started. service:ssh.shutdown service/supervisor.go:242
DEBU [PROC:1]    Service has started. service:ssh.node service/supervisor.go:242
DEBU [PROC:1]    Service has started. service:common.rotate service/supervisor.go:242
DEBU [PROC:1]    No signal pipe to import, must be first Teleport process. service/service.go:761
DEBU [PROC:1]    Service has started. service:register.node service/supervisor.go:242
DEBU [PROC:1]    Connected state: never updated. service/connect.go:99
INFO [PROC]      Connecting to the cluster worker7 with TLS client certificate. service/connect.go:127
DEBU [PROC]      Attempting to connect to Auth Server directly. service/connect.go:793
DEBU [PROC]      Attempting to connect to Auth Server through tunnel. service/connect.go:801
DEBU [CLIENT]    HTTPS client init(proxyAddr=10.254.9.135:3025, insecure=true) client/weblogin.go:307
WARNING: You are using insecure connection to SSH proxy https://10.254.9.135:3025
ERRO [PROC:1]    "Node failed to establish connection to cluster: 404 page not found\n." time/sleep.go:148

auth log:
ERRO [AUTH:1] "Failed to retrieve client pool. Client cluster worker7, target cluster teleportcluster, error: \nERROR REPORT:\nOriginal Error: *trace.NotFoundError key \"/authorities/host/worker7\" is not found\nStack Trace:\n\t/go/src/github.com/gravitational/teleport/lib/backend/memory/memory.go:186 github.com/gravitational/teleport/lib/backend/memory.(*Memory).Get\n\t/go/src/github.com/gravitational/teleport/lib/backend/report.go:159 github.com/gravitational/teleport/lib/backend.(*Reporter).Get\n\t/go/src/github.com/gravitational/teleport/lib/backend/wrap.go:89 github.com/gravitational/teleport/lib/backend.(*Wrapper).Get\n\t/go/src/github.com/gravitational/teleport/lib/services/local/trust.go:207 github.com/gravitational/teleport/lib/services/local.(*CA).GetCertAuthority\n\t/go/src/github.com/gravitational/teleport/lib/cache/cache.go:892 github.com/gravitational/teleport/lib/cache.(*Cache).GetCertAuthority\n\t/go/src/github.com/gravitational/teleport/lib/auth/middleware.go:546 github.com/gravitational/teleport/lib/auth.ClientCertPool\n\t/go/src/github.com/gravitational/teleport/lib/auth/middleware.go:253 github.com/gravitational/teleport/lib/auth.(*TLSServer).GetConfigForClient\n\t/opt/go/src/crypto/tls/handshake_server.go:141 crypto/tls.(*Conn).readClientHello\n\t/opt/go/src/crypto/tls/handshake_server.go:40 crypto/tls.(*Conn).serverHandshake\n\t/opt/go/src/crypto/tls/conn.go:1362 crypto/tls.(*Conn).Handshake\n\t/go/src/github.com/gravitational/teleport/lib/multiplexer/tls.go:141 github.com/gravitational/teleport/lib/multiplexer.(*TLSListener).detectAndForward\n\t/opt/go/src/runtime/asm_amd64.s:1375 runtime.goexit\nUser Message: key \"/authorities/host/worker7\" is not found\n." auth/middleware.go:261

[huifanglu2018]

auth log:
2021/01/22 13:55:19 http: TLS handshake error from 10.254.200.234:58309: remote error: tls: unknown certificate ERRO [AUTH:1] "Failed to retrieve client pool. Client cluster worker7, target cluster teleportcluster, error: \nERROR REPORT:\nOriginal Error: *trace.NotFoundError key \"/authorities/host/worker7\" is not found\nStack Trace:\n\t/go/src/github.com/gravitational/teleport/lib/backend/memory/memory.go:186 github.com/gravitational/teleport/lib/backend/memory.(*Memory).Get\n\t/go/src/github.com/gravitational/teleport/lib/backend/report.go:159 github.com/gravitational/teleport/lib/backend.(*Reporter).Get\n\t/go/src/github.com/gravitational/teleport/lib/backend/wrap.go:89 github.com/gravitational/teleport/lib/backend.(*Wrapper).Get\n\t/go/src/github.com/gravitational/teleport/lib/services/local/trust.go:207 github.com/gravitational/teleport/lib/services/local.(*CA).GetCertAuthority\n\t/go/src/github.com/gravitational/teleport/lib/cache/cache.go:892 github.com/gravitational/teleport/lib/cache.(*Cache).GetCertAuthority\n\t/go/src/github.com/gravitational/teleport/lib/auth/middleware.go:546 github.com/gravitational/teleport/lib/auth.ClientCertPool\n\t/go/src/github.com/gravitational/teleport/lib/auth/middleware.go:253 github.com/gravitational/teleport/lib/auth.(*TLSServer).GetConfigForClient\n\t/opt/go/src/crypto/tls/handshake_server.go:141 crypto/tls.(*Conn).readClientHello\n\t/opt/go/src/crypto/tls/handshake_server.go:40 crypto/tls.(*Conn).serverHandshake\n\t/opt/go/src/crypto/tls/conn.go:1362 crypto/tls.(*Conn).Handshake\n\t/go/src/github.com/gravitational/teleport/lib/multiplexer/tls.go:141 github.com/gravitational/teleport/lib/multiplexer.(*TLSListener).detectAndForward\n\t/opt/go/src/runtime/asm_amd64.s:1375 runtime.goexit\nUser Message: key \"/authorities/host/worker7\" is not found\n." auth/middleware.go:261 WARN [MXTLS:1] Handshake failed. error:remote error: tls: bad certificate multiplexer/tls.go:143

[huifanglu2018]

The guild seems not to work for me, I try so many possible mistake, it still cannot add the second node to cluster...
https://goteleport.com/teleport/docs/quickstart/#add-a-node-to-the-cluster
Is there a example configure for me ?

[webvictim]

@huifanglu2018 You may have some old credentials cached for some reason. Look at ps -ef | grep teleport and make sure there are no other Teleport processes running on the node you're trying to add, remove /var/lib/teleport completely and then run the teleport start command again.

[huifanglu2018]

@webvictim Thank you so much! It works after removing /var/lib/teleport. 👍