K3os: After a complete power off: the server is currently unable to handle the request

Created on 24 Nov 2020  路  9Comments  路  Source: rancher/k3os

Version (k3OS / kernel)

Node1 OS/kernel ========== 5.9.8-2-MANJARO Linux ( k3s version v1.19.3+k3s3 (0e4fbfef) )
Node2/3 k3os --version ========= k3os version v0.19.2-dev.4


Architecture

Node1/2/3 ==== x86_64

Describe the bug

Hi, I am creating 3 node HA cluster with embedded db option. I am having 3 nodes, Node1 is running on my system having k3s installed and running the --cluster-init command. Node2 and Node3 are k3os and I am using configs

Node2 CloudInit

ssh_authorized_keys:
- ssh-ed25519 hdjkldhkd... (either rsa or ssh-ed will be fine)
- "github:YourGithubUserName" (this will be used to fetch public key stored in your github account and it will be placed under ssh/authorised_keys in Node2/3)
k3os:
  data_sources:
  - aws
  - cdrom
  modules:
  - kvm
  - nvme
  sysctl:
    kernel.printk: "4 4 1 7"
    kernel.kptr_restrict: "1"
  dns_nameservers:
  - 192.168.1.1 (if you have your own DNS configured)
  - 8.8.8.8
  ntp_servers:
  - 0.in.pool.ntp.org
  - 1.in.pool.ntp.org
  server_url: https://192.168.1.X:6443  (IP of Node1)
  token: secrets

  k3s_args:
  - server
  - "--node-name=node2"

Node3 CloudInit

ssh_authorized_keys:
- ssh-ed25519 hdjkldhkd... (either rsa or ssh-ed will be fine)
- "github:YourGithubUserName" (this will be used to fetch public key stored in your github account and it will be placed under ssh/authorised_keys in Node2/3)
k3os:
  data_sources:
  - aws
  - cdrom
  modules:
  - kvm
  - nvme
  sysctl:
    kernel.printk: "4 4 1 7"
    kernel.kptr_restrict: "1"
  dns_nameservers:
  - 192.168.1.1 (if you have your own DNS configured)
  - 8.8.8.8
  ntp_servers:
  - 0.in.pool.ntp.org
  - 1.in.pool.ntp.org
  server_url: https://192.168.1.X:6443  (IP of Node1)
  token: secrets

  k3s_args:
  - server
  - "--node-name=node3"

HA cluster comes up fine in the first try but after reboot it is giving the error and behaving like this same issue

To Reproduce

Have 3 nodes, Node1 running k3s on manjaro (or I guess any os), Node2 and Node3 bootstrapped with cloudinit files given above.
after successful HACluster comes up, try to reboot the whole architecture systems, its giving the stated error and cluster is not re-created(formed).

Expected behavior

cluster should come up.

Actual behavior

cluster giving error "the server is currently unable to handle the request"

Additional context

relevant issue from k3s

kinbug
Source
picture dharmendrakariya
👍2

Most helpful comment

hi @dweomer

my workaround was to replace the k3s inside with the latest version (v1.19.4+ks31). I stopped / started in a random way and the cluster was healthy after a few minutes. I rebuild the ISO image and reinstalled all nodes.

yeah, i was hoping to push out a dev build with that very version. possibly today

picture dweomer on 24 Nov 2020
👍2

All 9 comments

hi,

same issue with it. Installing K3OS with embedded etcd works, until rebooting / poweroff. Have also commented this.

cu denny

linuxmail picture linuxmail on 24 Nov 2020

I was pretty sure that I had avoided this problem but I just rebooted my 3-node HA NUCs all at once and it is failing in the exact same way. Will try to get a build out soon-ish with the latest k3s that also has kine and etcd fixes.

dweomer picture dweomer on 24 Nov 2020

seeing a lot of errors in the logs:

WARNING: 2020/11/24 16:55:57 grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: context canceled". Reconnecting...
2020-11-24 16:55:57.094303 W | rafthttp: health check for peer 7a44c34067ffd15c could not connect: dial tcp 192.168.1.33:2380: connect: connection refused
raft2020/11/24 16:56:00 INFO: a155000d15c5b5b6 is starting a new election at term 258
raft2020/11/24 16:56:00 INFO: a155000d15c5b5b6 became candidate at term 259
raft2020/11/24 16:56:00 INFO: a155000d15c5b5b6 received MsgVoteResp from a155000d15c5b5b6 at term 259
raft2020/11/24 16:56:00 INFO: a155000d15c5b5b6 [logterm: 12, index: 38147148] sent MsgVote request to 7a44c34067ffd15c at term 259
raft2020/11/24 16:56:00 INFO: a155000d15c5b5b6 [logterm: 12, index: 38147148] sent MsgVote request to fa5fddcfd17240bf at term 259
time="2020-11-24T16:56:00.672321678Z" level=info msg="Cluster-Http-Server 2020/11/24 16:56:00 http: TLS handshake error from 192.168.1.32:37208: remote error: tls: bad certificate"
time="2020-11-24T16:56:00.896420546Z" level=info msg="Cluster-Http-Server 2020/11/24 16:56:00 http: TLS handshake error from 192.168.1.33:50348: remote error: tls: bad certificate"
2020-11-24 16:56:02.060708 E | etcdserver: publish error: etcdserver: request timed out
2020-11-24 16:56:02.094352 W | rafthttp: health check for peer 7a44c34067ffd15c could not connect: dial tcp 192.168.1.33:2380: connect: connection refused
2020-11-24 16:56:02.094562 W | rafthttp: health check for peer 7a44c34067ffd15c could not connect: dial tcp 192.168.1.33:2380: connect: connection refused
2020-11-24 16:56:02.094580 W | rafthttp: health check for peer fa5fddcfd17240bf could not connect: dial tcp 192.168.1.32:2380: connect: connection refused
2020-11-24 16:56:02.094590 W | rafthttp: health check for peer fa5fddcfd17240bf could not connect: dial tcp 192.168.1.32:2380: connect: connection refused
time="2020-11-24T16:56:06.043236062Z" level=info msg="Cluster-Http-Server 2020/11/24 16:56:06 http: TLS handshake error from 192.168.1.32:37212: remote error: tls: bad certificate"
time="2020-11-24T16:56:06.385941470Z" level=info msg="Cluster-Http-Server 2020/11/24 16:56:06 http: TLS handshake error from 192.168.1.33:50352: remote error: tls: bad certificate"
2020-11-24 16:56:07.094676 W | rafthttp: health check for peer 7a44c34067ffd15c could not connect: dial tcp 192.168.1.33:2380: connect: connection refused
2020-11-24 16:56:07.094729 W | rafthttp: health check for peer fa5fddcfd17240bf could not connect: dial tcp 192.168.1.32:2380: connect: connection refused
2020-11-24 16:56:07.094858 W | rafthttp: health check for peer 7a44c34067ffd15c could not connect: dial tcp 192.168.1.33:2380: connect: connection refused
2020-11-24 16:56:07.094879 W | rafthttp: health check for peer fa5fddcfd17240bf could not connect: dial tcp 192.168.1.32:2380: connect: connection refused
raft2020/11/24 16:56:08 INFO: a155000d15c5b5b6 is starting a new election at term 259
raft2020/11/24 16:56:08 INFO: a155000d15c5b5b6 became candidate at term 260
raft2020/11/24 16:56:08 INFO: a155000d15c5b5b6 received MsgVoteResp from a155000d15c5b5b6 at term 260
raft2020/11/24 16:56:08 INFO: a155000d15c5b5b6 [logterm: 12, index: 38147148] sent MsgVote request to 7a44c34067ffd15c at term 260
raft2020/11/24 16:56:08 INFO: a155000d15c5b5b6 [logterm: 12, index: 38147148] sent MsgVote request to fa5fddcfd17240bf at term 260
time="2020-11-24T16:56:11.413060455Z" level=info msg="Cluster-Http-Server 2020/11/24 16:56:11 http: TLS handshake error from 192.168.1.32:37216: remote error: tls: bad certificate"
time="2020-11-24T16:56:11.874578011Z" level=info msg="Cluster-Http-Server 2020/11/24 16:56:11 http: TLS handshake error from 192.168.1.33:50356: remote error: tls: bad certificate"
{"level":"warn","ts":"2020-11-24T16:56:12.094Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"passthrough:///https://127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
time="2020-11-24T16:56:12.094584665Z" level=info msg="Failed to test data store connection: context deadline exceeded"
WARNING: 2020/11/24 16:56:12 grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: context canceled". Reconnecting...
2020-11-24 16:56:12.094776 W | rafthttp: health check for peer 7a44c34067ffd15c could not connect: dial tcp 192.168.1.33:2380: connect: connection refused
2020-11-24 16:56:12.094971 W | rafthttp: health check for peer fa5fddcfd17240bf could not connect: dial tcp 192.168.1.32:2380: connect: connection refused
2020-11-24 16:56:12.095014 W | rafthttp: health check for peer fa5fddcfd17240bf could not connect: dial tcp 192.168.1.32:2380: connect: connection refused
2020-11-24 16:56:12.095023 W | rafthttp: health check for peer 7a44c34067ffd15c could not connect: dial tcp 192.168.1.33:2380: connect: connection refused
time="2020-11-24T16:56:16.781895127Z" level=info msg="Cluster-Http-Server 2020/11/24 16:56:16 http: TLS handshake error from 192.168.1.32:37220: remote error: tls: bad certificate"
raft2020/11/24 16:56:17 INFO: a155000d15c5b5b6 is starting a new election at term 260
raft2020/11/24 16:56:17 INFO: a155000d15c5b5b6 became candidate at term 261
raft2020/11/24 16:56:17 INFO: a155000d15c5b5b6 received MsgVoteResp from a155000d15c5b5b6 at term 261
raft2020/11/24 16:56:17 INFO: a155000d15c5b5b6 [logterm: 12, index: 38147148] sent MsgVote request to 7a44c34067ffd15c at term 261
raft2020/11/24 16:56:17 INFO: a155000d15c5b5b6 [logterm: 12, index: 38147148] sent MsgVote request to fa5fddcfd17240bf at term 261
2020-11-24 16:56:17.060854 E | etcdserver: publish error: etcdserver: request timed out
2020-11-24 16:56:17.094903 W | rafthttp: health check for peer 7a44c34067ffd15c could not connect: dial tcp 192.168.1.33:2380: connect: connection refused
2020-11-24 16:56:17.095085 W | rafthttp: health check for peer 7a44c34067ffd15c could not connect: dial tcp 192.168.1.33:2380: connect: connection refused
2020-11-24 16:56:17.095123 W | rafthttp: health check for peer fa5fddcfd17240bf could not connect: dial tcp 192.168.1.32:2380: connect: connection refused
2020-11-24 16:56:17.095153 W | rafthttp: health check for peer fa5fddcfd17240bf could not connect: dial tcp 192.168.1.32:2380: connect: connection refused
time="2020-11-24T16:56:17.369835065Z" level=info msg="Cluster-Http-Server 2020/11/24 16:56:17 http: TLS handshake error from 192.168.1.33:50360: remote error: tls: bad certificate"
raft2020/11/24 16:56:22 INFO: a155000d15c5b5b6 is starting a new election at term 261
raft2020/11/24 16:56:22 INFO: a155000d15c5b5b6 became candidate at term 262
raft2020/11/24 16:56:22 INFO: a155000d15c5b5b6 received MsgVoteResp from a155000d15c5b5b6 at term 262
raft2020/11/24 16:56:22 INFO: a155000d15c5b5b6 [logterm: 12, index: 38147148] sent MsgVote request to 7a44c34067ffd15c at term 262
raft2020/11/24 16:56:22 INFO: a155000d15c5b5b6 [logterm: 12, index: 38147148] sent MsgVote request to fa5fddcfd17240bf at term 262
2020-11-24 16:56:22.095004 W | rafthttp: health check for peer 7a44c34067ffd15c could not connect: dial tcp 192.168.1.33:2380: connect: connection refused
2020-11-24 16:56:22.095199 W | rafthttp: health check for peer fa5fddcfd17240bf could not connect: dial tcp 192.168.1.32:2380: connect: connection refused
2020-11-24 16:56:22.095222 W | rafthttp: health check for peer fa5fddcfd17240bf could not connect: dial tcp 192.168.1.32:2380: connect: connection refused
2020-11-24 16:56:22.095254 W | rafthttp: health check for peer 7a44c34067ffd15c could not connect: dial tcp 192.168.1.33:2380: connect: connection refused
time="2020-11-24T16:56:22.150479395Z" level=info msg="Cluster-Http-Server 2020/11/24 16:56:22 http: TLS handshake error from 192.168.1.32:37224: remote error: tls: bad certificate"
time="2020-11-24T16:56:22.862126958Z" level=info msg="Cluster-Http-Server 2020/11/24 16:56:22 http: TLS handshake error from 192.168.1.33:50364: remote error: tls: bad certificate"
{"level":"warn","ts":"2020-11-24T16:56:27.094Z","caller":"clientv3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"passthrough:///https://127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
time="2020-11-24T16:56:27.095056727Z" level=info msg="Failed to test data store connection: context deadline exceeded"
2020-11-24 16:56:27.095089 W | rafthttp: health check for peer 7a44c34067ffd15c could not connect: dial tcp 192.168.1.33:2380: connect: connection refused
WARNING: 2020/11/24 16:56:27 grpc: addrConn.createTransport failed to connect to {https://127.0.0.1:2379  <nil> 0 <nil>}. Err :connection error: desc = "transport: authentication handshake failed: context canceled". Reconnecting...
2020-11-24 16:56:27.095335 W | rafthttp: health check for peer fa5fddcfd17240bf could not connect: dial tcp 192.168.1.32:2380: connect: connection refused
2020-11-24 16:56:27.095382 W | rafthttp: health check for peer 7a44c34067ffd15c could not connect: dial tcp 192.168.1.33:2380: connect: connection refused
2020-11-24 16:56:27.095392 W | rafthttp: health check for peer fa5fddcfd17240bf could not connect: dial tcp 192.168.1.32:2380: connect: connection refused
time="2020-11-24T16:56:27.525220000Z" level=info msg="Cluster-Http-Server 2020/11/24 16:56:27 http: TLS handshake error from 192.168.1.32:37228: remote error: tls: bad certificate"
time="2020-11-24T16:56:28.350198803Z" level=info msg="Cluster-Http-Server 2020/11/24 16:56:28 http: TLS handshake error from 192.168.1.33:50368: remote error: tls: bad certificate"
raft2020/11/24 16:56:30 INFO: a155000d15c5b5b6 is starting a new election at term 262
raft2020/11/24 16:56:30 INFO: a155000d15c5b5b6 became candidate at term 263
raft2020/11/24 16:56:30 INFO: a155000d15c5b5b6 received MsgVoteResp from a155000d15c5b5b6 at term 263
raft2020/11/24 16:56:30 INFO: a155000d15c5b5b6 [logterm: 12, index: 38147148] sent MsgVote request to 7a44c34067ffd15c at term 263
raft2020/11/24 16:56:30 INFO: a155000d15c5b5b6 [logterm: 12, index: 38147148] sent MsgVote request to fa5fddcfd17240bf at term 263
2020-11-24 16:56:32.061171 E | etcdserver: publish error: etcdserver: request timed out
2020-11-24 16:56:32.095234 W | rafthttp: health check for peer 7a44c34067ffd15c could not connect: dial tcp 192.168.1.33:2380: connect: connection refused
2020-11-24 16:56:32.095436 W | rafthttp: health check for peer fa5fddcfd17240bf could not connect: dial tcp 192.168.1.32:2380: connect: connection refused
2020-11-24 16:56:32.095460 W | rafthttp: health check for peer 7a44c34067ffd15c could not connect: dial tcp 192.168.1.33:2380: connect: connection refused
2020-11-24 16:56:32.095481 W | rafthttp: health check for peer fa5fddcfd17240bf could not connect: dial tcp 192.168.1.32:2380: connect: connection refused
time="2020-11-24T16:56:32.890078087Z" level=info msg="Cluster-Http-Server 2020/11/24 16:56:32 http: TLS handshake error from 192.168.1.32:37232: remote error: tls: bad certificate"
dweomer picture dweomer on 24 Nov 2020

hi @dweomer

my workaround was to replace the k3s inside with the latest version (v1.19.4+ks31). I stopped / started in a random way and the cluster was healthy after a few minutes. I rebuild the ISO image and reinstalled all nodes.

linuxmail picture linuxmail on 24 Nov 2020

hi @dweomer

my workaround was to replace the k3s inside with the latest version (v1.19.4+ks31). I stopped / started in a random way and the cluster was healthy after a few minutes. I rebuild the ISO image and reinstalled all nodes.

yeah, i was hoping to push out a dev build with that very version. possibly today

dweomer picture dweomer on 24 Nov 2020
👍2

Please try https://github.com/rancher/k3os/releases/tag/v0.19.4-dev.5

dweomer picture dweomer on 24 Nov 2020
👍1

great!! will try and update the issue

dharmendrakariya picture dharmendrakariya on 25 Nov 2020

@dweomer Thanks. Tested with random poweroff nodes (all included) and it works !

linuxmail picture linuxmail on 25 Nov 2020

@dweomer hi, I checked, not getting any issue anymore. Thanks:) also closing this one.

dharmendrakariya picture dharmendrakariya on 26 Nov 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

larssb picture larssb  路  7Comments
MaximizeRr picture MaximizeRr  路  7Comments
runningman84 picture runningman84  路  3Comments
philipsparrow picture philipsparrow  路  4Comments
joharper54SEG picture joharper54SEG  路  5Comments