K3s: cluster-init: [ERROR]: probe I/O capabilities: can't allocate write buffer

[caleno]

Version:

Describe the bug
Starting a new cluster with embedded db fails with error message
02:00:30.442 [ERROR]: probe I/O capabilities: can't allocate write buffer
FATA[2019-11-25T14:36:08.565595250Z] starting kubernetes: failed to create task object

To Reproduce
Run: 'K3S_TOKEN=hilfe k3s server --cluster-init' (on a Centos 7 server (?))

Expected behavior
It should k3s cluster with dqlite as backend which I can join more servers nodes to.

Actual behavior
K3S_TOKEN=hilfe k3s server --cluster-init
INFO[2019-11-25T15:50:05.381319277+01:00] Starting k3s v1.0.0 (18bd921c)
03:23:31.137 [DEBUG]: data dir: /var/lib/rancher/k3s/server/db/state.dqlite
03:23:31.137 [DEBUG]: metadata1: version 191, term 0, voted for 0
03:23:31.137 [DEBUG]: metadata2: version 192, term 0, voted for 0
03:23:31.137 [DEBUG]: metadata: version 194, term 0, voted for 0
03:23:31.137 [ERROR]: probe I/O capabilities: can't allocate write buffer
FATA[2019-11-25T15:50:05.399366856+01:00] starting kubernetes: failed to create task object

Additional context
Ran same command on two different servers, same problem.
Server1: CentOS Linux release 7.4.1708 (Core)
Server2: CentOS Linux release 7.7.1908 (Core)

[bigl0af]

I'm having the exact same problem on Centos7 as well...

[ruskofd]

Same here, same CentOS 7 version

[gitrootid]

### Same here, inite Ubuntu 18.04.3 LTS and try to join with Ubuntu 16.04.6 LTS ,and
docker version
Client:
Version: 18.06.3-ce
API version: 1.38
Go version: go1.10.3
Git commit: d7080c1
Built: Wed Feb 20 02:27:18 2019
OS/Arch: linux/amd64
Experimental: false

Server:
Engine:
Version: 18.06.3-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: d7080c1
Built: Wed Feb 20 02:26:20 2019
OS/Arch: linux/amd64
Experimental: false

### log:

#$ K3S_TOKEN=SECRET k3s server --docker --server https://10.0.1.68:6443

INFO[0000] Preparing data dir /var/lib/rancher/k3s/data/2492a28b74abfbdbd3f943c8109a93e979669647bbbe95eadae43b60c667df69
INFO[2019-11-28T14:47:56.282408598+08:00] Starting k3s v1.0.0 (18bd921c)
00:13:44.749 [DEBUG]: data dir: /var/lib/rancher/k3s/server/db/state.dqlite
00:13:44.749 [DEBUG]: metadata1: version 0, term 0, voted for 0
00:13:44.749 [DEBUG]: metadata2: version 0, term 0, voted for 0
00:13:44.749 [DEBUG]: metadata: version 2, term 0, voted for 0
00:13:44.749 [ERROR]: probe I/O capabilities: can't allocate write buffer
FATA[2019-11-28T14:47:56.392257744+08:00] starting kubernetes: failed to create task object

[gitrootid]

the inite-cluster server log:

W1128 06:45:54.041551 10741 pod_container_deletor.go:75] Container "9a0d7df36e50318c52ceaf25089927468cd94d7df82e6a4bf627f789a291a6b8" not found in pod's containers
I1128 06:46:49.844151 10741 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I1128 06:47:49.850532 10741 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
I1128 06:47:56.319868 10741 log.go:172] http: TLS handshake error from 10.0.1.99:43590: remote error: tls: bad certificate
I1128 06:47:56.341660 10741 log.go:172] http: TLS handshake error from 10.0.1.99:43598: remote error: tls: bad certificate
I1128 06:48:49.859101 10741 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io

[caleno]

I managed to get it working on Ubuntu 18.04.3 LTS

[geertvanheusden]

@caleno Could you please explain us how you fixed it? ;-)

[davideshay]

Seeing the same issue on Ubuntu 18.0.4.3 LTS and Debian 10 (odroid N2 image) both on arm64.
@caleno, any more details on resolution?

[bushchannelstepper]

Also seeing this on Slackware 14.2 with OpenRC

[bushchannelstepper]

FWIW Alpine 3.11.2 extended (also OpenRC) is not displaying this issue

[ethan-reynolds]

Same thing here. I haven't been able to track down the source for the error message, but it looks like something in the libuv I/O library or the raft library?
I wasn't aware that k3s now defaults to dqlite over sqlite in non-ha clusters, but that's the assumption I reach based on the location of the raft related code in k3s.

running under Hyper-V, 2048MB RAM, static IP assigned.
initial cluster bootstrap works fine, attempting to use --cluster-init seems to corrupt/ modify something before the process exits.

CentOS Linux release 7.7.1908 (Core)
Linux centos2 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
SELinux status: disabled
firewalld disabled

$ curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="--node-external-ip 141.73.90.64" sh -s -
successful

$ systemctl stop k3s
successful

$ systemctl start k3s
successful

$ systemctl stop k3s
successful

$ k3s server --cluster-init --node-external-ip 141.73.90.64
INFO[2020-01-03T09:19:16.864609700-05:00] Starting k3s v1.0.1 (e94a3c60)
01:10:03.126 [DEBUG]: data dir: /var/lib/rancher/k3s/server/db/state.dqlite
01:10:03.126 [DEBUG]: metadata1: version 7, term 0, voted for 0
01:10:03.126 [DEBUG]: metadata2: version 8, term 0, voted for 0
01:10:03.126 [DEBUG]: metadata: version 10, term 0, voted for 0
01:10:03.126 [ERROR]: probe I/O capabilities: can't allocate write buffer
FATA[2020-01-03T09:19:16.896075600-05:00] starting kubernetes: failed to create task object

$ systemctl start k3s
Job for k3s.service failed because the control process exited with error code. See "systemctl status k3s.service" and "journalctl -xe" for details.

$ journalctl -xe
...
-- Unit k3s.service has begun starting up.
Jan 03 09:21:42 centos2 k3s[8887]: time="2020-01-03T09:21:42.578847900-05:00" level=info msg="Starting k3s v1.0.1 (e94a3c60)"
Jan 03 09:21:42 centos2 k3s[8887]: 01:12:28.844 [DEBUG]: data dir: /var/lib/rancher/k3s/server/db/state.dqlite
Jan 03 09:21:42 centos2 k3s[8887]: time="2020-01-03T09:21:42.614008500-05:00" level=fatal msg="starting kubernetes: failed to create task
Jan 03 09:21:42 centos2 systemd[1]: k3s.service: main process exited, code=exited, status=1/FAILURE
Jan 03 09:21:42 centos2 systemd[1]: Failed to start Lightweight Kubernetes.
...

[BruceWayneVsSuperMan]

@ethan-reynolds Same with you, on CentOS 7 version