Kind: Kind fails with error "failed to create cluster: failed to init node with kubeadm: exit status 1"

[cippaciong]

What happened:
Kind doesn't create a cluster when running kind create cluster but instead exits with error message:

kind create cluster --retain
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.15.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Creating kubeadm config 📜
 ✗ Starting control-plane 🕹️
Error: failed to create cluster: failed to init node with kubeadm: exit status 1

What you expected to happen:
Kind creates the cluster

How to reproduce it (as minimally and precisely as possible):

• Install kind
• Run kind create cluster
• Might be highly dependant on my environment and as such hard to reproduce

Anything else we need to know?:

• Kind installed from AUR package kind-bin using the official binary released on GitHub
• I was initially using btrfs as storage driver, now I switched to overlay2 but the error persists

Environment:

• kind version: (use kind version): 0.5.1
• Kubernetes version: (use kubectl version): 1.15.2
• Docker version: (use docker info): 19.03.2-ce
• OS (e.g. from /etc/os-release): Arch Linux

Logs
kind_logs.tar.gz

[BenTheElder]

for the logs you posted I'm going to say https://kind.sigs.k8s.io/docs/user/known-issues/#docker-on-btrfs

are you sure you've managed to switch to overlay2? the backing filesystem cannot be btrfs, kubernetes and containerd/docker in docker don't work with this. something like ext4 + overlay.

however additionally:

do you have enough free ram?
from docker info:

 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.599GiB

kubeadm reccommends 2gb, however on docker for mac one-node kind works with the minimum settings (1gb / 512mb swap).

Can you run kind with increased logging level and give the output of kind create cluster?

[cippaciong]

are you sure you've managed to switch to overlay2? the backing filesystem cannot be btrfs, kubernetes and containerd/docker in docker don't work with this. something like ext4 + overlay.

Just to make sure I understand it correctly: setting overlay2 in /etc/docker/daemon.json won't solve the issue if the underlying file system is btrfs right?
If that's the case then that's definitely the problem here since my root file system uses btrfs.

I would exclude a RAM problem since I have almost 4GB available.

[BenTheElder]

I think if you check docker's logs it will continue using the btrfs driver (?)
I don't have any experience with this driver myself, but it has a number of known issues with docker-in-docker and kubernetes.

[cippaciong]

I did a little bit of research yesterday and it looks like overlay2 doesn't support a btrfs backing filesystem, even though docker info showed that I was using overlay2 and all my images were gone (as expected) and new images were created in /var/lib/docker/overlay2.

I decided to create an ext4 file system "on top" of btrfs using a sparse file and mounted /var/lib/docker there. Now kind is able to create the cluster :slightly_smiling_face:.

As you said, the problem was entirely btrfs and apparently setting overlay2 as storage driver doesn't make any difference if the backing filesystem is btrfs.

If you think this information should be added to the relevant known-issue, I would gladly create a pull request, just let me know :slightly_smiling_face:.

[vk496]

Definitely this should be remarked in the docs. I didn't consider the btrfs problem because docker info reported overlay2, even if I was on btrfs backend in my host system.

Exactly the same problem, exactly the same solution :)

[BenTheElder]

https://kind.sigs.k8s.io/docs/user/known-issues/#docker-on-btrfs

[cippaciong]

@BenTheElder that's not enough apparently. Even setting "storage-driver": "overlay2", if the underlying file system is btrfs you will see overlay2 when running docker info but kind, or more in general docker in docker, won't work.

[BenTheElder]

@cippaciong would you mind updating that entry? It was written by a contributor, I haven't personally tried to run docker or kubernetes on btrfs because they both have known issues upstream.