Kind: [btrfs] Failed to create cluster with http://localhost:10248/healthz connection refused, timed out waiting for the condition

[maxguru]

What happened:

• cloned kind repo on Gentoo
• built and ran kind using make build && bin/kind create cluster
• control-plane failed to start

$ make build && bin/kind create cluster
go build -v -o /home/al/apps/kind-repo/bin/kind -trimpath -ldflags="-buildid= -w -X=sigs.k8s.io/kind/pkg/cmd/kind/version.GitCommit=aebedcf8f4bf68a139b9572c0f18656cc1a9e429"
go: downloading github.com/spf13/pflag v1.0.5
go: downloading github.com/spf13/cobra v1.0.0
go: downloading k8s.io/apimachinery v0.19.2
go: downloading github.com/mattn/go-isatty v0.0.12
go: downloading github.com/alessio/shellescape v1.2.2
go: downloading github.com/pkg/errors v0.9.1
go: downloading sigs.k8s.io/yaml v1.2.0
go: downloading gopkg.in/yaml.v3 v3.0.0-20200615113413-eeeca48fe776
go: downloading github.com/evanphx/json-patch v4.9.0+incompatible
go: downloading golang.org/x/sys v0.0.0-20200928205150-006507a75852
go: downloading github.com/BurntSushi/toml v0.3.1
go: downloading github.com/pelletier/go-toml v1.8.1
go: downloading gopkg.in/yaml.v2 v2.2.8
go: downloading github.com/evanphx/json-patch/v5 v5.1.0
unicode/utf16
unicode/utf8
internal/race
internal/unsafeheader
math/bits
golang.org/x/sys/internal/unsafeheader
unicode
runtime/internal/sys
sigs.k8s.io/kind/pkg/apis/config/defaults
sigs.k8s.io/kind/pkg/cluster/constants
encoding
internal/nettrace
runtime/internal/atomic
sync/atomic
internal/cpu
sigs.k8s.io/kind/pkg/log
internal/goversion
runtime/internal/math
internal/testlog
internal/bytealg
math
runtime
internal/reflectlite
sync
internal/singleflight
math/rand
errors
sort
internal/oserror
io
vendor/golang.org/x/net/dns/dnsmessage
strconv
syscall
hash
bytes
strings
crypto
reflect
path
bufio
regexp/syntax
internal/syscall/unix
internal/syscall/execenv
time
regexp
context
internal/poll
github.com/alessio/shellescape
encoding/binary
internal/fmtsort
k8s.io/apimachinery/pkg/util/sets
os
encoding/base64
crypto/sha1
golang.org/x/sys/unix
internal/lazyregexp
path/filepath
fmt
net
io/ioutil
os/exec
sigs.k8s.io/kind/pkg/fs
internal/goroot
encoding/csv
encoding/hex
k8s.io/apimachinery/pkg/util/errors
github.com/pkg/errors
os/user
go/token
flag
k8s.io/apimachinery/pkg/util/version
net/url
encoding/json
text/template/parse
github.com/BurntSushi/toml
gopkg.in/yaml.v3
math/big
gopkg.in/yaml.v2
archive/tar
go/scanner
sigs.k8s.io/kind/pkg/errors
sigs.k8s.io/kind/pkg/exec
sigs.k8s.io/kind/pkg/apis/config/v1alpha4
github.com/mattn/go-isatty
go/ast
sigs.k8s.io/kind/pkg/internal/env
sigs.k8s.io/kind/pkg/cluster/nodes
text/template
sigs.k8s.io/kind/pkg/cluster/internal/logs
sigs.k8s.io/kind/pkg/build/nodeimage/internal/container/docker
sigs.k8s.io/kind/pkg/cluster/nodeutils
github.com/evanphx/json-patch/v5
sigs.k8s.io/kind/pkg/internal/apis/config
github.com/spf13/pflag
go/parser
sigs.k8s.io/yaml
github.com/pelletier/go-toml
sigs.k8s.io/kind/pkg/cluster/internal/kubeadm
go/doc
sigs.k8s.io/kind/pkg/cluster/internal/loadbalancer
sigs.k8s.io/kind/pkg/internal/apis/config/encoding
sigs.k8s.io/kind/pkg/cluster/internal/kubeconfig/internal/kubeconfig
go/build
sigs.k8s.io/kind/pkg/internal/cli
github.com/spf13/cobra
sigs.k8s.io/kind/pkg/cmd
sigs.k8s.io/kind/pkg/cluster/internal/providers
sigs.k8s.io/kind/pkg/cluster/internal/create/actions
sigs.k8s.io/kind/pkg/cluster/internal/kubeconfig
sigs.k8s.io/kind/pkg/cluster/internal/create/actions/installcni
sigs.k8s.io/kind/pkg/cluster/internal/create/actions/kubeadminit
sigs.k8s.io/kind/pkg/cluster/internal/create/actions/installstorage
sigs.k8s.io/kind/pkg/cluster/internal/create/actions/kubeadmjoin
sigs.k8s.io/kind/pkg/cluster/internal/create/actions/waitforready
sigs.k8s.io/kind/pkg/cluster/internal/delete
sigs.k8s.io/kind/pkg/build/nodeimage/internal/kube
sigs.k8s.io/kind/pkg/cluster/internal/patch
sigs.k8s.io/kind/pkg/build/nodeimage
sigs.k8s.io/kind/pkg/cmd/kind/completion/bash
sigs.k8s.io/kind/pkg/cmd/kind/completion/fish
sigs.k8s.io/kind/pkg/cmd/kind/version
sigs.k8s.io/kind/pkg/cmd/kind/completion/zsh
sigs.k8s.io/kind/pkg/cmd/kind/completion
sigs.k8s.io/kind/pkg/cluster/internal/providers/common
sigs.k8s.io/kind/pkg/cluster/internal/create/actions/config
sigs.k8s.io/kind/pkg/cluster/internal/create/actions/loadbalancer
sigs.k8s.io/kind/pkg/cluster/internal/providers/docker
sigs.k8s.io/kind/pkg/cluster/internal/providers/podman
sigs.k8s.io/kind/pkg/cmd/kind/build/nodeimage
sigs.k8s.io/kind/pkg/cluster/internal/create
sigs.k8s.io/kind/pkg/cmd/kind/build
sigs.k8s.io/kind/pkg/cluster
sigs.k8s.io/kind/pkg/internal/runtime
sigs.k8s.io/kind/pkg/cmd/kind/delete/clusters
sigs.k8s.io/kind/pkg/cmd/kind/create/cluster
sigs.k8s.io/kind/pkg/cmd/kind/get/kubeconfig
sigs.k8s.io/kind/pkg/cmd/kind/export/logs
sigs.k8s.io/kind/pkg/cmd/kind/get/nodes
sigs.k8s.io/kind/pkg/cmd/kind/delete/cluster
sigs.k8s.io/kind/pkg/cmd/kind/load/image-archive
sigs.k8s.io/kind/pkg/cmd/kind/get/clusters
sigs.k8s.io/kind/pkg/cmd/kind/export/kubeconfig
sigs.k8s.io/kind/pkg/cmd/kind/load/docker-image
sigs.k8s.io/kind/pkg/cmd/kind/get
sigs.k8s.io/kind/pkg/cmd/kind/export
sigs.k8s.io/kind/pkg/cmd/kind/create
sigs.k8s.io/kind/pkg/cmd/kind/delete
sigs.k8s.io/kind/pkg/cmd/kind/load
sigs.k8s.io/kind/pkg/cmd/kind
sigs.k8s.io/kind/cmd/kind/app
sigs.k8s.io/kind
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.19.4) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✗ Starting control-plane 🕹️ 
ERROR: failed to create cluster: failed to init node with kubeadm: command "docker exec --privileged kind-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1
Command Output: I1126 16:01:11.135893     145 initconfiguration.go:200] loading configuration from "/kind/kubeadm.conf"
[config] WARNING: Ignored YAML document with GroupVersionKind kubeadm.k8s.io/v1beta2, Kind=JoinConfiguration
W1126 16:01:11.148256     145 configset.go:348] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[init] Using Kubernetes version: v1.19.4
[certs] Using certificateDir folder "/etc/kubernetes/pki"
I1126 16:01:11.148476     145 certs.go:103] creating a new certificate authority for ca
[certs] Generating "ca" certificate and key
[certs] Generating "apiserver" certificate and key
[certs] apiserver serving cert is signed for DNS names [kind-control-plane kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost] and IPs [10.96.0.1 172.25.0.2 127.0.0.1]
[certs] Generating "apiserver-kubelet-client" certificate and key
I1126 16:01:11.664734     145 certs.go:103] creating a new certificate authority for front-proxy-ca
[certs] Generating "front-proxy-ca" certificate and key
[certs] Generating "front-proxy-client" certificate and key
I1126 16:01:11.979101     145 certs.go:103] creating a new certificate authority for etcd-ca
[certs] Generating "etcd/ca" certificate and key
[certs] Generating "etcd/server" certificate and key
[certs] etcd/server serving cert is signed for DNS names [kind-control-plane localhost] and IPs [172.25.0.2 127.0.0.1 ::1]
[certs] Generating "etcd/peer" certificate and key
[certs] etcd/peer serving cert is signed for DNS names [kind-control-plane localhost] and IPs [172.25.0.2 127.0.0.1 ::1]
[certs] Generating "etcd/healthcheck-client" certificate and key
[certs] Generating "apiserver-etcd-client" certificate and key
I1126 16:01:12.524154     145 certs.go:69] creating new public/private key files for signing service account users
[certs] Generating "sa" key and public key
[kubeconfig] Using kubeconfig folder "/etc/kubernetes"
I1126 16:01:12.634313     145 kubeconfig.go:100] creating kubeconfig file for admin.conf
[kubeconfig] Writing "admin.conf" kubeconfig file
I1126 16:01:12.726130     145 kubeconfig.go:100] creating kubeconfig file for kubelet.conf
[kubeconfig] Writing "kubelet.conf" kubeconfig file
I1126 16:01:12.843671     145 kubeconfig.go:100] creating kubeconfig file for controller-manager.conf
[kubeconfig] Writing "controller-manager.conf" kubeconfig file
I1126 16:01:12.967218     145 kubeconfig.go:100] creating kubeconfig file for scheduler.conf
[kubeconfig] Writing "scheduler.conf" kubeconfig file
I1126 16:01:13.173147     145 kubelet.go:63] Stopping the kubelet
[kubelet-start] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[kubelet-start] Starting the kubelet
I1126 16:01:13.237027     145 manifests.go:96] [control-plane] getting StaticPodSpecs
[control-plane] Using manifest folder "/etc/kubernetes/manifests"
[control-plane] Creating static Pod manifest for "kube-apiserver"
I1126 16:01:13.237542     145 manifests.go:109] [control-plane] adding volume "ca-certs" for component "kube-apiserver"
I1126 16:01:13.237552     145 manifests.go:109] [control-plane] adding volume "etc-ca-certificates" for component "kube-apiserver"
I1126 16:01:13.237558     145 manifests.go:109] [control-plane] adding volume "k8s-certs" for component "kube-apiserver"
I1126 16:01:13.237563     145 manifests.go:109] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-apiserver"
I1126 16:01:13.237568     145 manifests.go:109] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-apiserver"
[control-plane] Creating static Pod manifest for "kube-controller-manager"
I1126 16:01:13.244295     145 manifests.go:135] [control-plane] wrote static Pod manifest for component "kube-apiserver" to "/etc/kubernetes/manifests/kube-apiserver.yaml"
I1126 16:01:13.244310     145 manifests.go:96] [control-plane] getting StaticPodSpecs
I1126 16:01:13.244521     145 manifests.go:109] [control-plane] adding volume "ca-certs" for component "kube-controller-manager"
I1126 16:01:13.244528     145 manifests.go:109] [control-plane] adding volume "etc-ca-certificates" for component "kube-controller-manager"
I1126 16:01:13.244532     145 manifests.go:109] [control-plane] adding volume "flexvolume-dir" for component "kube-controller-manager"
I1126 16:01:13.244536     145 manifests.go:109] [control-plane] adding volume "k8s-certs" for component "kube-controller-manager"
I1126 16:01:13.244540     145 manifests.go:109] [control-plane] adding volume "kubeconfig" for component "kube-controller-manager"
I1126 16:01:13.244544     145 manifests.go:109] [control-plane] adding volume "usr-local-share-ca-certificates" for component "kube-controller-manager"
I1126 16:01:13.244547     145 manifests.go:109] [control-plane] adding volume "usr-share-ca-certificates" for component "kube-controller-manager"
[control-plane] Creating static Pod manifest for "kube-scheduler"
I1126 16:01:13.245312     145 manifests.go:135] [control-plane] wrote static Pod manifest for component "kube-controller-manager" to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
I1126 16:01:13.245324     145 manifests.go:96] [control-plane] getting StaticPodSpecs
I1126 16:01:13.245546     145 manifests.go:109] [control-plane] adding volume "kubeconfig" for component "kube-scheduler"
I1126 16:01:13.246054     145 manifests.go:135] [control-plane] wrote static Pod manifest for component "kube-scheduler" to "/etc/kubernetes/manifests/kube-scheduler.yaml"
[etcd] Creating static Pod manifest for local etcd in "/etc/kubernetes/manifests"
I1126 16:01:13.246493     145 local.go:82] [etcd] wrote Static Pod manifest for a local etcd member to "/etc/kubernetes/manifests/etcd.yaml"
I1126 16:01:13.246501     145 waitcontrolplane.go:87] [wait-control-plane] Waiting for the API server to be healthy
I1126 16:01:13.247043     145 loader.go:375] Config loaded from file:  /etc/kubernetes/admin.conf
[wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s
I1126 16:01:13.247827     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:13.748305     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:14.249318     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 1 milliseconds
I1126 16:01:14.748344     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:15.248333     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:15.748303     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:16.248264     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:16.748159     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:17.248329     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:17.748291     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:18.248302     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:18.748320     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:19.248288     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:19.748346     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:20.248234     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:20.748318     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:21.248202     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:21.748266     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:22.248199     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:22.748244     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:23.248275     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:23.748260     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:24.248355     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:24.748255     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:25.248235     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:25.748257     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:26.248279     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:26.748246     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:27.248244     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:27.748243     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:28.248304     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:28.748297     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:29.248289     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:29.748252     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:30.248196     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:30.748273     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:31.248253     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:31.748264     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:32.248229     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:32.748276     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:33.248289     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:33.748380     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:34.248192     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:34.748274     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:35.248305     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:35.748302     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:36.248281     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:36.748330     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:37.248217     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:37.748242     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:38.248301     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:38.748300     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:39.248249     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:39.748293     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:40.248256     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:40.748318     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:41.248242     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:41.748311     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:42.248241     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:42.748265     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:43.248276     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:43.748250     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:44.248229     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:44.748195     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:45.248235     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:45.748231     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:46.248326     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:46.748270     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:47.248244     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:47.748198     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:48.248221     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:48.748259     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:49.248219     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:49.748288     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:50.248252     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:50.748268     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:51.248280     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:51.748328     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:52.248241     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:52.748232     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
[kubelet-check] Initial timeout of 40s passed.
I1126 16:01:53.248174     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I1126 16:01:53.748231     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:54.248297     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:54.748190     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:55.248275     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:55.748286     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:56.248257     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:56.748337     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:57.248288     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:57.748251     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:58.248297     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I1126 16:01:58.748271     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:59.248246     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:01:59.748273     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:00.248325     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:00.748320     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:01.248252     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:01.748259     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:02.248307     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:02.748280     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:03.248282     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:03.748288     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:04.248289     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:04.748267     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:05.248273     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:05.748293     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:06.248274     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:06.748265     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:07.248278     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:07.748283     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:08.248373     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I1126 16:02:08.748298     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:09.248271     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:09.748295     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:10.248239     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:10.748284     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:11.248254     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:11.748257     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:12.248234     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:12.748221     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:13.248254     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:13.748198     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:14.248308     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:14.748222     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:15.248207     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:15.748244     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:16.248172     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:16.748348     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:17.248241     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:17.748237     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:18.248245     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:18.748231     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:19.248228     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:19.748284     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:20.248217     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:20.748239     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:21.248249     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:21.748239     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:22.248245     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:22.748208     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:23.248199     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:23.748243     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:24.248205     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:24.748236     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:25.248189     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:25.748250     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:26.248272     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:26.748232     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:27.248250     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:27.748267     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:28.248224     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.
I1126 16:02:28.748246     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:29.248162     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:29.748193     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:30.248231     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:30.748252     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:31.248249     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:31.748278     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:32.248274     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:32.748233     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:33.248239     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:33.748244     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:34.248202     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:34.748227     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:35.248241     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:35.748166     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:36.248271     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:36.748215     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:37.248202     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:37.748253     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:38.248268     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:38.748264     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:39.248204     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:39.748262     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:40.248156     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:40.748259     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:41.248256     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:41.748260     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:42.248224     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:42.748224     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:43.248213     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:43.748216     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:44.248237     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:44.748246     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:45.248234     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:45.748247     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:46.248221     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:46.748206     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:47.248259     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:47.748213     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:48.248216     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:48.748207     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:49.248205     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:49.748189     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:50.248241     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:50.748224     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:51.248207     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:51.748241     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:52.248245     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:52.748270     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:53.248231     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:53.748266     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:54.248221     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:54.748198     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:55.248208     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:55.748248     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:56.248255     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:56.748257     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:57.248197     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:57.748224     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:58.248248     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:58.748222     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:59.248261     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:02:59.748239     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:03:00.248242     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:03:00.748196     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:03:01.248218     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:03:01.748188     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:03:02.248237     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:03:02.748197     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:03:03.248234     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:03:03.748236     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:03:04.248317     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:03:04.748256     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:03:05.248232     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:03:05.748246     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:03:06.248238     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:03:06.748207     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:03:07.248231     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:03:07.748182     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
I1126 16:03:08.248213     145 round_trippers.go:443] GET https://kind-control-plane:6443/healthz?timeout=10s  in 0 milliseconds
[kubelet-check] It seems like the kubelet isn't running or healthy.
[kubelet-check] The HTTP call equal to 'curl -sSL http://localhost:10248/healthz' failed with error: Get "http://localhost:10248/healthz": dial tcp [::1]:10248: connect: connection refused.

        Unfortunately, an error has occurred:
                timed out waiting for the condition

        This error is likely caused by:
                - The kubelet is not running
                - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled)

        If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands:
                - 'systemctl status kubelet'
                - 'journalctl -xeu kubelet'

        Additionally, a control plane component may have crashed or exited when started by the container runtime.
        To troubleshoot, list all containers using your preferred container runtimes CLI.

        Here is one example how you may list all Kubernetes containers running in cri-o/containerd using crictl:
                - 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock ps -a | grep kube | grep -v pause'
                Once you have found the failing container, you can inspect its logs with:
                - 'crictl --runtime-endpoint unix:///run/containerd/containerd.sock logs CONTAINERID'

couldn't initialize a Kubernetes cluster
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init.runWaitControlPlanePhase
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/init/waitcontrolplane.go:114
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:234
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:422
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.NewCmdInit.func1
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/init.go:149
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:842
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:950
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:887
k8s.io/kubernetes/cmd/kubeadm/app.Run
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50
main.main
        _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
        /usr/local/go/src/runtime/proc.go:204
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1374
error execution phase wait-control-plane
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run.func1
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:235
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).visitAll
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:422
k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow.(*Runner).Run
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/phases/workflow/runner.go:207
k8s.io/kubernetes/cmd/kubeadm/app/cmd.NewCmdInit.func1
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/cmd/init.go:149
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:842
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:950
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:887
k8s.io/kubernetes/cmd/kubeadm/app.Run
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/app/kubeadm.go:50
main.main
        _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubeadm/kubeadm.go:25
runtime.main
        /usr/local/go/src/runtime/proc.go:204
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1374

What you expected to happen:

• control-plane should have started

How to reproduce it (as minimally and precisely as possible):

• git clone [email protected]:kubernetes-sigs/kind.git kind-repo
• cd kind-repo
• make build && bin/kind create cluster

Anything else we need to know?:

• Bug might be related to issues #1709, #1735, #1891.
• Same problem occurs with tags v0.9.0, v0.8.1, v0.8.0, v0.7.0. I haven't tried anything older.

Environment:

• kind version: (use kind version): kind v0.10.0-alpha+aebedcf8f4bf68 go1.15.5 linux/amd64
• Kubernetes version: (use kubectl version): v1.19.3
• Docker version: (use docker info): 19.03.13
• OS (e.g. from /etc/os-release): Gentoo (recently updated)

[BenTheElder]

Please check the known issues page in the docs.

You can obtain more logs for us to debug with kind create cluster --retain and then kind export logs.

Can you share docker info?

Basically from the logs so far we know API server didn't come up healthy. There are any number of possible causes. The only known kind bug is related to ZFS (should be fixed at HEAD) but there are known broken environments (e.g. the firewalld issue)

[BenTheElder]

Will reopen when we have more details. As-is this is not actionable for ~one week now.
Sorry for your trouble. If you get time again sometime please share more debug logs and environment details.

[maxguru]

I'm sorry for the delay.

I looked at known problems but I don't see anything immediately obvious.

The docker info info seems to be already dumped with kind create cluster --retain.

See 345396429.tar.gz.

[BenTheElder]

It looks like you're running on btrfs, I'm not surprised to see that isn't working.

Kubernetes etc. are rarely used on these COW filesystems versus overlayfs on e.g. ext4.

and indeed there's a rather large panic in the kubelet.log starting with:

Nov 27 23:29:21 kind-control-plane kubelet[2131]: F1127 23:29:21.852946 2131 kubelet.go:1296] Failed to start ContainerManager failed to get rootfs info: failed to get device for dir "/var/lib/kubelet": could not find device with major: 0, minor: 30 in cached partitions map

[BenTheElder]

We do make a best effort to support it, but I don't think any of the maintainers use btrfs / have a test env handy and Kubernetes CI definitely doesn't run on it.

https://github.com/kubernetes-sigs/kind/commit/d198c7af5b12176c97408c46ee74e4baece4638f

I thought this was sufficient, apparently not 😞

[BenTheElder]

Actually, it looks like we're not passing /dev/mapper based on the inspect.json in your log export so the check to mount that must be broken.

You can manually configure this as a possible workaround while we work on fixing the check.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- extraMounts:
  - hostPath: /dev/mapper
    containerPath: /dev/mapper

like:

cat << EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- extraMounts:
  - hostPath: /dev/mapper
    containerPath: /dev/mapper
EOF

[BenTheElder]

Did you configure docker at all? It's really strange that you have storage driver = overlay2 while backing filesystem = btrfs.

https://docs.docker.com/storage/storagedriver/select-storage-driver/#supported-backing-filesystems

btrfs : btrfs is supported, overlay2/overlay support xfs, ext4.

The overlay2 but actually btrfs is tripping up our check to mount /dev/mapper, but I'm surprised this is even a working docker environment.

EDIT: relevant excerpt from docke- info.txt in the log export you uploaded:

 Storage Driver: overlay2
  Backing Filesystem: btrfs
  Supports d_type: true
  Native Overlay Diff: false

[maxguru]

I'm a bit confused. Are we talking about btrfs storage driver or something else? Because my rootfs is btrfs. I didn't mess with changing the default storage driver in docker. So /var/lib/docker is on a btrfs formatted partition. I guess that's what 'backing filesystem' means.

I've been running docker containers like this for a long time and didn't have any issues before. Should I move it to ext4?

I tried the provided config file and it seems to be working:

$ ./kind create cluster --config kind-dev-mapper-test.yml 
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.19.1) 🖼
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/
$ kubectl cluster-info --context kind-kind
Kubernetes master is running at https://127.0.0.1:39709
KubeDNS is running at https://127.0.0.1:39709/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

Thanks for your help!

[BenTheElder]

I'm a bit confused. Are we talking about btrfs storage driver or something else? Because my rootfs is btrfs. I didn't mess with changing the default storage driver in docker. So /var/lib/docker is on a btrfs formatted partition. I guess that's what 'backing filesystem' means.

So usually when docker is on btrfs backing filesystem it will use the btrfs driver, or at least that's the documented behavior which kind is expecting. I'm trying to figure out if your system is configured otherwise or if docker's docs are misleading, I didn't expect to see overlay driver on btrfs.

If this is working fine we shouldn't make you change your environment config! We should improve kind to detect this case as also requiring /dev/mapper passthrough just like when zfs or btrfs are the docker driver. 😅

[maxguru]

My rootfs used to be ext4. I moved it over to btrfs a while back. I didn't realize I needed to change anything.

[BenTheElder]

Ohh interesting. This may have confused docker into using overlay on btrfs even though their docs don't show that to be supported. It seems it works though?

So we should maybe still alter the check in kind for auto-mounting /dev/mapper 😅

[maxguru]

Right, it was working fine for maybe a year except the issue with kind. I recreated /var/lib/docker and it seems to have switched to btrfs storage driver. Kind works now without the special config file.

[Napoleon0621]

@BenTheElder Hi Ben,
I'm trying to create a multi-node cluster and had that same timeout error during "Joining worker nodes" stage.
It seems that it has less to do with btrfs, because I can successfully create a cluster with 5 nodes, but failed at larger scale, ex.10 nodes.
The error message is almost the same with maxguru's, except for I did had a time delay connecting with api server range from hundred ms to thousand ms.

[BenTheElder]

@Napoleon0621 please file a distinct issue in the future so we can keep bug tracking organized.

If you're failing at a certain node count you likely just don't have enough resources to support this many. IOPs is one that frequently bites people that otherwise have plenty of ram and cpu. You can try putting etcd in memory (search the issue tracker) but eventually any machine will run out of IO and parts of Kubernetes dependencies are heavy on it.

[maxguru]

I was able to bring up 10 node cluster without any issues myself.

[aojea]

/close

I was able to bring up 10 node cluster without any issues myself.

I think that the original problem was solved, we can open new ones to have the issues organised so users can look for the information or find it easily

[k8s-ci-robot]

@aojea: Closing this issue.

In response to this:

/close

I was able to bring up 10 node cluster without any issues myself.

I think that the original problem was solved, we can open new ones to have the issues organised so user can look for the information

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[BenTheElder]

@aojea this is not really fixed, we need a more intelligent check for mounting mapper, even if docker doesn't officially list this as supported it seems to happen to multiple users AND continue working otherwise.

[aojea]

is this related @BenTheElder ?
https://github.com/kubernetes/kubernetes/issues/95826

[BenTheElder]

@aojea the kubernetes PR could be an issue but the problem here is this https://github.com/kubernetes-sigs/kind/issues/1945#issuecomment-737728387

we currently detect when to mount /dev/mapper based on docker's storage driver. we need to detect based on the backing filesystem as well (which is more annoying to parse it's like a list of status objects IIRC ...)