Kind: Failure in joining worker nodes while creating kind cluster

[leelavg]

What happened:

• Not able to create a cluster with 3 worker nodes:

~/Downloads/k8s $ kind create cluster --image kindest/node:v1.18.2 --config kind-config.yml                                                                                                   
Creating cluster "kind" ...                                                                                                                                                                   
 ✓ Ensuring node image (kindest/node:v1.18.2) 🖼                                                                                                                                               
 ✓ Preparing nodes 📦 📦 📦 📦                                                                                                                                                                
 ✓ Writing configuration 📜                                                                                                                                                                   
 ✓ Starting control-plane 🕹️                                                                                                                                                                   
 ✓ Installing CNI 🔌                                                                                                                                                                          
 ✓ Installing StorageClass 💾                                                                                                                                                                 
 ✗ Joining worker nodes 🚜                                                                                                                                                                    
ERROR: failed to create cluster: failed to join node with kubeadm: command "docker exec --privileged kind-worker3 kubeadm join --config /kind/kubeadm.conf --ignore-preflight-errors=all --v=6
" failed with error: exit status 1                                                                                                                                                            
Command Output: W0605 03:48:00.642098     459 join.go:346] [preflight] WARNING: JoinControlPane.controlPlane settings will be ignored when control-plane flag is not set.                     
I0605 03:48:00.642208     459 join.go:371] [preflight] found NodeName empty; using OS hostname as NodeName
[Remaining output truncated]

What you expected to happen:

• kind cluster should be created successfully

How to reproduce it (as minimally and precisely as possible):

• create a tar ball from image kindest/node:v1.18.2@sha256:7b27a6d0f2517ff88ba444025beae41491b016bc6af573ba467b70c5e8e0d85f
• load image from tar ball into docker and tag it appropriately
• create kind cluster $ kind create cluster --image kindest/node:v1.18.2 --config kind-config.yml
• observe the error while joining the worker nodes
• below is the kind-config.yml
  ```kind: Cluster
  apiVersion: kind.x-k8s.io/v1alpha4
  nodes:
• role: control-plane
• role: worker
• role: worker
• role: worker

**Anything else we need to know?**:
- able to create cluster with **no** worker nodes successfully
- able to create cluster with worker nodes before system upgrade
- done a fresh install of `golang, docker, kind` post upgrade to narrow out any issues due to system upgrade

**Environment:**

- kind version: (use `kind version`):
`kind v0.8.1 go1.14.2 linux/amd64`

- Kubernetes version: (use `kubectl version`):
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"clean", BuildDate:"2020-05-20T12:52:00Z", GoVersion:"go1.13.9", Compiler:"gc", Platform:"linux/amd64"}
The connection to the server localhost:8080 was refused - did you specify the right host or port?

- Docker version: (use `docker info`):
```Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: journald
 Cgroup Driver: systemd
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: /usr/libexec/docker/docker-init
 containerd version: 
 runc version: fbdbaf85ecbc0e077f336c03062710435607dbf1
 init version: 
 Security Options:
  seccomp
   Profile: default
  selinux
 Kernel Version: 5.6.13-300.fc32.x86_64
 Operating System: Fedora 32 (Workstation Edition)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 14.86GiB
 Name: glv
 ID: SRED:GURW:YTDW:CF6L:TDLK:IGRO:TGGB:TJZC:SAYZ:BXSA:OLCQ:UJ42
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: true

• OS (e.g. from /etc/os-release):
  NAME=Fedora VERSION="32 (Workstation Edition)" ID=fedora VERSION_ID=32 VERSION_CODENAME="" PLATFORM_ID="platform:f32" PRETTY_NAME="Fedora 32 (Workstation Edition)" ANSI_COLOR="0;34" LOGO=fedora-logo-icon CPE_NAME="cpe:/o:fedoraproject:fedora:32" HOME_URL="https://fedoraproject.org/" DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f32/system-administrators-guide/" SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help" BUG_REPORT_URL="https://bugzilla.redhat.com/" REDHAT_BUGZILLA_PRODUCT="Fedora" REDHAT_BUGZILLA_PRODUCT_VERSION=32 REDHAT_SUPPORT_PRODUCT="Fedora" REDHAT_SUPPORT_PRODUCT_VERSION=32 PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy" VARIANT="Workstation Edition" VARIANT_ID=workstation
• GO version:
  go version go1.14.2 linux/amd64

Please let me know if any additional info is required.

[BenTheElder]

Your host may not have enough resources.

Can you create a single cluster with one control plane and one worker?
Can you upload the full logs?

[BenTheElder]

I see you're on fedora 32, Did you look at our open issues at all? https://github.com/kubernetes-sigs/kind/issues/1547

[leelavg]

• before raising the issue I looked at known issues, github issues but searched for joining worker nodes etc
• suspecting this issue is related to recent upgrade, i mentioned related info in questionnaire
• my bad not searching specifically for fedora/OS, anyways thanks for pointing in the right direction and changing firewalld.conf solved the issue.