Kind: [Podman] cluster cannot be created due to `create session key: operation not permitted`

[AkihiroSuda]

What happened:
$ sudo -i KIND_EXPERIMENTAL_PROVIDER=podman ~/gopath/bin/kind create cluster failed

What you expected to happen:
It should work

How to reproduce it (as minimally and precisely as possible):

$ sudo -i KIND_EXPERIMENTAL_PROVIDER=podman ~/gopath/bin/kind create cluster                                                                                                                             
using podman due to KIND_EXPERIMENTAL_PROVIDER                                                                                                                                                                                                       
enabling experimental podman provider                                                                                                                                                                                                                
Creating cluster "kind" ...                                                                                                                                                                                                                          
 ✓ Ensuring node image (kindest/node:v1.19.3) 🖼                                                                                                                                                                                                      
 ✓ Preparing nodes 📦                                                                                                                                                                                                                                
 ✓ Writing configuration 📜                                                                                                                                                                                                                          
 ✗ Starting control-plane 🕹️                                                                                                                                                                                                                          
ERROR: failed to create cluster: failed to init node with kubeadm: command "podman exec --privileged kind-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1
...

$ sudo -i podman exec kind-control-plane journalctl -f
...
Nov 17 20:11:45 kind-control-plane kubelet[111]: E1117 20:11:45.992469     111 remote_runtime.go:113] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to create containerd task: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: join session keyring: create session key: operation not permitted: unknown
Nov 17 20:11:45 kind-control-plane kubelet[111]: E1117 20:11:45.992517     111 kuberuntime_sandbox.go:69] CreatePodSandbox for pod "kube-controller-manager-kind-control-plane_kube-system(7b2df35f618f1892763012ad45e50505)" failed: rpc error: code = Unknown desc = failed to create containerd task: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: join session keyring: create session key: operation not permitted: unknown
Nov 17 20:11:45 kind-control-plane kubelet[111]: E1117 20:11:45.992527     111 kuberuntime_manager.go:730] createPodSandbox for pod "kube-controller-manager-kind-control-plane_kube-system(7b2df35f618f1892763012ad45e50505)" failed: rpc error: code = Unknown desc = failed to create containerd task: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: join session keyring: create session key: operation not permitted: unknown
Nov 17 20:11:45 kind-control-plane kubelet[111]: E1117 20:11:45.992579     111 pod_workers.go:191] Error syncing pod 7b2df35f618f1892763012ad45e50505 ("kube-controller-manager-kind-control-plane_kube-system(7b2df35f618f1892763012ad45e50505)"), skipping: failed to "CreatePodSandbox" for "kube-controller-manager-kind-control-plane_kube-system(7b2df35f618f1892763012ad45e50505)" with CreatePodSandboxError: "CreatePodSandbox for pod \"kube-controller-manager-kind-control-plane_kube-system(7b2df35f618f1892763012ad45e50505)\" failed: rpc error: code = Unknown desc = failed to create containerd task: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: join session keyring: create session key: operation not permitted: unknown"
...

Anything else we need to know?:

It works fine with Docker.

Environment:

• kind version: (use kind version): kind v0.10.0-alpha go1.15.5 linux/amd64 (050064b358ef09a00362ec79361081d05db4c214 )
• Kubernetes version: (use kubectl version): Default (v1.19.3)
• Docker Podman version: 2.2.0-dev (containers/podman@65880e556314d1d8cb3cf12e20666b823eab8c76), with crun 0.15.1 and conmon 2.0.21
• OS (e.g. from /etc/os-release): Ubuntu 20.10, with cgroup v2 enabled

[BenTheElder]

cc @amwat @aojea

[BenTheElder]

sounds similar to https://github.com/kinvolk/kube-spawn/issues/326
/area provider/podman
also https://github.com/containers/podman/issues/7013#issuecomment-660646089 is interesting.

I don't currently have a podman development environment spun up, will punt to @amwat / @aojea for now.

aside: TIL https://github.com/opencontainers/runc/issues/453

[aojea]

@giuseppe @mheon does this rings a bell?

starting container process caused: process_linux.go:459: container init caused: join session keyring: create session key: operation not permitted: unknown"

[mheon]

I have never seen that one before - I'm assuming this is with the runc runtime?

[BenTheElder]

Yes this is runc / containerd "inside" the "node" container.

[aojea]

podman 2.1.1 works for me
Testing podman from master fails due to this https://github.com/containers/podman/issues/8444

 ✓ Installing StorageClass 💾 
ERROR: failed to create cluster: network details should only be one line, got 0 lines

, but it goes through the phase

✗ Starting control-plane 🕹️

@AkihiroSuda is it possible that the failure is related to use cgroupsv2?

OS (e.g. from /etc/os-release): Ubuntu 20.10, with cgroup v2 enabled

[danpawlik]

podman 2.1.1 works for me
Testing podman from master fails due to this containers/podman#8444

 ✓ Installing StorageClass 💾 
ERROR: failed to create cluster: network details should only be one line, got 0 lines

, but it goes through the phase

✗ Starting control-plane

@AkihiroSuda is it possible that the failure is related to use cgroupsv2?

OS (e.g. from /etc/os-release): Ubuntu 20.10, with cgroup v2 enabled

I got similar issue on Fedora 33 with podman 2.2.1 with kind node v.1.19.0 with kind binary in v0.9.0.
So I built new kind binary base on commit 'e36bf401e3d979' and the issue disappear.

$ kind --version
kind version 0.10.0-alpha+e36bf401e3d979
...
$ KIND_EXPERIMENTAL_PROVIDER=podman kind create cluster --image kindest/node:v1.20.0 --wait 2m --name my-test

using podman due to KIND_EXPERIMENTAL_PROVIDER
enabling experimental podman provider
Creating cluster "my-test" ...
 ✓ Ensuring node image (kindest/node:v1.20.0) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
 ✓ Waiting ≤ 2m0s for control-plane = Ready ⏳
 • Ready after 28s 💚