Minikube: Podman v2 breaks the podman driver

[afbjorklund]

The podman driver is still broken, with v2.0.4 and 2.0.5:

new client: new client: Error creating new ssh host from driver: Error getting ssh port for driver: get ssh host-port: convert host-port '\x00' to number: strconv.Atoi: parsing "": invalid syntax

This is a regression/follow-up from previous #8587 (2.0.1)

The workaround is still to use the v1.9.3 release instead.

The goal is to remove the need for current warning #8783

❗ Using podman 2 is not supported yet. your version is "2.0.5". minikube might not work. use at your own risk.

---

Note that podman 1 is no longer supported upstream:
https://podman.io/blogs/2020/08/01/deprecate-and-remove-varlink-notice.html

We also plan to drop support from the RHEL8.4 release, spring 2021

And that RHEL 8 does not support docker, only podman.
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html-single/building_running_and_managing_containers/index#running-containers-without-docker_building-running-and-managing-containers

For RHEL 8, Docker is not included and not supported by Red Hat

This basically means that you can not run KIC on Red Hat Enterprise Linux.

Fedora still includes Docker (Moby), but doesn't work because of cgroups v2.

[afbjorklund]

This basically means that you can not run KIC on Red Hat Enterprise Linux.

Arguably nobody cares, since Red Hat users tend to use OpenShift anyway...

Fedora still includes Docker (Moby), but doesn't work because of cgroups v2.

Currently this means that you need to use KVM, or change the grub defaults.

[afbjorklund]

Was logging the wrong error, due to the auto-restart feature hiding stuff. :angry:

Root cause is that the NetworkSettings.Ports changed from array to map:

 sudo -n podman1 container inspect -f "{{.NetworkSettings.Ports}}" minikube
[{42439 8443 tcp 127.0.0.1} {41827 22 tcp 127.0.0.1} {37637 2376 tcp 127.0.0.1} {46263 5000 tcp 127.0.0.1}]
 sudo -n podman2 container inspect -f "{{.NetworkSettings.Ports}}" minikube
map[22/tcp:[{127.0.0.1 41827}] 2376/tcp:[{127.0.0.1 37637}] 5000/tcp:[{127.0.0.1 46263}] 8443/tcp:[{127.0.0.1 42439}]]

https://github.com/containers/podman/commit/639b809c80dc2fce76c4d24459d06c469777868f

This was to make it more compatible with Docker, while also breaking Podman v1.

So we need to use the docker code for 2.0.1+, but the old podman code for 1.x+2.0.0.

 sudo -n podman container inspect -f "{{range .NetworkSettings.Ports}}{{if eq .ContainerPort 22}}{{.HostPort}}{{end}}{{end}}" minikube
ERRO[0000] Error printing inspect output: Template parsing error: template: image:1:40: executing "image" at <.ContainerPort>: can't evaluate field ContainerPort in type []define.InspectHostPort
 sudo podman container inspect -f '{{(index (index .NetworkSettings.Ports "22/tcp") 0).HostPort}}' minikube
41827

https://github.com/containers/podman/issues/5380

[afbjorklund]

Seems to be mostly working now, but it still fails to remove the volume sometimes...

❗  Using podman 2 is not supported yet. your version is "2.0.6". minikube might not work. use at your own risk.
✨  Using the podman (experimental) driver based on user configuration
👍  Starting control plane node minikube in cluster minikube
🔥  Creating podman container (CPUs=2, Memory=7900MB) ...
🤦  StartHost failed, but will try again: creating host: create: creating: setting up container node: creating volume for minikube container: sudo -n podman volume create minikube --label name.minikube.sigs.k8s.io=minikube --label created_by.minikube.sigs.k8s.io=true: exit status 125
stdout:

stderr:
Error: volume with name minikube already exists: volume already exists

🔄  Restarting existing podman container for "minikube" ...
😿  Failed to start podman container. Running "minikube delete" may fix it: podman inspect ip minikube: sudo -n podman container inspect -f {{.NetworkSettings.IPAddress}} minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container minikube


❌  Exiting due to GUEST_PROVISION: Failed to start host: podman inspect ip minikube: sudo -n podman container inspect -f  minikube: exit status 125
stdout:

stderr:
Error: error inspecting object: no such container minikube

The workaround is to remove it manually, if so: sudo podman volume rm minikube

[afbjorklund]

From what I can see, CentOS 8.2.2004 still featured Podman 1.6.3
(CentOS 8.0.1905 and CentOS 8.0.1911 both had various "1.0.0")

And the RHEL 8.3 Beta still has Podman "2.0.0" (might end up 2.0.5 ?)
For some reason it (podman2) is not in "CentOS Stream" yet, though ?

[afbjorklund]

Verified that is still works with the new podman 2.1(only RC so far)