Podman: "Could not get runtime: unknown event logger type" on Fedora 31 Beta Silverblue

[Zlopez]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
Various podman commands fails with Error: could not get runtime: unknown event logger type: on Fedora 31 Beta Silverblue

Steps to reproduce the issue:

1. podman ps

Describe the results you received:
Error: could not get runtime: unknown event logger type:

Describe the results you expected:
List of containers

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Error: could not get runtime: unknown event logger type:

but output of podman --version:
podman version 1.6.1

Output of podman info --debug:

Error: could not get runtime: unknown event logger type:

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.6.1-2.fc31.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):
Physical machine

[mheon]

Running as root, or rootless?

Can you provide your libpod.conf? As root, /etc/containers/libpod.conf or /usr/share/libpod.conf; without root, $HOME/.config/containers/libpod.conf.

[Zlopez]

I'm running rootless podman, here is the libpod.conf

olume_path = "/var/home/zlopez/.local/share/containers/storage/volumes"
image_default_transport = "docker://"
runtime = "/usr/bin/crun"
runtime_path = ["/usr/bin/runc", "/usr/sbin/runc", "/usr/local/bin/runc", "/usr/local/sbin/runc", "/sbin/runc", "/bin/runc", "/usr/lib/cri-o-runc/sbin/runc"]
conmon_path = ["/usr/libexec/podman/conmon", "/usr/libexec/crio/conmon", "/usr/local/lib/podman/conmon", "/usr/local/libexec/crio/conmon", "/usr/bin/conmon", "/usr/sbin/conmon", "/usr/lib/crio/bin/conmon"]
conmon_env_vars = ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"]
cgroup_manager = "cgroupfs"
init_path = "/usr/libexec/podman/catatonit"
static_dir = "/var/home/zlopez/.local/share/containers/storage/libpod"
tmp_dir = "/run/user/1000/libpod/tmp"
max_log_size = -1
no_pivot_root = false
cni_config_dir = "/etc/cni/net.d/"
cni_plugin_dir = ["/usr/libexec/cni", "/usr/lib/cni", "/usr/local/lib/cni", "/opt/cni/bin"]
infra_image = "k8s.gcr.io/pause:3.1"
infra_command = "/pause"
enable_port_reservation = true
label = true
network_cmd_path = ""
num_locks = 0 
events_logger = ""
events_logfile_path = ""
detach_keys = ""
SDNotify = false
cgroup_check = true

[mheon]

Your config seems to be missing two fields that I have related to events.

events_logger = "file"
EventsLogFilePath = ""

Try adding those to the end your your user's libpod.conf

[Zlopez]

I tried to add the two lines and got this

Error: could not get runtime: error decoding configuration file /home/zlopez/.config/containers/libpod.conf: Near line 26 (last key parsed 'events_logger'): Key 'events_logger' has already been defined.

[mheon]

Ahh - they're already in there, between num_locks and detach_keys.

You can remove the new ones you added and just modify the existing ones - events_logger to "file" should be sufficient

[Zlopez]

Added "file" to events_logger and got this

Error: could not get runtime: no valid executable found for OCI runtime runc: invalid argument

[mheon]

At this point, I would recommend removing $HOME/.config/containers/libpod.conf entirely, so Podman will automatically re-generate it from the root defaults - that should get you a working system.

@rhatdan FYI - it looks like the migration code ran, but he doesn't have the new runtime paths sections (older Podman generated this config), so the migration to crun failed.

[Zlopez]

@mheon This did the trick, thanks.

[GoodMirek]

Same issue here, happened after upgrade from FC30 Workstation (dnf based) to FC31 Workstation (dnf based).
After removing $HOME/.config/containers/libpod.conf entirely, i can run podman version, but cannot start any of my old pet containers (e.g. toolbox) and have to remove and recreate them.

[Zlopez]

@GoodMirek There was some regression in podman (https://github.com/containers/toolbox/issues/277) which unfortunately prevented toolbox enter on old containers. I needed to create a new one when this issue was fixed.

[GoodMirek]

@Zlopez Thanks for your very fast response. Actually, I had to remove all my old pet containers, not only toolbox ones, e.g. also pgadmin4. Sure I can manage that and I intentionally test beta versions to find issues, but if that happened to regular users after upgrade from FC30, once FC31 is Generally Available, it would not be the best user experience.

[Zlopez]

@GoodMirek I absolutely agree with you.

There are two issues that will be unfortunate for any user upgrading to F31. This one, where you need to regenerate $HOME/.config/containers/libpod.conf and the regression in podman create which will prevent you to use any previously created container. I personally thought that this was solved together with containers/toolbox#277, but it seems that this affects every user who does update from F30 to F31. I'm not sure if there is time to actually do anything about it when the release of F31 is closing up.

[mheon]

We're planning a new Podman release with solutions to most migration
issues. Hopefully these issues are serious enough for a freeze exception
and we'll be able to get it into the F31 final compose. If not, it'll be a
very early update.

On Wed, Oct 16, 2019, 03:40 Zlopez [e-mail] wrote:

@GoodMirek https://github.com/GoodMirek I absolutely agree with you.

There are two issues that will be unfortunate for any user upgrading to
F31. This one, where you need to regenerate
$HOME/.config/containers/libpod.conf and the regression in podman create
which will prevent you to use any previously created container. I
personally thought that this was solved together with
containers/toolbox#277 https://github.com/containers/toolbox/issues/277,
but it seems that this affects every user who does update from F30 to F31.
I'm not sure if there is time to actually do anything about it when the
release of F31 is closing up.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/containers/libpod/issues/4210?email_source=notifications&email_token=AB3AOCE3RXDMSELUVUPGWETQO3ANRA5CNFSM4I6BMWYKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEBLO2NA#issuecomment-542567732,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB3AOCBTOPME6BIGRUXVQ7LQO3ANRANCNFSM4I6BMWYA
.

[GoodMirek]

On a different instance (FC31 Silverblue Kinoite), after deleting $HOME/.config/containers/libpod.conf , I get the following error now:

podman ps -a
ERRO[0001] Error retrieving container 9aee254d7eb0058257520d088547de8d123109b3b9396ec0888917a2aa501a7a from the database: container 9aee254d7eb0058257520d088547de8d123109b3b9396ec0888917a2aa501a7a was created with OCI runtime runc, but that runtime is not available in the current configuration: internal libpod error

Error message has been truncated - other containers followed with the same error.
Commands podman rm -a and podman rmi -a fail with the same error.

[mheon]

That is fixed in 1.6.2

[mheon]

1.6.2 should now be in the Fedora 31 repos. I believe it will be available in the default install, and will not require a day 1 upgrade.

For folks finding this from a search engine, if you encounter difficulty launching containers after an upgrade to F31, we recommend the following:

• Whenever possible, remove containers created in older Fedora versions and recreate with F31 Podman - this is inconvenient, but the safest option
• If you have containers you do not wish to recreate, you can run podman system migrate --runtime crun to convert them to work on Fedora 31. There is, however, a chance that some containers will not work after the migration.
• If you encounter issues where Podman will not launch due to an unknown event logger, we recommend removing $HOME/.config/containers/libpod.conf (assuming you have not modified your copy). Podman will regenerate the configuration file with sane defaults for F31.

Closing this as such. Feel free to open new issues with any further migration difficulties that appear.

[fbruetting]

Still had this issue in Podman v1.6.2 and needed to delete libpod.conf :/

[rgolangh]

F31 upgrade, podman v1.6.2, libpod.conf removed, running rootless, and now trying to start a previously stopped container:

08:25 $ podman  start pg11
Error: unable to start container "pg11": systemd cgroup flag passed, but systemd support for managing cgroups is not available: OCI runtime error

Inspecting:

08:28 $ podman inspect pg11 | rg cgroup -A 2 -B 2
            "--log-level",
            "error",
            "--cgroup-manager",
            "cgroupfs",
            "--tmpdir",
            "/run/user/1000/libpod/tmp",

This was working on F30.

[mheon]

Can you check in inspect if the container's runtime is set to crun or runc? I suspect your container is still on the runc runtime, and you'll need to run podman system migrate --runtime crun to get it on crun.

[prototact]

I have a similar issue in Fedora 31, where podman is working under root but as user I receive the following error:

'''
Error: could not get runtime: unknown event logger type:
'''

I ran the command to migrate to the new runtime but the problem persists. Also, there is no libpod.conf.

[rgolangh]

Can you check in inspect if the container's runtime is set to crun or runc? I suspect your container is still on the runc runtime, and you'll need to run podman system migrate --runtime crun to get it on crun.

I couldn't get the migrate to do it. It simply ignored the migrate. Eventually I had to recreate the container.

[mheon]

@orpheusgame You almost certainly have a libpod.conf - rootless Podman will automatically create one if it doesn't exist. Are you sure that there is no ~/.config/containers/libpod.conf?

[prototact]

@mheon yes turns out there was one in the location you mentioned from fedora 30 (i upgraded yesterday). i removed it and podman regenerated one on its own. now everything is fine, no sudoers group required. good stuff.

[jboero]

Same issues here even after all workarounds in this thread. I don't want to add fuel to the fire here but if Podman is going to snuff out Docker, it's clearly not ready yet and maybe the crippling cgroup changes need to go back into Beta (or Rawhide) until it is. OpenShift? Broke. Containers? Broke. This not a good strategy.

 jboero  xps  ~  $  podman ps
CONTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES
 jboero  xps  ~  $  docker ps
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
CONTAINER ID  IMAGE  COMMAND  CREATED  STATUS  PORTS  NAMES
 jboero  xps  ~  $  oc cluster up
Getting a Docker client ...
Checking if image openshift/origin-control-plane:v3.11 is available ...
error: unexpected error inspecting image openshift/origin-control-plane:v3.11
 jboero  xps  ~  $ 

Workaround
In the end, I needed to default back to Docker by adding systemd.unified_cgroup_hierarchy=0 kernel param. In fact grubby is also broken now so I needed to manually add it in /etc/default/grub and grub2-mkconfig -o /boot/grub2/grub.cfg and reboot. Then docker (now moby v18) works again and so does OpenShift / origin.

[mheon]

That doesn't sound like any of the issues I've seen in this thread. A reproducer or bug would be greatly appreciated - we can only fix things when they're reported.

[mheon]

Also, for reference, the migration issue originally referenced here ("unknown event logger type") should be fixed in the upcoming Podman 1.7.0. Existing installations that used the old migration code will still need to manually remove libpod.conf, but new installations should not see this problem.

[rhatdan]

Docker and Kubernetes was blocking the move to cgroup V2 for many years. The move to Cgroup V2 was to spur these upstreams to move, and if you follow what is going on in Kubernetes and runc world, this is happening. I do not believe these projects would have moved to V2 unless a distro started defaulting to V2, and Fedora is supposed to be the leader in new technology.