Toolbox: Unable to start container after update **podman 2:3.1.2-1.fc34 -> 2:3.2.0-0.1.rc1.fc34**

[maksimetsa]

Describe the bug
Unable to start container after update podman 2:3.1.2-1.fc34 -> 2:3.2.0-0.1.rc1.fc34

[ma3yta@localhost ~]$ podman start --attach dotnet --log-level debug (Click to expand)

INFO[0000] podman filtering at log level debug

DEBU[0000] Called start.PersistentPreRunE(podman start --attach dotnet --log-level debug)

DEBU[0000] Merged system config "/usr/share/containers/containers.conf"

DEBU[0000] Using conmon: "/usr/bin/conmon"

DEBU[0000] Initializing boltdb state at /var/home/ma3yta/.local/share/containers/storage/libpod/bolt_state.db

DEBU[0000] Using graph driver overlay

DEBU[0000] Using graph root /var/home/ma3yta/.local/share/containers/storage

DEBU[0000] Using run root /run/user/1000/containers

DEBU[0000] Using static dir /var/home/ma3yta/.local/share/containers/storage/libpod

DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp

DEBU[0000] Using volume path /var/home/ma3yta/.local/share/containers/storage/volumes

DEBU[0000] Set libpod namespace to ""

DEBU[0000] Not configuring container store

DEBU[0000] Initializing event backend journald

DEBU[0000] configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument

DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument

DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument

DEBU[0000] Using OCI runtime "/usr/bin/crun"

DEBU[0000] Default CNI network name podman is unchangeable

INFO[0000] Setting parallel job count to 25

INFO[0000] podman filtering at log level debug

DEBU[0000] Called start.PersistentPreRunE(podman start --attach dotnet --log-level debug)

DEBU[0000] overlay storage already configured with a mount-program

DEBU[0000] Merged system config "/usr/share/containers/containers.conf"

DEBU[0000] overlay storage already configured with a mount-program

DEBU[0000] Using conmon: "/usr/bin/conmon"

DEBU[0000] Initializing boltdb state at /var/home/ma3yta/.local/share/containers/storage/libpod/bolt_state.db

DEBU[0000] Using graph driver overlay

DEBU[0000] Using graph root /var/home/ma3yta/.local/share/containers/storage

DEBU[0000] Using run root /run/user/1000/containers

DEBU[0000] Using static dir /var/home/ma3yta/.local/share/containers/storage/libpod

DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp

DEBU[0000] Using volume path /var/home/ma3yta/.local/share/containers/storage/volumes

DEBU[0000] overlay storage already configured with a mount-program

DEBU[0000] Set libpod namespace to ""

DEBU[0000] [graphdriver] trying provided driver "overlay"

DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs

DEBU[0000] backingFs=btrfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false

DEBU[0000] Initializing event backend journald

DEBU[0000] configured OCI runtime runc initialization failed: no valid executable found for OCI runtime runc: invalid argument

DEBU[0000] configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument

DEBU[0000] configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument

DEBU[0000] Using OCI runtime "/usr/bin/crun"

DEBU[0000] Default CNI network name podman is unchangeable

INFO[0000] Setting parallel job count to 25

DEBU[0000] overlay: mount_data=,lowerdir=/var/home/ma3yta/.local/share/containers/storage/overlay/l/56QB4BQZ3IUKLISNMQW3Q4337D:/var/home/ma3yta/.local/share/containers/storage/overlay/l/YQU2LM4PUIJC5HZBXPALDHS5CB,upperdir=/var/home/ma3yta/.local/share/containers/storage/overlay/821be7b681919e10e491f72dd4cb2738860d609bfffa10ac88c652558a203070/diff,workdir=/var/home/ma3yta/.local/share/containers/storage/overlay/821be7b681919e10e491f72dd4cb2738860d609bfffa10ac88c652558a203070/work,context="system_u:object_r:container_file_t:s0:c460,c465"

DEBU[0000] mounted container "b2a64907dbefb2a22240b94bc88bd0c92540895b7a392370efa0fc7dc601dceb" at "/var/home/ma3yta/.local/share/containers/storage/overlay/821be7b681919e10e491f72dd4cb2738860d609bfffa10ac88c652558a203070/merged"

DEBU[0000] Created root filesystem for container b2a64907dbefb2a22240b94bc88bd0c92540895b7a392370efa0fc7dc601dceb at /var/home/ma3yta/.local/share/containers/storage/overlay/821be7b681919e10e491f72dd4cb2738860d609bfffa10ac88c652558a203070/merged

DEBU[0000] Workdir "/" resolved to host path "/var/home/ma3yta/.local/share/containers/storage/overlay/821be7b681919e10e491f72dd4cb2738860d609bfffa10ac88c652558a203070/merged"

DEBU[0000] Not modifying container b2a64907dbefb2a22240b94bc88bd0c92540895b7a392370efa0fc7dc601dceb /etc/passwd

DEBU[0000] Not modifying container b2a64907dbefb2a22240b94bc88bd0c92540895b7a392370efa0fc7dc601dceb /etc/group

DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode subscription

DEBU[0000] Setting CGroups for container b2a64907dbefb2a22240b94bc88bd0c92540895b7a392370efa0fc7dc601dceb to user.slice:libpod:b2a64907dbefb2a22240b94bc88bd0c92540895b7a392370efa0fc7dc601dceb

DEBU[0000] set root propagation to "rslave"

DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d

DEBU[0000] Created OCI spec for container b2a64907dbefb2a22240b94bc88bd0c92540895b7a392370efa0fc7dc601dceb at /var/home/ma3yta/.local/share/containers/storage/overlay-containers/b2a64907dbefb2a22240b94bc88bd0c92540895b7a392370efa0fc7dc601dceb/userdata/config.json

DEBU[0000] /usr/bin/conmon messages will be logged to syslog

DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c b2a64907dbefb2a22240b94bc88bd0c92540895b7a392370efa0fc7dc601dceb -u b2a64907dbefb2a22240b94bc88bd0c92540895b7a392370efa0fc7dc601dceb -r /usr/bin/crun -b /var/home/ma3yta/.local/share/containers/storage/overlay-containers/b2a64907dbefb2a22240b94bc88bd0c92540895b7a392370efa0fc7dc601dceb/userdata -p  -n dotnet --exit-dir /run/user/1000/libpod/tmp/exits --full-attach -s -l k8s-file:/var/home/ma3yta/.local/share/containers/storage/overlay-containers/b2a64907dbefb2a22240b94bc88bd0c92540895b7a392370efa0fc7dc601dceb/userdata/ctr.log --log-level debug --syslog --conmon-pidfile /run/user/1000/containers/overlay-containers/b2a64907dbefb2a22240b94bc88bd0c92540895b7a392370efa0fc7dc601dceb/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/ma3yta/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg b2a64907dbefb2a22240b94bc88bd0c92540895b7a392370efa0fc7dc601dceb]"

INFO[0000] Running conmon under slice user.slice and unitName libpod-conmon-b2a64907dbefb2a22240b94bc88bd0c92540895b7a392370efa0fc7dc601dceb.scope

DEBU[0000] Received: -1

DEBU[0000] Cleaning up container b2a64907dbefb2a22240b94bc88bd0c92540895b7a392370efa0fc7dc601dceb

DEBU[0000] Network is already cleaned up, skipping...

DEBU[0000] unmounted container "b2a64907dbefb2a22240b94bc88bd0c92540895b7a392370efa0fc7dc601dceb"

Error: unable to start container b2a64907dbefb2a22240b94bc88bd0c92540895b7a392370efa0fc7dc601dceb: opening file `` for writing: No such file or directory: OCI not found

Expected behaviour
Containers works fine after update podman 2:3.1.2-1.fc34 -> 2:3.2.0-0.1.rc1.fc34

Actual behaviour

 podman start --attach dotnet --log-level debug
 Error: unable to start container b2a64907dbefb2a22240b94bc88bd0c92540895b7a392370efa0fc7dc601dceb: opening file `` for writing: No such file or directory: OCI not found

Works fine when create a new container but doesn't work for old containers. **Output of `toolbox --version` (v0.0.90+)** `toolbox version 0.0.99.1` **Toolbox package info (`rpm -q toolbox`)** ``toolbox-0.0.99.1-1.fc34.x86_64`` **Output of `podman version`**

Version:      3.2.0-rc1
API Version:  3.2.0-rc1
Go Version:   go1.16.3
Built:        Thu May  6 04:03:46 2021
OS/Arch:      linux/amd64

Podman package info (rpm -q podman)
podman-3.2.0-0.1.rc1.fc34.x86_64

Info about your OS
Fedora Silverblue 34

[maksimetsa]

moved issue to podman repo https://github.com/containers/podman/issues/10274

[HarryMichal]

Thank you for reporting the issue, @Ma3yTa!

[anthr76]

FYI this regression seems to have made it's way back :)