Toolbox: "Error: failed to start container fedora-toolbox-32" on Fedora Silverblue
Fedora Silverblue 32, 32.20200712.0 (2020-07-12T00:42:58Z)
$USER = rugk
logs
$ toolbox enter
#Error: failed to start container fedora-toolbox-32
$ podman logs fedora-toolbox-32
toolbox: running as real user ID 0
toolbox: resolved absolute path for /usr/bin/toolbox to /usr/bin/toolbox
toolbox: TOOLBOX_PATH is /usr/bin/toolbox
toolbox: XDG_RUNTIME_DIR is unset
toolbox: XDG_RUNTIME_DIR set to /run/user/1000
toolbox: creating /run/.toolboxenv
toolbox: redirecting /etc/host.conf to /run/host/etc/host.conf
toolbox: redirecting /etc/hosts to /run/host/etc/hosts
toolbox: redirecting /etc/resolv.conf to /run/host/etc/resolv.conf
toolbox: binding /etc/machine-id to /run/host/etc/machine-id
toolbox: creating /run/systemd/journal
toolbox: binding /run/systemd/journal to /run/host/run/systemd/journal
toolbox: creating /sys/fs/selinux
toolbox: binding /sys/fs/selinux to /usr/share/empty
toolbox: creating /var/lib/flatpak
toolbox: binding /var/lib/flatpak to /run/host/var/lib/flatpak
toolbox: creating /var/log/journal
toolbox: binding /var/log/journal to /run/host/var/log/journal
toolbox: creating /var/mnt
toolbox: binding /var/mnt to /run/host/var/mnt
toolbox: redirecting /etc/localtime to /run/host/monitor/localtime
toolbox: redirecting /etc/timezone to /run/host/monitor/timezone
toolbox: making /media a symbolic link to /run/media
toolbox: making /mnt a symbolic link to /var/mnt
id: 'rugk': no such user
toolbox: making /home a symlink
toolbox: adding user rugk with UID 1000
useradd: Warning: missing or non-executable shell '/bin/zsh'
toolbox: removing password for user rugk
passwd: Note: deleting a password also unlocks the password.
toolbox: removing password for user root
passwd: Note: deleting a password also unlocks the password.
toolbox: setting KCM as the default Kerberos credential cache
toolbox: finished initializing container
toolbox: going to sleep
$ podman --log-level debug start fedora-toolbox-32
INFO[0000] podman filtering at log level debug
DEBU[0000] Called start.PersistentPreRunE(podman --log-level debug start fedora-toolbox-32)
DEBU[0000] Ignoring libpod.conf EventsLogger setting "/var/home/rugk/.config/containers/containers.conf". Use "journald" if you want to change this setting and remove libpod.conf files.
DEBU[0000] Reading configuration file "/usr/share/containers/containers.conf"
DEBU[0000] Merged system config "/usr/share/containers/containers.conf": &{{[] [] containers-default-0.14.4 [] private enabled [CAP_AUDIT_WRITE CAP_CHOWN CAP_DAC_OVERRIDE CAP_FOWNER CAP_FSETID CAP_KILL CAP_MKNOD CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETFCAP CAP_SETGID CAP_SETPCAP CAP_SETUID CAP_SYS_CHROOT] [] [] [] [] [] true [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] false false false private k8s-file -1 slirp4netns false 2048 private /usr/share/containers/seccomp.json 65536k private host 65536} {true systemd [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] [/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] ctrl-p,ctrl-q true /run/user/1000/libpod/tmp/events/events.log file [/usr/share/containers/oci/hooks.d] docker:// /pause k8s.gcr.io/pause:3.2 /usr/libexec/podman/catatonit shm false 2048 /usr/bin/crun map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] missing false [] [crun runc] [crun] [kata kata-runtime kata-qemu kata-fc] {false false false false false false} /etc/containers/policy.json false 3 /var/home/rugk/.local/share/containers/storage/libpod 10 /run/user/1000/libpod/tmp /var/home/rugk/.local/share/containers/storage/volumes} {[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] podman /etc/cni/net.d/}}
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /var/home/rugk/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /var/home/rugk/.local/share/containers/storage
DEBU[0000] Using run root /run/user/1000/containers
DEBU[0000] Using static dir /var/home/rugk/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp
DEBU[0000] Using volume path /var/home/rugk/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
DEBU[0000] Initializing event backend file
DEBU[0000] using runtime "/usr/bin/runc"
DEBU[0000] using runtime "/usr/bin/crun"
WARN[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument
DEBU[0000] using runtime "/usr/bin/crun"
INFO[0000] Setting parallel job count to 49
DEBU[0000] overlay: mount_data=lowerdir=/var/home/rugk/.local/share/containers/storage/overlay/l/HHFMQSI45LPHONFF5SO75XX7BS:/var/home/rugk/.local/share/containers/storage/overlay/l/RYRUJGFXQY6SCM2AR2OXNWYPQP,upperdir=/var/home/rugk/.local/share/containers/storage/overlay/7d5b61e2bf03f015c6e164a34926fa52ab94518850fe3e6a0204b6fb59ef2b35/diff,workdir=/var/home/rugk/.local/share/containers/storage/overlay/7d5b61e2bf03f015c6e164a34926fa52ab94518850fe3e6a0204b6fb59ef2b35/work,context="system_u:object_r:container_file_t:s0:c539,c801"
DEBU[0000] mounted container "462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d" at "/var/home/rugk/.local/share/containers/storage/overlay/7d5b61e2bf03f015c6e164a34926fa52ab94518850fe3e6a0204b6fb59ef2b35/merged"
DEBU[0000] Created root filesystem for container 462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d at /var/home/rugk/.local/share/containers/storage/overlay/7d5b61e2bf03f015c6e164a34926fa52ab94518850fe3e6a0204b6fb59ef2b35/merged
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret
DEBU[0000] Setting CGroups for container 462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d to user.slice:libpod:462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d
DEBU[0000] set root propagation to "rslave"
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d
DEBU[0000] Created OCI spec for container 462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d at /var/home/rugk/.local/share/containers/storage/overlay-containers/462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d/userdata/config.json
DEBU[0000] /usr/bin/conmon messages will be logged to syslog
DEBU[0000] running conmon: /usr/bin/conmon args="[--api-version 1 -c 462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d -u 462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d -r /usr/bin/crun -b /var/home/rugk/.local/share/containers/storage/overlay-containers/462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d/userdata -p /run/user/1000/containers/overlay-containers/462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d/userdata/pidfile -n fedora-toolbox-32 --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket -s -l k8s-file:/var/home/rugk/.local/share/containers/storage/overlay-containers/462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d/userdata/ctr.log --log-level debug --syslog --conmon-pidfile /run/user/1000/containers/overlay-containers/462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/rugk/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg /usr/bin/crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d]"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied
DEBU[0000] Received: -1
DEBU[0000] Cleaning up container 462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] unmounted container "462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d"
Error: unable to start container "462c77582f51221cb9dc8c98f800ec0ec99842a3147aeb6325142dee18349e9d": setrlimit `RLIMIT_NPROC`: Operation not permitted: OCI runtime permission denied error
podman container list lists no container.
rugk
All 17 comments
Repulling comntainer…
$ podman run --it --rm registry.fedoraproject.org/fedora:32 /bin/bash
Error: unknown flag: --it
$ podman run --rm registry.fedoraproject.org/fedora:32 /bin/bash
Trying to pull registry.fedoraproject.org/fedora:32...
Getting image source signatures
Copying blob dd9f43919ba0 done
Copying config 00ff39a8bf done
Writing manifest to image destination
Storing signatures
…did not work either.
I.e. toolbox enter still does not work.
rugk
on 12 Jul 2020
Hey @rugk! It seems you're still running the Shell Toolbox (v0.0.18) even though your current deployment of Silverblue should already contain v0.0.92. Can you, please, try to upgrade your machine again and try to use the new Toolbox??
HarryMichal
on 13 Jul 2020
toolbox --version returns 0.0.92.
Does not work, though. (still the same deployment AFAIK)
rugk
on 13 Jul 2020
Hmm, odd. v0.0.18 prepends toolbox: to every log message, while the newer version uses a special logging package that uses a different format.
I have the same deployment and a newly created container is running alright.
According to the output of podman logs fedora-toolbox-32 the container uses the old Toolbox. Can you show me the output of podman inspect -t container fedora-toolbox-32??
HarryMichal
on 13 Jul 2020
Without knowing anything about the system, maybe the logs are still from an old toolbox version and it did not write any new logs?
podman inspect -t container fedora-toolbox-32
https://gist.github.com/rugk/6bd2a15378f9811e8db9cb0f5c964e96
IIRC I created that container a few months ago, not that long ago.
rugk
on 13 Jul 2020
Without knowing anything about the system, maybe the logs are still from an old toolbox version and it did not write any new logs?
Good point! That actually may be the reason.
So, this happens with an old container... Can the issue be reproduced with a new container??
HarryMichal
on 14 Jul 2020
I have the same issue with Fedora 31 Workstation
5.7.7-100.fc31.x86_64
Access to all of my existing toolbox containers are failing with toolbox: failed to start container «containerName»
tools --version says that is not a recognized option here. This was working yesterday. System changes since then was only dnf update, which was:
Packages Altered:
Upgrade librados2-2:14.2.10-1.fc31.x86_64 @updates
Upgraded librados2-2:14.2.9-2.fc31.x86_64 @@System
Upgrade librbd1-2:14.2.10-1.fc31.x86_64 @updates
Upgraded librbd1-2:14.2.9-2.fc31.x86_64 @@System
In case there was no reboot yesterday, the previous update did:
Packages Altered:
Upgrade LibRaw-0.19.5-3.fc31.x86_64 @updates
Upgraded LibRaw-0.19.5-1.fc31.x86_64 @@System
Upgrade dkms-2.8.2-1.fc31.noarch @updates
Upgraded dkms-2.8.1-4.20200214git5ca628c.fc31.noarch @@System
I could supply the output of podman inspect, but since I have 8 separate containers failing the same way, all at the same time, I doubt it is a container specific problem. Unless a toolbox run --container «name» sudo dnf -y upgrade triggered the problem. That was run yesterday on all of the containers. toolsbox list shows status "Exited (143) 10 hours ago", which is when the host system was shut down.
EDIT: So far, a newly created container is working correctly
mMerlin
on 14 Jul 2020
After building up a new container with the same (very close) configuration/content as one of the failing cases, I did podman inspect on each of them, and compared the outputs. The interesting differences are:
old failing new working
[ [
{ {
"State": { "State": {
"OciVersion": "1.0.1-dev", | "OciVersion": "1.0.2-dev",
}, },
"MountLabel": "system_u:object_r:container_file_t:s0:c26,c243", | "MountLabel": "system_u:object_r:container_file_t:s0:c361,c604",
"Config": { "Config": {
"Env": [ "Env": [
"container=oci", | "container=podman",
], ],
}, },
"HostConfig": { "HostConfig": {
"IpcMode": "host", | "IpcMode": "private",
"MemorySwappiness": -1, | "MemorySwappiness": 0,
"Ulimits": [ | "Ulimits": [],
{ <
"Name": "RLIMIT_NOFILE", <
"Soft": 524288, <
"Hard": 524288 <
}, <
{ <
"Name": "RLIMIT_NPROC", <
"Soft": 63377, <
"Hard": 63377 <
} <
], <
} }
} }
] ]
tools --version says that is not a recognized option here.
Obviously yes, because your command is wrong… :upside_down_face: :wink:
rugk
on 14 Jul 2020
:)
% toolbox --version
toolbox: unrecognized option '--version'
Try 'toolbox --help' for more information.
Doesn't work either
:(
mMerlin
on 15 Jul 2020
Thank you @mMerlin for all the information! It is very helpful! With this I know, this is caused by Podman itself. I found some issues in Podman's bug tracker: https://github.com/containers/podman/issues/6857, https://github.com/containers/podman/issues/6389.
Based on the changelog entry for Podman v2.0.2, the issue should be already fixed.
* Fixed a bug where NPROC and NOFILE rlimits could be improperly set for rootless Podman containers, causing them to fail to start.
Can you confirm that the issue is resolved or is it still present?
:)
% toolbox --version toolbox: unrecognized option '--version' Try 'toolbox --help' for more information.Doesn't work either
:(
On Fedora 31 is still Toolbox v0.0.18 which does not have the option --version available. Instead, you can check the version with rpm -q toolbox.
HarryMichal
on 15 Jul 2020
Apparently not fixed with versions I have available. Starting from:
% podman --version
podman version 2.0.1
% rpm -q toolbox
toolbox-0.0.18-2.fc31.noarch
% sudo dnf upgrade podman --enablerepo=updates-testing --best
% dnf history info 196
…
Command Line : upgrade podman --enablerepo=updates-testing --best
Comment :
Packages Altered:
Upgrade podman-2:2.0.2-1.fc31.x86_64 @updates-testing
Upgraded podman-2:2.0.1-1.fc31.x86_64 @@System
Upgrade podman-docker-2:2.0.2-1.fc31.noarch @updates-testing
Upgraded podman-docker-2:2.0.1-1.fc31.noarch @@System
Upgrade podman-plugins-2:2.0.2-1.fc31.x86_64 @updates-testing
Upgraded podman-plugins-2:2.0.1-1.fc31.x86_64 @@System
% podman --version
podman version 2.0.2
With a system reboot for good measure, the old containers are still failing to start. No change in the symptoms.
mMerlin
on 15 Jul 2020
Looks like on F31 the update did not fix the issue according to Bodhi: https://bodhi.fedoraproject.org/updates/FEDORA-2020-c510ceb7db. I encourage you to comment on the update and give it a negative karma.
@rugk, I believe the same could be done for the F32 update: https://bodhi.fedoraproject.org/updates/FEDORA-2020-19924b556e.
Apart from this, I'd possibly comment on one of the mentioned issue tickets to make it known that the bug has not been fixed.
HarryMichal
on 15 Jul 2020
I added a reproducer to the upstream issue.
HarryMichal
on 15 Jul 2020
So it seems like we should sort it out in Podman, right?
Duplicate of https://github.com/containers/podman/issues/6389 and https://github.com/containers/podman/issues/6857
debarshiray
on 23 Jul 2020
Sorry how to fix this? Tried dnf downgrade podman but still cant enter, fedora-toolbox-32
Thanks
biji
on 21 Sep 2020
687
biji
on 4 Mar 2021
Related issues
abitrolly
·
8Comments
allanday
·
3Comments
evelineraine
·
5Comments
hferreiro
·
4Comments
cgwalters
·
9Comments
Most helpful comment
After building up a new container with the same (very close) configuration/content as one of the failing cases, I did podman inspect on each of them, and compared the outputs. The interesting differences are: