Podman: Unable to start toolbox containers in Fedora Silverblue 31

Created on 13 Sep 2019 · 36Comments · Source: containers/podman

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
Unable to start fedora-toolbox-31 (or any other toolbox container) in Fedora Silverblue 31.

Steps to reproduce the issue:

toolbox create to create a new toolbox on Fedora Silverblue 31
toolbox enter to enter the toolbox. This generates an error: toolbox: failed to start container fedora-toolbox-31
podman --log-level debug start fedora-toolbox-31 to debug the issue generates a warning (see description)`

Describe the results you received:

The following output is presented when debuging with:

podman --log-level debug start fedora-toolbox-31

INFO[0000] running as rootless                          
DEBU[0000] using conmon: "/usr/libexec/podman/conmon"   
DEBU[0000] Initializing boltdb state at /var/home/returntrip/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/returntrip/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /var/home/returntrip/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/returntrip/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] overlay: mount_data=lowerdir=/var/home/returntrip/.local/share/containers/storage/overlay/l/HDZBVTBU4P5YJODD2W5VAHROK6:/var/home/returntrip/.local/share/containers/storage/overlay/l/UNHS2OY3XUUJNRAEMMDBKQJA23,upperdir=/var/home/returntrip/.local/share/containers/storage/overlay/03373eec54ee786f0ae0abee768212b4bd5122d531fbe298d9713b1b44e723f6/diff,workdir=/var/home/returntrip/.local/share/containers/storage/overlay/03373eec54ee786f0ae0abee768212b4bd5122d531fbe298d9713b1b44e723f6/work,context="system_u:object_r:container_file_t:s0:c687,c748" 
DEBU[0000] mounted container "2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c" at "/var/home/returntrip/.local/share/containers/storage/overlay/03373eec54ee786f0ae0abee768212b4bd5122d531fbe298d9713b1b44e723f6/merged" 
DEBU[0000] Created root filesystem for container 2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c at /var/home/returntrip/.local/share/containers/storage/overlay/03373eec54ee786f0ae0abee768212b4bd5122d531fbe298d9713b1b44e723f6/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container 2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c to user.slice:libpod:2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c 
DEBU[0000] set root propagation to "rslave"             
DEBU[0000] Created OCI spec for container 2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c at /var/home/returntrip/.local/share/containers/storage/overlay-containers/2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c/userdata/config.json 
DEBU[0000] /usr/libexec/podman/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/libexec/podman/conmon    args="[--api-version 1 -s -c 2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c -u 2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c -r /usr/bin/crun -b /var/home/returntrip/.local/share/containers/storage/overlay-containers/2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c/userdata -p /run/user/1000/overlay-containers/2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c/userdata/pidfile -l k8s-file:/var/home/returntrip/.local/share/containers/storage/overlay-containers/2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog --conmon-pidfile /run/user/1000/overlay-containers/2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/returntrip/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000 --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c]"
INFO[0000] Running conmon under slice user.slice and unitName libpod-conmon-2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c.scope 
WARN[0000] Failed to add conmon to systemd sandbox cgroup: write unix @: sendmsg: broken pipe 
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Cleaning up container 2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] unmounted container "2083b2d1cc90d630152ae3a61e0f9cd68266ce985e5ff282f7e54811183b065c" 
ERRO[0000] unable to start container "fedora-toolbox-31": error reading container (probably exited) json message: EOF

Describe the results you expected:

User should be able to access toolbox without errors

Additional information you deem important (e.g. issue happens only occasionally):
Discussed with Rishi and he suggested to file an issue here

Output of podman version:

podman version 1.5.1-dev

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.13rc1
  podman version: 1.5.1-dev
host:
  BuildahVersion: 1.10.1
  Conmon:
    package: podman-1.5.1-2.17.dev.gitce64c14.fc31.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.0, commit: 118fcdfca36d706f766bad2663b11bd2c41bf2e7'
  Distribution:
    distribution: fedora
    version: "31"
  MemFree: 7146295296
  MemTotal: 16778080256
  OCIRuntime:
    package: crun-0.8-1.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun 0.8
      spec: 1.0.0
      +SYSTEMD +SELINUX +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 7985950720
  SwapTotal: 7985950720
  arch: amd64
  cpus: 16
  eventlogger: journald
  hostname: rauros.figura.io
  kernel: 5.3.0-0.rc6.git0.1.fc31.x86_64
  os: linux
  rootless: true
  uptime: 1h 55m 38.6s (Approximately 0.04 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/returntrip/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /var/home/returntrip/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 1
  RunRoot: /run/user/1000
  VolumePath: /var/home/returntrip/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.5.1-2.17.dev.gitce64c14.fc31.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):
Phisical machine. Rebased FSB30 to FSB31

kinbug

Source

returntrip

👍1

Most helpful comment

made a new release: https://bodhi.fedoraproject.org/updates/FEDORA-2019-4b4957bbc6

giuseppe on 18 Nov 2019

👍3

All 36 comments

Hm. This is using crun, not runc. Not seeing any direct errors out of Conmon aside from the CGroup failures - can you check journalctl for anything from Conmon? The real error may be printed there

mheon on 13 Sep 2019

@mheon here you go:

Sep 13 18:15:06 rauros.figura.io systemd[1]: Incoming private connection from unprivileged client, refusing: Operation not permitted
Sep 13 18:15:06 rauros.figura.io conmon[33800]: conmon 41166c85cc2192782d95 <ndebug>: failed to write to /proc/self/oom_score_adj: Permission denied
Sep 13 18:15:06 rauros.figura.io conmon[33801]: conmon 41166c85cc2192782d95 <ninfo>: attach sock path: /run/user/1000/libpod/tmp/socket/41166c85cc2192782d95d70c0a870bf9927c64ea516254457f664a4ff80a1f64/attach
Sep 13 18:15:06 rauros.figura.io conmon[33801]: conmon 41166c85cc2192782d95 <ninfo>: addr{sun_family=AF_UNIX, sun_path=/run/user/1000/libpod/tmp/socket/41166c85cc2192782d95d70c0a870bf9927c64ea516254457f664a4ff80a1f64/attach}
Sep 13 18:15:06 rauros.figura.io conmon[33801]: conmon 41166c85cc2192782d95 <ninfo>: ctl fifo path: /var/home/returntrip/.local/share/containers/storage/overlay-containers/41166c85cc2192782d95d70c0a870bf9927c64ea516254457f664a4ff80a1f64/userdata/ctl
Sep 13 18:15:06 rauros.figura.io conmon[33801]: conmon 41166c85cc2192782d95 <error>: Failed to mkfifo at /var/home/returntrip/.local/share/containers/storage/overlay-containers/41166c85cc2192782d95d70c0a870bf9927c64ea516254457f664a4ff80a1f64/userdata/ctl: File exists
Sep 13 18:15:06 rauros.figura.io podman[33801]: 2019-09-13 18:15:06.132351348 +0200 CEST m=+0.055784416 container cleanup 41166c85cc2192782d95d70c0a870bf9927c64ea516254457f664a4ff80a1f64 (image=registry.fedoraproject.org/f31/fedora-toolbox:31, name=fedora-toolbox-31)

returntrip on 13 Sep 2019

@giuseppe PTAL
Error seems to be writing to /proc/self/oom_score_adj?

rhatdan on 13 Sep 2019

Sep 13 18:15:06 rauros.figura.io conmon[33801]: conmon 41166c85cc2192782d95 <error>: Failed to mkfifo at /var/home/returntrip/.local/share/containers/storage/overlay-containers/41166c85cc2192782d95d70c0a870bf9927c64ea516254457f664a4ff80a1f64/userdata/ctl: File exists

That was probably what was fatal

@returntrip Can you check and see if that file still exists?

mheon on 13 Sep 2019

(It should have been cleaned up when the container exited - if not, that's probably the problem)

mheon on 13 Sep 2019

@mheon removed ~/.local/share/containers/ to start fresh and recreated the toolbox but the issue is the same:

Sep 13 19:22:29 rauros.figura.io systemd[1]: Incoming private connection from unprivileged client, refusing: Operation not permitted
Sep 13 19:22:29 rauros.figura.io conmon[39312]: conmon a5f8742e74888002ed88 <ndebug>: failed to write to /proc/self/oom_score_adj: Permission denied
Sep 13 19:22:29 rauros.figura.io conmon[39313]: conmon a5f8742e74888002ed88 <ninfo>: attach sock path: /run/user/1000/libpod/tmp/socket/a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c/attach
Sep 13 19:22:29 rauros.figura.io conmon[39313]: conmon a5f8742e74888002ed88 <ninfo>: addr{sun_family=AF_UNIX, sun_path=/run/user/1000/libpod/tmp/socket/a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c/attach}
Sep 13 19:22:29 rauros.figura.io conmon[39313]: conmon a5f8742e74888002ed88 <ninfo>: ctl fifo path: /var/home/returntrip/.local/share/containers/storage/overlay-containers/a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c/userdata/ctl
Sep 13 19:22:29 rauros.figura.io conmon[39313]: conmon a5f8742e74888002ed88 <error>: Failed to mkfifo at /var/home/returntrip/.local/share/containers/storage/overlay-containers/a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c/userdata/ctl: File exists
Sep 13 19:22:29 rauros.figura.io podman[39313]: 2019-09-13 19:22:29.402860873 +0200 CEST m=+0.050886213 container cleanup a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c (image=registry.fedoraproject.org/f31/fedora-toolbox:31, name=fedora-toolbox-31)

sudo ls -l /var/home/returntrip/.local/share/containers/storage/overlay-containers/a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c/userdata/ctl

prw-r--r--. 1 returntrip returntrip 0 Sep 13 19:22 /var/home/returntrip/.local/share/containers/storage/overlay-containers/a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c/userdata/ctl

returntrip on 13 Sep 2019

if I delete that file manually I get this:

Sep 13 19:27:54 rauros.figura.io systemd[1]: Incoming private connection from unprivileged client, refusing: Operation not permitted
Sep 13 19:27:54 rauros.figura.io conmon[39497]: conmon a5f8742e74888002ed88 <ndebug>: failed to write to /proc/self/oom_score_adj: Permission denied
Sep 13 19:27:54 rauros.figura.io conmon[39498]: conmon a5f8742e74888002ed88 <ninfo>: attach sock path: /run/user/1000/libpod/tmp/socket/a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c/attach
Sep 13 19:27:54 rauros.figura.io conmon[39498]: conmon a5f8742e74888002ed88 <ninfo>: addr{sun_family=AF_UNIX, sun_path=/run/user/1000/libpod/tmp/socket/a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c/attach}
Sep 13 19:27:54 rauros.figura.io conmon[39498]: conmon a5f8742e74888002ed88 <ninfo>: ctl fifo path: /var/home/returntrip/.local/share/containers/storage/overlay-containers/a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c/userdata/ctl
Sep 13 19:27:54 rauros.figura.io conmon[39498]: conmon a5f8742e74888002ed88 <ninfo>: terminal_ctrl_fd: 12
Sep 13 19:27:54 rauros.figura.io conmon[39498]: conmon a5f8742e74888002ed88 <error>: Failed to create container: exit status 1

returntrip on 13 Sep 2019

Sep 13 19:27:54 rauros.figura.io conmon[39498]: conmon a5f8742e74888002ed88 <error>: Failed to create container: exit status 1

Well that's wonderfully nonspecific.
@giuseppe Any ideas here? crun doesn't seem to be giving us an error

mheon on 13 Sep 2019

how have you created the user session? Have you logged in with the user or used sudo?

There is a known issue with cgroups v2 and Podman that prevents the creation of containers if the current cgroup is not owned by your user. It is fixed upstream with afd0818326aa37f03a3bc74f0269a06a403db16d that automatically creates the systemd scope if necessary

giuseppe on 17 Sep 2019

could you try using systemd-run --scope --user podman --log-level debug start fedora-toolbox-31 ?

giuseppe on 17 Sep 2019

@giuseppe I have the same issue and when I run the command I get the following output:

Running scope as unit: run-r09c9b5df086343a9a2fd227722aad8b8.scope
INFO[0000] running as rootless                          
DEBU[0000] using conmon: "/usr/libexec/podman/conmon"   
DEBU[0000] Initializing boltdb state at /home/taaem/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/taaem/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /home/taaem/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/taaem/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] overlay: mount_data=lowerdir=/home/taaem/.local/share/containers/storage/overlay/l/YUGGEBJHTPB3S7ZST3SBMDPWWP:/home/taaem/.local/share/containers/storage/overlay/l/BU3HQSQE3HS4SOAL2TQM4I5BJU,upperdir=/home/taaem/.local/share/containers/storage/overlay/07069f11388dcd8c7ce99401ae431be6c70f23f3afd51afd47477eafdd1ca6c5/diff,workdir=/home/taaem/.local/share/containers/storage/overlay/07069f11388dcd8c7ce99401ae431be6c70f23f3afd51afd47477eafdd1ca6c5/work,context="system_u:object_r:container_file_t:s0:c66,c513" 
DEBU[0000] mounted container "9605fe496bc27a179d03fd0de7fe06f0f5a967d73b1bb0b347cb8286b59ffc3b" at "/home/taaem/.local/share/containers/storage/overlay/07069f11388dcd8c7ce99401ae431be6c70f23f3afd51afd47477eafdd1ca6c5/merged" 
DEBU[0000] Created root filesystem for container 9605fe496bc27a179d03fd0de7fe06f0f5a967d73b1bb0b347cb8286b59ffc3b at /var/home/taaem/.local/share/containers/storage/overlay/07069f11388dcd8c7ce99401ae431be6c70f23f3afd51afd47477eafdd1ca6c5/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container 9605fe496bc27a179d03fd0de7fe06f0f5a967d73b1bb0b347cb8286b59ffc3b to libpod_parent:libpod:9605fe496bc27a179d03fd0de7fe06f0f5a967d73b1bb0b347cb8286b59ffc3b 
DEBU[0000] set root propagation to "rslave"             
DEBU[0000] Created OCI spec for container 9605fe496bc27a179d03fd0de7fe06f0f5a967d73b1bb0b347cb8286b59ffc3b at /home/taaem/.local/share/containers/storage/overlay-containers/9605fe496bc27a179d03fd0de7fe06f0f5a967d73b1bb0b347cb8286b59ffc3b/userdata/config.json 
DEBU[0000] /usr/libexec/podman/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/libexec/podman/conmon    args="[--api-version 1 -s -c 9605fe496bc27a179d03fd0de7fe06f0f5a967d73b1bb0b347cb8286b59ffc3b -u 9605fe496bc27a179d03fd0de7fe06f0f5a967d73b1bb0b347cb8286b59ffc3b -r /usr/bin/runc -b /home/taaem/.local/share/containers/storage/overlay-containers/9605fe496bc27a179d03fd0de7fe06f0f5a967d73b1bb0b347cb8286b59ffc3b/userdata -p /run/user/1000/overlay-containers/9605fe496bc27a179d03fd0de7fe06f0f5a967d73b1bb0b347cb8286b59ffc3b/userdata/pidfile -l k8s-file:/home/taaem/.local/share/containers/storage/overlay-containers/9605fe496bc27a179d03fd0de7fe06f0f5a967d73b1bb0b347cb8286b59ffc3b/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog --conmon-pidfile /run/user/1000/overlay-containers/9605fe496bc27a179d03fd0de7fe06f0f5a967d73b1bb0b347cb8286b59ffc3b/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/taaem/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000 --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg cgroupfs --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg runc --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 9605fe496bc27a179d03fd0de7fe06f0f5a967d73b1bb0b347cb8286b59ffc3b]"
INFO[0000] Running conmon under slice /libpod_parent and unitName libpod-conmon-9605fe496bc27a179d03fd0de7fe06f0f5a967d73b1bb0b347cb8286b59ffc3b.scope 
WARN[0000] Failed to add conmon to systemd sandbox cgroup: read unix @->/run/systemd/private: read: connection reset by peer 
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: -1                                 
DEBU[0000] Cleaning up container 9605fe496bc27a179d03fd0de7fe06f0f5a967d73b1bb0b347cb8286b59ffc3b 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] unmounted container "9605fe496bc27a179d03fd0de7fe06f0f5a967d73b1bb0b347cb8286b59ffc3b" 
ERRO[0000] unable to start container "fedora-toolbox-31": time="2019-09-17T15:00:08-04:00" level=error msg="systemd cgroup flag passed, but systemd support for managing cgroups is not available\n"
systemd cgroup flag passed, but systemd support for managing cgroups is not available: OCI runtime error

taaem on 17 Sep 2019

@giuseppe I was running toolbox/podman as a user without sudo.

I guess I could override https://koji.fedoraproject.org/koji/buildinfo?buildID=1379339 and see if the issue is solved for me?

Output of systemd-run --scope --user podman --log-level debug start fedora-toolbox-31:

systemd-run --scope --user podman --log-level debug start fedora-toolbox-31
Running scope as unit: run-rf4cb382bfa374051baf2944a824dc73f.scope
INFO[0000] running as rootless                          
DEBU[0000] using conmon: "/usr/libexec/podman/conmon"   
DEBU[0000] Initializing boltdb state at /var/home/returntrip/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/returntrip/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /var/home/returntrip/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/returntrip/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] overlay: mount_data=lowerdir=/var/home/returntrip/.local/share/containers/storage/overlay/l/XRK4QZLJQPEHYIF75UOZ4Z4AEA:/var/home/returntrip/.local/share/containers/storage/overlay/l/2KWKCKSL6L5L3REF3EY2EONXBD,upperdir=/var/home/returntrip/.local/share/containers/storage/overlay/1e78d62f890fb00ca60d00ff51c0a17e78940b1540562bc38d03efe7ca034264/diff,workdir=/var/home/returntrip/.local/share/containers/storage/overlay/1e78d62f890fb00ca60d00ff51c0a17e78940b1540562bc38d03efe7ca034264/work,context="system_u:object_r:container_file_t:s0:c381,c1014" 
DEBU[0000] mounted container "a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c" at "/var/home/returntrip/.local/share/containers/storage/overlay/1e78d62f890fb00ca60d00ff51c0a17e78940b1540562bc38d03efe7ca034264/merged" 
DEBU[0000] Created root filesystem for container a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c at /var/home/returntrip/.local/share/containers/storage/overlay/1e78d62f890fb00ca60d00ff51c0a17e78940b1540562bc38d03efe7ca034264/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c to user.slice:libpod:a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c 
DEBU[0000] set root propagation to "rslave"             
DEBU[0000] Created OCI spec for container a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c at /var/home/returntrip/.local/share/containers/storage/overlay-containers/a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c/userdata/config.json 
DEBU[0000] /usr/libexec/podman/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/libexec/podman/conmon    args="[--api-version 1 -s -c a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c -u a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c -r /usr/bin/crun -b /var/home/returntrip/.local/share/containers/storage/overlay-containers/a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c/userdata -p /run/user/1000/overlay-containers/a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c/userdata/pidfile -l k8s-file:/var/home/returntrip/.local/share/containers/storage/overlay-containers/a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog --conmon-pidfile /run/user/1000/overlay-containers/a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/returntrip/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000 --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c]"
INFO[0000] Running conmon under slice user.slice and unitName libpod-conmon-a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c.scope 
WARN[0000] Failed to add conmon to systemd sandbox cgroup: read unix @->/run/systemd/private: read: connection reset by peer 
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Cleaning up container a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] unmounted container "a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c" 
ERRO[0000] unable to start container "fedora-toolbox-31": error reading container (probably exited) json message: EOF

Output of journalctl:

Sep 18 00:34:04 rauros.figura.io polkitd[1465]: Registered Authentication Agent for unix-process:26393:2803234 (system bus name :1.898 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_GB.UTF-8)
Sep 18 00:34:04 rauros.figura.io systemd[1605]: Started /usr/bin/podman --log-level debug start fedora-toolbox-31.
Sep 18 00:34:04 rauros.figura.io systemd[1]: Incoming private connection from unprivileged client, refusing: Operation not permitted
Sep 18 00:34:04 rauros.figura.io conmon[26423]: conmon a5f8742e74888002ed88 <ndebug>: failed to write to /proc/self/oom_score_adj: Permission denied
Sep 18 00:34:04 rauros.figura.io conmon[26424]: conmon a5f8742e74888002ed88 <ninfo>: attach sock path: /run/user/1000/libpod/tmp/socket/a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c/attach
Sep 18 00:34:04 rauros.figura.io conmon[26424]: conmon a5f8742e74888002ed88 <ninfo>: addr{sun_family=AF_UNIX, sun_path=/run/user/1000/libpod/tmp/socket/a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c/attach}
Sep 18 00:34:04 rauros.figura.io conmon[26424]: conmon a5f8742e74888002ed88 <ninfo>: ctl fifo path: /var/home/returntrip/.local/share/containers/storage/overlay-containers/a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c/userdata/ctl
Sep 18 00:34:04 rauros.figura.io conmon[26424]: conmon a5f8742e74888002ed88 <error>: Failed to mkfifo at /var/home/returntrip/.local/share/containers/storage/overlay-containers/a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c/userdata/ctl: File exists
Sep 18 00:34:04 rauros.figura.io polkitd[1465]: Unregistered Authentication Agent for unix-process:26393:2803234 (system bus name :1.898, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_GB.UTF-8) (disconnected from bus)
Sep 18 00:34:04 rauros.figura.io podman[26424]: 2019-09-18 00:34:04.96040169 +0200 CEST m=+0.051012634 container cleanup a5f8742e74888002ed888f4358ff358de43c46213601c7de6c436619b1afbf8c (image=registry.fedoraproject.org/f31/fedora-toolbox:31, name=fedora-toolbox-31)
Sep 18 00:34:04 rauros.figura.io systemd[1605]: run-rf4cb382bfa374051baf2944a824dc73f.scope: Succeeded.

returntrip on 18 Sep 2019

I went ahead and tested with podman-1.6.0-2.gitca5ff03.fc31.x86_64. To override podman I had to also override crun-0.9.1-1.fc31.x86_64

When I run toolbox enterjust hangs and systemd-run --scope --user podman --log-level debug start fedora-toolbox-31also hangs with this output:

Running scope as unit: run-r7a54502fbe2f48169d5514bd649458af.scope
INFO[0000] running as rootless                          
DEBU[0000] using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/returntrip/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/returntrip/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /var/home/returntrip/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/returntrip/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] overlay: mount_data=lowerdir=/var/home/returntrip/.local/share/containers/storage/overlay/l/ZICV27Z6WQJPPN4QQJB3FOLR4G:/var/home/returntrip/.local/share/containers/storage/overlay/l/342GDVZC4MZJG43YFEUVOS7YPM,upperdir=/var/home/returntrip/.local/share/containers/storage/overlay/77163a25bb1018335083ca47011e1ce84890a9d417872aa49e76b8dc9c3f0843/diff,workdir=/var/home/returntrip/.local/share/containers/storage/overlay/77163a25bb1018335083ca47011e1ce84890a9d417872aa49e76b8dc9c3f0843/work,context="system_u:object_r:container_file_t:s0:c85,c257" 
DEBU[0000] mounted container "d2d3e2bb8c6235c4b3fc0f9dd5f1f8e02c21a924122c8000e6e036df66b3f52f" at "/var/home/returntrip/.local/share/containers/storage/overlay/77163a25bb1018335083ca47011e1ce84890a9d417872aa49e76b8dc9c3f0843/merged" 
DEBU[0000] Created root filesystem for container d2d3e2bb8c6235c4b3fc0f9dd5f1f8e02c21a924122c8000e6e036df66b3f52f at /var/home/returntrip/.local/share/containers/storage/overlay/77163a25bb1018335083ca47011e1ce84890a9d417872aa49e76b8dc9c3f0843/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container d2d3e2bb8c6235c4b3fc0f9dd5f1f8e02c21a924122c8000e6e036df66b3f52f to user.slice:libpod:d2d3e2bb8c6235c4b3fc0f9dd5f1f8e02c21a924122c8000e6e036df66b3f52f 
DEBU[0000] set root propagation to "rslave"             
DEBU[0000] Created OCI spec for container d2d3e2bb8c6235c4b3fc0f9dd5f1f8e02c21a924122c8000e6e036df66b3f52f at /var/home/returntrip/.local/share/containers/storage/overlay-containers/d2d3e2bb8c6235c4b3fc0f9dd5f1f8e02c21a924122c8000e6e036df66b3f52f/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -s -c d2d3e2bb8c6235c4b3fc0f9dd5f1f8e02c21a924122c8000e6e036df66b3f52f -u d2d3e2bb8c6235c4b3fc0f9dd5f1f8e02c21a924122c8000e6e036df66b3f52f -r /usr/bin/crun -b /var/home/returntrip/.local/share/containers/storage/overlay-containers/d2d3e2bb8c6235c4b3fc0f9dd5f1f8e02c21a924122c8000e6e036df66b3f52f/userdata -p /run/user/1000/overlay-containers/d2d3e2bb8c6235c4b3fc0f9dd5f1f8e02c21a924122c8000e6e036df66b3f52f/userdata/pidfile -l k8s-file:/var/home/returntrip/.local/share/containers/storage/overlay-containers/d2d3e2bb8c6235c4b3fc0f9dd5f1f8e02c21a924122c8000e6e036df66b3f52f/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog --conmon-pidfile /run/user/1000/overlay-containers/d2d3e2bb8c6235c4b3fc0f9dd5f1f8e02c21a924122c8000e6e036df66b3f52f/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/returntrip/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000 --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg d2d3e2bb8c6235c4b3fc0f9dd5f1f8e02c21a924122c8000e6e036df66b3f52f]"
DEBU[0000] disabling SD notify                          
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: 12073                              
INFO[0000] Got Conmon PID as 12068                      
DEBU[0000] Created container d2d3e2bb8c6235c4b3fc0f9dd5f1f8e02c21a924122c8000e6e036df66b3f52f in OCI runtime 
DEBU[0000] Starting container d2d3e2bb8c6235c4b3fc0f9dd5f1f8e02c21a924122c8000e6e036df66b3f52f with command [toolbox --verbose init-container --home /home/returntrip --home-link --monitor-host --shell /bin/bash --uid 1000 --user returntrip]

Logs from journalctl

Sep 18 14:53:52 rauros.figura.io systemd[1609]: Started /usr/bin/podman --log-level debug start fedora-toolbox-31.
Sep 18 14:53:52 rauros.figura.io conmon[12067]: conmon d2d3e2bb8c6235c4b3fc <ndebug>: failed to write to /proc/self/oom_score_adj: Permission denied
Sep 18 14:53:52 rauros.figura.io conmon[12068]: conmon d2d3e2bb8c6235c4b3fc <ninfo>: attach sock path: /run/user/1000/libpod/tmp/socket/d2d3e2bb8c6235c4b3fc0f9dd5f1f8e02c21a924122c8000e6e036df66b3f52f/attach
Sep 18 14:53:52 rauros.figura.io conmon[12068]: conmon d2d3e2bb8c6235c4b3fc <ninfo>: addr{sun_family=AF_UNIX, sun_path=/run/user/1000/libpod/tmp/socket/d2d3e2bb8c6235c4b3fc0f9dd5f1f8e02c21a924122c8000e6e036df66b3f52f/attach}
Sep 18 14:53:52 rauros.figura.io conmon[12068]: conmon d2d3e2bb8c6235c4b3fc <ninfo>: ctl fifo path: /var/home/returntrip/.local/share/containers/storage/overlay-containers/d2d3e2bb8c6235c4b3fc0f9dd5f1f8e02c21a924122c8000e6e036df66b3f52f/userdata/ctl
Sep 18 14:53:52 rauros.figura.io conmon[12068]: conmon d2d3e2bb8c6235c4b3fc <ninfo>: terminal_ctrl_fd: 12
Sep 18 14:53:52 rauros.figura.io systemd[1609]: Started libcrun container.
Sep 18 14:53:52 rauros.figura.io conmon[12068]: conmon d2d3e2bb8c6235c4b3fc <ndebug>: container PID: 12073
Sep 18 14:53:52 rauros.figura.io conmon[12068]: conmon d2d3e2bb8c6235c4b3fc <ndebug>: couldn't find cb for pid 12072
Sep 18 14:53:52 rauros.figura.io podman[12038]: 2019-09-18 14:53:52.671048345 +0200 CEST m=+0.119450862 container init d2d3e2bb8c6235c4b3fc0f9dd5f1f8e02c21a924122c8000e6e036df66b3f52f (image=registry.fedoraproject.org/f31/fedora-toolbox:31, name=fedora-toolbox-31)

If I interrupt the hanging command and I try a rerun I get:

Running scope as unit: run-r9d5594fdf6be4abaaae060612a420b8c.scope
INFO[0000] running as rootless                          
DEBU[0000] using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/returntrip/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/returntrip/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /var/home/returntrip/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/returntrip/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] Created root filesystem for container 3e81817353354090119babcb2a33a9bd6bf9ea9b9c62b6be2d979012c5688f74 at /var/home/returntrip/.local/share/containers/storage/overlay/2b75bcfdf56a4df2caa9935d125a02ff20c0f0e16f53092a19a36527538a45ca/merged 
DEBU[0000] Starting container 3e81817353354090119babcb2a33a9bd6bf9ea9b9c62b6be2d979012c5688f74 with command [toolbox --verbose init-container --home /home/returntrip --home-link --monitor-host --shell /bin/bash --uid 1000 --user returntrip] 
2019-09-18T13:08:41.000103373Z: bind socket to '/run/user/1000/crun/3e81817353354090119babcb2a33a9bd6bf9ea9b9c62b6be2d979012c5688f74/notify/notify': Address already in use
ERRO[0000] unable to start container "fedora-toolbox-31": `/usr/bin/crun start 3e81817353354090119babcb2a33a9bd6bf9ea9b9c62b6be2d979012c5688f74` failed: exit status 1

returntrip on 18 Sep 2019

I hope you don't have NOTIFY_SOCKET in your environment. :)

debarshiray on 18 Sep 2019

@debarshiray Yeah it is set :|. But now I have removed the newer crun and podman versions, so will test again tomorrow with NOTIFY_SOCKET unset and the rest.. Thx.

returntrip on 18 Sep 2019

I too am seeing this issue, and after applying overrides for podman and crun, and unsetting NOTIFY_SOCKET, containers still fail to start.

Versions:
podman 1.6.0-2.gitca5ff03.fc31
crun 0.9.1-1.fc31

conmon version 2.0.0
commit: ae68c1c29156934457e2649990c241a5e0b0251b

Logs:

INFO[0000] running as rootless                          
DEBU[0000] using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/evan/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/evan/.local/share/containers/storage 
DEBU[0000] Using run root /tmp/1000                     
DEBU[0000] Using static dir /home/evan/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/evan/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] overlay: mount_data=lowerdir=/home/evan/.local/share/containers/storage/overlay/l/A6ZQYUBMXOV5BFSH5OFYXP6OXM:/home/evan/.local/share/containers/storage/overlay/l/YQZ3CD5TJRRAEAT4AM5RCW2DQH,upperdir=/home/evan/.local/share/containers/storage/overlay/cd89760b5ea482b29591fde866d8adc8ae8041db068fcf18c068a62f62b7abe4/diff,workdir=/home/evan/.local/share/containers/storage/overlay/cd89760b5ea482b29591fde866d8adc8ae8041db068fcf18c068a62f62b7abe4/work,context="system_u:object_r:container_file_t:s0:c423,c497" 
DEBU[0000] mounted container "c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640" at "/home/evan/.local/share/containers/storage/overlay/cd89760b5ea482b29591fde866d8adc8ae8041db068fcf18c068a62f62b7abe4/merged" 
DEBU[0000] Created root filesystem for container c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640 at /var/home/evan/.local/share/containers/storage/overlay/cd89760b5ea482b29591fde866d8adc8ae8041db068fcf18c068a62f62b7abe4/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640 to user.slice:libpod:c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640 
DEBU[0000] set root propagation to "rslave"             
DEBU[0000] Created OCI spec for container c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640 at /home/evan/.local/share/containers/storage/overlay-containers/c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -s -c c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640 -u c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640 -r /usr/bin/crun -b /home/evan/.local/share/containers/storage/overlay-containers/c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640/userdata -p /tmp/1000/overlay-containers/c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640/userdata/pidfile -l k8s-file:/home/evan/.local/share/containers/storage/overlay-containers/c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog --conmon-pidfile /tmp/1000/overlay-containers/c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/evan/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /tmp/1000 --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640]"
DEBU[0000] disabling SD notify                          
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: -1                                 
DEBU[0000] Cleaning up container c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] unmounted container "c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640" 
ERRO[0000] unable to start container "fedora-toolbox-31": statfs '/var/home/evan/.local/share/containers/storage/overlay/cd89760b5ea482b29591fde866d8adc8ae8041db068fcf18c068a62f62b7abe4/merged/etc/profile.d/toolbox.sh': No such file or directory: OCI runtime command not found error

Sep 19 00:29:40 rob systemd[1403]: libpod-0b9fc319c19a5c7778c71ddb89a7bfc16d23967c71fc7f7504042a507c4b8172.scope: Succeeded.
Sep 19 00:29:40 rob conmon[10285]: conmon 0b9fc319c19a5c7778c7 <error>: Failed to create container: exit status 1
Sep 19 00:29:40 rob systemd[1403]: Started libcrun container.
Sep 19 00:29:40 rob conmon[10285]: conmon 0b9fc319c19a5c7778c7 <ninfo>: terminal_ctrl_fd: 12
Sep 19 00:29:40 rob conmon[10285]: conmon 0b9fc319c19a5c7778c7 <ninfo>: ctl fifo path: /home/evan/.local/share/containers/storage/overlay-containers/0b9fc319c19a5c7778c71ddb89a7bfc16d23967c71fc7f7504042a507c4b8172/userdata/ctl
Sep 19 00:29:40 rob conmon[10285]: conmon 0b9fc319c19a5c7778c7 <ninfo>: addr{sun_family=AF_UNIX, sun_path=/run/user/1000/libpod/tmp/socket/0b9fc319c19a5c7778c71ddb89a7bfc16d23967c71fc7f7504042a507c4b8172/attach}
Sep 19 00:29:40 rob conmon[10285]: conmon 0b9fc319c19a5c7778c7 <ninfo>: attach sock path: /run/user/1000/libpod/tmp/socket/0b9fc319c19a5c7778c71ddb89a7bfc16d23967c71fc7f7504042a507c4b8172/attach
Sep 19 00:29:40 rob conmon[10284]: conmon 0b9fc319c19a5c7778c7 <ndebug>: failed to write to /proc/self/oom_score_adj: Permission denied
Sep 19 00:29:40 rob systemd[1403]: Started /usr/bin/podman --log-level debug start fedora-toolbox-31.

evan-a-a on 19 Sep 2019

👍1

That looks more like toolbox than Podman, given it's a not found in the
entrypoint...

On Thu, Sep 19, 2019, 01:07 Evan Anderson notifications@github.com wrote:

I too am seeing this issue, and after applying overrides for podman and
crun, and unsetting NOTIFY_SOCKET, containers still fail to start.
Versions:
podman 1.6.0-0.7.dev.git408f278.fc32
crun 0.9.1-1.fc31

conmon version 2.0.0
commit: ae68c1c29156934457e2649990c241a5e0b0251b

Logs:

INFO[0000] running as rootless
DEBU[0000] using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/evan/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/evan/.local/share/containers/storage
DEBU[0000] Using run root /tmp/1000
DEBU[0000] Using static dir /home/evan/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp
DEBU[0000] Using volume path /home/evan/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] Initializing event backend journald
DEBU[0000] using runtime "/usr/bin/crun"
DEBU[0000] using runtime "/usr/bin/runc"
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
DEBU[0000] overlay: mount_data=lowerdir=/home/evan/.local/share/containers/storage/overlay/l/A6ZQYUBMXOV5BFSH5OFYXP6OXM:/home/evan/.local/share/containers/storage/overlay/l/YQZ3CD5TJRRAEAT4AM5RCW2DQH,upperdir=/home/evan/.local/share/containers/storage/overlay/cd89760b5ea482b29591fde866d8adc8ae8041db068fcf18c068a62f62b7abe4/diff,workdir=/home/evan/.local/share/containers/storage/overlay/cd89760b5ea482b29591fde866d8adc8ae8041db068fcf18c068a62f62b7abe4/work,context="system_u:object_r:container_file_t:s0:c423,c497"
DEBU[0000] mounted container "c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640" at "/home/evan/.local/share/containers/storage/overlay/cd89760b5ea482b29591fde866d8adc8ae8041db068fcf18c068a62f62b7abe4/merged"
DEBU[0000] Created root filesystem for container c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640 at /var/home/evan/.local/share/containers/storage/overlay/cd89760b5ea482b29591fde866d8adc8ae8041db068fcf18c068a62f62b7abe4/merged
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret
DEBU[0000] Setting CGroups for container c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640 to user.slice:libpod:c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640
DEBU[0000] set root propagation to "rslave"
DEBU[0000] Created OCI spec for container c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640 at /home/evan/.local/share/containers/storage/overlay-containers/c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640/userdata/config.json
DEBU[0000] /usr/bin/conmon messages will be logged to syslog
DEBU[0000] running conmon: /usr/bin/conmon args="[--api-version 1 -s -c c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640 -u c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640 -r /usr/bin/crun -b /home/evan/.local/share/containers/storage/overlay-containers/c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640/userdata -p /tmp/1000/overlay-containers/c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640/userdata/pidfile -l k8s-file:/home/evan/.local/share/containers/storage/overlay-containers/c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog --conmon-pidfile /tmp/1000/overlay-containers/c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/evan/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /tmp/1000 --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640]"
DEBU[0000] disabling SD notify
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: -1
DEBU[0000] Cleaning up container c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] unmounted container "c7910a4cf17dae8ec611467cbfa63de9a8448820fe851f48aba9818583524640"
ERRO[0000] unable to start container "fedora-toolbox-31": statfs '/var/home/evan/.local/share/containers/storage/overlay/cd89760b5ea482b29591fde866d8adc8ae8041db068fcf18c068a62f62b7abe4/merged/etc/profile.d/toolbox.sh': No such file or directory: OCI runtime command not found error

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/containers/libpod/issues/4024?email_source=notifications&email_token=AB3AOCFPJGTFQQ4C3YXHSJTQKMCKPA5CNFSM4IWRM2O2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7CHDHY#issuecomment-532967839,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB3AOCHRJHRPV4D2NEQ73NTQKMCKPANCNFSM4IWRM2OQ
.

mheon on 19 Sep 2019

@debarshiray thanks, that made it work :)

For other with this issue, NEW toolboxes work with unset NOTIFY_SOCKET and these packages overridden:

crun-0.9.1-1.fc31.x86_64
podman-1.6.0-2.gitca5ff03.fc31.x86_64

returntrip on 19 Sep 2019

gnome-session bug tracked here: https://bugzilla.redhat.com/show_bug.cgi?id=1753328

giuseppe on 19 Sep 2019

That looks more like toolbox than Podman, given it's a not found in the entrypoint...

Did you mean this:

ERRO[0000] unable to start container "fedora-toolbox-31": statfs '/var/home/evan/.local/share/containers/storage/overlay/cd89760b5ea482b29591fde866d8adc8ae8041db068fcf18c068a62f62b7abe4/merged/etc/profile.d/toolbox.sh': No such file or directory: OCI runtime command not found error

That's not the entrypoint though. It's a configuration file that's either bind mounted or copied into the container depending on which version of Toolbox created the container.

Maybe we should just write this issue off as the NOTIFY_SOCKET problem? :)

debarshiray on 19 Sep 2019

For other with this issue, NEW toolboxes work with unset NOTIFY_SOCKET and these packages overridden:
crun-0.9.1-1.fc31.x86_64
podman-1.6.0-2.gitca5ff03.fc31.x86_64

That worked for me ... once.

After a new container has stopped, it fails to start again:

$ systemd-run --scope --user podman --log-level debug start pkg
Running scope as unit: run-r2a26d1c35ba0408a9b528d8c11d04d9b.scope
INFO[0000] running as rootless                          
DEBU[0000] using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/fmuellner/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/fmuellner/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /var/home/fmuellner/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/fmuellner/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] overlay: mount_data=lowerdir=/var/home/fmuellner/.local/share/containers/storage/overlay/l/6AS62C2FPSH3SYDVN2QGKLAMDT:/var/home/fmuellner/.local/share/containers/storage/overlay/l/YVON65E27VEP6KMCXEPSJ5D6T3,upperdir=/var/home/fmuellner/.local/share/containers/storage/overlay/f5db83a87ed505bae645773ef5d2ddf4506a40527e8ff4d2278e7ef210a37746/diff,workdir=/var/home/fmuellner/.local/share/containers/storage/overlay/f5db83a87ed505bae645773ef5d2ddf4506a40527e8ff4d2278e7ef210a37746/work,context="system_u:object_r:container_file_t:s0:c69,c512" 
DEBU[0000] mounted container "5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222" at "/var/home/fmuellner/.local/share/containers/storage/overlay/f5db83a87ed505bae645773ef5d2ddf4506a40527e8ff4d2278e7ef210a37746/merged" 
DEBU[0000] Created root filesystem for container 5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222 at /var/home/fmuellner/.local/share/containers/storage/overlay/f5db83a87ed505bae645773ef5d2ddf4506a40527e8ff4d2278e7ef210a37746/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container 5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222 to user.slice:libpod:5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222 
DEBU[0000] set root propagation to "rslave"             
DEBU[0000] Created OCI spec for container 5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222 at /var/home/fmuellner/.local/share/containers/storage/overlay-containers/5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -s -c 5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222 -u 5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222 -r /usr/bin/crun -b /var/home/fmuellner/.local/share/containers/storage/overlay-containers/5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222/userdata -p /run/user/1000/overlay-containers/5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222/userdata/pidfile -l k8s-file:/var/home/fmuellner/.local/share/containers/storage/overlay-containers/5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog --conmon-pidfile /run/user/1000/overlay-containers/5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/fmuellner/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000 --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222]"
DEBU[0000] disabling SD notify                          
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Cleaning up container 5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] unmounted container "5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222" 
ERRO[0000] unable to start container "pkg": error reading container (probably exited) json message: EOF

fmuellner on 19 Sep 2019

Luckily cleaning out ~/.local/share/containers/storage/overlay-containers//userdata/ fixes the last error ... because of course the recreated toolboxes don't work in F30, so downgrading is no longer an option ...

fmuellner on 19 Sep 2019

For other with this issue, NEW toolboxes work with unset NOTIFY_SOCKET and these packages overridden:
crun-0.9.1-1.fc31.x86_64
podman-1.6.0-2.gitca5ff03.fc31.x86_64

That worked for me ... once.

After a new container has stopped, it fails to start again:

$ systemd-run --scope --user podman --log-level debug start pkg
Running scope as unit: run-r2a26d1c35ba0408a9b528d8c11d04d9b.scope
INFO[0000] running as rootless                          
DEBU[0000] using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/fmuellner/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/fmuellner/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /var/home/fmuellner/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/fmuellner/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] overlay: mount_data=lowerdir=/var/home/fmuellner/.local/share/containers/storage/overlay/l/6AS62C2FPSH3SYDVN2QGKLAMDT:/var/home/fmuellner/.local/share/containers/storage/overlay/l/YVON65E27VEP6KMCXEPSJ5D6T3,upperdir=/var/home/fmuellner/.local/share/containers/storage/overlay/f5db83a87ed505bae645773ef5d2ddf4506a40527e8ff4d2278e7ef210a37746/diff,workdir=/var/home/fmuellner/.local/share/containers/storage/overlay/f5db83a87ed505bae645773ef5d2ddf4506a40527e8ff4d2278e7ef210a37746/work,context="system_u:object_r:container_file_t:s0:c69,c512" 
DEBU[0000] mounted container "5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222" at "/var/home/fmuellner/.local/share/containers/storage/overlay/f5db83a87ed505bae645773ef5d2ddf4506a40527e8ff4d2278e7ef210a37746/merged" 
DEBU[0000] Created root filesystem for container 5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222 at /var/home/fmuellner/.local/share/containers/storage/overlay/f5db83a87ed505bae645773ef5d2ddf4506a40527e8ff4d2278e7ef210a37746/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container 5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222 to user.slice:libpod:5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222 
DEBU[0000] set root propagation to "rslave"             
DEBU[0000] Created OCI spec for container 5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222 at /var/home/fmuellner/.local/share/containers/storage/overlay-containers/5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -s -c 5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222 -u 5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222 -r /usr/bin/crun -b /var/home/fmuellner/.local/share/containers/storage/overlay-containers/5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222/userdata -p /run/user/1000/overlay-containers/5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222/userdata/pidfile -l k8s-file:/var/home/fmuellner/.local/share/containers/storage/overlay-containers/5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog --conmon-pidfile /run/user/1000/overlay-containers/5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/fmuellner/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000 --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222]"
DEBU[0000] disabling SD notify                          
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Cleaning up container 5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] unmounted container "5e2bf791560226349c527b1b87f7dfcc6dc20e01f234faaef8ed504a13415222" 
ERRO[0000] unable to start container "pkg": error reading container (probably exited) json message: EOF

@fmuellner yeah confirmed :(

returntrip on 19 Sep 2019

Woah - the downgrade broke things?

Do both F30 and F31 have the same Podman version?

mheon on 19 Sep 2019

Argh - userdata fixed it? It's probably Conmon refusing to recreate the ctl file then.

mheon on 19 Sep 2019

Do both F30 and F31 have the same Podman version?

I don't know, I quickly switched back to F31. The error was about crun missing.

Argh - userdata fixed it?

Interestingly enough, I forgot to override podman/crun again when switching back to F31. So this fixed the issue with the podman/crun versions in the current image.

fmuellner on 19 Sep 2019

Hi,

I'm running Fedora Silverblue 31 (base commit 80944945d229dc557bc8c2b9e49c7bfbe055cc7b5537bcf8b9aa6893061e1fc4) witch contains the following packages

podman-2:1.6.2-2.fc31.x86_64
crun-0.10.5-2.fc31.x86_64
conmon-2:2.0.2-1.fc31.x86_64
toolbox-0.0.16-1.fc31.noarch

and I'm also unable to start a toolbox.

toolbox enter
No toolbox containers found. Create now? [y/N] y
Image required to create toolbox container.
Download registry.fedoraproject.org/f31/fedora-toolbox:31 (500MB)? [y/N]: y
toolbox: failed to start container fedora-toolbox-31

NOTIFY_SOCKET is not set and I've done rm -rf ~/.local/share/containers ~/.config/containers ~/.config/toolbox before trying to create a new toolbox.

I found nothing conmon related in journalctl's output, I only found these lines:

nov. 18 10:22:49 work-laptop podman[11296]: 2019-11-18 10:22:49.875183241 +0100 CET m=+23.976717159 image pull  
nov. 18 10:22:50 work-laptop podman[11479]: 2019-11-18 10:22:50.183431794 +0100 CET m=+0.061530635 container create 617a8c010139271a8983957f4fbb0d81049e8d4371143e9e34a794fe7f221321 (image=registry.fedoraproject.org/f31/fedora-toolbox:31, name=fedora-toolbox-31)
nov. 18 10:22:51 work-laptop systemd[1231]: Started libcrun container.
nov. 18 10:22:51 work-laptop systemd[1231]: libpod-617a8c010139271a8983957f4fbb0d81049e8d4371143e9e34a794fe7f221321.scope: Succeeded.
nov. 18 10:22:51 work-laptop podman[11553]: 2019-11-18 10:22:51.401371887 +0100 CET m=+0.046879652 container cleanup 617a8c010139271a8983957f4fbb0d81049e8d4371143e9e34a794fe7f221321 (image=registry.fedoraproject.org/f31/fedora-toolbox:31, name=fedora-toolbox-31)

Here is the output of systemd-run --scope --user podman --log-level debug start fedora-toolbox-31

Running scope as unit: run-rb77b8c5cc38e40e0bcc68f159ddcae94.scope
DEBU[0000] using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/yann/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/yann/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /var/home/yann/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/yann/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] using runtime "/usr/bin/runc"                
INFO[0000] running as rootless                          
DEBU[0000] using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/yann/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/yann/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /var/home/yann/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/yann/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] overlay: mount_data=lowerdir=/var/home/yann/.local/share/containers/storage/overlay/l/IXELAXBEWYHCTRWWW2GWCQGBBL:/var/home/yann/.local/share/containers/storage/overlay/l/XBJEYEAZIDMRFQRUPSPX3GTHBC,upperdir=/var/home/yann/.local/share/containers/storage/overlay/bb73298c9bfadd94beba4db080bdc5397544c03bd37ed233cb0179178bb50f71/diff,workdir=/var/home/yann/.local/share/containers/storage/overlay/bb73298c9bfadd94beba4db080bdc5397544c03bd37ed233cb0179178bb50f71/work,context="system_u:object_r:container_file_t:s0:c587,c710" 
DEBU[0000] mounted container "617a8c010139271a8983957f4fbb0d81049e8d4371143e9e34a794fe7f221321" at "/var/home/yann/.local/share/containers/storage/overlay/bb73298c9bfadd94beba4db080bdc5397544c03bd37ed233cb0179178bb50f71/merged" 
DEBU[0000] Created root filesystem for container 617a8c010139271a8983957f4fbb0d81049e8d4371143e9e34a794fe7f221321 at /var/home/yann/.local/share/containers/storage/overlay/bb73298c9bfadd94beba4db080bdc5397544c03bd37ed233cb0179178bb50f71/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container 617a8c010139271a8983957f4fbb0d81049e8d4371143e9e34a794fe7f221321 to user.slice:libpod:617a8c010139271a8983957f4fbb0d81049e8d4371143e9e34a794fe7f221321 
DEBU[0000] set root propagation to "rslave"             
DEBU[0000] Created OCI spec for container 617a8c010139271a8983957f4fbb0d81049e8d4371143e9e34a794fe7f221321 at /var/home/yann/.local/share/containers/storage/overlay-containers/617a8c010139271a8983957f4fbb0d81049e8d4371143e9e34a794fe7f221321/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -s -c 617a8c010139271a8983957f4fbb0d81049e8d4371143e9e34a794fe7f221321 -u 617a8c010139271a8983957f4fbb0d81049e8d4371143e9e34a794fe7f221321 -r /usr/bin/crun -b /var/home/yann/.local/share/containers/storage/overlay-containers/617a8c010139271a8983957f4fbb0d81049e8d4371143e9e34a794fe7f221321/userdata -p /run/user/1000/overlay-containers/617a8c010139271a8983957f4fbb0d81049e8d4371143e9e34a794fe7f221321/userdata/pidfile -l k8s-file:/var/home/yann/.local/share/containers/storage/overlay-containers/617a8c010139271a8983957f4fbb0d81049e8d4371143e9e34a794fe7f221321/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog --conmon-pidfile /run/user/1000/overlay-containers/617a8c010139271a8983957f4fbb0d81049e8d4371143e9e34a794fe7f221321/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/yann/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000 --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg 617a8c010139271a8983957f4fbb0d81049e8d4371143e9e34a794fe7f221321]"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: -1                                 
DEBU[0000] Cleaning up container 617a8c010139271a8983957f4fbb0d81049e8d4371143e9e34a794fe7f221321 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] unmounted container "617a8c010139271a8983957f4fbb0d81049e8d4371143e9e34a794fe7f221321" 
ERRO[0000] unable to start container "fedora-toolbox-31": creating file '/var/home/yann/.local/share/containers/storage/overlay/bb73298c9bfadd94beba4db080bdc5397544c03bd37ed233cb0179178bb50f71/merged/mnt': Is a directory: OCI runtime error

yann-soubeyrand on 18 Nov 2019

Seems like the issue is in crun: https://github.com/containers/toolbox/issues/330.

yann-soubeyrand on 18 Nov 2019

@yann-soubeyrand: Yep. There's a workaround to downgrade crun until a fixed version finally makes its way into Fedora (and then Silverblue): https://github.com/containers/toolbox/issues/330#issuecomment-554292574

It's a bummer that it takes so long for what can easily be considered — especially for Silverblue — a critical fix. Hopefully tests can catch this kind of thing in the future?

_Edit:_ I mean "takes so long" to see toolbox working on Silverblue again. It would be nice if composes could have a hotfix for when something like this happens. We could've had crun downgraded in the next compose, for example, so rpm-ostree update would make podman & toolbox work again (with an older crun for the time being), instead of having to wait for the new version with the bug to be fixed, released, built, checked, and then included.

Software development, of course, takes time and I'm happy we have awesome developers who can identify and fix this relatively quickly. :+1:

garrett on 18 Nov 2019

made a new release: https://bodhi.fedoraproject.org/updates/FEDORA-2019-4b4957bbc6

giuseppe on 18 Nov 2019

👍3

made a new release: https://bodhi.fedoraproject.org/updates/FEDORA-2019-4b4957bbc6

Thanks tested and given Karma.

returntrip on 18 Nov 2019

seems to be working for me!

dustymabe on 18 Nov 2019

Thank you! toolbox works again!

stephanmol on 20 Nov 2019

Hello,

I'm running a fresh SB 31 install, and I am hitting the same error.

Notably something different, is that toolbox was working before. Then it stopped working.

I'm not sure what changed, perhaps it was a reboot after an rpm-ostree update? Or maybe just a long running session, and at one point I closed all the toolbox shells and then tried to enter a new one... I'm really not sure.

I thought about opening a new issue, but the symptoms seem to be the same as this one.

crun version

$ rpm -qa | grep crun
crun-0.10.6-1.fc31.x86_64

Running manually with podman

$ podman --log-level debug start fedora-toolbox-31
DEBU[0000] using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/ramblurr/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/ramblurr/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /var/home/ramblurr/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/ramblurr/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
INFO[0000] running as rootless                          
DEBU[0000] using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/ramblurr/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/ramblurr/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /var/home/ramblurr/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/ramblurr/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] overlay: mount_data=lowerdir=/var/home/ramblurr/.local/share/containers/storage/overlay/l/D3NXUWR7Y46VJCUIHS4IL6X7AI:/var/home/ramblurr/.local/share/containers/storage/overlay/l/S44FSYXRO7Z2FPOVNDJJ6UGPIC,upperdir=/var/home/ramblurr/.local/share/containers/storage/overlay/00269047d4a5354444b06f8e984a230f870f779f999538db926ea2a90e77597b/diff,workdir=/var/home/ramblurr/.local/share/containers/storage/overlay/00269047d4a5354444b06f8e984a230f870f779f999538db926ea2a90e77597b/work,context="system_u:object_r:container_file_t:s0:c310,c939" 
DEBU[0000] mounted container "d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98" at "/var/home/ramblurr/.local/share/containers/storage/overlay/00269047d4a5354444b06f8e984a230f870f779f999538db926ea2a90e77597b/merged" 
DEBU[0000] Created root filesystem for container d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98 at /var/home/ramblurr/.local/share/containers/storage/overlay/00269047d4a5354444b06f8e984a230f870f779f999538db926ea2a90e77597b/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98 to user.slice:libpod:d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98 
DEBU[0000] set root propagation to "rslave"             
DEBU[0000] Created OCI spec for container d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98 at /var/home/ramblurr/.local/share/containers/storage/overlay-containers/d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -s -c d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98 -u d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98 -r /usr/bin/crun -b /var/home/ramblurr/.local/share/containers/storage/overlay-containers/d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98/userdata -p /run/user/1000/overlay-containers/d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98/userdata/pidfile -l k8s-file:/var/home/ramblurr/.local/share/containers/storage/overlay-containers/d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog --conmon-pidfile /run/user/1000/overlay-containers/d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/ramblurr/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000 --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98]"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

Running using systemd:

$ systemd-run --scope --user podman --log-level debug start fedora-toolbox-31
Running scope as unit: run-rc2bb5fa719624f758453d23762a46569.scope
DEBU[0000] using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/ramblurr/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/ramblurr/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /var/home/ramblurr/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/ramblurr/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
INFO[0000] running as rootless                          
DEBU[0000] using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /var/home/ramblurr/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /var/home/ramblurr/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000                
DEBU[0000] Using static dir /var/home/ramblurr/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /var/home/ramblurr/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] Initializing event backend journald          
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] overlay: mount_data=lowerdir=/var/home/ramblurr/.local/share/containers/storage/overlay/l/D3NXUWR7Y46VJCUIHS4IL6X7AI:/var/home/ramblurr/.local/share/containers/storage/overlay/l/S44FSYXRO7Z2FPOVNDJJ6UGPIC,upperdir=/var/home/ramblurr/.local/share/containers/storage/overlay/00269047d4a5354444b06f8e984a230f870f779f999538db926ea2a90e77597b/diff,workdir=/var/home/ramblurr/.local/share/containers/storage/overlay/00269047d4a5354444b06f8e984a230f870f779f999538db926ea2a90e77597b/work,context="system_u:object_r:container_file_t:s0:c310,c939" 
DEBU[0000] mounted container "d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98" at "/var/home/ramblurr/.local/share/containers/storage/overlay/00269047d4a5354444b06f8e984a230f870f779f999538db926ea2a90e77597b/merged" 
DEBU[0000] Created root filesystem for container d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98 at /var/home/ramblurr/.local/share/containers/storage/overlay/00269047d4a5354444b06f8e984a230f870f779f999538db926ea2a90e77597b/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98 to user.slice:libpod:d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98 
DEBU[0000] set root propagation to "rslave"             
DEBU[0000] Created OCI spec for container d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98 at /var/home/ramblurr/.local/share/containers/storage/overlay-containers/d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -s -c d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98 -u d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98 -r /usr/bin/crun -b /var/home/ramblurr/.local/share/containers/storage/overlay-containers/d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98/userdata -p /run/user/1000/overlay-containers/d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98/userdata/pidfile -l k8s-file:/var/home/ramblurr/.local/share/containers/storage/overlay-containers/d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98/userdata/ctr.log --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket --log-level debug --syslog --conmon-pidfile /run/user/1000/overlay-containers/d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /var/home/ramblurr/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000 --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg journald --exit-command-arg container --exit-command-arg cleanup --exit-command-arg d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98]"
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: -1                                 
DEBU[0000] Cleaning up container d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98 
DEBU[0000] Network is already cleaned up, skipping...   
DEBU[0000] unmounted container "d58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98" 
ERRO[0000] unable to start container "fedora-toolbox-31": container 'd58ddb73de71d2ccdbbfc05b8f1b162426ae93adc3ea509e6147ed687f8e1f98' already exists: OCI runtime error

Ramblurr on 13 Dec 2019

looks like the fedora-toolbox-31 container wasn't correctly deleted from the OCI runtime.

Can you check what you have under /run/user/1000/crun?

giuseppe on 13 Dec 2019

Ugh, a reboot fixed the issue. Could this have happened when I podman kill a toolbox container?

Sometimes when I open a new Terminal tab (when the active tab is a toolbox shell), the new tab shows the toolbox prompt, then freezes. My only recourse is to close the tab. I then podman kill the container.

Ramblurr on 13 Dec 2019

Was this page helpful?

0 / 5 - 0 ratings