Podman: Error: stat /sys/fs/cgroup/systemd/org/freedesktop/ConsoleKit/Session1: no such file or directory

[Noah-Huppert]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Any podman command exits with the error:

Error: stat /sys/fs/cgroup/systemd/org/freedesktop/ConsoleKit/Session1: no such file or directory

and the status 125.

Steps to reproduce the issue:

1. Have podman 1.6.3 on Void Linux

2. Run any podman command (ex: podman ls)

3. Error

Describe the results you received:

Every command exits with status 125 and the error above.

Describe the results you expected:

Podman to complete action for command.

Additional information you deem important (e.g. issue happens only occasionally):

• Happens every time
• I'm not using a system with systemd
• Podman had been working before I updated (Last updated ~1-2 months ago)

Output of podman version:

podman version 1.6.3

Output of podman info --debug:

Error: stat /sys/fs/cgroup/systemd/org/freedesktop/ConsoleKit/Session1: no such file or directory

But buildah info --debug doesn't error:

{
    "debug": {
        "buildah version": "1.11.4",
        "compiler": "gc",
        "git commit": "",
        "go version": "go1.13.3"
    },
    "host": {
        "CgroupVersion": "v1",
        "Distribution": {
            "distribution": "\"void\"",
            "version": "unknown"
        },
        "MemTotal": 16611033088,
        "MenFree": 14549090304,
        "SwapFree": 0,
        "SwapTotal": 0,
        "arch": "amd64",
        "cpus": 8,
        "hostname": "apollo",
        "kernel": "5.3.14_1",
        "os": "linux",
        "rootless": true,
        "uptime": "3m 12.94s"
    },
    "store": {
        "ContainerStore": {
            "number": 22
        },
        "GraphDriverName": "vfs",
        "GraphOptions": null,
        "GraphRoot": "/home/noah/.local/share/containers/storage",
        "GraphStatus": {},
        "ImageStore": {
            "number": 31
        },
        "RunRoot": "/run/user/1000"
    }
}

Package info (e.g. output of rpm -q podman or apt list podman):

% xq podman
pkgver: podman-1.6.3_1
short_desc: Simple management tool for containers and images
architecture: x86_64
build-date: 2019-11-17 06:45 UTC
filename-sha256: 0143062169ce4be4e48d71535cc293a2c4c1475afc5c55103b6c9bdad63d9053
filename-size: 17MB
homepage: https://podman.io/
installed_size: 52MB
license: Apache-2.0
maintainer: Cameron Nemo <[email protected]>
repository: http://mirror.clarkson.edu/voidlinux/current
shlib-requires:
    libpthread.so.0
    libgpgme.so.11
    libassuan.so.0
    libgpg-error.so.0
    libseccomp.so.2
    librt.so.1
    libdevmapper.so.1.02
    libc.so.6
source-revisions: podman:38b0579768
depends:
    runc>=0
    conmon>=0
    cni-plugins>=0
    slirp4netns>=0
    containers.image>=0
    glibc>=2.29_1
    libgpgme>=1.12.0_2
    libassuan>=2.0.1_1
    libgpg-error>=1.6_1
    libseccomp>=2.0.0_1
    device-mapper>=2.02.110_1

Additional environment details (AWS, VirtualBox, physical, etc.):

Void Linux with runit on a Lenovo X1 7th generation.

% uname -a
Linux apollo 5.3.14_1 #1 SMP PREEMPT Fri Nov 29 14:39:11 UTC 2019 x86_64 GNU/Linux

% lsb_release -a
LSB Version:    1.0
Distributor ID: VoidLinux
Description:    Void Linux
Release:    rolling
Codename:   void

[mheon]

@giuseppe Looks like we're trying to hit systemd over dbus on a system without systemd

[Noah-Huppert]

Is there a way to manually tell podman not to use systemd?

I saw something about a --systemd flag somewhere so I tried running with --systemd=false but the error still occurred.

[mheon]

That's specific to systemd in a container - won't apply here.

Are you running as root, or without? Can you check your libpod.conf (usually lives in ~/.config/containers/libpod.conf if you're not root, or /etc/containers/libpod.conf and /usr/share/containers/libpod.conf if you are root) and see what cgroup_manager is set to?

[mheon]

I don't think root Podman should ever need to talk directly to systemd. Rootless does some systemd-specific configuration, but I think it should be optional - we shouldn't need it to work.

[Noah-Huppert]

I believe I am running rootless.

% cat ~/.config/containers/libpod.conf | grep cgroup_manager
cgroup_manager = "cgroupfs"

[rhatdan]

That looks like your are running root? '# "?
Are you running with cgroup V2? I think Podman/crun will attempt to talk to systemd on a cgroup V2 machine. The other area might be on the event manager. Changing it to file might fix this.

[Noah-Huppert]

@rhatdan ah the # I added in just so if the commands were copy pasted they wouldn't run, out of habit. I have edited my past comments to use % to reduce confusion. I am running as a regular user account.

I think I have cgroup V2 on my machine, I can mount with type cgroup2:

% sudo mount -t cgroup2 none /mnt

I'm not quite sure how to change the event manager. I see a events_logger option in libpod.conf which is set to file but nothing about an event manager.

[rhatdan]

podman --events-backend=file
But if it is using cgroup V2 then we are going to have problems. Podman uses systemd to setup the device cgroup. If you boot with cgroup V1 it should work without requiring systemd.
@giuseppe WDYT?

[mheon]

Per buildah info he's on CGroups v1

[mheon]

events_logger in libpod.conf should be set to file - if it's set to journal, that would connect to systemd.

I'm wondering if this is our linger support check, though.

[Noah-Huppert]

Changing the events backend to file did not seem to work:

% podman --events-backend=file ls
Error: stat /sys/fs/cgroup/systemd/org/freedesktop/ConsoleKit/Session1: no such file or directory

Yea I couldn't confirm for sure if I was using cgroups 1 or 2, but as @mheon mentioned it looks like I'm on v1.

[Noah-Huppert]

Here are my ~/.config/containers/{libpod.conf,storage.conf} files. https://gist.github.com/Noah-Huppert/cc8859c62be4535a458a0be3d624b05d

If any more debug information would be useful, or if there are any leads I could try and chase down, let me know.

[giuseppe]

the only two settings I can think of are cgroup_manager and --events-backend=file.

In theory it should work also with cgroups v2 on a system without systemd (without allowing the usage of resources, as on a cgroups v1 system), but I've personally never tried so.

Probably the easiest is to debug it on a system without systemd installed. @Noah-Huppert how have you installed the other dependencies? Are they already available on Void Linux or you needed to build them by yourself?

[mheon]

@Noah-Huppert Can you run a Podman command with --log-level=debug enabled and provide the output? Would help to know exactly where things are blowing up on account systemd not being present

[Noah-Huppert]

Yea sure, I'll post logs soon.

I installed podman through Void Linux's package manager. Here is the package build configuration: https://github.com/void-linux/void-packages/blob/master/srcpkgs/podman/template

If I had to guess I would say that the package manager is probably not setting up one of the dependencies required for rootless. I will verify if this is true in a few days when I also post the log.

[Noah-Huppert]

Log level debug:

% podman --log-level=debug --events-backend=file --cgroup-manager=cgroupfs ls
time="2019-12-14T14:09:14-05:00" level=debug msg="Reading configuration file \"/home/noah/.config/containers/libpod.conf\""
time="2019-12-14T14:09:14-05:00" level=debug msg="Merged system config \"/home/noah/.config/containers/libpod.conf\": &{{false false false true true true} 0 {   [] [] []} /home/noah/.local/share/containers/storage/volumes docker://  runc map[crun:[/usr/bin/crun /usr/local/bin/crun] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc]] [] [] [] [/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon] [PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] cgroupfs  /home/noah/.local/share/containers/storage/libpod /run/user/1000/libpod/tmp -1 false /etc/cni/net.d/ [/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin]  []   k8s.gcr.io/pause:3.1 /pause true true  2048  file  ctrl-p,ctrl-q false}"
time="2019-12-14T14:09:14-05:00" level=warning msg="The cgroups manager is set to systemd but there is no systemd user session available"
time="2019-12-14T14:09:14-05:00" level=warning msg="For using systemd, you may need to login using an user session"
time="2019-12-14T14:09:14-05:00" level=warning msg="Alternatively, you can enable lingering with: `loginctl enable-linger 1000` (possibly as root)"
time="2019-12-14T14:09:14-05:00" level=warning msg="Falling back to --cgroup-manager=cgroupfs and --events-backend=file"
time="2019-12-14T14:09:14-05:00" level=debug msg="Using conmon: \"/usr/libexec/podman/conmon\""
time="2019-12-14T14:09:14-05:00" level=debug msg="Initializing boltdb state at /home/noah/.local/share/containers/storage/libpod/bolt_state.db"
time="2019-12-14T14:09:14-05:00" level=debug msg="Using graph driver vfs"
time="2019-12-14T14:09:14-05:00" level=debug msg="Using graph root /home/noah/.local/share/containers/storage"
time="2019-12-14T14:09:14-05:00" level=debug msg="Using run root /run/user/1000"
time="2019-12-14T14:09:14-05:00" level=debug msg="Using static dir /home/noah/.local/share/containers/storage/libpod"
time="2019-12-14T14:09:14-05:00" level=debug msg="Using tmp dir /run/user/1000/libpod/tmp"
time="2019-12-14T14:09:14-05:00" level=debug msg="Using volume path /home/noah/.local/share/containers/storage/volumes"
time="2019-12-14T14:09:14-05:00" level=debug msg="Set libpod namespace to \"\""
time="2019-12-14T14:09:14-05:00" level=debug msg="[graphdriver] trying provided driver \"vfs\""
time="2019-12-14T14:09:14-05:00" level=debug msg="Initializing event backend file"
time="2019-12-14T14:09:14-05:00" level=warning msg="Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument"
time="2019-12-14T14:09:14-05:00" level=debug msg="using runtime \"/usr/bin/runc\""
time="2019-12-14T14:09:14-05:00" level=error msg="stat /sys/fs/cgroup/systemd/org/freedesktop/ConsoleKit/Session1: no such file or directory"

I noticed two things in the logs:

1. The cgroups manager is set to systemd but there is no systemd user session available

   Even though I am passing the --cgroup-manager=cgroupfs option it still is trying to use systemd.

2. Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument

   It is trying to use crun which is required to use rootless with cgroup V2.
   However according to buildah info I am using cgroup V1 so falling back to runc shouldn't be an issue.

I have also confirmed, by looking back at some older scripts, that before I updated I could use podman without sudo. I wasn't just imagining it :sweat_smile: . So something in my environment, the Void Linux podman package, or podman probably changed.

[rhatdan]

Is there crun and systemd defined in ~/.config/.containers/libpod.conf?

[Noah-Huppert]

Is there crun and systemd defined in ~/.config/.containers/libpod.conf?

There is no mention of systemd in my libpod.conf. Which is really odd b/c the debug output always says WARN[0000] The cgroups manager is set to systemd ....

My runtime is set to runc. A path for crun is defined in the runtimes section of my libpod.conf, but this has always been in there. Removing runtimes.crun does not fix the problem.

[mheon]

time="2019-12-14T14:09:14-05:00" level=debug msg="using runtime \"/usr/bin/runc\""

We are using runc - don't worry about the crun warning, that's expected if crun isn't installed. As long as the using runtime... line says runc we're using runc.

It seems like libpod has completed initialization (or is near to completing it) when we make this query.

[mheon]

The other error messages, though, the systemd ones - those apparently trigger unconditionally when Podman is run as root when a DBus connection is not available, even when we are already configured to not use journald logging and events. I'm not sure why that is - maybe a mistake?

[mheon]

Hm. We're running as rootless, though. Which means that at least part of the config file isn't being respected here.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@Noah-Huppert @mheon Is this still an issue?

[Noah-Huppert]

@rhatdan

Yes:

% podman version
Error: stat /sys/fs/cgroup/systemd/org/freedesktop/ConsoleKit/Session1: no such file or directory
% sudo podman version
Version:            1.7.0
RemoteAPI Version:  1
Go Version:         go1.13.5
OS/Arch:            linux/amd64

[Noah-Huppert]

What does the stale issue marker signify? Does this cause the issue to get placed lower in a queue? Should I bump this issue every 30 days to prevent this?

[rhatdan]

No it just means no one has commented on it in 30 days. The goal is to wake us up to relook at the issue or close it, if it is no longer relevant or not something we will fix.

[rhatdan]

@giuseppe Were you able to recreate this?
@Noah-Huppert Does this happen if you use runc as your OCI Runtime?

[giuseppe]

@rhatdan I've not tried as the issue requires a system without systemd

[Noah-Huppert]

@rhatdan I believe I am using runc as my OCI runtime:

% sudo podman info
host:
... SNIP ...
  OCIRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
... SNIP ...

[rhatdan]

Well then you could try crun, but I doubt that will fix the issue.

[rhatdan]

@Noah-Huppert Are you still working on this issue?

[Noah-Huppert]

@rhatdan My fix has been adding alias podman="sudo podman" to my shell profile...

I was hoping someone with more in depth knowledge of this issue would be able to provide a fix since it breaks podman on what I assume are all non-systemd systems. I had attempted to look around the source code but I couldn't figure out where this error was being thrown, given me never having touched the code before.

[rhatdan]

Did you ever try crun? I am guessing the runc was attempting to connect to dbus.

[svireli]

Hi,
I'm experiencing the same issue. Running with sudo fixes it for now.
I'm using void linux (without systemd).

BuildahVersion: 1.13.1
CgroupVersion: v1
runc version spec: 1.0.1-dev
Version: 1.8.0
RemoteAPI Version: 1
Go Version: go1.13.7
OS/Arch: linux/amd64

[shanemcd]

I am seeing something similar to this while trying to run (rootless) Podman inside of a privileged container running under Docker for Mac:

$ podman --events-backend=file --cgroup-manager=cgroupfs run -ti alpine ash
Error: stat /sys/fs/cgroup/systemd/docker/a6ae1e9add09166a5180fcf35e8945a80fbe9647eddb32d8838410bf24a12e87: no such file or directory

[mheon]

You need to mount /sys/fs/cgroup into the outer container to resolve that

On Wed, Mar 4, 2020, 10:13 Shane McDonald [e-mail] wrote:

I am seeing something similar to this while trying to run (rootless)
Podman inside of a privileged container running under Docker for Mac:

$ podman --events-backend=file --cgroup-manager=cgroupfs run -ti alpine ash
Error: stat /sys/fs/cgroup/systemd/docker/a6ae1e9add09166a5180fcf35e8945a80fbe9647eddb32d8838410bf24a12e87: no such file or directory

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/containers/libpod/issues/4648?email_source=notifications&email_token=AB3AOCBJ3K3BZIN46I4QIPLRFZV23A5CNFSM4JV33QQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENYLOOY#issuecomment-594589499,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB3AOCD462TVKTDFOKFUCMDRFZV23ANCNFSM4JV33QQQ
.

[mheon]

The complication is that said mount must be from the Linux VM, not the Mac
host's filesystem

On Wed, Mar 4, 2020, 10:24 Matthew Heon [e-mail] wrote:

You need to mount /sys/fs/cgroup into the outer container to resolve that

On Wed, Mar 4, 2020, 10:13 Shane McDonald [e-mail]
wrote:

I am seeing something similar to this while trying to run (rootless)
Podman inside of a privileged container running under Docker for Mac:

$ podman --events-backend=file --cgroup-manager=cgroupfs run -ti alpine ash
Error: stat /sys/fs/cgroup/systemd/docker/a6ae1e9add09166a5180fcf35e8945a80fbe9647eddb32d8838410bf24a12e87: no such file or directory

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/containers/libpod/issues/4648?email_source=notifications&email_token=AB3AOCBJ3K3BZIN46I4QIPLRFZV23A5CNFSM4JV33QQ2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOENYLOOY#issuecomment-594589499,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AB3AOCD462TVKTDFOKFUCMDRFZV23ANCNFSM4JV33QQQ
.

[shanemcd]

Thanks for the quick response. Getting closer. Now I'm seeing:

there might not be enough IDs available in the namespace (requested 0:42 for /etc/shadow): lchown /etc/shadow: invalid argument

Dont want to hijack this issue, so I'll poke elsewhere. Thanks again.

[vrothberg]

@Noah-Huppert did you try running rootless but with crun instead of runc?

[svireli]

@vrothberg I guess crun needs systemd, I can't even build it without systemd libraries.

[giuseppe]

@vrothberg I guess crun needs systemd, I can't even build it without systemd libraries.

you can configure it with ./configure --disable-systemd

[Noah-Huppert]

Thanks @giuseppe I was literally just building crun and ran into the same problem. Working on building a package to install for crun on my distro now.

[Noah-Huppert]

@vrothberg I just tried using podman rootless with crun, no change:

% crun --version
crun version 0.13
commit: e79e4de4ac16da0ce48777afb72c6241de870525
spec: 1.0.0
+SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL

% cat /etc/containers/libpod.conf | grep -F 'runtime = "'
#runtime = "runc"
runtime = "crun"



md5-155d39e3c3cd47338804f3cd9aa17f5c



% sudo podman info
host:
  BuildahVersion: 1.13.1
  CgroupVersion: v1
  Conmon:
    package: Unknown
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.10, commit: unknown'
  Distribution:
    distribution: '"void"'
    version: unknown
  MemFree: 756760576
  MemTotal: 16608944128
  OCIRuntime:
    name: crun
    package: Unknown
    path: /usr/bin/crun
    version: |-
      crun version 0.13
      commit: e79e4de4ac16da0ce48777afb72c6241de870525
      spec: 1.0.0
      +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 8
  eventlogger: file
  hostname: apollo
  kernel: 5.4.20_1
  os: linux
  rootless: false
  uptime: 13h 0m 20s (Approximately 0.54 days)
registries:
  search:
  - docker.io
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 39
  GraphDriverName: overlay
  GraphOptions: {}
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 101
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes



md5-155d39e3c3cd47338804f3cd9aa17f5c



% podman ps
Error: stat /sys/fs/cgroup/systemd/org/freedesktop/ConsoleKit/Session1: no such file or directory

[ryudice]

I'm having this issue on ubuntu 18.04 as well. Was trying to avoid rootful builds and stumble upon this...

[shanemcd]

Circling back to this in case in helps anyone else. There were a few things going on here.

The following steps are in addition to mounting /sys/fs/cgroup as mentioned above.

First, we have the following in our Dockerfile:

RUN echo -e 'cgroup_manager = "cgroupfs"\nevents_logger = "file"' > /etc/containers/libpod.conf

Next, /home/<user>/.local/share/containers needs to be marked as a VOLUME in the Dockerfile. Without this we were getting a weird "command not found" for the ENTRYPOINT.

Then, we have containers that do some black magic in the entrypoint in order to deal with OpenShift's UID behavior. We were writing out a custom /etc/passwd, which was enough for what we were doing before, but now we need to write out the /etc/sub*id files.

Lastly, we needed podman system migrate after writing the /etc/sub*id files.

The following gets rootless podman working for me inside of a docker container:

#!/bin/bash

if [ `id -u` -ge 500 ] || [ -z "${CURRENT_UID}" ]; then

cat << EOF > /etc/passwd
root:x:0:0:root:/root:/bin/bash
myuser:x:`id -u`:`id -g`:,,,:/home/myuser:/bin/bash
EOF

cat <<EOF > /etc/subuid
myuser:100000:50001
EOF

cat <<EOF > /etc/subgid
myuser:100000:50001
EOF

fi

# Required to get rootless podman working after
# writing out the sub*id files above
podman system migrate

exec $@

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

Has anyone tried with the latest podman. Now we have containers.conf and a lot of these settings can be set by default there. Including cgroup_manager and events_logger.

[jfilak]

Results from the latest Fedora 32 with podman built from this git repo with a non-root user:

$ bin/podman --version
podman version 2.1.0-dev
$ bin/podman run -it --rm fedora bash
Error: creating cgroup directory `/sys/fs/cgroup/cpu/user.slice/user-jfilak.slice/[email protected]/user.slice/libpod-9c1142723b03fde3636de6ef49e70cfed1eb538bb14732b3f70811cacdbf9a70.scope`: No such file or directory: OCI runtime command not found error
$ bin/podman --events-backend=file --cgroup-manager=cgroupfs run -it --rm fedora bash
[root@dded3b35dfea /]# exit
$ bin/podman --cgroup-manager=cgroupfs run -it --rm fedora bash
[root@b15589b716f8 /]# exit
$ bin/podman --events-backend=file run -it --rm fedora bash
Error: creating cgroup directory `/sys/fs/cgroup/cpu/user.slice/user-jfilak.slice/[email protected]/user.slice/libpod-9c1142723b03fde3636de6ef49e70cfed1eb538bb14732b3f70811cacdbf9a70.scope`: No such file or directory: OCI runtime command not found error

It was enough to add the parameter --cgroup-manager=cgroupfs.

[rhatdan]

@giuseppe Any update on this one?

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@shanemcd Do you still have this problem. As far as running podman inside of a container, you can configure containers.conf to fix those problems with event-logger and cgroup manager. I think this is fixed, or fixable at this point. I am going to close, reopen if I am mistaken. Or open a new issue with new data.