Podman: RUN fails in rootless podman build when cgroups is v1

Created on 15 Jul 2020 · 21Comments · Source: containers/podman

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

When cgroups is v1 podman build fails on RUN step with the following:

ERRO[0000] systemd cgroup flag passed, but systemd support for managing cgroups is not available 
systemd cgroup flag passed, but systemd support for managing cgroups is not available
error running container: error creating container for [/bin/sh -c touch /file]: : exit status 1
Error: error building at STEP "RUN touch /file": error while running runtime: exit status 1

Cannot reproduce with buildah bud, so filing it here.

Steps to reproduce the issue:

On a system running an up-to-date Fedora 32:

Check that podman build is working okay:

$ podman build --no-cache -f -
FROM fedora:latest
RUN touch /file
STEP 1: FROM fedora:latest
STEP 2: RUN touch /file
STEP 3: COMMIT
--> 85f1922ea05
85f1922ea05bfe11e035caefa54d31f22a8c10559e7f4ca3fe9215ea2b31e163

Configure system to use cgroups v1 and reboot.

sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"

Run podman build again:

$ podman build --no-cache -f -
FROM fedora:latest                          
RUN touch /file
STEP 1: FROM fedora:latest
STEP 2: RUN touch /file
ERRO[0000] systemd cgroup flag passed, but systemd support for managing cgroups is not available 
systemd cgroup flag passed, but systemd support for managing cgroups is not available
error running container: error creating container for [/bin/sh -c touch /file]: : exit status 1
Error: error building at STEP "RUN touch /file": error while running runtime: exit status 1

buildah bud works:

$ buildah bud --no-cache -f -
FROM fedora:latest
RUN touch /file
STEP 1: FROM fedora:latest
STEP 2: RUN touch /file
STEP 3: COMMIT
--> 03c30b816e2
03c30b816e266f69814b26c377a85c4419a4c26086b799f29c13bf79e6f4f527

Describe the results you received:

Build fails on RUN step.

Describe the results you expected:

Build should succeed.

Additional information you deem important (e.g. issue happens only occasionally):

Tested this on a fresh Fedora 32 installation.

The above is working with podman 1.8.2, but fails after upgrading to podman 2.0.2

Output of podman version:

Version:      2.0.2
API Version:  1
Go Version:   go1.14.3
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

When cgroups is v1

host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.18-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.18, commit: 6e8799f576f11f902cd8a8d8b45b2b2caf636a85'
  cpus: 8
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: localhost-live
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.7.8-200.fc32.x86_64
  linkmode: dynamic
  memFree: 2097209344
  memTotal: 4090630144
  ociRuntime:
    name: runc
    package: runc-1.0.0-144.dev.gite6555cc.fc32.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10+dev
      commit: fbdbaf85ecbc0e077f336c03062710435607dbf1
      spec: 1.0.1-dev
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.1-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.1
      commit: bbf27c5acd4356edb97fa639b4e15e0cd56a39d5
      libslirp: 4.2.0
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 2147479552
  swapTotal: 2147479552
  uptime: 39.76s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/csomh/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/csomh/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 2
  runRoot: /run/user/1000/containers
  volumePath: /home/csomh/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.14.3
  OsArch: linux/amd64
  Version: 2.0.2

When cgroups is v2:

host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.18-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.18, commit: 6e8799f576f11f902cd8a8d8b45b2b2caf636a85'
  cpus: 8
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: localhost-live
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.7.8-200.fc32.x86_64
  linkmode: dynamic
  memFree: 1938636800
  memTotal: 4090626048
  ociRuntime:
    name: crun
    package: crun-0.14.1-1.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.14.1
      commit: 598ea5e192ca12d4f6378217d3ab1415efeddefa
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.1-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.1
      commit: bbf27c5acd4356edb97fa639b4e15e0cd56a39d5
      libslirp: 4.2.0
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 2147479552
  swapTotal: 2147479552
  uptime: 1m 50.78s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/csomh/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/csomh/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /home/csomh/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.14.3
  OsArch: linux/amd64
  Version: 2.0.2

Package info (e.g. output of rpm -q podman or apt list podman):

$ rpm -q podman
podman-2.0.2-1.fc32.x86_64
$ rpm -q buildah
buildah-1.15.0-1.fc32.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

Was able to reproduce both on physical machine as in VM.

Buildah kinbug

Source

csomh

All 21 comments

After upgrading podman to 2.0.2, we encounter the same problem on Ubuntu (podman seems to use cgroups v1 by default).

$ podman info --debug
host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.18, commit: '
  cpus: 16
  distribution:
    distribution: ubuntu
    version: "20.04"
  eventLogger: file
  hostname: xps15
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.4.0-40-generic
  linkmode: dynamic
  memFree: 8379371520
  memTotal: 33285279744
  ociRuntime:
    name: runc
    package: 'runc: /usr/sbin/runc'
    path: /usr/sbin/runc
    version: 'runc version spec: 1.0.1-dev'
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.4
      commit: unknown
      libslirp: 4.2.0
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 999288832
  swapTotal: 1023406080
  uptime: 17h 32m 57.92s (Approximately 0.71 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/rg/.config/containers/storage.conf
  containerStore:
    number: 23
    paused: 0
    running: 0
    stopped: 23
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.9.0
        fuse-overlayfs: version 0.7.6
        FUSE library version 3.9.0
        using FUSE kernel interface version 7.31
  graphRoot: /home/rg/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 174
  runRoot: /run/user/1000/containers
  volumePath: /home/rg/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.14.4
  OsArch: linux/amd64
  Version: 2.0.2

weatherfrog on 15 Jul 2020

@TomSweeneyRedHat @nalind PTAL

mheon on 15 Jul 2020

After some code digging the error seems to be due to adding --systemd-cgroup to the runtime. But not sure if the condition is wrong (would need to check for cgroupVersion) or cgroupManager is wrong when cgroup is v1.

Using the same flag with buildah bud will result in the same error:

$ buildah bud --runtime-flag systemd-cgroup --no-cache -f -
from fedora:latest
run touch /file

STEP 1: FROM fedora:latest
STEP 2: run touch /file
ERRO[0000] systemd cgroup flag passed, but systemd support for managing cgroups is not available 
systemd cgroup flag passed, but systemd support for managing cgroups is not available
error running container: error creating container for [/bin/sh -c touch /file]: : exit status 1
error building at STEP "RUN touch /file": error while running runtime: exit status 1
ERRO exit status 1

But anyways, this is just a hint, please take it with a grain of salt.

csomh on 15 Jul 2020

Setting cgroup_manager="cgroupfs" in libpod.conf solves the issue.

csomh on 15 Jul 2020

❤1

That should be automatically set for rootless cgroupsv1 systems

mheon on 15 Jul 2020

Seeing this on RHEL8 gating tests.

edsantiago on 15 Jul 2020

Eeek. Alright, we're probably incorrectly setting systemd cgroups on v1 systemd, then.

mheon on 15 Jul 2020

Setting cgroup_manager="cgroupfs" in libpod.conf solves the issue.

Confirmed, thank you.

weatherfrog on 16 Jul 2020

Please do not use libpod.conf, Set it in containers.conf and remove libpod.conf.

rhatdan on 16 Jul 2020

Please do not use libpod.conf, Set it in containers.conf and remove libpod.conf.

Thanks @rhatdan! What is the reason for this? man podman is still referencing libpod.conf and barely mentions containers.conf. What am I missing?

csomh on 17 Jul 2020

Yikes that is a pretty serious omission.

rhatdan on 17 Jul 2020

@csomh PTAL https://github.com/containers/podman/pull/7009

rhatdan on 17 Jul 2020

👍1

@mheon, did we ever have a spot where we set cgroup_manager="cgroupfs" for rootless folks? I took a quick look through the Podman code and didn't find that it was getting set to anything outside of the testing code.

TomSweeneyRedHat on 18 Jul 2020

It should be decided by c/common and containers.conf

mheon on 18 Jul 2020

https://github.com/containers/common/blob/master/pkg/config/config.go#L516

rhatdan on 20 Jul 2020

Confirmed, reproduced on a fresh RHEL8 install. It's not just Buildah, all podman run commands are also failing for me.

mheon on 21 Jul 2020

I think I know what's going on - that code for c/common seems bunk, it doesn't work on Cgroups v1 systems.

mheon on 21 Jul 2020

https://github.com/containers/common/pull/231 to fix
@edsantiago @vrothberg @rhatdan FYI - we'll need this landed and then backported so we can cut 2.0.3

mheon on 21 Jul 2020

containers/common#231 to fix
@edsantiago @vrothberg @rhatdan FYI - we'll need this landed and then backported so we can cut 2.0.3

https://github.com/containers/common/pull/235

vrothberg on 22 Jul 2020

:+1: problem is gone with podman 2.0.3. Thank you guys!

weatherfrog on 24 Jul 2020

Seems I still have the problem with podman-2.0.4-1.fc31.x86_64 (but the fix of @weatherfrog in ~/.config/libpod.conf works).

If it's fine with a fresh install (which I haven't tested), maybe it would be efficient to fix existing config.

tisc0 on 13 Aug 2020

Was this page helpful?

0 / 5 - 0 ratings

Related issues

Rootless podman error: could not get runtime: open /proc/31678/ns/user: no such file or directory

faulesocke · 65Comments

Error creating /run/user/0/containers on podman login

rossturk · 54Comments

Trying to use --init fails with "container-init binary not found on the host: stat /usr/libexec/podman/catatonit: no such file or directory"

jlebon · 58Comments

"Error: invalid configuration, cannot specify resource limits without cgroups v2 and --cgroup-manager=systemd"

avikivity · 92Comments

Error: stat /sys/fs/cgroup/systemd/org/freedesktop/ConsoleKit/Session1: no such file or directory

Noah-Huppert · 51Comments