Podman: 'podman build' fails where 'buildah bud' works

[acmel]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

Fails:

podman build -t my_image .

Works:

buildah bud -t my_image .

Then I can go and use my_image with 'podman run', etc.

Steps to reproduce the issue:

[perfbuilder@five fedora]$ mkdir b
[perfbuilder@five fedora]$ cd b
[perfbuilder@five b]$ vim Dockerfile
[perfbuilder@five b]$ cat Dockerfile
FROM fedora:32
RUN dnf -y install make
[perfbuilder@five b]$ podman build -t my_image .
STEP 1: FROM fedora:32
STEP 2: RUN dnf -y install make
2020-09-01T20:48:12.000711043Z: sd-bus call: Permission denied
error running container: error creating container for [/bin/sh -c dnf -y install make]: : exit status 1
Error: error building at STEP "RUN dnf -y install make": error while running runtime: exit status 1
[perfbuilder@five b]$
[perfbuilder@five b]$
[perfbuilder@five b]$
[perfbuilder@five b]$ buildah bud -t my_image .
STEP 1: FROM fedora:32
STEP 2: RUN dnf -y install make
Fedora 32 openh264 (From Cisco) - x86_64 3.2 kB/s | 5.1 kB 00:01
Fedora Modular 32 - x86_64 3.5 MB/s | 4.9 MB 00:01
Fedora Modular 32 - x86_64 - Updates 1.9 MB/s | 3.7 MB 00:01
Fedora 32 - x86_64 - Updates 2.6 MB/s | 22 MB 00:08
Fedora 32 - x86_64 4.8 MB/s | 70 MB 00:14
Dependencies resolved.

Installed:
gc-8.0.4-3.fc32.x86_64 guile22-2.2.6-4.fc32.x86_64 libtool-ltdl-2.4.6-33.fc32.x86_64 make-1:4.2.1-16.fc32.x86_64

Complete!
STEP 3: COMMIT my_image
Getting image source signatures
Copying blob 0ccfac04663b skipped: already exists
Copying blob 66344faa701b done
Copying config c06773a370 done
Writing manifest to image destination
Storing signatures
--> c06773a3700
c06773a37001dcd4cb891ebe22dbb62065e5d2bdfecf1e1c9e34d1f6eb09d5b1
[perfbuilder@five b]$ podman images | grep my_image
localhost/my_image latest c06773a37001 4 minutes ago 466 MB
[perfbuilder@five b]$

If I now try with 'podman build' it will work, i.e. find it in the cache, etc:

perfbuilder@five b]$ podman build -t my_image .
STEP 1: FROM fedora:32
STEP 2: RUN dnf -y install make
--> Using cache c06773a37001dcd4cb891ebe22dbb62065e5d2bdfecf1e1c9e34d1f6eb09d5b1
STEP 3: COMMIT my_image
--> c06773a3700
c06773a37001dcd4cb891ebe22dbb62065e5d2bdfecf1e1c9e34d1f6eb09d5b1
[perfbuilder@five b]$

If I now remove the image:

[perfbuilder@five b]$ podman rmi my_image
Untagged: localhost/my_image:latest
Deleted: c06773a37001dcd4cb891ebe22dbb62065e5d2bdfecf1e1c9e34d1f6eb09d5b1
[perfbuilder@five b]$

And try again it'll fail, see with --log-level warn, there are some messages
that may help someone more experienced than me with podman to figure this out:

[perfbuilder@five b]$ podman --log-level warn build -t my_image .
WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available
WARN[0000] For using systemd, you may need to login using an user session
WARN[0000] Alternatively, you can enable lingering with: loginctl enable-linger 1001 (possibly as root)
WARN[0000] Falling back to --cgroup-manager=cgroupfs
WARN[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument
WARN[0000] The cgroupv2 manager is set to systemd but there is no systemd user session available
WARN[0000] For using systemd, you may need to login using an user session
WARN[0000] Alternatively, you can enable lingering with: loginctl enable-linger 1001 (possibly as root)
WARN[0000] Falling back to --cgroup-manager=cgroupfs
WARN[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument
STEP 1: FROM fedora:32
STEP 2: RUN dnf -y install make
2020-09-01T20:59:34.000618935Z: sd-bus call: Permission denied
error running container: error creating container for [/bin/sh -c dnf -y install make]: : exit status 1
Error: error building at STEP "RUN dnf -y install make": error while running runtime: exit status 1
[perfbuilder@five b]$

In the past I tried using that loginctl enable-linger 1001, can't recall what was the result,
but IIRC looked like a dead end, didn't help.

Describe the results you received:

Failure to build a simple image.

Describe the results you expected:

The image to be have been built just like with 'buildah bud'

Additional information you deem important (e.g. issue happens only occasionally):

Its the version in fedora:32, happens all the time.

Output of podman version:

podman version 2.0.5

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.15.1
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.19-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.19, commit: 5dce9767526ed27f177a8fa3f281889ad509fea7'
  cpus: 24
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: five
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1001
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.9.0-rc3
  linkmode: dynamic
  memFree: 2142887936
  memTotal: 33675350016
  ociRuntime:
    name: crun
    package: crun-0.14.1-3.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.14.1
      commit: 598ea5e192ca12d4f6378217d3ab1415efeddefa
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /tmp/run-1001/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 16907759616
  swapTotal: 16915623936
  uptime: 24h 7m 34.57s (Approximately 1.00 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/perfbuilder/.config/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 1
    stopped: 3
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/perfbuilder/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 156
  runRoot: /var/tmp/run-1001/containers
  volumePath: /home/perfbuilder/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Wed Dec 31 21:00:00 1969
  GitCommit: ""
  GoVersion: go1.14.6
  OsArch: linux/amd64
  Version: 2.0.5

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.0.5-1.fc32.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

No

Additional environment details (AWS, VirtualBox, physical, etc.):

physical

[TomSweeneyRedHat]

@acmel thanks for the issue. My first guess is the --layers option. By default, Podman uses --layers=true. Could you try:
' podman build --layers=false -t my_image .`

If you've time, it would also be interesting to see if buildah with layers with true fails buildah bud --layers=true -t my_image .

But all that said, I'm not sure why --layers would trip up this Dockerfile, it seems rather straight forward.

[acmel]

[perfbuilder@five b]$ podman images | grep my_image
[perfbuilder@five b]$ podman build --layers=false -t my_image .
STEP 1: FROM fedora:32
STEP 2: RUN dnf -y install make
2020-09-02T01:20:31.000195784Z: sd-bus call: Permission denied
error running container: error creating container for [/bin/sh -c dnf -y install make]: : exit status 1
Error: error building at STEP "RUN dnf -y install make": error while running runtime: exit status 1
[perfbuilder@five b]$ cat Dockerfile
FROM fedora:32
RUN dnf -y install make
[perfbuilder@five b]$

[QiWang19]

Could you try podman build --cgroup-manager=cgroupfs -t my_image .
issue #6368 has encountered with the same error sd-bus call: Permission denied

[acmel]

Yeah, that did the trick, see below.

Now please hand hold me, am I missing some kernel config option? There is this cgroup transition, I'm probably missing something really obvious here :-\

perfbuilder@five b]$ podman build --cgroup-manager=cgroupfs -t my_image .
STEP 1: FROM fedora:32
STEP 2: RUN dnf -y install make
Fedora 32 openh264 (From Cisco) - x86_64 3.1 kB/s | 5.1 kB 00:01
Fedora Modular 32 - x86_64 1.2 MB/s | 4.9 MB 00:04
Fedora Modular 32 - x86_64 - Updates 2.7 MB/s | 3.7 MB 00:01
Fedora 32 - x86_64 - Updates 2.6 MB/s | 22 MB 00:08
Fedora 32 - x86_64 3.8 MB/s | 70 MB 00:18

Dependencies resolved.

Package Architecture Version Repository Size

Installing:
make x86_64 1:4.2.1-16.fc32 fedora 494 k
Installing dependencies:
gc x86_64 8.0.4-3.fc32 fedora 113 k
guile22 x86_64 2.2.6-4.fc32 fedora 6.6 M
libtool-ltdl x86_64 2.4.6-33.fc32 fedora 37 k

Transaction Summary

Install 4 Packages

Total download size: 7.2 M
Installed size: 46 M
Downloading Packages:
(1/4): libtool-ltdl-2.4.6-33.fc32.x86_64.rpm 51 kB/s | 37 kB 00:00
(2/4): gc-8.0.4-3.fc32.x86_64.rpm 24 kB/s | 113 kB 00:04
(3/4): make-4.2.1-16.fc32.x86_64.rpm 16 kB/s | 494 kB 00:31

(4/4): guile22-2.2.6-4.fc32.x86_64.rpm 203 kB/s | 6.6 MB 00:33

Total 217 kB/s | 7.2 MB 00:34
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : libtool-ltdl-2.4.6-33.fc32.x86_64 1/4
Installing : gc-8.0.4-3.fc32.x86_64 2/4
Installing : guile22-2.2.6-4.fc32.x86_64 3/4
Running scriptlet: guile22-2.2.6-4.fc32.x86_64 3/4
Installing : make-1:4.2.1-16.fc32.x86_64 4/4
Running scriptlet: make-1:4.2.1-16.fc32.x86_64 4/4
Verifying : gc-8.0.4-3.fc32.x86_64 1/4
Verifying : guile22-2.2.6-4.fc32.x86_64 2/4
Verifying : libtool-ltdl-2.4.6-33.fc32.x86_64 3/4
Verifying : make-1:4.2.1-16.fc32.x86_64 4/4

Installed:
gc-8.0.4-3.fc32.x86_64 guile22-2.2.6-4.fc32.x86_64 libtool-ltdl-2.4.6-33.fc32.x86_64 make-1:4.2.1-16.fc32.x86_64

Complete!
STEP 3: COMMIT my_image
--> 7e9ce9a742e
7e9ce9a742e3d6f0843a5204014818b5bce294653741fbab4a4d14a2b7d82487
[perfbuilder@five b]$

[QiWang19]

@acmel thanks for checking, In the /etc/containers/containers.conf or /etc/containers/libpod.conf, is the cgroup_manager = "systemd" configured?

[acmel]

I couldn't find any containers.conf or libpod.conf in this system, its a fedora:32 with:

[perfbuilder@five b]$ rpm -q podman
podman-2.0.5-1.fc32.x86_64
[perfbuilder@five b]$ rpm -qa | grep libpod
[perfbuilder@five b]

[acmel]

I take that back, I didn't find it in /etc, it is at:

[perfbuilder@five b]$ grep cgroup_manager /usr/share/containers/containers.conf

cgroup_manager = "systemd"

[perfbuilder@five b]$

[acmel]

So if I change in that file it to:

$ grep cgroup_manager /usr/share/containers/containers.conf
cgroup_manager = "cgroupfs"
$

I works, equivalent to using --cgroup-manager=cgroupfs, thanks!