Podman: Cannot start containers on cgroupsv1 due to seccomp issue

[marusak]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description
(Not sure if this is the right place or this is more distro specific, but lets try here :) )

When I use Fedora 33 with cgroupsV1 I cannot start container.

Steps to reproduce the issue:

grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"

...reboot...

$ sudo podman run -dit --name swamped-crate busybox:latest sh
Error: container_linux.go:349: starting container process caused "error adding seccomp rule for syscall socket: permission denied": OCI permission denied

For now I see this only on system with podman-2.2.1 (other systems use older version).
(or to tell the whole truth, it is system which has this one transaction. System without this dnf transaction works fine.

Installing:
 kernel                         x86_64    5.9.14-200.fc33      updates    8.3 k
Upgrading:
 conmon                         x86_64    2:2.0.21-3.fc33      fedora      46 k
 containernetworking-plugins    x86_64    0.8.7-1.fc33         fedora     9.3 M
 containers-common              x86_64    1:1.2.0-10.fc33      updates     79 k
 crun                           x86_64    0.16-1.fc33          updates    155 k
 glibc                          x86_64    2.32-2.fc33          updates    3.5 M
 glibc-all-langpacks            x86_64    2.32-2.fc33          updates     19 M
 glibc-common                   x86_64    2.32-2.fc33          updates    1.8 M
 podman                         x86_64    2:2.2.1-1.fc33       updates     11 M
Installing dependencies:
 kernel-core                    x86_64    5.9.14-200.fc33      updates     32 M
 kernel-modules                 x86_64    5.9.14-200.fc33      updates     30 M

)

Describe the results you received:

Error: container_linux.go:349: starting container process caused "error adding seccomp rule for syscall socket: permission denied": OCI permission denied

Describe the results you expected:
Container would be started.

Output of podman version:

Version:      2.2.1
API Version:  2.1.0
Go Version:   go1.15.5
Built:        Tue Dec  8 15:37:50 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.18.0
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-3.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 0f53fb68333bdead5fe4dc5175703e22cf9882ab'
  cpus: 8
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: foo
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.9.13-200.fc33.x86_64
  linkmode: dynamic
  memFree: 14401789952
  memTotal: 33439072256
  ociRuntime:
    name: crun
    package: crun-0.16-1.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.16
      commit: eb0145e5ad4d8207e84a327248af76663d4e50dd
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.fc33.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 21084758016
  swapTotal: 21084758016
  uptime: 18h 8m 31.21s (Approximately 0.75 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/mmarusak/.config/containers/storage.conf
  containerStore:
    number: 11
    paused: 0
    running: 1
    stopped: 10
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.3.0-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.3
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/mmarusak/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 31
  runRoot: /run/user/1000/containers
  volumePath: /home/mmarusak/.local/share/containers/storage/volumes
version:
  APIVersion: 2.1.0
  Built: 1607438270
  BuiltTime: Tue Dec  8 15:37:50 2020
  GitCommit: ""
  GoVersion: go1.15.5
  OsArch: linux/amd64
  Version: 2.2.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.2.1-1.fc33.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

Yes

[rhatdan]

If you disable seccomp does it work?
Is there anything in the audit.log?

@giuseppe PTAL

[giuseppe]

it is a too old runc that doesn't know how to deal with the updated seccomp profile.

Either update runc or use crun, which is the default on Fedora as OCI runtime

[rhatdan]

@giuseppe is the version of runc we are shipping in Fedora too old?

[metal3d]

Same issue here (Fedora 33):

$ cat /proc/cmdline 
BOOT_IMAGE=(hd0,gpt2)/vmlinuz-5.9.13-200.fc33.x86_64 root=UUID=5a84ca16-19c6-4d9e-9510-478713eb282e ro rootflags=subvol=root resume=UUID=a927b95e-4988-4dbb-8452-d6ffb8499d15 rhgb quiet systemd.unified_cgroup_hierarchy=0

$ podman run --rm -it debian bash
Error: container_linux.go:349: starting container process caused "error adding seccomp rule for syscall socket: permission denied": OCI permission denied

[giuseppe]

what is the runc version you are using?

I've tried with this version and it works on Fedora 33:

# rpm -qf /usr/bin/runc
runc-1.0.0-279.dev.gitdedadbf.fc33.x86_64
# runc --version
runc version 1.0.0-rc92+dev
commit: c9a9ce0286785bef3f3c3c87cd1232e535a03e15
spec: 1.0.2-dev

[daniviga]

@marusak @metal3d maybe you have an old containerd installed from docker-ce repos?

Can you try this

sudo dnf swap containerd.io runc

as suggested by @giuseppe you can check the current setup via

$ rpm -qf /usr/bin/runc
containerd.io-XYZ.fc28.x86_64