podman system service consuming 100% CPU

[dmesser]

/kind bug

Description

On my Fedora 33 machine, running podman 2.1.1, I have 5 containers running, controlled via systemd user service which are autostarted at boot (loginctl enable-linger). I noticed since some time now that two podman system services consume all available CPU time on my 2 core system.

Steps to reproduce the issue:

1. Run podman user containers, start via generated systemd units

2. observe system utilization via top

Describe the results you received:

top - 12:08:28 up 1 day, 11:26,  2 users,  load average: 4.57, 4.41, 4.33
Tasks: 148 total,   1 running, 147 sleeping,   0 stopped,   0 zombie
%Cpu(s): 98.8 us,  0.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.7 hi,  0.2 si,  0.0 st
MiB Mem :   3832.2 total,    686.9 free,   1246.6 used,   1898.7 buff/cache
MiB Swap:  12155.0 total,  11891.7 free,    263.2 used.   2949.3 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   3399 root      20   0 1616032   9268     84 S  99.0   0.2   2069:47 /usr/bin/podman system service
   3405 dmesser   20   0 1690020  12664   2308 S  98.7   0.3   2069:41 /usr/bin/podman system service
   1921 dmesser   20   0  755456  26900  18316 S   0.3   0.7   0:33.02 grafana-server --homepath=/usr/share/grafana --config=/etc/grafana/grafa+
   1975 dmesser   20   0  972604 216612  22764 S   0.3   5.5  12:06.23 influxd
      1 root      20   0  108532   7860   4392 S   0.0   0.2   0:05.71 /usr/lib/systemd/systemd --switched-root --system --deserialize 29
      2 root      20   0       0      0      0 S   0.0   0.0   0:00.08 [kthreadd]
      3 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [rcu_gp]
      4 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [rcu_par_gp]
      6 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [kworker/0:0H-kblockd]
      9 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [mm_percpu_wq]
     10 root      20   0       0      0      0 S   0.0   0.0   0:00.34 [ksoftirqd/0]
     11 root      20   0       0      0      0 I   0.0   0.0   0:01.91 [rcu_sched]
     12 root      rt   0       0      0      0 S   0.0   0.0   0:00.04 [migration/0]
     13 root      20   0       0      0      0 S   0.0   0.0   0:00.00 [cpuhp/0]
     14 root      20   0       0      0      0 S   0.0   0.0   0:00.00 [cpuhp/1]
     15 root      rt   0       0      0      0 S   0.0   0.0   0:00.14 [migration/1]
     16 root      20   0       0      0      0 S   0.0   0.0   0:01.62 [ksoftirqd/1]
     18 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [kworker/1:0H-kblockd]
     19 root      20   0       0      0      0 S   0.0   0.0   0:00.00 [kdevtmpfs]
     20 root       0 -20       0      0      0 I   0.0   0.0   0:00.00 [netns]
     21 root      20   0       0      0      0 S   0.0   0.0   0:00.00 [rcu_tasks_kthre]
     22 root      20   0       0      0      0 S   0.0   0.0   0:00.00 [rcu_tasks_rude_]

Describe the results you expected:

Low to none CPU utilization from podman system components.

Additional information you deem important (e.g. issue happens only occasionally):

The issue already appeared on Fedora 32, and continues to happen after upgrading to Fedora 33.

Output of podman version:

$ podman version
Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.15.2
Built:        Wed Oct  7 18:21:20 2020
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.16.1
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.21-3.fc33.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: 0f53fb68333bdead5fe4dc5175703e22cf9882ab'
  cpus: 2
  distribution:
    distribution: fedora
    version: "33"
  eventLogger: journald
  hostname: <redacted>
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.8.18-300.fc33.x86_64
  linkmode: dynamic
  memFree: 696098816
  memTotal: 4018405376
  ociRuntime:
    name: crun
    package: crun-0.15-5.fc33.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.15
      commit: 56ca95e61639510c7dbd39ff512f80f626404969
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.4-4.dev.giteecccdb.fc33.x86_64
    version: |-
      slirp4netns version 1.1.4+dev
      commit: eecccdb96f587b11d7764556ffacfeaffe4b6e11
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.0
  swapFree: 12469395456
  swapTotal: 12745433088
  uptime: 35h 27m 45.72s (Approximately 1.46 days)
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/dmesser/.config/containers/storage.conf
  containerStore:
    number: 5
    paused: 0
    running: 5
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.2.0-1.fc33.x86_64
      Version: |-
        fusermount3 version: 3.9.3
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.3
        using FUSE kernel interface version 7.31
  graphRoot: /home/dmesser/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 4
  runRoot: /run/user/1000/containers
  volumePath: /home/dmesser/.local/share/containers/storage/volumes
version:
  APIVersion: 2.0.0
  Built: 1602087680
  BuiltTime: Wed Oct  7 18:21:20 2020
  GitCommit: ""
  GoVersion: go1.15.2
  OsArch: linux/amd64
  Version: 2.1.1

Package info (output of rpm -q podman):

podman-2.1.1-12.fc33.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide?

No

Additional environment details (AWS, VirtualBox, physical, etc.):

Hetzner VM

[xomachine]

It might be a duplicate of https://github.com/containers/podman/issues/7946

Could you test against the latest master?

[SpyTec]

I've got the same issue on v2.1.1. @xomachine could you or anyone point me on how to install from master? Are there builds we can use directly or do we have to build it ourselves?

[rhatdan]

This is fixed in master. Closing. We should have a new release next week, BTW.