Podman: Error: setrlimit `RLIMIT_NOFILE`: Invalid argument: OCI runtime error

[mybigman]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

unable to create containers

Steps to reproduce the issue:

1. podman run -it --rm --name test alpine:latest

Describe the results you received:
Error: setrlimit RLIMIT_NOFILE: Invalid argument: OCI runtime error

Describe the results you expected:
The container to start

Additional information you deem important (e.g. issue happens only occasionally):
podman run --ulimit host -it --rm --name test alpine:latest
Using --ulimit host solves the issue.

Output of podman version:

Version:      2.0.1
API Version:  1
Go Version:   go1.14.3
Built:        Thu Jan  1 10:00:00 1970
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.18-1.fc32.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.18, commit: 6e8799f576f11f902cd8a8d8b45b2b2caf636a85'
  cpus: 128
  distribution:
    distribution: fedora
    version: "32"
  eventLogger: file
  hostname: hostname.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.7.6-201.fc32.x86_64
  linkmode: dynamic
  memFree: 268024225792
  memTotal: 270138871808
  ociRuntime:
    name: crun
    package: crun-0.14-2.fc32.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.14
      commit: ebc56fc9bcce4b3208bb0079636c80545122bf58
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  os: linux
  remoteSocket:
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.1-1.fc32.x86_64
    version: |-
      slirp4netns version 1.1.1
      commit: bbf27c5acd4356edb97fa639b4e15e0cd56a39d5
      libslirp: 4.2.0
      SLIRP_CONFIG_VERSION_MAX: 2
  swapFree: 34359734272
  swapTotal: 34359734272
  uptime: 27m 59.31s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 8
    paused: 0
    running: 0
    stopped: 8
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.1-1.fc32.x86_64
      Version: |-
        fusermount3 version: 3.9.1
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.9.1
        using FUSE kernel interface version 7.31
  graphRoot: /home/user/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 9
  runRoot: /run/user/1000/containers
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 10:00:00 1970
  GitCommit: ""
  GoVersion: go1.14.3
  OsArch: linux/amd64
  Version: 2.0.1

Package info (e.g. output of rpm -q podman or apt list podman):

podman-2.0.1-1.fc32.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):
Physical Machine

ulimit -u --soft
8388608

ulimit -u --hard
8388608

[vrothberg]

Thanks for opening the issue, @mybigman! We're working to fix the issue. Cc @giuseppe @mheon @rhatdan

[mheon]

This looks sufficiently different from the PID limit that I'm not sure the proposed fix for that will fix this

[mheon]

It's also rather unusual that you're seeing this with NOFILE, and not the pids limit.

It looks like it is set to a non-standard value here...

[rhatdan]

Could you show the podman inspect $CTR and look at the ulimit settings?

Do you have a containers.conf on your system with some settings?

[mybigman]

@rhatdan I searched the system and there at two containers.conf

~/.config/containers/containers.conf < I did not create this file and renaming this file made no difference.

default_ulimits = [
  "nproc=200000:400000",
]

/usr/share/containers/containers.conf < I have not made any changes to this file and everything appears to be commented out.

I can only do an inspect on a container that's ran with --ulimit host since I can no longer create containers without it or an existing container that was built prior to the issue.

built with --ulimit host

"Ulimits": [],

existing container

 "Ulimits": [
    {
        "Name": "RLIMIT_NOFILE",
        "Soft": 524288,
        "Hard": 524288
    },
    {
        "Name": "RLIMIT_NPROC",
        "Soft": 4194304,
        "Hard": 4194304
    }
],

[rhatdan]

If you remove the file from your homedir, you still can not create a container?

[mybigman]

Correct... without this file I still cant create the container.

[rhatdan]

If you create the container, with the create command and then inspect it you are seeing ulimits in the config?

podman --version

[mybigman]

Doing podman create alpine:latest ls has the following in the ulimits.

"Ulimits": [
    {
        "Name": "RLIMIT_NOFILE",
        "Soft": 524288,
        "Hard": 1024
    },
    {
        "Name": "RLIMIT_NPROC",
        "Soft": 4194304,
        "Hard": 4194304
    }
],

Podman version is in first post.

podman version 2.0.1

[rhatdan]

Could you download and try https://koji.fedoraproject.org/koji/buildinfo?buildID=1537752

[rhatdan]

It should shortly be in updates-testing.
https://bodhi.fedoraproject.org/updates/FEDORA-2020-19924b556e

# dnf -y update --enablerepo=updates-testing podman

Sadly it was not there when I just tried.

[mybigman]

Installed the 2.0.2 version from testing and works as expected. Thanks for persevering.

ulimits

 "Ulimits": [
    {
        "Name": "RLIMIT_NOFILE",
        "Soft": 1024,
        "Hard": 524288
    },
    {
        "Name": "RLIMIT_NPROC",
        "Soft": 4194304,
        "Hard": 4194304
    }
],

[rugk]

This issue is still happening in Podman v2.0.2, although it was supposed to be fixed there.

For more information see:
https://github.com/containers/toolbox/issues/500#issuecomment-658424664

[HarryMichal]

I also hit this issue on Fedora Silverblue Rawhide. I encounter it mainly when I try to enter a toolbox container created with an older version of Podman (pre v2.0.2). New steps to reproduce the issue:

1. podman run --ulimit=nofile=524288:524288 --ulimit=nproc=63173:63173 alpine echo Hello World

One of the --ulimit options can be dropped, I included them because they used to be set both.

The ^ command fails with:

Error: setrlimit `RLIMIT_NPROC`: Operation not permitted: OCI runtime permission denied error
````

Output of `rpm -q podman`:

podman-2.1.0-0.70.dev.git3d33590.fc33.x86_64

Output of `podman version`:

Version: 2.1.0-dev
API Version: 1
Go Version: go1.14.3
Built: Thu Jan 1 01:00:00 1970
OS/Arch: linux/amd64

Output of `podman info --debug`:

host:
arch: amd64
buildahVersion: 1.16.0-dev
cgroupVersion: v2
conmon:
package: conmon-2.0.19-0.5.dev.giteff699e.fc33.x86_64
path: /usr/bin/conmon
version: 'conmon version 2.0.19-dev, commit: 3c47d3797172bffa8ab02661ac4805b593cfb4ba'
cpus: 4
distribution:
distribution: fedora
version: "33"
eventLogger: file
hostname: harry-ntb
idMappings:
gidmap:
- container_id: 0
host_id: 1001
size: 1
- container_id: 1
host_id: 100000
size: 65536
uidmap:
- container_id: 0
host_id: 1000
size: 1
- container_id: 1
host_id: 100000
size: 65536
kernel: 5.8.0-0.rc5.1.fc33.x86_64
linkmode: dynamic
memFree: 2233049088
memTotal: 16632356864
ociRuntime:
name: crun
package: crun-0.14-1.fc33.x86_64
path: /usr/bin/crun
version: |-
crun version 0.14
commit: ebc56fc9bcce4b3208bb0079636c80545122bf58
spec: 1.0.0
+SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
os: linux
remoteSocket:
path: /run/user/1000/podman/podman.sock
rootless: true
slirp4netns:
executable: /usr/bin/slirp4netns
package: slirp4netns-1.1.4-2.dev.git4c6befe.fc33.x86_64
version: |-
slirp4netns version 1.1.4+dev
commit: 4c6befe05c3137232cf06a5c2879daf4c20be6b1
libslirp: 4.3.1
SLIRP_CONFIG_VERSION_MAX: 3
swapFree: 8401186816
swapTotal: 8401186816
uptime: 20h 6m 17.41s (Approximately 0.83 days)
registries:
search:
- registry.fedoraproject.org
- registry.access.redhat.com
- registry.centos.org
- docker.io
store:
configFile: /var/home/harry/.config/containers/storage.conf
containerStore:
number: 14
paused: 0
running: 3
stopped: 11
graphDriverName: overlay
graphOptions:
overlay.mount_program:
Executable: /usr/bin/fuse-overlayfs
Package: fuse-overlayfs-1.1.0-6.dev.git50ab2c2.fc33.x86_64
Version: |-
fusermount3 version: 3.9.2
fuse-overlayfs: version 1.1.0
FUSE library version 3.9.2
using FUSE kernel interface version 7.31
graphRoot: /var/home/harry/.local/share/containers/storage
graphStatus:
Backing Filesystem: extfs
Native Overlay Diff: "false"
Supports d_type: "true"
Using metacopy: "false"
imageStore:
number: 12
runRoot: /run/user/1000
volumePath: /var/home/harry/.local/share/containers/storage/volumes
version:
APIVersion: 1
Built: 0
BuiltTime: Thu Jan 1 01:00:00 1970
GitCommit: ""
GoVersion: go1.14.3
OsArch: linux/amd64
Version: 2.1.0-dev
```

[mheon]

Is this Silverblue?

[mheon]

And can you include a full podman info?

[HarryMichal]

I updated my previous comment with the needed info.

[mheon]

This has always been broken - I went back to v1.9.3 and that command is equally nonfunctional. This does not seem like a Podman issue. We briefly made containers with bad rlimits, and those containers were and still are unusable. New containers will work properly.