Podman: Running a pod as rootless user with systemd: "setting rlimit type 7: operation not permitted"

Created on 18 Jun 2020  路  34Comments  路  Source: containers/podman

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

According to #5605: I try to run a pod with systemd as a rootless user and one redis container inside, I get the same error and some additional output if a extend the logging:

Jun 18 12:16:58 hostsystem systemd[1]: Started Podman pod-redis-pod.service.
Jun 18 12:16:58 hostsystem systemd[1]: Starting Podman container-redis.service...
Jun 18 12:16:59 hostsystem podman[24348]: Error: unable to start container "redis": container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"process_linux.go:353: setting rlimits for ready process caused \\\"error setting rlimit type 7: operation not permitted\\\"\"": OCI runtime permission denied error
Jun 18 12:16:59 hostsystem systemd[1]: container-redis.service: Control process exited, code=exited, status=125/n/a
Jun 18 12:16:59 hostsystem systemd[1]: container-redis.service: Failed with result 'exit-code'.
Jun 18 12:16:59 hostsystem systemd[1]: Failed to start Podman container-redis.service.
Jun 18 12:16:59 hostsystem systemd[1]: container-redis.service: Service RestartSec=100ms expired, scheduling restart.
Jun 18 12:16:59 hostsystem systemd[1]: container-redis.service: Scheduled restart job, restart counter is at 5.
Jun 18 12:16:59 hostsystem systemd[1]: Stopped Podman container-redis.service.
Jun 18 12:16:59 hostsystem systemd[1]: Stopping Podman pod-redis-pod.service...

Steps to reproduce the issue:

  1. Create pod and container:
podman pod create -n redis-pod -p 6379
podman create --name=redis -d --pod=redis-pod localhost/redis:6.0.4
podman generate systemd -f -n redis-pod
  1. Modify unit files like this:
[Unit]
Description=Podman pod-redis-pod.service
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target
Requires=container-redis.service
Before=container-redis.service

[Service]
User=rootlessuser
Group=rootlessuser
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
ExecStart=/usr/bin/podman --log-level debug --syslog true start 2cf7fac7e0d9-infra
ExecStop=/usr/bin/podman stop -t 10 2cf7fac7e0d9-infra
PIDFile=/run/user/2662/containers/overlay-containers/d16d7cfc632aaac4c2a6e118a21ff8c8031af9fccc9cd1ae42280d7dd66e9df0/userdata/conmon.pid
KillMode=none
Type=forking

[Install]
WantedBy=multi-user.target default.target

[Unit]
Description=Podman container-redis.service
Documentation=man:podman-generate-systemd(1)
Wants=network.target
After=network-online.target
RefuseManualStart=yes
RefuseManualStop=yes
BindsTo=pod-redis-pod.service
After=pod-redis-pod.service

[Service]
User=rootlessuser
Group=rootlessuser
Environment=PODMAN_SYSTEMD_UNIT=%n
Restart=on-failure
ExecStart=/usr/bin/podman start redis
ExecStop=/usr/bin/podman stop -t 10 redis
PIDFile=/run/user/2662/containers/overlay-containers/2e0e9d201129d04071b7cedb8e6d828c7877684979b9a66e8f4280749538c9c3/userdata/conmon.pid
KillMode=none
Type=forking

[Install]
WantedBy=multi-user.target default.target
  1. Copy unit files to /etc/systemd/system/ and enable them.

Describe the results you received:

Here is a debug log which is created by systemd running the above unit. It is one try of some where systemd tries to start the pod.

tl;dr

Jun 18 12:16:56 hostsystem systemd[1]: Starting Podman container-redis.service...
Jun 18 12:16:56 hostsystem podman[23798]: Error: unable to start container "redis": container_linux.go:344: starting container process caused "process_linux.go:424: container init caused \"process_linux.go:353: setting rlimits for ready process caused \\\"error setting rlimit type 7: operation not permitted\\\"\"": OCI runtime permission denied error
Jun 18 12:16:56 hostsystem systemd[1]: container-redis.service: Control process exited, code=exited, status=125/n/a
Jun 18 12:16:56 hostsystem systemd[1]: container-redis.service: Failed with result 'exit-code'.
Jun 18 12:16:56 hostsystem systemd[1]: Failed to start Podman container-redis.service.
Jun 18 12:16:57 hostsystem systemd[1]: container-redis.service: Service RestartSec=100ms expired, scheduling restart.
Jun 18 12:16:57 hostsystem systemd[1]: container-redis.service: Scheduled restart job, restart counter is at 3.
Jun 18 12:16:57 hostsystem systemd[1]: Stopped Podman container-redis.service.
Jun 18 12:16:57 hostsystem systemd[1]: Stopping Podman pod-redis-pod.service...
Jun 18 12:16:57 hostsystem podman[23900]: d16d7cfc632aaac4c2a6e118a21ff8c8031af9fccc9cd1ae42280d7dd66e9df0
Jun 18 12:16:57 hostsystem systemd[1]: pod-redis-pod.service: Succeeded.
Jun 18 12:16:57 hostsystem systemd[1]: Stopped Podman pod-redis-pod.service.
Jun 18 12:16:57 hostsystem systemd[1]: pod-redis-pod.service: Found left-over process 23169 (fuse-overlayfs) in control group while starting unit. Ignoring.
Jun 18 12:16:57 hostsystem systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jun 18 12:16:57 hostsystem systemd[1]: pod-redis-pod.service: Found left-over process 23722 (slirp4netns) in control group while starting unit. Ignoring.
Jun 18 12:16:57 hostsystem systemd[1]: This usually indicates unclean termination of a previous run, or service implementation deficiencies.
Jun 18 12:16:57 hostsystem systemd[1]: Starting Podman pod-redis-pod.service...

After systemd stopped restarting, this was logged by the pod:

$ podman logs 2cf7fac7e0d9-infra
Shutting down, got signal: Terminated
Shutting down, got signal: Terminated
Shutting down, got signal: Terminated
Shutting down, got signal: Terminated
Shutting down, got signal: Terminated

Describe the results you expected:

A running pod with a redis container inside.

Additional information you deem important (e.g. issue happens only occasionally):

This issue happens everytime with 1.9.3.

Output of podman version:

Version:            1.9.3
RemoteAPI Version:  1
Go Version:         go1.11.6
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  gitCommit: ""
  goVersion: go1.11.6
  podmanVersion: 1.9.3
host:
  arch: amd64
  buildahVersion: 1.14.9
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.16, commit: '
  cpus: 24
  distribution:
    distribution: debian
    version: "10"
  eventLogger: file
  hostname: hostsystem
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 2662
      size: 1
    - container_id: 1
      host_id: 1083040
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 2662
      size: 1
    - container_id: 1
      host_id: 1083040
      size: 65536
  kernel: 4.19.0-9-amd64
  memFree: 58359988224
  memTotal: 66994249728
  ociRuntime:
    name: runc
    package: 'runc: /usr/sbin/runc'
    path: /usr/sbin/runc
    version: |-
      runc version 1.0.0~rc6+dfsg1
      commit: 1.0.0~rc6+dfsg1-3
      spec: 1.0.1
  os: linux
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.0.0
      commit: unknown
      libslirp: 4.2.0
  swapFree: 7999582208
  swapTotal: 7999582208
  uptime: 48h 29m 16.66s (Approximately 2.00 days)
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/rootlessuser/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 0
    stopped: 2
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: 'fuse-overlayfs: /usr/bin/fuse-overlayfs'
      Version: |-
        fusermount3 version: 3.4.1
        fuse-overlayfs: version 0.7.6
        FUSE library version 3.4.1
        using FUSE kernel interface version 7.27
  graphRoot: /home/rootlessuser/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 3
  runRoot: /run/user/2662/containers
  volumePath: /home/rootlessuser/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

Listing... Done
podman/unknown,now 1.9.3~1 amd64 [installed]
podman/unknown 1.9.3~1 arm64
podman/unknown 1.9.3~1 armhf
podman/unknown 1.9.3~1 ppc64el

Additional environment details (AWS, VirtualBox, physical, etc.):

I do not have this problem if I use podman 1.9.1 with the same setting.

In Release v1.9.2 you fixed a bug: "Fixed a bug where rootless Podman was setting resource limits on cgroups v2 systems that were not using systemd-managed cgroups (and thus did not support resource limits), resulting in containers failing to start" Maybe this could be the issue, that podman now tries to use cgroupv2 which is not enabled if it is running wiht systemd? My idea now is to enable cgroupv2 and use crun.

I also tried this workaround (with some changes) and it worked for me. But it is not the way I want to use podman.

kinbug stale-issue
Source
picture rei-ber

All 34 comments

@vrothberg PTAL

rhatdan picture rhatdan on 18 Jun 2020

Can you try create the container with --cgroups=no-conmon and retry?

vrothberg picture vrothberg on 18 Jun 2020

@giuseppe WDYT?

vrothberg picture vrothberg on 18 Jun 2020

Can you try create the container with --cgroups=no-conmon and retry?

Got the same result with the same setting, except adding your suggestion to create-command of redis container.

And I also changed to cgroupv2 and now facing on this issue #6084

rei-ber picture rei-ber on 18 Jun 2020

This looks like a cgroups issue. Assigning to @giuseppe who's the specialist for all-things cgroups :)

vrothberg picture vrothberg on 22 Jun 2020

It doesn't look like a cgroups issue.

I think it can be the same root case of https://github.com/containers/libpod/issues/6389

We are forcing some rlimits, and the OCI runtime cannot satisfy them (in this case runc).

giuseppe picture giuseppe on 22 Jun 2020

Thanks, @giuseppe!

@rei-ber , could try the mentioned workarounds/solutions in #6389. https://github.com/containers/libpod/issues/6389#issuecomment-634755801 could be one.

vrothberg picture vrothberg on 22 Jun 2020

in this case, the issue is with NOFILE instead of NPROC

giuseppe picture giuseppe on 22 Jun 2020

Thanks, @giuseppe!

@rei-ber , could try the mentioned workarounds/solutions in #6389. #6389 (comment) could be one.

Well, I add this to ~/.config/containers/containers.conf:

default_ulimits = [
 "nofile=4098:4098",
]

and it worked more or less. First I got this error in journalctl while I enabled debuglog and syslog in start command of podman:

Jun 22 14:28:45 hostsystem podman[10157]: [conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

After podman system reset and creating a new containers.conf with the settings above in users' home, it worked. I did not made any changes under /etc/security/limits.conf So my default hard limit for this user is: 

$ ulimit -Hn
1048576

But I recognize another behavior. I have another container in the same pod with redis inside. The other container is created with --ulimit host. After starting the pod, the infra container and redis container try to start, but they fail due to the failed to write to /proc/self/oom_score_adj: Permission denied error. If I remove --ulimit host from the other container specification and have nofile ulimit set via containers.conf it works. Why does it not work with --ulimit host? I would expect that it takes the config from containers.conf?!

rei-ber picture rei-ber on 22 Jun 2020

If I remove --ulimit host from the other container specification and have nofile ulimit set via containers.conf it works. Why does it not work with --ulimit host? I would expect that it takes the config from containers.conf?!

@giuseppe @rhatdan WDYT?

vrothberg picture vrothberg on 23 Jun 2020

--ulimit host takes the same ulimits that are in place for the podman process.

So it might be the --ulimit host are different when you create the container than what are really available when you try to run it

giuseppe picture giuseppe on 23 Jun 2020

Any chance that you could try this out with podman v2.0, creating the containers from scratch. I know there was a bug in podman 1.9.* that was creating ulimit requests in the container config, when they were not asked for.

rhatdan picture rhatdan on 23 Jun 2020

Any chance that you could try this out with podman v2.0, creating the containers from scratch. I know there was a bug in podman 1.9.* that was creating ulimit requests in the container config, when they were not asked for.

These days I can't try it with v2.0 on the machine where I have this issue; I will try it in a few days. But I want to share my results with the same setup on another machine with you. I used an OpenSuSe Leap 15.1 with the same container and pod I described above and ran it with systemd. The result was, that it worked. But there are some differences in Versions of kernel, fuse-overlayfs etc.

$ podman info --debug
debug:
  compiler: gc
  gitCommit: ""
  goVersion: go1.12.17
  podmanVersion: 1.9.3
host:
  arch: amd64
  buildahVersion: 1.14.9
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.10-lp151.2.1.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.10, commit: unknown'
  cpus: 8
  distribution:
    distribution: '"opensuse-leap"'
    version: "15.1"
  eventLogger: file
  hostname: hostname
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 100
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 2662
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.5.10-1.g3925fb5-default
  memFree: 423849984
  memTotal: 6130479104
  ociRuntime:
    name: runc
    package: runc-1.0.0~rc6-lp151.1.2.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc6
      spec: 1.0.1-dev
  os: linux
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-0.4.5-lp151.2.9.1.x86_64
    version: |-
      slirp4netns version 0.4.5
      commit: unknown
      libslirp: 4.3.0-git
  swapFree: 4294963200
  swapTotal: 4294963200
  uptime: 7h 15m 21s (Approximately 0.29 days)
registries:
  search:
  - docker.io
store:
  configFile: /home/rootlessuser/.config/containers/storage.conf
  containerStore:
    number: 2
    paused: 0
    running: 2
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.7.6-lp151.5.1.x86_64
      Version: |-
        fuse-overlayfs: version 0.7.6
        FUSE library version 3.6.1
        using FUSE kernel interface version 7.29
  graphRoot: /home/rootlessuser/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 4
  runRoot: /tmp/run-2662/containers
  volumePath: /home/rootlessuser/.local/share/containers/storage/volumes

$ podman version
Version:            1.9.3
RemoteAPI Version:  1
Go Version:         go1.12.17
Built:              Mon May 25 02:00:00 2020
OS/Arch:            linux/amd64

$ ulimit -Ha
ulimit -Ha
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 23256
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 4096
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 23256
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

$ ulimit -Sa
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 23256
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 23256
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

After a clean installation of podman and building container and pod from scratch, without any containers.conf in home dir, I get this limits:

$ podman inspect redis --format '{{ .HostConfig.Ulimits }}'
[{RLIMIT_NOFILE 4096 4096} {RLIMIT_NPROC 23256 23256}]
$ podman inspect ba7d52822080-infra --format '{{ .HostConfig.Ulimits }}'
[{RLIMIT_NOFILE 1024 1024}]

The infra container gets a different rlimit for nofile; why? I would expect it gets the same as the redis container.

So, it could be a bug of another component?!

rei-ber picture rei-ber on 24 Jun 2020

I get the same error
Error: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:378: setting rlimits for ready process caused \\\"error setting rlimit type 7: invalid argument\\\"\"": OCI runtime error

with a simple podman run --rm -it hello-world as non-root.

podman run --rm -it --ulimit host hello-world works.

the-antz picture the-antz on 24 Jun 2020
👍1

Your saying you are getting this same issue with a container created with podman v2.0?

rhatdan picture rhatdan on 25 Jun 2020

Your saying you are getting this same issue with a container created with podman v2.0?

I can't tell for the-antz but I got the same issue with containers created with podman v2.0

ps:

I "podman system reset" plus " rm -rf ~/.local/share/containers/" plus rebooted and still have the issue
Is it enought to say "container created with podman v2.0" ?

Eluminae picture Eluminae on 25 Jun 2020

What does
$ podman inspect ba7d52822080-infra --format '{{ .HostConfig.Ulimits }}'
Show?

rhatdan picture rhatdan on 25 Jun 2020

Cgroup V1 or V2?
Do you have limits set in /etc/security/limits.conf? Or the equivalent for your platform?

rhatdan picture rhatdan on 25 Jun 2020

I'm getting the same error in voidlinux, as a workaround I'm calling podman with the --ulimit=host parameter.

My /etc/security/limits.conf is only commented code and /etc/security/limits.d is empty.

It was working fine in podman 1.9.3 until today that I updated to 2.0.0

ulimit -Sa
Maximum size of core files created                           (kB, -c) 0
Maximum size of a process鈥檚 data segment                     (kB, -d) unlimited
Maximum size of files created by the shell                   (kB, -f) unlimited
Maximum size that may be locked into memory                  (kB, -l) 64
Maximum resident set size                                    (kB, -m) unlimited
Maximum number of open file descriptors                          (-n) 1024
Maximum stack size                                           (kB, -s) 8192
Maximum amount of cpu time in seconds                   (seconds, -t) unlimited
Maximum number of processes available to a single user           (-u) 63440
Maximum amount of virtual memory available to the shell      (kB, -v) unlimited

ulimit -Ha
Maximum size of core files created                           (kB, -c) unlimited
Maximum size of a process鈥檚 data segment                     (kB, -d) unlimited
Maximum size of files created by the shell                   (kB, -f) unlimited
Maximum size that may be locked into memory                  (kB, -l) 64
Maximum resident set size                                    (kB, -m) unlimited
Maximum number of open file descriptors                          (-n) 4096
Maximum stack size                                           (kB, -s) unlimited
Maximum amount of cpu time in seconds                   (seconds, -t) unlimited
Maximum number of processes available to a single user           (-u) 63440
Maximum amount of virtual memory available to the shell      (kB, -v) unlimited

podman version
Version:      2.0.0
API Version:  1
Go Version:   go1.14.4
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

podman info --debug
host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v1
  conmon:
    package: Unknown
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.17, commit: unknown'
  cpus: 8
  distribution:
    distribution: '"void"'
    version: unknown
  eventLogger: file
  hostname: x1
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.6.19_1
  linkmode: dynamic
  memFree: 12201861120
  memTotal: 16751316992
  ociRuntime:
    name: runc
    package: Unknown
    path: /usr/bin/runc
    version: 'runc version spec: 1.0.1-dev'
  os: linux
  remoteSocket:
    path: /tmp/run-1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.0.1
      commit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4
      libslirp: 4.3.0
  swapFree: 0
  swapTotal: 0
  uptime: 25m 37.58s
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
store:
  configFile: /home/juan/.config/containers/storage.conf
  containerStore:
    number: 4
    paused: 0
    running: 0
    stopped: 4
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/juan/.data/containers/storage
  graphStatus: {}
  imageStore:
    number: 1
  runRoot: /tmp/run-1000/containers
  volumePath: /home/juan/.data/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.14.4
  OsArch: linux/amd64
  Version: 2.0.0
juantascon picture juantascon on 25 Jun 2020

Your saying you are getting this same issue with a container created with podman v2.0?

Yes. I also tried podman system reset and manually wiped all local images and container related configs to make it recreate everything with v2, that didn't help.

Cgroup V1 or V2?

I have no clue, sry. How do I check that?

Do you have limits set in /etc/security/limits.conf? Or the equivalent for your platform?

No, all entries are commented out.

podman inspect hello-world-container-name --format '{{ .HostConfig.Ulimits }}' outputs
[{RLIMIT_NOFILE 4096 1024} {RLIMIT_NPROC 32768 32768}]

edit: this is on voidlinux, too

the-antz picture the-antz on 25 Jun 2020

I have also wiped all podman data and restarted podman and I'm getting the same issue
killall podman\ pause; sudo rm -rf .data/containers/ /tmp/run-1000/{containers,libpod,runc,netns}

juantascon picture juantascon on 25 Jun 2020

Cgroup V1 or V2?

I have no clue, sry. How do I check that?

e.g.: podman info | grep cgroupVersion

rei-ber picture rei-ber on 25 Jun 2020 
podman info | grep cgroupVersion
  cgroupVersion: v1
juantascon picture juantascon on 25 Jun 2020 
podman info | grep cgroupVersion
  cgroupVersion: v1

same here

the-antz picture the-antz on 25 Jun 2020

same here, same disto. Void podman seems broken :D

Eluminae picture Eluminae on 25 Jun 2020

I believe for v2.0.X this will be fixed with: https://github.com/containers/libpod/pull/6786.

rhafer picture rhafer on 29 Jun 2020

I believe for v2.0.X this will be fixed with: #6786.

Does it really fix it? I mean, I understand this bug as follows: a rootless container is started by systemd and podman tries to set limits which is not allowed due to permission problems of rootless user. Or does it fail, as you commented in pull request, because of already set limits of the user? Then there is another point: every user already has a limit set by system. Or do you mean the limits set by podman? If yes: where podman sets this limits?
Is it clear what I mean? I try to understand the main reason of this bug, because I did not get it really ;)

rei-ber picture rei-ber on 29 Jun 2020

Does it really fix it?

I think so. yes.

I mean, I understand this bug as follows: a rootless container is started by systemd and podman tries to set limits which is not allowed due to permission problems of rootless user.

It doesn't fail because of a permission problem. non-root users are allowed to call setrlimit(). The problem was that the ordering of the values for the setrlimit() call was wrong and that caused the softlimit to be higher than the hardlimit I think. This is what is triggering that "error setting rlimit type 7: invalid argument error.

Or does it fail, as you commented in pull request, because of already set limits of the user?

I think it will only fail if the user's current soft limit is set to something different than the hard limit.

Then there is another point: every user already has a limit set by system. Or do you mean the limits set by podman? If yes: where podman sets this limits?
Is it clear what I mean? I try to understand the main reason of this bug, because I did not get it really ;)

The main reason is the wrong ordering of the arguments to the AddProcessRlimits("RLIMIT_NPROC", max, current) calls. Which later endup in a call to setrlimit() with the wrong ordering.

rhafer picture rhafer on 30 Jun 2020

Alright, makes sense; thanks for your explanation :-)

rei-ber picture rei-ber on 30 Jun 2020

I am having the same issue with my WSL2 installation. Ubuntu 20.04, podman version 2.0.1

I purged everything on my system and reinstalled many times.

Whats funny is it used to work like a dream on an earlier version, but when I upgraded to this version its been giving me this same error.

$ podman run --rm -it hello-world
Error: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:378: setting rlimits for ready process caused \\\"error setting rlimit type 7: invalid argument\\\"\"": OCI runtime error

And with root:

$ sudo podman run --rm -it hello-world
Error: systemd cgroup flag passed, but systemd support for managing cgroups is not available: OCI runtime error

The --ulimit=host flag fixes things when I run rootless. But oddly enough doesnt change anything when running with root.

I will wait for the suggested fix above of course, but wanted to just toss in my experience here aswell. In the meantime maybe ill try to downgrade to an earlier version.

Thanks!

daltschu22 picture daltschu22 on 3 Jul 2020

Rootful should change to use cgroupfs for --cgroup-manager. You can permanently change this in containers.conf.
There is a fix in the works for the rootless issues on podman trying to set pids-limit.

https://github.com/containers/libpod/pull/6842

rhatdan picture rhatdan on 4 Jul 2020
👍1

A friendly reminder that this issue had no activity for 30 days.

github-actions[bot] picture github-actions[bot] on 4 Aug 2020

the fix got merged. Let's close this

giuseppe picture giuseppe on 4 Aug 2020

Thanks, now it works for me with podman 2.0.4 :+1:

rei-ber picture rei-ber on 6 Aug 2020
Was this page helpful?
0 / 5 - 0 ratings

Related issues

Conan-Kudo picture Conan-Kudo  路  91Comments
johanbrandhorst picture johanbrandhorst  路  65Comments
jlebon picture jlebon  路  58Comments
vrischmann picture vrischmann  路  50Comments
AkihiroSuda picture AkihiroSuda  路  50Comments