Podman: Service startup crash in a container run with podman as a result of permission issue

[FelixErnst]

/kind bug

Description

Running the docker.io/rocker/rstudio image runs into a problem. An error occurs on the startup of the service with a report of a permission problem.

Recreating the docker.io/rocker/rstudio image with buildah runs into the same problem, when installing the .deb for the service file.

See more detailed informations here: https://github.com/rocker-org/rocker-versioned/issues/187

Steps to reproduce the issue:

1. podman run -dit --env PASSWORD=bioc -p 8788:8787 --name rocker docker.io/rocker/rstudio

2. podman attach rocker

3. Repetitive output of the following error

3 Jan 2020 21:33:46 [rserver] ERROR system error 1 (Operation not permitted); OCCURRED AT: rstudio::core::Error rstudio::core::system::setResourceLimit(rstudio::core::system::ResourceLimit, RLimitType, RLimitType) /var/lib/jenkins/workspace/IDE/open-source-pipeline/v1.2-patch/src/cpp/core/system/PosixSystem.cpp:1496; LOGGED FROM: int main(int, char* const*) /var/lib/jenkins/workspace/IDE/open-source-pipeline/v1.2-patch/src/cpp/server/ServerMain.cpp:478
rserver[9447]: ERROR system error 1 (Operation not permitted); OCCURRED AT: rstudio::core::Error rstudio::core::system::setResourceLimit(rstudio::core::system::ResourceLimit, RLimitType, RLimitType) /var/lib/jenkins/workspace/IDE/open-source-pipeline/v1.2-patch/src/cpp/core/system/PosixSystem.cpp:1496; LOGGED FROM: int main(int, char* const*) /var/lib/jenkins/workspace/IDE/open-source-pipeline/v1.2-patch/src/cpp/server/ServerMain.cpp:478
rsession: no process found

Describe the results you received:

A different behavior from running the container with docker run

Describe the results you expected:

The same behavior as running the container with docker run. However, I am not sure whether this is an issue with podman, buildah or the service itself doing something better left undone.

Additional information you deem important (e.g. issue happens only occasionally):

A build process recreating the image with buildah throws the same error when installing the .deb

Output of podman version:

Version:            1.6.2
RemoteAPI Version:  1
Go Version:         go1.13.1
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.13.1
  podman version: 1.6.2
host:
  BuildahVersion: 1.11.3
  CgroupVersion: v2
  Conmon:
    package: conmon-2.0.2-1.fc31.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.2, commit: 186a550ba0866ce799d74006dab97969a2107979'
  Distribution:
    distribution: fedora
    version: "31"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  MemFree: 13252202496
  MemTotal: 16791990272
  OCIRuntime:
    name: crun
    package: crun-0.10.6-1.fc31.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 0.10.6
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +YAJL
  SwapFree: 5368705024
  SwapTotal: 5368705024
  arch: amd64
  cpus: 1
  eventlogger: journald
  hostname: linux.fritz.box
  kernel: 5.3.16-300.fc31.x86_64
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.0-20.1.dev.gitbbd6f25.fc31.x86_64
    Version: |-
      slirp4netns version 0.4.0-beta.3+dev
      commit: bbd6f25c70d5db2a1cd3bfb0416a8db99a75ed7e
  uptime: 9m 58.69s
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - quay.io
store:
  ConfigFile: /home/felix/.config/containers/storage.conf
  ContainerStore:
    number: 14
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.7.2-2.fc31.x86_64
      Version: |-
        fusermount3 version: 3.6.2
        fuse-overlayfs: version 0.7.2
        FUSE library version 3.6.2
        using FUSE kernel interface version 7.29
  GraphRoot: /home/felix/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 6
  RunRoot: /run/user/1000/containers
  VolumePath: /home/felix/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.6.2-2.fc31.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

Running Fedora 31 in a Virtualbox. Access to exposed port is available from Virtualbox host and client, but results in an empty result, since the service crashes on startup.

[rhatdan]

Are you running rootless mode?
Any chance this is an SELinux issue?

[FelixErnst]

Yes, it is rootless. How to diagnose whether it is an SELinux issue?

Other containers run fine (Including the docker.io/rocker/r-ver and rebuild image using buildah)

[rhatdan]

Could you try rootfull? It might be user namespace not allowing some interation.

For SELinux just do a sudo setenforce 0 and then run the tests. If it succeeds, then it is SELinux blocking it.

If it works as rootfull, then it is probably something with user namespace and we need to dig deeper. Not all things that can be done as root can be done in rootless mode.

[FelixErnst]

SELinux: setting sudo setenforce 0 doesn't change the behavior

running podman as root does work.

[rhatdan]

Running Podman as root is the exact same as using Docker, which is why it works their.

The issue looks like it is trying to set the resource limits higher then is allowed for the user.

setResourceLimit(rstudio::core::system::ResourceLimit, RLimitType, RLimitType)

Running podman as non root, does not alllow users to get more privileges then they normally have. So the lack of CAP_SYS_RESOURCE is the issue. If you know what the resource limit is being set, and it is lower then the user is allowed it should work. If it is higher, then I don't believe you can get this to work.

[FelixErnst]

I will try to find out. However, I tried 12g of memory and 8 cpus which is the setup of the VirtualBox and it didn't work. So my gut feeling right would be that, this is not about how much but how the resource limit is set, if the function actually crashes from the trying to achieve the deduced function.

thanks for all the infos.

[rhatdan]

Maybe this syscall is just not allowed to non privileged users. If we new the value that the container is attempting to set, then we would know if this is blocked.

It looks like if the limit being set is above the HARD limit for the user, it will be blocked.

man ulimit
...
ulimit [-HSabcdefiklmnpqrstuvxPT [limit]]
              Provides  control  over the resources available to the shell and to processes started by it, on systems that allow such control.  The -H and -S options specify that the hard or soft limit is set for the given resource.
              A hard limit cannot be increased by a non-root user once it is set; a soft limit may be increased up to the value of the hard limit.  If neither -H nor -S is specified, both the soft and hard limits are set.  The value
              of  limit  can  be  a number in the unit specified for the resource or one of the special values hard, soft, or unlimited, which stand for the current hard limit, the current soft limit, and no limit, respectively.  If
              limit is omitted, the current value of the soft limit of the resource is printed, unless the -H option is given.  When more than one resource is specified, the limit name and unit are printed before the  value.   Other
              options are interpreted as follows:

[FelixErnst]

In the original issue I tagged a few people from RStudio. Maybe they can offer some inside, which resource setting is causing the crash.

I am sorry for splitting this issue, but that how it went. I hope thats ok for you. I will let you know, of something develops on the other side.

[mheon]

I think there's a user limits file somewhere where root can define default ulimits for the whole system or specific users - if all else fails, you could likely adjust your user's configuration to what RStudio expects. This would apply to everything run with your user, not just the container, though.

[rhatdan]

It is set by pam_limits using the /etc/security/limits.conf file.

[giuseppe]

I think you are hitting: https://github.com/rstudio/rstudio/blob/master/src/cpp/server/ServerMain.cpp#L541

4096 is not too high, so I think you should be fine with your default configuration.

Can you please try again with --ulimit=host to podman run?

podman run -dit --ulimit=host --env PASSWORD=bioc -p 8788:8787 --name rocker docker.io/rocker/rstudio

[FelixErnst]

After reading the other issue I though so myself

podman run -dit --ulimit=host --env PASSWORD=bioc -p 8788:8787 --name rocker docker.io/rocker/rstudio

works.

I close this issue. Thanks for all the help and pointing me in the right direction.