Runc: Missing pieces running rootless containers on RHEL 7.4 + Documentation

[foldingbeauty]

Running containers in a rootless manner isn't straight forward on RHEL 7.4 as I excepted. I willing to write some extra documentation about getting rootless containers up-and-running on RHEL 7.4 if I can find the last piece of information how to solve the issue I have.

How to reproduce:

• Provision a Node and install a clean RHEL 7.4
• Install the latest runc (1.0.0-rc5)
• Make some adjustments to the Kernel to enable user namespaces (namespace.unpriv_enable=1 user_namespace.enable=1)
• Change the settings to allow to create user namespaces (sudo echo "user.max_user_namespaces=15000" >> /etc/sysctl.conf)
• Run the runc test script to check if we all good @ https://github.com/opencontainers/runc/blob/master/script/check-config.sh
• Create a rootless container using the instruction @ https://github.com/opencontainers/runc/blob/master/README.md#rootless-containers
• runc --root /tmp/runc run mycontainerid

I got an error:

container_linux.go:348: starting container process caused "process_linux.go:301: running exec setns process for init caused \"exit status 47\""

I think I'm close. Any one having a glue what I miss? I also disabled SELinux to be sure.

If I got this working I will change the documentation and test-script to make sure all RHEL users can experience the awesome power of rootless containers!

[ghost]

RHEL-7.4 is too old for rootless. When RHEL-7.6 is out (soon), it should work fine. I did test rootless throughout over there using the following tests that run as non-root.

https://fedorapeople.org/cgit/caiqian/public_git/runctst.git/tree/runctst.py

[giuseppe]

I've used rootless containers on RHEL-7.5 and it should work fine. The only limitation is that you can map only one user inside the user namespace, or you need to manually install newuidmap/newgidmap as it is not present on RHEL.

[davidMcneil]

@giuseppe Could you explain more how to get rootless containers to work in RHEL-7.5? My system configuration is as follows:

> more /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)
> uname -r
3.10.0-862.11.6.el7.x86_64
> more /proc/cmdline
BOOT_IMAGE=/vmlinuz-3.10.0-862.11.6.el7.x86_64 root=/dev/mapper/cl-root ro crashkernel=auto namespace.unpriv_enable=1 user_namespace.enable=1 rd.lvm.lv=cl/root rd.lvm.lv=cl/swap rhgb quiet
> sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      31

When trying to run a container I get container_linux.go:337: starting container process caused "process_linux.go:309: running exec setns process for init caused \"exit status 39\""

Should running rootless containers be possible with this system configuration?

[giuseppe]

@davidMcneil you'd need to install newuidmap/newgidmap separately, as they are not present on RHEL 7.5, but that is not supported.

[rhatdan]

@davidMcneil We are working to get these packaged and shipping in RHEL7, but they come from a newer shadow-utils package, that the maintainers do not want to back port to RHEL7 because it is too risky. We are working on a mechanism to ship the two binaries in a separate package, but this is still being negotiated.

[AkihiroSuda]

@giuseppe wrote a blog about this 👍 https://www.scrivano.org/2018/10/12/rootless-podman-from-upstream-on-centos-7/

[AkihiroSuda]

Also there is @vbatts 's newxidmap RPM and doc: https://copr.fedorainfracloud.org/coprs/vbatts/shadow-utils-newxidmap/

[ssbarnea]

Any idea when this woud work without extra steps? Following instructions from @AkihiroSuda allowed me to finally do a successfil pull as a non-root user on CentOS-7, but was clearly a messy process.

[tnguyen14]

@rhatdan is there any update on shipping newuidmap/newgidmap as separate packages?
@giuseppe is there a way to run rootless without those 2 binaries on RHEL 7.6?

[rhatdan]

The newuidmap and newgidmap will be in RHEL7.7 version of shadow-utils.

[tnguyen14]

Ah okay. Is there any way to have rootless container to work on 7.6 without newuidmap/newgidmap?

[giuseppe]

@giuseppe is there a way to run rootless without those 2 binaries on RHEL 7.6?

recent Podman releases support single UID mapping, so if you are running without these binaries rootless containers will still work but you'll have only one UID/GID available in the user namespace

[cyphar]

@giuseppe This is the runc repo not podman. :wink:

@tnguyen14 From memory, 7.6 should have all the kernel bits necessary -- but it'd probably just be easier to try it out and tell us if it works (I don't have access to a RHEL subscription).

[tnguyen14]

@cyphar by try it out, do you mean building newgidmap/newuidmap manually?

[cyphar]

@tnguyen14 Yes (sorry I didn't see your question earlier).

[AkihiroSuda]

This is more straightforward with the latest RHEL/CentOS 7. Closable?

[nighter7]

I'm trying to run podman inside a container which will run in openshift using docker.

The use case is to create a pod with 2 containers, container A having business logic running and the container B having podman as sidecar.

Container A will use podman in Container B to pull docker executable images, run them and get the output, parse it and continue with the business logic.

so far running the container outside openshift, in docker I'm stuck with mounting net related path even using --net=host:

[root@86a6be3ae891 /]# podman --storage-driver vfs run hello-world
ERRO[0000] unable to write pod event: "write unixgram @00093->/run/systemd/journal/socket: sendmsg: no such file or directory"

ERRO[0000] Error preparing container eb2829b33026d0065b5dd75111f15031b197542546cd9337cc7bd83e0ede0ec6: error creating network namespace for container eb2829b33026d0065b5dd75111f15031b197542546cd9337cc7bd83e0ede0ec6: mount --make-rshared /var/run/netns failed: "operation not permitted"

Error: failed to mount shm tmpfs "/var/lib/containers/storage/vfs-containers/eb2829b33026d0065b5dd75111f15031b197542546cd9337cc7bd83e0ede0ec6/userdata/shm": operation not permitted

--

Just for giving a try I moved the image to openshift but It seems openshift is “modifying/restricting” /proc/sys/user/max_user_namespaces setting its value to 0:

bash-4.4$ podman info
cannot clone: Invalid argument
user namespaces are not enabled in /proc/sys/user/max_user_namespaces
Error: could not get runtime: cannot re-exec process
bash-4.4$ cat /proc/sys/user/max_user_namespaces
0

Running that in docker, it has a value different that 0:

[root@b17073d73435 /]# cat /proc/sys/user/max_user_namespaces
64060

Any clue???
As mentioned, the scenario is
podman in docker

openshfit 3.11