Podman: Is there any chance to run rootless podman container inside another one?

[psmolkin]

/kind feature

Description

I tried to run rootless podman inside another privileged container. But namespaces mapping doesn't work.

Steps to reproduce the issue:

1. Run Fedora 30
2. Install podman
3. # podman run --privileged --detach --name=test --net=host --security-opt label=disable --security-opt seccomp=unconfined --device /dev/fuse:rw quay.io/podman/testing sh -c 'tail -f /dev/null' # podman exec -it test bash
4. # groupadd -g 1001 test
5. ```
   # useradd -g 1001 -u 1001 test
   ````
6. # cat /etc/sub?id test:100000:65536 test:100000:65536
7. # su test
8. podman unshare cat /proc/self/uid_map WARN[0000] using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding subids 0 1001 1

Describe the results you received:
rootless single mapping

Describe the results you expected:
Something like this

podman unshare cat /proc/self/uid_map
         0       1001     1
         1       100000   65536

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:            1.5.1
RemoteAPI Version:  1
Go Version:         go1.12.7
OS/Arch:            linux/amd64

Output of podman info --debug:

WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument
WARN[0000] using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding subids
ERRO[0000] unable to write system event: "write unixgram @000ae->/run/systemd/journal/socket: sendmsg: no such file or directory"
debug:
  compiler: gc
  git commit: ""
  go version: go1.12.7
  podman version: 1.5.1
host:
  BuildahVersion: 1.10.1
  Conmon:
    package: podman-1.5.1-3.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.0, commit: d728afa06cd2df86a27f32a4692c7099a56acc97-dirty'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 320118784
  MemTotal: 2552766464
  OCIRuntime:
    package: runc-1.0.0-93.dev.gitb9b6cc6.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: e3b4c1108f7d1bf0d09ab612ea09927d9b59b4e3
      spec: 1.0.1-dev
  SwapFree: 2191708160
  SwapTotal: 2206199808
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: 172.17.0.183
  kernel: 5.2.14-200.fc30.x86_64
  os: linux
  rootless: true
  uptime: 3h 20m 18.65s (Approximately 0.12 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/test/.config/containers/storage.conf
  ContainerStore:
    number: 0
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /home/test/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: overlayfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 0
  RunRoot: /tmp/run-1001
  VolumePath: /home/test/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

Name         : podman
Epoch        : 2
Version      : 1.5.1
Release      : 3.fc30
Architecture : x86_64
Size         : 54 M
Source       : podman-1.5.1-3.fc30.src.rpm
Repository   : @System
From repo    : updates
Summary      : Manage Pods, Containers and Container Images
URL          : https://podman.io/
License      : ASL 2.0

Additional environment details (AWS, VirtualBox, physical, etc.):
Hyper-V

[mheon]

This should be theoretically possible, but I don't think anyone has successfully achieved it.

@giuseppe @rhatdan There seems to be a fair bit of interest in this, so we might want to look into what it would take / writing a tutorial on how to do it.

[TomSweeneyRedHat]

We're probably a bit closer with upstream/1.6.0 with crun in play, but I think there are still some hiccups.

[giuseppe]

looks like newuidmap/newgidmap don't get enough privileges to setup the namespace.

What is the result of getcap /usr/bin/newuidmap?

In case that is empty, you may try with chmod +s /usr/bin/newgidmap /usr/bin/newgidmap

I am afraid the new*map programs miss the file capabilities, either because of the way Fedora images are built, or because they don't work correctly within overlayfs

[psmolkin]

@giuseppe Yes I already tried to do this, based on your other comment . But unfortunately that didn’t change anything.

[giuseppe]

I've tried similar steps to yours and it seems to work fine:

#podman run --privileged  --name=test --net=host --security-opt label=disable --security-opt seccomp=unconfined --device /dev/fuse:rw  --rm -ti fedora sh
# yum install -y podman crun
# chmod +s /usr/bin/newgidmap /usr/bin/newgidmap
# groupadd -g 1001 test && useradd -g 1001 -u 1001 test
# su test
$ podman --cgroup-manager cgroupfs unshare cat /proc/self/uid_map
         0       1001          1
         1     100000      65536

so it must be something else going wrong

[giuseppe]

is there any pause process running inside the container? Could you try podman system migrate && odman --cgroup-manager cgroupfs unshare cat /proc/self/uid_map as rootless?

[giuseppe]

is there any pause process running inside the container? Could you try podman system migrate && odman --cgroup-manager cgroupfs unshare cat /proc/self/uid_map as rootless?

@psmolkin had a chance to try it out?

[psmolkin]

@giuseppe
I apologize for the long reply.
So. I tried to install crun and change default runtime at /usr/share/containers/libpod.conf
I tried to do the same in the container. And ran system migrate but nothing changed

$ podman system migrate && podman --log-level debug --cgroup-manager cgroupfs unshare cat /proc/self/uid_map
WARN[0000] using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding subids
INFO[0000] running as rootless
DEBU[0000] using conmon: "/usr/libexec/podman/conmon"
DEBU[0000] Initializing boltdb state at /home/test/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/test/.local/share/containers/storage
DEBU[0000] Using run root /tmp/run-1001
DEBU[0000] Using static dir /home/test/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /tmp/run-1001/libpod/tmp
DEBU[0000] Using volume path /home/test/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs
DEBU[0000] backingFs=overlayfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
DEBU[0000] Initializing event backend journald
DEBU[0000] using runtime "/usr/bin/crun"
DEBU[0000] using runtime "/usr/bin/runc"
         0       1001          1
$ cat /etc/sub?id
test:100000:65536
test:100000:65536

[johanbrandhorst]

I'm also trying to get to work, with the aim to eventually be able to run automated test suites that start local containers within an unprivileged docker or podman container. I'm able to get this far, with both archlinux/base and fedora bases.

After installing podman and confirming podman info works, this is what I get when trying to run a container:

# podman run --rm -it ubuntu
ERRO[0000] unable to write system event: "write unixgram @00045->/run/systemd/journal/socket: sendmsg: no such file or directory" 
Trying to pull docker.io/library/ubuntu...
Getting image source signatures
Copying blob 5667fdb72017 done
Copying blob d83811f270d5 done
Copying blob ee671aafb583 done
Copying blob 7fc152dfb3a6 done
Copying config 2ca708c1c9 done
Writing manifest to image destination
Storing signatures
ERRO[0007] unable to write pod event: "write unixgram @00045->/run/systemd/journal/socket: sendmsg: no such file or directory" 
ERRO[0007] error creating network namespace for container fc189c2fb049f6d0955773f86245d7394e0a35181ca97c23782e4b17f8f66fba: mount --make-rshared /var/run/netns failed: "operation not permitted" 
ERRO[0007] unable to write pod event: "write unixgram @00045->/run/systemd/journal/socket: sendmsg: no such file or directory" 
Error: failed to mount shm tmpfs "/home/REDACTED/.local/share/containers/storage/vfs-containers/fc189c2fb049f6d0955773f86245d7394e0a35181ca97c23782e4b17f8f66fba/userdata/shm": operation not permitted

The basic steps I'm following:

1. Install podman on local, bare metal machine
2. Start a container with easy podman installation available (archlinux/base, fedora).
3. Install podman
4. Configure podman to use vfs since I was getting overlay errors
   Sidenote: I think this is because my bare metal podman installation is configured with vfs.
   The error I'm seeing is:
   # podman info ERRO[0000] 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay" Error: could not get runtime: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay": backing file system is unsupported for this graph driver
5. Run container from within container (see log above)

Am I missing something? I'm testing this locally with podman on bare metal, but the environment I'm _really_ targeting is docker on CircleCI.

[mheon]

You'll probably want to run the outer container with either --privileged or --security-opt seccomp=unconfined

[mheon]

(I think seccomp will block the mount calls otherwise)

[johanbrandhorst]

Thanks for the tip, that does unfortunately defeat the whole point :(. Is there any chance this will be possible without --privileged eventually?

[mheon]

Not without changes to the Seccomp profile (and potentially other things) - Seccomp blocks a lot of things (like the mount calls I mentioned) that we need to continue setup.

[johanbrandhorst]

https://stackoverflow.com/a/56856410 might be useful for this discussion too.

[rhatdan]

Could you try to remove seccomp. The seccomp.json that Docker ships blocks the mount syscall, even if it was deemed safe by the kernel. IE non privileged mount is allowed for procfs/sysfs/bind mounts and fuse-mounts for non privileged users but it requires the mount syscall.

The seccomp.json that we ship with Podman allows the mount syscall. You might need a couple of other syscalls that Docker blocks.

Might be other issues as well.

You could try to run podman within podman and see if this works.

[github-actions[bot]]

This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days.

[mheon]

We may want a tracker issue for this. I think we have 3-4 open issues about this.

[vrothberg]

We may want a tracker issue for this. I think we have 3-4 open issues about this.

@rhatdan, I believe you're working on this at the moment. Would you open a tracker issue?

[chengkuangan]

I'm also trying to get to work, with the aim to eventually be able to run automated test suites that start local containers within an unprivileged docker or podman container. I'm able to get this far, with both archlinux/base and fedora bases.

After installing podman and confirming podman info works, this is what I get when trying to run a container:

# podman run --rm -it ubuntu
ERRO[0000] unable to write system event: "write unixgram @00045->/run/systemd/journal/socket: sendmsg: no such file or directory" 
Trying to pull docker.io/library/ubuntu...
Getting image source signatures
Copying blob 5667fdb72017 done
Copying blob d83811f270d5 done
Copying blob ee671aafb583 done
Copying blob 7fc152dfb3a6 done
Copying config 2ca708c1c9 done
Writing manifest to image destination
Storing signatures
ERRO[0007] unable to write pod event: "write unixgram @00045->/run/systemd/journal/socket: sendmsg: no such file or directory" 
ERRO[0007] error creating network namespace for container fc189c2fb049f6d0955773f86245d7394e0a35181ca97c23782e4b17f8f66fba: mount --make-rshared /var/run/netns failed: "operation not permitted" 
ERRO[0007] unable to write pod event: "write unixgram @00045->/run/systemd/journal/socket: sendmsg: no such file or directory" 
Error: failed to mount shm tmpfs "/home/REDACTED/.local/share/containers/storage/vfs-containers/fc189c2fb049f6d0955773f86245d7394e0a35181ca97c23782e4b17f8f66fba/userdata/shm": operation not permitted

The basic steps I'm following:

1. Install podman on local, bare metal machine
2. Start a container with easy podman installation available (archlinux/base, fedora).
3. Install podman
4. Configure podman to use vfs since I was getting overlay errors
   Sidenote: I think this is because my bare metal podman installation is configured with vfs.
   The error I'm seeing is:
   # podman info ERRO[0000] 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay" Error: could not get runtime: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay": backing file system is unsupported for this graph driver
5. Run container from within container (see log above)

Am I missing something? I'm testing this locally with podman on bare metal, but the environment I'm _really_ targeting is docker on CircleCI.

I am using Docker, I do this in my Dockerfile. I do a build for go and libpod from scratch during docker build and also set the events_logger to file. The error gone away. But I have another issue similar to this reported issue.

RUN sed -i 's/# events_logger = "journald"/events_logger = "file"/g' $GOPATH/src/github.com/containers/libpod/libpod.conf

RUN cp /var/go/src/github.com/containers/libpod/libpod.conf /etc/containers/

[rhatdan]

Currently this requires a privileged container and it requires you to mount a different volume on /var/lib/containers/

[FlorianLudwig]

Currently this requires a privileged container and it requires you to mount a different volume on /var/lib/containers/

So I would assume this to work:

podman run --privileged --rm -ti --net=host --security-opt label=disable --security-opt seccomp=unconfined -v ~/tmp_container:/var/lib/containers/ fedora:31 sh -c "dnf install -y podman && podman info"

but it doesn't:

ERRO[0000] 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay" 
Error: could not get runtime: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay": backing file system is unsupported for this graph driver

[rhatdan]

Actually to get this to work, you would need to use fuse-overlay, since you are not allowed to use overlay as non root.
YOu could also try with a storage driver of vfs, and see if this works.

[rhatdan]

This works
podman run -ti --cap-add SYS_ADMIN --device /dev/fuse quay.io/podman/stable podman info

[psmolkin]

@rhatdan thanks for updates!
Which host systems should I use?

[rhatdan]

Don't really understand the question?

[FlorianLudwig]

@rhatdan

thank you for your patients :) !

podman run -ti --cap-add SYS_ADMIN --device /dev/fuse quay.io/podman/stable podman info

does indeed work! But trying to execute anything fails, with networking errors:

podman run -ti --cap-add SYS_ADMIN --device /dev/fuse quay.io/podman/stable podman run hello-world
Trying to pull docker.io/library/hello-world...

Getting image source signatures
Copying blob 1b930d010525 done
Copying config fce289e99e done
Writing manifest to image destination
Storing signatures
ERRO[0005] Error adding network: operation not permitted 
ERRO[0005] Error while adding to cni lo network: operation not permitted 
Error: error configuring network namespace for container 4f6cdd985ee9c0adeec364425ad8f19bbc07de00cf0ca2b3773dc61aba7cc256: operation not permitted

Or:

$ podman run --net=none -ti --security-opt label=disable --security-opt seccomp=unconfined -v ~/tmp_container:/var/lib/containers/ --cap-add SYS_ADMIN --device /dev/fuse quay.io/podman/stable podman run --net=none hello-world
ERRO[0000] Error deleting network: neither iptables nor ip6tables usable 
ERRO[0000] Error while removing pod from CNI network "podman": neither iptables nor ip6tables usable 
ERRO[0000] Error refreshing container 120d2572059087516d0bac18a8cdbab99d86a3419e1d8228c2cbfc830bad00ef: neither iptables nor ip6tables usable 
ERRO[0000] Error deleting network: neither iptables nor ip6tables usable 
ERRO[0000] Error while removing pod from CNI network "podman": neither iptables nor ip6tables usable 
ERRO[0000] Error refreshing container 2177e54067c62c702d7e0675287f23d0e7c8b23d45960740f30183f543001ab2: neither iptables nor ip6tables usable 
ERRO[0000] Error deleting network: neither iptables nor ip6tables usable 
ERRO[0000] Error while removing pod from CNI network "podman": neither iptables nor ip6tables usable 
ERRO[0000] Error refreshing container 2fc722295834ee74c63a0b62fe49f7d6699d5ddc78c210ff6638086785b0c38f: neither iptables nor ip6tables usable 
ERRO[0000] Error deleting network: neither iptables nor ip6tables usable 
ERRO[0000] Error while removing pod from CNI network "podman": neither iptables nor ip6tables usable 
ERRO[0000] Error refreshing container 3bc002079e65317e9dffe81cc3490787a01f50c45e8923f9b8e0e40daf54fb56: neither iptables nor ip6tables usable 
ERRO[0000] Error deleting network: neither iptables nor ip6tables usable 
ERRO[0000] Error while removing pod from CNI network "podman": neither iptables nor ip6tables usable 
ERRO[0000] Error refreshing container 4a546bc670f756c94bc3a8ca275025076da9ea473efafea9e3a3d63dcd7ac40d: neither iptables nor ip6tables usable 
ERRO[0000] Error deleting network: neither iptables nor ip6tables usable 
ERRO[0000] Error while removing pod from CNI network "podman": neither iptables nor ip6tables usable 
ERRO[0000] Error refreshing container 946dfcb6cfd0ddf496e6b9e7e910c3735f9344a17d1a6f8bb425175b90a9f7ad: neither iptables nor ip6tables usable 
ERRO[0000] Error deleting network: neither iptables nor ip6tables usable 
ERRO[0000] Error while removing pod from CNI network "podman": neither iptables nor ip6tables usable 
ERRO[0000] Error refreshing container c9e7e82bb08ba5e6eac12db1fe8062cb57c45c40a7038c583f8b0a024d9e0361: neither iptables nor ip6tables usable 
Error: setrlimit `RLIMIT_NOFILE`: Operation not permitted: OCI runtime permission denied error

Tried --net=none and --net=host both fail with an iptables error.

EDIT: Copied the wrong second example and clarified my question.

[FlorianLudwig]

I just released that the error that is actually the problem is setrlimit not the networking.

Looking into the code:

https://github.com/containers/libpod/blob/b3a3fc3ef6750b01d6182ea43f1fb0921966408b/cmd/podman/main_local.go#L266

It looks like it tries to set the limit to 1048576. So I tried with ulimit -n 1048576 and podman --ulimit=host but made no change:

$ ulimit -n 1048576
$ podman run --ulimit=host --net=host -ti --security-opt label=disable --security-opt seccomp=unconfined -v ~/tmp_container:/var/lib/containers/ --cap-add SYS_ADMIN --device /dev/fuse quay.io/podman/stable podman run --net=host hello-world
...
Error: setrlimit `RLIMIT_NPROC`: Operation not permitted: OCI runtime permission denied error

$ podman run --ulimit=host --net=host -ti --security-opt label=disable --security-opt seccomp=unconfined -v ~/tmp_container:/var/lib/containers/ --cap-add SYS_ADMIN --device /dev/fuse quay.io/podman/stable ulimit -Hn
1048576

Trying with --storage-driver vfs results in the same error.

[FlorianLudwig]

One more note: quay.io/podman/stable contains podman version 1.6.2! Updating to 1.8.2 did not resolve the issue, just created more error logs:

ERRO[0000] unable to write pod event: "write unixgram @00292->/run/systemd/journal/socket: sendmsg: no such file or directory" 

Current state

Here the furthest I have come so far to running podman inside podman:

☒ Rootless

$ podman run --ulimit=host --net=host -ti --security-opt label=disable --security-opt seccomp=unconfined -v ~/tmp_container:/var/lib/containers/ --cap-add SYS_ADMIN --device /dev/fuse quay.io/podman/stable sh -c "dnf update -y; podman run --net=host hello-world"
Error: setrlimit `RLIMIT_NOFILE`: Operation not permitted: OCI runtime permission denied error

☑ Rootfull

# sudo podman run --ulimit=host --net=host -ti --security-opt label=disable --security-opt seccomp=unconfined -v ~/tmp_container:/var/lib/containers/ --privileged --device /dev/fuse quay.io/podman/stable sh -c "dnf update -y; podman run --cgroup-manager=cgroupfs --net=host hello-world"

• --cgroup-manager=cgroupfs needed
• podman 1.8.2 needed

[rhatdan]

Non root users are not able to change the rlimits higher then what they are allocated. This is controlled by the linux kernel not anything podman can do about it. If you need huge rlimits then the user account has to default to them, or you need to run as root.

[FlorianLudwig]

@rhatdan I do not want to set any rlimits at all. Running:

podman run --net=host -ti --security-opt label=disable --security-opt seccomp=unconfined -v ~/tmp_container:/var/lib/containers/ --cap-add SYS_ADMIN --device /dev/fuse quay.io/podman/stable sh -c "dnf update -y; podman run --net=host hello-world"

Fails with:

Error: setrlimit `RLIMIT_NOFILE`: Operation not permitted: OCI runtime permission denied error

I thought I might workaround this by setting a higher rlimit on my "host" podman but that didn't change anything.

[rhatdan]

I think it would be best right now to work on the upstream image. I am actually building it locally and will push what I have done so far. The problem I have is that I have discovered a couple of bugs while going through this process.

[pothos]

This is working now, very exciting:

podman run -ti --security-opt seccomp=unconfined --security-opt label=disable --cap-add SYS_ADMIN --env STORAGE_DRIVER=vfs quay.io/podman/stable sh -c "dnf update -y; podman run hello-world"

Even multiple Matryoshka layers, in the spirit of subhurds from GNU Hurd :smile:

podman run -ti --security-opt seccomp=unconfined --security-opt label=disable --cap-add SYS_ADMIN --env STORAGE_DRIVER=vfs quay.io/podman/stable sh -c 'podman run -ti --security-opt seccomp=unconfined --security-opt label=disable --cap-add SYS_ADMIN --env STORAGE_DRIVER=vfs quay.io/podman/stable sh -c "dnf update -y; podman run hello-world"'

[bbodiya-akamai]

This is excellent! I was also able to confirm that this works with docker when the host kernel supports deferred deletion. I did need to add SYS_RESOURCE in addition to SYS_ADMIN though. Otherwise I got a setrlimit error like Florian above.

docker run --rm -ti --security-opt seccomp=unconfined --security-opt label=disable --cap-add SYS_ADMIN --cap-add SYS_RESOURCE --env STORAGE_DRIVER=vfs quay.io/podman/stable sh -c "podman run hello-world"

And of course for completeness:

docker run --rm -ti --security-opt seccomp=unconfined --security-opt label=disable --cap-add SYS_ADMIN --cap-add SYS_RESOURCE --env STORAGE_DRIVER=vfs quay.io/podman/stable sh -c 'podman run -ti --security-opt seccomp=unconfined --security-opt label=disable --cap-add SYS_ADMIN --env STORAGE_DRIVER=vfs quay.io/podman/stable sh -c "podman run hello-world"'

[c-goes]

@pothos Indeed it's very exciting, still it's "just" rootful Podman inside rootless Podman, not what the issue asked for.

This error occurs for rootless in rootless, for your example:

> podman run -ti --rm --security-opt seccomp=unconfined --security-opt label=disable --cap-add SYS_ADMIN --env STORAGE_DRIVER=vfs quay.io/podman/stable sh -c "dnf update -y; sudo -u podman --login"
.....
[podman@7515995098f9 ~]$ STORAGE_DRIVER=vfs podman run hello-world
Error: mount `proc` to '/proc': Operation not permitted: OCI runtime permission denied error

[rhatdan]

Yes running rootless podman inside of rootfull podman is definitely possible, and most likely more security then rootfull podman inside of rootfull podman

Rootless podman inside of rootless podman, would be very difficult to make work,because of the multiple user namespaces.

[c-goes]

You are right. I mistook 'privileged' for 'rootfull'.

[ashokponkumar]

This is working now, very exciting:

podman run -ti --security-opt seccomp=unconfined --security-opt label=disable --cap-add SYS_ADMIN --env STORAGE_DRIVER=vfs quay.io/podman/stable sh -c "dnf update -y; podman run hello-world"

Even multiple Matryoshka layers, in the spirit of subhurds from GNU Hurd 😄

podman run -ti --security-opt seccomp=unconfined --security-opt label=disable --cap-add SYS_ADMIN --env STORAGE_DRIVER=vfs quay.io/podman/stable sh -c 'podman run -ti --security-opt seccomp=unconfined --security-opt label=disable --cap-add SYS_ADMIN --env STORAGE_DRIVER=vfs quay.io/podman/stable sh -c "dnf update -y; podman run hello-world"'

When I tried the same, I got the below error:
Error: error opening "/var/lib/shared/vfs-images/images.lock": no such file or directory

I had to sudo the command to avoid this error. Is that expected?

[c-goes]

I had to sudo the command to avoid this error. Is that expected?

The first container can be rootless. Which version of podman do you use?

[ashokponkumar]

I had to sudo the command to avoid this error. Is that expected?

The first container can be rootless. Which version of podman do you use?

I was in 1.6.4. Same results with 1.9.2 too. Will try with version 2 too.

[njam]

For completeness sake adding the command to run podman _rootful in rootless_ using fuse overlay storage (instead of vfs):

podman run -ti --rm \
  --security-opt seccomp=unconfined \
  --security-opt label=disable \
  --cap-add SYS_ADMIN \
  --device /dev/fuse \
  --mount=type=tmpfs,destination=/var/lib/containers \
  quay.io/podman/stable podman run docker.io/library/alpine cat /etc/os-release

Works with the latest podman 2.0.5

[lpil]

@njam I tried your command and got the following error: Error: setrlimit RLIMIT_NOFILE: Operation not permitted: OCI runtime permission denied error.

[rhatdan]

What does your ulimit -a say?

[lpil]

$ ulimit -a
Maximum size of core files created                           (kB, -c) 0
Maximum size of a process’s data segment                     (kB, -d) unlimited
Maximum size of files created by the shell                   (kB, -f) unlimited
Maximum size that may be locked into memory                  (kB, -l) 64
Maximum resident set size                                    (kB, -m) unlimited
Maximum number of open file descriptors                          (-n) 1024
Maximum stack size                                           (kB, -s) 8192
Maximum amount of cpu time in seconds                   (seconds, -t) unlimited
Maximum number of processes available to a single user           (-u) 50570
Maximum amount of virtual memory available to the shell      (kB, -v) unlimited

[rhatdan]

And how do those number compare to the ulimits specified in the container?

[devurandom]

For completeness sake adding the command to run podman _rootful in rootless_ using fuse overlay storage (instead of vfs):

podman run -ti --rm \
  --security-opt seccomp=unconfined \
  --security-opt label=disable \
  --cap-add SYS_ADMIN \
  --device /dev/fuse \
  --mount=type=tmpfs,destination=/var/lib/containers \
  quay.io/podman/stable podman run docker.io/library/alpine cat /etc/os-release

Could the recently described Podman remote client be an alternative solution to this, similar to how people mount docker.sock into their containers?

[ashokponkumar]

For completeness sake adding the command to run podman _rootful in rootless_ using fuse overlay storage (instead of vfs):

podman run -ti --rm \
  --security-opt seccomp=unconfined \
  --security-opt label=disable \
  --cap-add SYS_ADMIN \
  --device /dev/fuse \
  --mount=type=tmpfs,destination=/var/lib/containers \
  quay.io/podman/stable podman run docker.io/library/alpine cat /etc/os-release

Works with the latest podman 2.0.5

Any tips on how this would translate to an equivalent deployment/podspec for the base container?

I have a requirement to run the below statement inside a ubi8 container.

podman run --rm -v dir:/workspace cloudfoundry/cnb:cflinuxfs3 /cnb/lifecycle/detector

(https://github.com/konveyor/move2kube/blob/30cd9c7eae72f888e56a3c49a5195797257996c7/internal/containerizer/cnb/containerruntimeprovider.go#L125).

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[donngchao]

same problem with "podman version 1.6.4"

[zhangguanzhang]

same problem with "podman version 1.6.4"

you could try with the latest version

[rhatdan]

This should only be tested versus the main branch. It will not work on older versions especially one as old as 1.6

[rhatdan]

Since we stated above this works on the main branch I am going to close. If you have other issues please open with a repeater.

[lucasbasquerotto]

Since we stated above this works on the main branch I am going to close. If you have other issues please open with a repeater.

@rhatdan Is there a way to run a rootless podman container inside another one, as stated in the title, without changing capabilities, security options, running in privileged mode, adding a fuse or similar device or changing the storage driver? Just a plain container execution, like:

host$ podman run image-with-podman
container$ podman run hello-world

The examples of commands that I saw were doing one or more of the changes I described above when running the container, not a plain run, but I may have overlook something, or maybe what I'm asking is not possible considering the linux kernel as it is (for now, at least).

[rhatdan]

I would doubt it is possible without modifying the security attributes. One key thing would be the size of the user namespace within the second container would need to be smaller then the primary.

$ podman run --user podman -ti quay.io/podman/stable sh
Trying to pull quay.io/podman/stable:latest...
Getting image source signatures
Copying blob 043d48cdf45e done  
Copying blob c336b26111dc done  
Copying blob d00bed0626fb done  
Copying blob b3227b79d9b6 done  
Copying blob 52cf2739ef0a done  
Copying config 5bc5c15126 done  
Writing manifest to image destination
Storing signatures
sh-5.0$ id
uid=1000(podman) gid=1000(podman) groups=1000(podman)
sh-5.0$ podman info
Error: cannot setup namespace using newuidmap: exit status 1

You could start playing here. The think would be to limit the /etc/subuid and /etc/subgid files inside of the podman/stable container to a smaller range then the rootless user on the hosts range. That might get you further.

[mgoltzsche]

The next thing that stops me from running a nested rootless podman (2.2.1) is that the inner container's file system is always empty which results in the following error:

$ podman run --rm --privileged -u podman:podman --network host quay.io/podman/stable podman run docker.io/alpine echo hello from nested container
Trying to pull docker.io/library/alpine:latest...
Getting image source signatures
Copying blob sha256:801bfaa63ef2094d770c809815b9e2b9c1194728e5e754ef7bc764030e140cea
Copying config sha256:389fef7118515c70fd6c0e0d50bb75669942ea722ccb976507d7b087e54d5a23
Writing manifest to image destination
Storing signatures
Error: executable file `echo` not found in $PATH: No such file or directory: OCI not found

(I also mounted the container to verify that the file system is actually empty.)

Enabling the podman debug log also doesn't show any other error than the usual oom_score_adj permission warning.

Though this also happens when I run podman with sudo and/or let the inner container run as root.
When using docker to run the outer container all of these scenarios work well.

[rhatdan]

Rootful worked for me.

# podman -v
podman version 2.2.1
# podman run --rm --privileged -u podman:podman quay.io/podman/stable podman run --rm docker.io/alpine echo hello from nested container
Trying to pull docker.io/alpine...
Getting image source signatures
Copying blob sha256:801bfaa63ef2094d770c809815b9e2b9c1194728e5e754ef7bc764030e140cea
Copying config sha256:389fef7118515c70fd6c0e0d50bb75669942ea722ccb976507d7b087e54d5a23
Writing manifest to image destination
Storing signatures
hello from nested container

[rhatdan]

Rootless gets the following error:

$ podman run --rm --privileged -u podman:podman quay.io/podman/stable podman run --rm docker.io/alpine echo hello from nested container
Error: cannot setup namespace using newuidmap: exit status 1

[mgoltzsche]

The newuidmap error in rootless mode vanishes if you assign a bigger subuid/subgid range on your host as you pointed out previously, e.g.:

sudo sh -c "echo $(id -un):100000:200000 >> /etc/subuid"
sudo sh -c "echo $(id -gn):100000:200000 >> /etc/subgid"

The file system error in my environment happens due to #8849: The error disappears when I

• mount a host directory as storage directory into the container and
• set --security-opt seccomp=unconfined.

Though, when using docker to run the outer container, this is not necessary since --privileged is sufficient.
I'd expect the same behaviour from podman.