Podman: Podman in podman as rootless user?

[dsalaza4]

/kind feature

Description

I am trying to run a podman container inside another podman container (just like docker in docker) in rootless mode.
I was not able to find any helpful information on the web :disappointed:
Actually, I would appreciate if someone can confirm if this is actually possible.

Steps to reproduce the issue:

I am running podman in Ubuntu 19.04.

1. Create the dockerfile for the first container

FROM fedora:latest

RUN dnf install -y podman

1. Create a tmp folder in home directory

mkdir tmp

1. Build and run first container

podman build -t test-podman .
podman run -it -v tmp:/var/lib/containers:rw localhost/test-podman

1. Once inside the container, I try to list podman's images and get this:

[root@9abfedbf98b2 /]# podman images
ERRO[0000] 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay" 
Error: Could not get runtime: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay": backing file system is unsupported for this graph driver

Describe the results you expected:

Podman images command should properly list the images

Additional information you deem important (e.g. issue happens only occasionally):

If i run the container with the --privileged flag (which I'm not sure if gives root permissons or just full permissions over my user, as I run it from my user):

podman run --privileged -it -v tmp:/var/lib/containers:rw localhost/test-podman

I get an error when listing the images, if try to list them again, everything seems to be working fine:

[root@6a1ea78b60ab /]# podman images
ERRO[0000] unable to write system event: "write unixgram @0007d->/run/systemd/journal/socket: sendmsg: no such file or directory" 
REPOSITORY                      TAG      IMAGE ID       CREATED        SIZE
docker.io/library/hello-world   latest   fce289e99eb9   8 months ago   6.14 kB
[root@6a1ea78b60ab /]# podman images
REPOSITORY                      TAG      IMAGE ID       CREATED        SIZE
docker.io/library/hello-world   latest   fce289e99eb9   8 months ago   6.14 kB

Also, if i try to run a container, I get:

[root@6a1ea78b60ab /]# podman run -it hello-world
ERRO[0000] unable to write pod event: "write unixgram @00085->/run/systemd/journal/socket: sendmsg: no such file or directory" 
Error: systemd cgroup flag passed, but systemd support for managing cgroups is not available: OCI runtime error

Output of podman version:

Ubuntu 19.04:

podman version 1.5.0

Fedora 30 (fedora:latest):

podman version 1.5.1

Output of podman info --debug:

Ubuntu 19.04

debug:
  compiler: gc
  git commit: ""
  go version: go1.10.4
  podman version: 1.5.0
host:
  BuildahVersion: 1.10.1
  Conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.0, commit: unknown'
  Distribution:
    distribution: ubuntu
    version: "19.04"
  MemFree: 370855936
  MemTotal: 8097402880
  OCIRuntime:
    package: 'containerd.io: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8
      commit: 425e105d5a03fabd737a126ad93d62a9eeede87f
      spec: 1.0.1-dev
  SwapFree: 906227712
  SwapTotal: 1023406080
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: dsalazar-ThinkPad-X1C-5th-W10DG
  kernel: 5.0.0-25-generic
  os: linux
  rootless: true
  uptime: 1h 57m 55.84s (Approximately 0.04 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
store:
  ConfigFile: /home/dsalazar/.config/containers/storage.conf
  ContainerStore:
    number: 19
  GraphDriverName: vfs
  GraphOptions: null
  GraphRoot: /home/dsalazar/.local/share/containers/storage
  GraphStatus: {}
  ImageStore:
    number: 7
  RunRoot: /run/user/1000
  VolumePath: /home/dsalazar/.local/share/containers/storage/volumes

Fedora 30

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.7
  podman version: 1.5.1
host:
  BuildahVersion: 1.10.1
  Conmon:
    package: podman-1.5.1-3.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.0, commit: d728afa06cd2df86a27f32a4692c7099a56acc97-dirty'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 323915776
  MemTotal: 8097402880
  OCIRuntime:
    package: runc-1.0.0-93.dev.gitb9b6cc6.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: e3b4c1108f7d1bf0d09ab612ea09927d9b59b4e3
      spec: 1.0.1-dev
  SwapFree: 906227712
  SwapTotal: 1023406080
  arch: amd64
  cpus: 4
  eventlogger: journald
  hostname: 6a1ea78b60ab
  kernel: 5.0.0-25-generic
  os: linux
  rootless: false
  uptime: 1h 57m 1.09s (Approximately 0.04 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 2
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mountopt=nodev,metacopy=on
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 1
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

Ubuntu 19.04:

podman/disco,now 1.5.0-2~ubuntu19.04~ppa8 amd64 [installed]

Fedora 30:

podman-1.5.1-3.fc30.x86_64

[mheon]

I think you'll have to use the VFS storage driver inside the second container; I don't think fuse-overlay enjoys being layered.

Also, you'll probably only have the use of 1 UID/GID in the inner container, and you'll have to use the ignore chown errors storage option to make that possible. It may be feasible to get newuidmap/newgidmap running inside the rootless container with appropriate mappings to make the inner rootless container use the same UIDs/GIDs as the other container, but I don't know if that's ever been tried before.

Seccomp is definitely going to prevent you from doing this, so you'll have to disable it on the outer container with --security-opt seccomp=unconfined

[mheon]

In short, I think this has been tried, and it can probably be made to work - but there will be serious limitations, I think.

[giuseppe]

this is not a supported use case (yet), but if you want to play with it:

• --privileged is needed. It doesn't give any root permission. Rootless podman never has that privileges (except for setting up the user namespace using newuidmap/newgidmap, but that must be configured by root anyway).
  -- you will either need to use --cgroup-manager=cgroupfs or try to bind mount /var/run from the host.
  -- you'll need to bind mount the storage directory. Overlay on overlay is known to not work

[dsalaza4]

I've been playing around with the flags @giuseppe suggested, but wasn't able to get it to work. I'm leaving this issue open.

[smekkley]

Running root podman in a rootless podman is not possible at the moment, it's also not possible to run root podman in a root container without --privileged, due to the fact that root podman requires to create tmpfs (if using vfs) and to write to /sys/fs/cgroup.

Podman can be run in podman by the following, but there is no benefit to do this, because running privileged with root podman is almost equivalent of host root.
# sudo podman run --privileged -it fedora /bin/bash
# podman --cgroup-manager=cgroupfs run --storage-driver=vfs --net=host -it alpine /bin/ash
For less privileged way with buildah, you can take a look at. https://developers.redhat.com/blog/2019/04/04/build-and-run-buildah-inside-a-podman-container/
It's possible because buildah supports chroot mode.

[rhatdan]

I think the only access you need is SYS_ADMIN capability, rather then --privileged. But I have not tried. And SYS_ADMIN is one of the most powerful capabilities.

Bottom line podman command will attempt to do some mount commands and other functions that are not allowed without CAP_SYS_ADMIN

[smekkley]

I tried with CAP_SYS_ADMIN but it failed to write to /sys/fs/cgroup.
Buildah chroot mode doesn't experience this issue, because it doesn't create a container but only chroot, thus root access within the container is necessary.

I imagine podman in podman use case is only for testing, though it'll be nice to have zero impacts on the host OS.
If the buildah-run option supported the most options of podman-run, I'd say it's also good enough. For example, buildah-run doesn't support passing variable.

Of course if it's possible, running the rootless podman inside rootless podman is the best, but I think we are still far from it considering the fact that we can't run rootless podman in root podman yet without privileged option.

[rhatdan]

Could you give me some examples of what you would like to run within a root container?
Now as I think of it, we could theoretically run rooless podman within a root running podman container. If it was configured correctly.
@giuseppe WDYT?

[smekkley]

I'm trying to run /bin/ash from alpine, like the example above.
Regardless of what this message means, I think rootless podman requires user namespace support, which normal root podman doesn't provide, unless it's privileged.

$ podman ps
cannot clone: Operation not permitted
Error: could not get runtime: cannot re-exec process

In summary, I think the current state is...

1. privileged root podman = supports root and rootless podman inside.
2. non privileged root podman = only chroot buildah is supported.
3. privileged/non-privileged rootless podman = only chroot buildah is supported.

[rhatdan]

I think that we can convince rootless podman to run within a non privileged root podman, but their needs to be a bit of configuration to make sure it uses fuse-overlayfs and has proper settings on its cgroup-manager and others.

When you run rootless podman as non root user, then you get less access then you get with a rootfull non privileged container.

@giuseppe Do you agree, this should be possible?

[smekkley]

Yes, I think ability to run rootless podman within a root non-privileged container would be great. It would also give an option to run podman in other container engines without being privileged. It would also probably mean, with buildah we no longer have to use chroot mode.

[axelthimm]

Could you give me some examples of what you would like to run within a root container?
Now as I think of it, we could theoretically run rooless podman within a root running podman container. If it was configured correctly.
@giuseppe WDYT?

A real world example is when you want to integrate image creation in CI/CD workflows which are typically governed by container runners these days. For example when trying to use podman/buildah within the context of a gitlab runner the CI scripts run in a container that will use buildah to construct a new container. For security reasons the outer container should be rootless.

I'm currently trying to implement this and it looks like I cannot get any setup with an external rootless container to work. :/

[smekkley]

Building an image with buildah in a rootless container is possible by following the link I posted.

$ podman run --rm -it fedora /bin/bash
# yum install -y buildah
# buildah --storage-driver=vfs from --name myalpine alpine
# buildah --storage-driver=vfs run --isolation=chroot myalpine /bin/ash

But this buildah has to be run as root in the rootless container.

Ability to run rootless podman in a rootless podman container is very slick, if that's what you are trying to achieve. @axelthimm

[axelthimm]

Thanks @smekkley, that seems to work nicely! I was following the link that you posted that was pulling buildah and ended with

sh-5.0# buildah run --isolation=chroot myalpine ls /
error running subprocess: error bind mounting /sys from host into mount namespace: no such file or directory
                              ERRO exit status 1                                

I'm not sure how this differs from the recipe you sent in the previous comment, but I'll try to model the gitlab runner accordingly.

Of course, in the long run having rootless-in-rootless would be the perfect solution, but already having the outer container run rootless is a huge win and at least satisfies the use case I described.

Thanks!

[rhatdan]

I would use quay.io/buildah/stable for your buildah inside of a rootless podman container.
It will work much better then using vfs and is supported by the buidlah team.

[smekkley]

Probably you are missing -v /dev/fuse:/dev/fuse and option or configuration to use fuse-overlayfs, or fuse might be too old on your system. I put vfs as an example because it's more portable. With vfs, you don't have to expose anything from the host, it'll probably succeed as long as you use the right buildah version.

[rhatdan]

BTW, I was doing some experimenting with podman in a locked down Podman container, on Friday. Made some progress, and sent information to podman.io mailing list. Will continue working and diagnosing what is going on next week.

[smekkley]

Any update on this? It'd be even great if podman was runnable in --add-cap=ALL environment, because it's far better than --privileged or mouting docker.socket in case of docker. I'm happy to test if it's in some branch.

[mheon]

I think we might work with --cap-add=ALL and --security-opt seccomp=unconfined? The issue is getting things working without either of those.

[smekkley]

Was supporting --isolation=chroot option an option in case of podman? What's nice about this option in buildah is that you are sure you only need root inside a container because it's just a chroot wrapper against the extracted container image directory.
In the future, it'd be nice to run rootless podman inside podman in a such way, but I guess it's challenging.

[axelthimm]

I tried podman in rootless podman, but it doesn't work, yet.

[rhatdan]

Yes, we are continuing to work on this.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

This is all tied to the containers.conf PR.

[vrothberg]

Closing in favor of https://github.com/containers/libpod/issues/4131. They're all tied together, so we can consolidate the issues a bit.