Podman: RFE: ability for rootless podman container to transparently act as the user running it, incl. managing their rootless podman

[jistr]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind feature

Use case:

On a very slim system like Fedora Silverblue, i'd like to use a rootless podman container as my development environment shell (to have additional tools which are not available on Silverblue host, and bind mount my $HOME or a subdirectory of it). This shell container should be able to manage my rootless podman, so that i can work on projects that require running containers.

Describe the results you received:

The first part -- mapping the host and container user transparently (e.g. UID 1000 on host being UID 1000 in container) -- is already achievable via using custom --uidmap / --gidmap parameters. The approach is nicely described in a commit message in Toolbox project.

With the uidmap/gidmap i can be user 1000 both in the container and on the host, so mounting $HOME into the container works transparently. With bind mounting $XDG_RUNTIME_DIR i can query running podman containers using podman ps and i see them, including the container i'm running the command from. However, once i try to run podman images, i get a permission error:

Error: Could not get runtime: mkdir /home/jistr/.local/share/containers/storage/overlay/l: permission denied

Describe the results you expected:

I'd like to be able to run podman images from the container and get a list of images in my rootless podman on the host, while keeping the requirement that i act as the same UID inside and outside the container, and i'm not using podman as root.

I guess we'd have to persuade the rootless podman in the container to not try to enter another user namespace, because we are already inside the namespace meant for rootless containers of the given user?

If what i'm asking for is impossible or very impractical, is there an alternative usage pattern for achieving what's described in "use case" section above?

Additional information you deem important (e.g. issue happens only occasionally):

I've been experimenting with running the container like this for now:

UID_MAX=65536
UID_HIGHER_START=$((UID+1))
UID_HIGHER_COUNT=$((UID_MAX-UID))

podman run \
    -e XDG_RUNTIME_DIR \
    -v $XDG_RUNTIME_DIR:$XDG_RUNTIME_DIR \
    \
    -v /etc/passwd:/etc/passwd:ro \
    -v /etc/group:/etc/group:ro \
    -v /etc/nsswitch.conf:/etc/nsswitch.conf:ro \
    -v /etc/security/limits.d:/etc/security/limits.d:ro \
    -v /usr/lib/passwd:/usr/lib/passwd \
    -v /usr/lib/group:/usr/lib/group \
    --uidmap $UID:0:1 \
    --uidmap 0:1:$UID \
    --uidmap $UID_HIGHER_START:$UID_HIGHER_START:$UID_HIGHER_COUNT \
    --gidmap $UID:0:1 \
    --gidmap 0:1:$UID \
    --gidmap $UID_HIGHER_START:$UID_HIGHER_START:$UID_HIGHER_COUNT \
    --user $UID:$UID \
    \
    -v $HOME:$HOME \
    -v /var/home:/var/home \
    --privileged \
    --rm \
    --net host \
    --pid host \
    --security-opt label=disable \
    --volume /var/run/dbus/system_bus_socket:/var/run/dbus/system_bus_socket \
    --volume /dev/bus:/dev/bus \
    --volume /dev/dri:/dev/dri \
    --volume /dev/fuse:/dev/fuse \
    -ti \
    localhost/testimage \
    bash

Output of podman version:

Same on host and in the container:

Version:            1.2.0
RemoteAPI Version:  1
Go Version:         go1.11.5
OS/Arch:            linux/amd64

Output of podman info --debug:

The output below is from the host. In the container the command crashes with the same error as when running podman images.

debug:
  compiler: gc
  git commit: ""
  go version: go1.11.5
  podman version: 1.2.0
host:
  BuildahVersion: 1.7.2
  Conmon:
    package: podman-1.2.0-2.git3bd528e.fc29.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: d88bb0e63cb70f9787a8e410716924f380af361f'
  Distribution:
    distribution: fedora
    version: "29"
  MemFree: 14722519040
  MemTotal: 25196142592
  OCIRuntime:
    package: runc-1.0.0-85.dev.gitdd22a84.fc29.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc6+dev
      commit: 1d3f73d4086533a858613bc4b6af2b5e882f4730
      spec: 1.0.1-dev
  SwapFree: 8589930496
  SwapTotal: 8589930496
  arch: amd64
  cpus: 4
  hostname: split
  kernel: 5.0.7-200.fc29.x86_64
  os: linux
  rootless: true
  uptime: 19h 51m 39.87s (Approximately 0.79 days)
insecure registries:
  registries: []
registries:
  registries:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/jistr/.config/containers/storage.conf
  ContainerStore:
    number: 3
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /home/jistr/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 7
  RunRoot: /run/user/1000
  VolumePath: /home/jistr/.local/share/containers/storage/volumes

Additional environment details (AWS, VirtualBox, physical, etc.):

Fedora Silverblue workstation, physical

[rhatdan]

This is what toolbox does?

[mheon]

The container-in-container functionality is very difficult to do properly - I'd recommend looking into what @baude and @jwhonce have been doing with podman-remote, which will simplify it greatly by not requiring any mounts beyond a single socket.

For the rest, I think we've been discussing an easy way to match the UID running rootless podman to the UID in the container (not mapping it to root). Toolbox does it, but we've seen a lot of interest in it, and it's complex to set up now.

[jistr]

@rhatdan I just tried, and it seems toolbox suffers from exactly the same problem:

@ ~
$ ./toolbox enter

@ ~
$ podman images
bash: podman: command not found...

@ ~
$ sudo dnf -qy install podman

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

Failed to set locale, defaulting to C

@ ~
$ podman images
WARN[0000] using rootless single mapping into the namespace. This might break some images. Check /etc/subuid and /etc/subgid for adding subids 
Error: Could not get runtime: mkdir /home/jistr/.local/share/containers/storage/overlay/l: permission denied

@mheon Indeed i guess podman-remote could be a way to go. It seems to be very recent work and i can't google much info about it, but if i'm scanning correctly and the CLI interface is meant to be identical to podman, and alias podman=podman-remote inside a container would work to unify how to use the CLI, then that would be very helpful. (Otherwise any automation aimed to work with both podman and podman-remote would have to account for differences between the CLIs.) Btw for a full dev env workflow (either via toolbox or custom-built containers), having a similar buildah-remote would be a very useful feature too i think... :).

Thanks for the replies :).

[rhatdan]

I would like you to simplify what you are asking for? Do you want to run podman commands inside of a container but continue to run them in rootless mode.

I would figure this would mostly work in toolbox, no idea why the sudu command fails.

@debarshiray Care to take a look.

Podman-remote might work but you need to leak the varlink socket from the homedir into the container, otherwise the container will fail to run within the container. You need a mechanism to contact varlink outside the container to make it work.

Running rootless podman inside of a rootless podman container could be difficult. You should be able to run it as root inside of the container.

Rootless would be an issue, because it would need a subset of /etc/subuid and /etc/subuid defined inside of the container based on the your /etc/subuid outside of the container, and the kernel might not allow it.

[jistr]

I would like you to simplify what you are asking for? Do you want to run podman commands inside of a container but continue to run them in rootless mode.

Yes exactly. A rootless container spawning more rootless containers on the host. All containers running under the same host user account in rootless mode.

I would figure this would mostly work in toolbox, no idea why the sudu command fails.

@debarshiray Care to take a look.

Podman-remote might work but you need to leak the varlink socket from the homedir into the container, otherwise the container will fail to run within the container. You need a mechanism to contact varlink outside the container to make it work.

Sharing the socket would be fine. I'd only be worried about potential CLI incompatibilities between podman and podman-remote. It would be really good if automation designed to work with podman would work as-is with podman-remote too.

Running rootless podman inside of a rootless podman container could be difficult. You should be able to run it as root inside of the container.

For my use case (similar to toolbox) this is not very applicable. But for now i'm working around the issue vice versa to what you suggested above. On the host i start a container with sudo podman run ... --user 1000:1000 ..., this puts me in a context where i get my extra apps from the container, my $UID is preserved but no UID/GID re-mapping has taken place yet. From there i can use rootless podman normally it seems.

Rootless would be an issue, because it would need a subset of /etc/subuid and /etc/subuid defined inside of the container based on the your /etc/subuid outside of the container, and the kernel might not allow it.

Yea i thought it could be that... The rootless podman on the host is doing subuid/subgid mapping, and the rootless podman in the container is probably trying to do that again and failing. I tried hacking it by putting jistr:0:65536 subuid/subgid entries into the container in a feeble attempt to prevent the 2nd level re-mapping, but that didn't work either :).

[rhatdan]

The primary goal of podman-remote is to emulate all of the commands of podman for a MAC or Windows box, but there are several commands we will not support since they might not make sense for a remote system. podman-remote varlink for example. But for most of the commands you should be fine. We need to make sure podman-remote->podman varlink is a stable API, but we have not fully released it yet.

@baude What do you think of this use case. How could we inject the socket to use podman varlink into a container.

[jistr]

no idea why the sudu command fails.

@debarshiray Care to take a look.

One more thing here to prevent a misunderstanding and needless debugging. The sudo dnf -qy install podman command did work fine, only podman images after it failed. (The message Failed to set locale, defaulting to C was non-fatal. I passed -q into the dnf command to shorten the output for pasting, so it just didn't print the usual installation messages, but it did install podman fine.)

[mheon]

I think we'd generally recommend a -v to volume-mount in the socket.

[baude]

well it depends on how you want to pull this all together ... varlink can do tcp sockets too. im lacking a clear picture in my head how this works. i have certainly done this though

[rhatdan]

I don't think we want to use TCP from host to host. I don't think bridge varlink would work, although I have no idea how this works anyways. This would seem to be more of a socket activated podman command.

[baude]

draw me a picture?

[mheon]

@baude Container, rootless, wants to run more Podman commands, also rootless, on same host that container runs on

[rhatdan]

@baude You are in silverblue, so you are not encouraged to install packages on the host. So the desktop team invented toolbox for this. It is a command that wraps podman to launch a privileged container via podman with your UID==UID inside the container. From your perspective it looks like a normal desktop session. You can do sudo dnf install etc. If you try to execute podman inside this environment it blows up.
User wants to execute podman containers while inside of a toolbox container (Inside podman).

Rootless podman will not really work here, since your User namespaces start to conflict. If however you had a podman-remote command that was able to communicate with the podman varlink in the users homedir, you could feel like you are running regular podman commands.

[rhatdan]

To make this work we would need to have "podman varlink/systemd" listening somewhere inside of the container so that podman-remote could find it.

@haraldh FYI

[debarshiray]

@jistr First of all, you are better off just using Toolbox instead of trying to put together all the podman bits by hand. :) It targets the exact problem that you are trying to solve, and is nothing but a wrapper around podman.

If you look closely at Toolbox, you'll see that you can use the toolbox command from inside a toolbox container. So you can do:

host $  toolbox enter --release 30
container $  toolbox enter --release 29
container $  toolbox enter --release 30
...

The toolbox containers have the toolbox command bind mounted at /usr/bin/toolbox. When running inside a container, it tunnels all the buildah and podman invocations to the host and wires up the file descriptors to pretend as if everything executed right there inside the container. This tunnelling works over the session's D-Bus socket and spawns a small daemon to handle the invocation on the host. Coincidentally, the Flatpak project already had the infrastructure to do this (bus name: org.freedesktop.Flatpak, interface: org.freedesktop.Flatpak.Development) fronted by the flatpak-spawn --host command, so Toolbox just piggy backs on that.

You'd need toolbox Git master for this to work best. Earlier versions didn't bind mount the command inside the container, so you used to have to place a copy of the command yourself inside the container.

So, you can use a similar principle to make rootless podman itself work inside a container. Either by using podman-remote or flatpak-spawn --host podman run ....

For Fedora Silverblue, we expect developers will spend a good chunk of their time inside a container, and we intend to provide pre-canned aliases for some common commands to reduce the cognitive load of using such an OS. I can imagine aliases for things like flatpak, rpm-ostree, podman, buildah, etc..

Not everything is in place today, because we need to work out some details, but we'll get there soon. :)

[rhatdan]

Nice work @debarshiray

[jistr]

@debarshiray Thanks, the trick with flatpak-spawn looks useful :).

Re just using toolbox instead -- i could perhaps do that for my main shell container (i didn't have a smooth experience with toolbox previously but i can give it another shot). I will have additional use cases where i'd like to sandbox the apps inside a bit more, somewhat like Flatpak does, or on the other hand perhaps give them additional bind mounts. I suspect i will also prefer pre-built container images (in one case i have to compile from source), rather than "here's an empty Fedora, install what you need" approach (but i think it's good approach for toolbox, as it makes it easily usable for wider community). So i suspect i might roll my own approach for particular use cases, but i would use much the same techniques as toolbox to make the app inside run transparently or semi-transparently as the user (--uidmap etc.). I'm still just exploring all this... :)

Thanks all :)

[rhatdan]

podman run --userns=keep-id
Solves this also. Reopen if you disagree.

[petersenna]

@rhatdan Are the preferred methods for calling podman from a toolbox still flatpak-spawn --host podman or podman-remote ?

[rhatdan]

@petersenna I have no idea @debarshiray Might?

[debarshiray]

@harrymichal is already looking at providing a podman wrapper inside toolbox containers that transparently forward the invocation to the host.

At the moment, this will most likely use the Flatpak infrastructure (bus name: org.freedesktop.Flatpak, interface: org.freedesktop.Flatpak.Development). It is a tiny little daemon that already exists and doesn't pull in too many dependencies, so it's a good fit to get going. However, one can also roll their own thing. It sounds like magic, but it's pretty simple, really.

I haven't kept up with the latest podman-remote developments, but it's basically the same idea really. There's nothing forcing you to use D-Bus or Varlink or anything as long as you can somehow set up a communication channel from the container to the host and back.