Podman: container with --userns=keep-id in a pod wont start

[johrstrom]

/kind bug

Description

I'm not sure if this is a bug or feauture, but I'd like to run a container in a pod with the --userns=keep-id flag, but there seems to be some issues in mounting file systems. I've provided a trivial way to reproduce below.

If this is not appropriate let me know, I mean if containers in pods where never meant to maintain the user ids, you can close this. I can get the container to run in the pod without the --userns flag obviously, but one main use cases I have is that I really need to maintain file attributes.

Steps to reproduce the issue:

1. podman pod create --name usens-test

2. podman run --pod=usens-test --rm -it --userns=keep-id centos:7 sleep 10

Describe the results you received:

[jeff@localhost ood-images]$ podman run --pod=usens-test --rm -it --userns=keep-id centos:7 sleep 10
Error: time="2019-09-11T12:16:30-04:00" level=warning msg="exit status 1"
time="2019-09-11T12:16:30-04:00" level=error msg="container_linux.go:346: starting container process caused \"process_linux.go:446: container init caused \\\"rootfs_linux.go:58: mounting \\\\\\\"sysfs\\\\\\\" to rootfs \\\\\\\"/home/jeff/.local/share/containers/storage/overlay/dc1b91f0e19cf3f60d6b54de699fe464e397c66870123ef1620b0a6f1d268a27/merged\\\\\\\" at \\\\\\\"/sys\\\\\\\" caused \\\\\\\"operation not permitted\\\\\\\"\\\"\"\n"
container_linux.go:346: starting container process caused "process_linux.go:446: container init caused \"rootfs_linux.go:58: mounting \\\"sysfs\\\" to rootfs \\\"/home/jeff/.local/share/containers/storage/overlay/dc1b91f0e19cf3f60d6b54de699fe464e397c66870123ef1620b0a6f1d268a27/merged\\\" at \\\"/sys\\\" caused \\\"operation not permitted\\\"\"": OCI runtime error

Describe the results you expected:
I would expect the same behaviour in starting the container with or with out adding it to a pod. That is, regardless of whether I use --pod=foo the --userns=keep-id works as expected.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

podman version 1.5.1

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.7
  podman version: 1.5.1
host:
  BuildahVersion: 1.10.1
  Conmon:
    package: podman-1.5.1-3.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.0, commit: d728afa06cd2df86a27f32a4692c7099a56acc97-dirty'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 286699520
  MemTotal: 8361414656
  OCIRuntime:
    package: runc-1.0.0-93.dev.gitb9b6cc6.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8+dev
      commit: e3b4c1108f7d1bf0d09ab612ea09927d9b59b4e3
      spec: 1.0.1-dev
  SwapFree: 7833382912
  SwapTotal: 8497655808
  arch: amd64
  cpus: 4
  eventlogger: file
  hostname: localhost.localdomain
  kernel: 5.1.18-300.fc30.x86_64
  os: linux
  rootless: true
  uptime: 286h 51m 37.34s (Approximately 11.92 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /home/jeff/.config/containers/storage.conf
  ContainerStore:
    number: 3
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mount_program=/usr/bin/fuse-overlayfs
  GraphRoot: /home/jeff/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 23
  RunRoot: /tmp/1000
  VolumePath: /home/jeff/.local/share/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.5.1-3.fc30.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):

This is fedora 30 in a virtualbox.

[mheon]

I think we're sharing user namespaces within the pod - which means that you'd have to do --userns=keep-id when you first create the pod. Except the option isn't supported at a pod level yet... Hm.

[johrstrom]

OK cool. Again, I don't know if this is a feature or bug or wontdo. Just let me know if you need more info or if there's a workaround. I'm not super good with cgroups, that seems to be the only thing you can pass into the pod command.

[mheon]

We're talking about a major rework of the pod command that will allow most of the options from podman run (which will be set on the infra container, and shared for all other containers in the pod). But I don't really have a perspective on when it'll happen - probably not for a few months.

[github-actions[bot]]

This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days.

[mheon]

We should make a more general tracker issue for adding additional options support to podman pod create

[johrstrom]

Whatever works for you folks. I think I see now this is a feature, but as long as you know this is a feature people would want (I think others would want it, I do anyhow) I'm fine with you tracking in an entirely new ticket. Especially as you start to design, scope and put work into upgrading the pod create api.

[rhatdan]

@giuseppe Could you take a look at this?

[giuseppe]

what about the idea here: https://github.com/containers/libpod/issues/2808#issuecomment-549759625 ?

These are two similar issues, that we can tackle together

[github-actions[bot]]

This issue had no activity for 30 days. In the absence of activity or the "do-not-close" label, the issue will be automatically closed within 7 days.

[duritong]

I still think this should be doable by podman and think the issue should stay re-opened (if work is not tracked somewhere else).

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[vrothberg]

Closing this issue as we can combine it with https://github.com/containers/libpod/issues/2808#issuecomment-549759625.

[orbsfoc]

Is there a work around for this? With 1.9.0 i still get the same error.

[rhatdan]

Not yet, keep discussion in #2808 (comment).

[eyusupov]

Looks like #2808 added network options, but not userns.

[mheon]

I'm going to reopen given that.

[rhatdan]

@giuseppe PTAL

[rhatdan]

@giuseppe Is some of the work you have done with a container joining another containers usernamespace help out here?

[giuseppe]

@giuseppe Is some of the work you have done with a container joining another containers usernamespace help out here?

the reproducer works fine with the last crun. In any case, I don't think there is anything we can do from Podman, the issue must be fixed in the OCI runtime.

[ensc]

There is still a difference of --userns=keep-id between pod and non-pod. E.g.

podman pod create --name usens-test
podman run --pod=usens-test --user=root --rm -it --userns=keep-id busybox nc -l -p 22

fails with bind: Permission denied while a plain

podman run                  --user=root --rm -it --userns=keep-id busybox nc -l -p 22

works.

[podman-2.0.6-1.fc32.x86_64]