Podman: fusermount3: mount failed: Operation not permitted

[fabrei]

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

I try to run container as rootless, but fusermount3 throws a permission denied error. The container runs without any problems when I execute the following commands as root.

Steps to reproduce the issue:

1. su - <rootless-username>

2. podman pull nginx

3. podmann run nginx

Describe the results you received:
fusermount3 respectively fuse-overlayfs throw a permission denied error.

Describe the results you expected:
I expect that the nginx server is running in a blocking shell as it does as user root.

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:            1.8.0
RemoteAPI Version:  1
Go Version:         go1.12.12
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.12
  podman version: 1.8.0
host:
  BuildahVersion: 1.13.1
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.10-lp151.2.1.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.10, commit: unknown'
  Distribution:
    distribution: '"opensuse-leap"'
    version: "15.1"
  IDMappings:
    gidmap:
    - container_id: 0
      host_id: 2662
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 2662
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  MemFree: 1372459008
  MemTotal: 33723113472
  OCIRuntime:
    name: runc
    package: runc-1.0.0~rc6-lp151.1.2.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc6
      spec: 1.0.1-dev
  SwapFree: 876081152
  SwapTotal: 1076883456
  arch: amd64
  cpus: 8
  eventlogger: file
  hostname: <hostname>
  kernel: 4.12.14-lp151.28.44-default
  os: linux
  rootless: true
  slirp4netns:
    Executable: /usr/bin/slirp4netns
    Package: slirp4netns-0.4.4-lp151.2.6.1.x86_64
    Version: |-
      slirp4netns version 0.4.4
      commit: unknown
      libslirp: 4.2.0
  uptime: 174h 8m 12.41s (Approximately 7.25 days)
registries:
  search:
  - docker.io
store:
  ConfigFile: /home/<rootless-username>/.config/containers/storage.conf
  ContainerStore:
    number: 1
  GraphDriverName: overlay
  GraphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-0.7.6-lp151.5.1.x86_64
      Version: |-
        fusermount3 version: 3.6.1
        fuse-overlayfs: version 0.7.6
        FUSE library version 3.6.1
        using FUSE kernel interface version 7.29
  GraphRoot: /home/<rootless-username>/.local/share/containers/storage
  GraphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  ImageStore:
    number: 1
  RunRoot: /var/tmp/run-2662/containers
  VolumePath: /home/<rootless-username>/.local/share/containers/storage/volumes

Output of podman --log-level debug run nginx

WARN[0000] The cgroups manager is set to systemd but there is no systemd user session available
WARN[0000] For using systemd, you may need to login using an user session
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 2662` (possibly as root)
WARN[0000] Falling back to --cgroup-manager=cgroupfs and --events-backend=file
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/<rootless-username>/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/<rootless-username>/.local/share/containers/storage
DEBU[0000] Using run root /var/tmp/run-2662/containers
DEBU[0000] Using static dir /home/<rootless-username>/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /var/tmp/run-2662/libpod/tmp
DEBU[0000] Using volume path /home/<rootless-username>/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] Not configuring container store
DEBU[0000] Initializing event backend file
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument
DEBU[0000] using runtime "/usr/bin/runc"
WARN[0000] The cgroups manager is set to systemd but there is no systemd user session available
WARN[0000] For using systemd, you may need to login using an user session
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 2662` (possibly as root)
WARN[0000] Falling back to --cgroup-manager=cgroupfs and --events-backend=file
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/<rootless-username>/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/<rootless-username>/.local/share/containers/storage
DEBU[0000] Using run root /var/tmp/run-2662/containers
DEBU[0000] Using static dir /home/<rootless-username>/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /var/tmp/run-2662/libpod/tmp
DEBU[0000] Using volume path /home/<rootless-username>/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] No store required. Not opening container store.
DEBU[0000] Initializing event backend file
DEBU[0000] using runtime "/usr/bin/runc"
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument
DEBU[0000] Failed to add podman to systemd sandbox cgroup: dial unix /run/user/0/bus: connect: permission denied
INFO[0000] running as rootless
WARN[0000] The cgroups manager is set to systemd but there is no systemd user session available
WARN[0000] For using systemd, you may need to login using an user session
WARN[0000] Alternatively, you can enable lingering with: `loginctl enable-linger 2662` (possibly as root)
WARN[0000] Falling back to --cgroup-manager=cgroupfs and --events-backend=file
DEBU[0000] Using conmon: "/usr/bin/conmon"
DEBU[0000] Initializing boltdb state at /home/<rootless-username>/.local/share/containers/storage/libpod/bolt_state.db
DEBU[0000] Using graph driver overlay
DEBU[0000] Using graph root /home/<rootless-username>/.local/share/containers/storage
DEBU[0000] Using run root /var/tmp/run-2662/containers
DEBU[0000] Using static dir /home/<rootless-username>/.local/share/containers/storage/libpod
DEBU[0000] Using tmp dir /var/tmp/run-2662/libpod/tmp
DEBU[0000] Using volume path /home/<rootless-username>/.local/share/containers/storage/volumes
DEBU[0000] Set libpod namespace to ""
DEBU[0000] [graphdriver] trying provided driver "overlay"
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs
DEBU[0000] backingFs=extfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false
DEBU[0000] Initializing event backend file
WARN[0000] Error initializing configured OCI runtime crun: no valid executable found for OCI runtime crun: invalid argument
DEBU[0000] using runtime "/usr/bin/runc"
DEBU[0000] parsed reference into "[overlay@/home/<rootless-username>/.local/share/containers/storage+/var/tmp/run-2662/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]docker.io/library/nginx:latest"
DEBU[0000] parsed reference into "[overlay@/home/<rootless-username>/.local/share/containers/storage+/var/tmp/run-2662/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@ed21b7a8aee9cc677df6d7f38a641fa0e3c05f65592c592c9f28c42b3dd89291"
DEBU[0000] exporting opaque data as blob "sha256:ed21b7a8aee9cc677df6d7f38a641fa0e3c05f65592c592c9f28c42b3dd89291"
DEBU[0000] Using slirp4netns netmode
DEBU[0000] No hostname set; container's hostname will default to runtime default
DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json"
DEBU[0000] created OCI spec and options for new container
DEBU[0000] Allocated lock 10 for container 82f7638f6e62df90eb2ed64f0fafaaf159e45258b49d5491df8a37a1a0baaf99
DEBU[0000] parsed reference into "[overlay@/home/<rootless-username>/.local/share/containers/storage+/var/tmp/run-2662/containers:overlay.mount_program=/usr/bin/fuse-overlayfs]@ed21b7a8aee9cc677df6d7f38a641fa0e3c05f65592c592c9f28c42b3dd89291"
DEBU[0000] exporting opaque data as blob "sha256:ed21b7a8aee9cc677df6d7f38a641fa0e3c05f65592c592c9f28c42b3dd89291"
DEBU[0000] created container "82f7638f6e62df90eb2ed64f0fafaaf159e45258b49d5491df8a37a1a0baaf99"
DEBU[0000] container "82f7638f6e62df90eb2ed64f0fafaaf159e45258b49d5491df8a37a1a0baaf99" has work directory "/home/<rootless-username>/.local/share/containers/storage/overlay-containers/82f7638f6e62df90eb2ed64f0fafaaf159e45258b49d5491df8a37a1a0baaf99/userdata"
DEBU[0000] container "82f7638f6e62df90eb2ed64f0fafaaf159e45258b49d5491df8a37a1a0baaf99" has run directory "/var/tmp/run-2662/containers/overlay-containers/82f7638f6e62df90eb2ed64f0fafaaf159e45258b49d5491df8a37a1a0baaf99/userdata"
DEBU[0000] New container created "82f7638f6e62df90eb2ed64f0fafaaf159e45258b49d5491df8a37a1a0baaf99"
DEBU[0000] container "82f7638f6e62df90eb2ed64f0fafaaf159e45258b49d5491df8a37a1a0baaf99" has CgroupParent "/libpod_parent/libpod-82f7638f6e62df90eb2ed64f0fafaaf159e45258b49d5491df8a37a1a0baaf99"
DEBU[0000] Not attaching to stdin
DEBU[0000] overlay: mount_data=lowerdir=/home/<rootless-username>/.local/share/containers/storage/overlay/l/RNNWVPCNDKT76IAPCDYC32OSQQ:/home/<rootless-username>/.local/share/containers/storage/overlay/l/K2POVXUJZDBCDE4BYX7CISJTWS:/home/<rootless-username>/.local/share/containers/storage/overlay/l/2TMLDBMKS66G3DKMYIOZKMRS6G,upperdir=/home/<rootless-username>/.local/share/containers/storage/overlay/5be0ce71f5b60dd62afcf3589dda1362bf4c6ba061e289105a8c9ddf09da07dc/diff,workdir=/home/<rootless-username>/.local/share/containers/storage/overlay/5be0ce71f5b60dd62afcf3589dda1362bf4c6ba061e289105a8c9ddf09da07dc/work
DEBU[0000] Made network namespace at /var/tmp/run-2662/netns/cni-1a881b8b-9a57-89f9-21cf-cfccdf243531 for container 82f7638f6e62df90eb2ed64f0fafaaf159e45258b49d5491df8a37a1a0baaf99
ERRO[0000] error unmounting /home/<rootless-username>/.local/share/containers/storage/overlay/5be0ce71f5b60dd62afcf3589dda1362bf4c6ba061e289105a8c9ddf09da07dc/merged: invalid argument
DEBU[0000] failed to mount container "82f7638f6e62df90eb2ed64f0fafaaf159e45258b49d5491df8a37a1a0baaf99": error creating overlay mount to /home/<rootless-username>/.local/share/containers/storage/overlay/5be0ce71f5b60dd62afcf3589dda1362bf4c6ba061e289105a8c9ddf09da07dc/merged: using mount program /usr/bin/fuse-overlayfs: fusermount3: mount failed: Operation not permitted
fuse-overlayfs: cannot mount: Operation not permitted
: exit status 1
DEBU[0000] slirp4netns command: /usr/bin/slirp4netns --disable-host-loopback --mtu 65520 --enable-sandbox -c -e 3 -r 4 --netns-type=path /var/tmp/run-2662/netns/cni-1a881b8b-9a57-89f9-21cf-cfccdf243531 tap0
DEBU[0000] Tearing down network namespace at /var/tmp/run-2662/netns/cni-1a881b8b-9a57-89f9-21cf-cfccdf243531 for container 82f7638f6e62df90eb2ed64f0fafaaf159e45258b49d5491df8a37a1a0baaf99
DEBU[0000] Cleaning up container 82f7638f6e62df90eb2ed64f0fafaaf159e45258b49d5491df8a37a1a0baaf99
DEBU[0000] Network is already cleaned up, skipping...
DEBU[0000] Container 82f7638f6e62df90eb2ed64f0fafaaf159e45258b49d5491df8a37a1a0baaf99 storage is already unmounted, skipping...
DEBU[0000] ExitCode msg: "error mounting storage for container 82f7638f6e62df90eb2ed64f0fafaaf159e45258b49d5491df8a37a1a0baaf99: error creating overlay mount to /home/<rootless-username>/.local/share/containers/storage/overlay/5be0ce71f5b60dd62afcf3589dda1362bf4c6ba061e289105a8c9ddf09da07dc/merged: using mount program /usr/bin/fuse-overlayfs: fusermount3: mount failed: operation not permitted\nfuse-overlayfs: cannot mount: operation not permitted\n: exit status 1"
ERRO[0000] error mounting storage for container 82f7638f6e62df90eb2ed64f0fafaaf159e45258b49d5491df8a37a1a0baaf99: error creating overlay mount to /home/<rootless-username>/.local/share/containers/storage/overlay/5be0ce71f5b60dd62afcf3589dda1362bf4c6ba061e289105a8c9ddf09da07dc/merged: using mount program /usr/bin/fuse-overlayfs: fusermount3: mount failed: Operation not permitted
fuse-overlayfs: cannot mount: Operation not permitted
: exit status 1

The permissions look like this

$ ll $(which fusermount3)
-rwsr-xr-x 1 root trusted 31504 27. Aug 2019  /usr/bin/fusermount3
$ ll $(which fuse-overlayfs)
-rwxr-xr-x 1 root root 85768 16. Mär 14:03 /usr/bin/fuse-overlayfs

I already changed the group permission of this two binaries to the <rootless-username> to try if it works, but it did not..

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.8.0-lp151.3.13.1.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):
It runs on a virtual machine with OpenSuse Leap 15.1 and I connect to the machine as root with ssh. I already removed the packages and reinstalled them. And I already removed .local/share/containers/ as well as .config/containers/storage.conf. Nothing worked..

[rhatdan]

@vrothberg @giuseppe PTAL
podman unshare cat /proc/self/uid_map

[fabrei]

$ podman unshare cat /proc/self/uid_map
         0       2662          1
         1     165536      65536
$ podman unshare cat /proc/self/gid_map
         0       2662          1
         1     165536      65536

$ cat /etc/subuid
<user-1>:100000:65536
<user-2>:100000000:100000001
<rootless-username>:165536:65536

$ cat /etc/subgid
<user-1>:100000:65536
<user-2>:100000000:100000001
<rootless-username>:165536:65536

The UID and GID of <rootless-username> is 2662. So this configuration seems correct in my opinion.

I read this instructions for setup: https://github.com/containers/libpod/blob/master/docs/tutorials/rootless_tutorial.md So I expect rootless podman works out of the box. Another friend of mine, installed podman on an ubuntu machine and it worked immediately. So we are both a bit confused..

[giuseppe]

the kernel is too old: kernel: 4.12.14-lp151.28.44-default

FUSE in a user namespace needs at least Linux 4.18

[fabrei]

Well..I did not expect this..Thanks for your help :-)

[j3mdamas]

@giuseppe so this means that rootless podman does not work on CentOS 7 / RHEL 7, since the kernels are maximum 3.10 for all the versions. Is this correct?
That's a bit unfortunate because nowhere in the documentation, either here or here. In fact, in both of them, there are specific references to those OS's, so I was hopeful they would work.
Is there any workaround without having to update to CentOS 8 / RHEL 8?
Thanks!

[j3mdamas]

I see it now on the main readme of fuse-overlayfs Also, please note that, when using fuse-overlayfs from a user namespace (for example, when using rootless podman) a Linux Kernel > v4.18.0 is required. taken from here

I tried to use the VFS driver instead, and it works, but it is rooted (i.e. I can change root-only files with a non-privileged user).

This post from RedHat seems to point to the possibility of making rootless podman work on RHEL 7.6, so I wonder if it's a limitation or there is a work around

[mheon]

I believe the patches required to do this were backported to RHEL 7.7 (or 7.8? I forget which). We are definitely shipping fuse-overlay in RHEL 7, and we would not do this if it were not supported.

[j3mdamas]

Thanks for the quick reply @mheon. I have CentOS 7.6 and I was looking forward to be able to make it work there without upgrading to 7.7, which the post from Redhat really gave me hope.

Do you know how can I find the patched versions of fuse-overlayfs and fuse3-libs? The one I have right now is the following:

Name        : fuse-overlayfs
Arch        : x86_64
Version     : 1.3.0
Release     : 1.el7
Size        : 124 k
Repo        : installed
From repo   : devel_kubic_libcontainers_stable
Summary     : FUSE overlay+shiftfs implementation for rootless containers
URL         : https://github.com/containers/fuse-overlayfs
License     : GPLv3+
Description : FUSE overlay+shiftfs implementation for rootless containers.

Name        : fuse3-libs
Arch        : x86_64
Version     : 3.6.1
Release     : 2.el7
Size        : 270 k
Repo        : installed
From repo   : EPEL
Summary     : File System in Userspace (FUSE) v3 libraries
URL         : http://fuse.sf.net
License     : LGPLv2+
Description : Devel With FUSE it is possible to implement a fully functional filesystem in a
            : userspace program. This package contains the FUSE v3 libraries.

[mheon]

I believe you need the 7.7 kernel at a minimum - this required kernel backports.

[giuseppe]

any reason to stick to such an old version of RHEL? I'd suggest switching to RHEL 8 to take advantage of the latest updates in the container tools.

If you are not able to move to RHEL 8, then you'll need at least the kernel from RHEL 7.8