Podman: Ubuntu LXC guest atop Fedora LXC host :: Getting "device not found, try 'modprobe fuse' first fuse-overlayfs: cannot mount: No such file or directory"...

[nmvega]

Hello Friends:

Preamble: The following described problem doesn't happen when I run the command shown as root, but alas I don't want to run it as root. =:)

So I have an Ubuntu 20.04 LTS LXC GUEST container atop a bare-metal Fedora-30 HOST.

Note. I usually use Fedora-XX as my LXC GUEST containers, but can't in this case because Ubuntu is required to run certain things.

So on the Fedora-30 HOST, I have the fuse driver loaded:

user@fedora$ lsmod | grep fuse
fuse 139264  1

And the Ubuntu 20.04 LTS GUEST sees it, too:

user@ubuntu$ lsmod | grep -i fuse
fuse 139264  1

However, I'm getting the following error which I don't know how to correct or debug:

user@ubuntu$ podman build -t registry.example.com/myGroup/myProject/hello:1.0 .
STEP 1: FROM python:3
Getting image source signatures
Copying blob ee776f0e36af done  
Copying blob e9afc4f90ab0 done  
Copying blob af14b6c2f878 done  
Copying blob 5573c4b30949 done  
Copying blob 11a88e764313 done  
Copying blob 989e6b19a265 done  
Copying blob 513c90a1afc3 done  
Copying blob df9b9e95bdb9 done  
Copying blob 86c9edb54464 done  
Copying config 7f5b6ccd03 done  
Writing manifest to image destination
Storing signatures
ERRO[0053] error unmounting /home/user/.local/share/containers/storage/overlay/99a9be71fbc8d0a188ad13e08d6dbd5a8fced07dd8f328a66bd90eb63610a495/merged:
invalid argument 
Error: error mounting new container: error mounting build container "63d55aedc44bb0dc0910481fe5f567caece7e576ca66e3cf9e4fca2a70b951ab":
error creating overlay mount to /home/user/.local/share/containers/storage/overlay/99a9be71fbc8d0a188ad13e08d6dbd5a8fced07dd8f328a66bd90eb63610a495/merged:
using mount program /bin/fuse-overlayfs: fuse:
device not found, try 'modprobe fuse' first
fuse-overlayfs: cannot mount: No such file or directory
: exit status 1

user@ubuntu$

I included podman info output below. Any ideas?

Thank you in advance! =:)

user@ubuntu$ podman info

host:
  arch: amd64
  buildahVersion: 1.15.0
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.0.18, commit: '
  cpus: 12
  distribution:
    distribution: ubuntu
    version: "20.04"
  eventLogger: file
  hostname: vps11
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 5000
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 5000
      size: 1
    - container_id: 1
      host_id: 165536
      size: 65536
  kernel: 5.6.13-100.fc30.x86_64
  linkmode: dynamic
  memFree: 30670012416
  memTotal: 67376635904
  ociRuntime:
    name: runc
    package: 'containerd.io: /usr/bin/runc'
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10
      commit: dc9208a3303feef5b3839f4323d9beb36df0a9dd
      spec: 1.0.1-dev
  os: linux
  remoteSocket:
    path: /run/user/5000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /bin/slirp4netns
    package: Unknown
    version: |-
      slirp4netns version 1.0.0
      commit: unknown
      libslirp: 4.2.0
  swapFree: 0
  swapTotal: 0
  uptime: 158h 8m 52.93s (Approximately 6.58 days)
registries:
  search:
  - docker.io
  - registry.fedoraproject.org
  - registry.access.redhat.com
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /bin/fuse-overlayfs
      Package: Unknown
      Version: |-
        fusermount3 version: 3.9.0
        fuse-overlayfs: version 0.7.6
        FUSE library version 3.9.0
        using FUSE kernel interface version 7.31
  graphRoot: /home/user/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: extfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 1
  runRoot: /run/user/5000/containers
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 0
  BuiltTime: Thu Jan  1 00:00:00 1970
  GitCommit: ""
  GoVersion: go1.14.2
  OsArch: linux/amd64
  Version: 2.0.2

[mheon]

Do you have a /dev/fuse inside the LXC container that is attempting to run Podman, and does the user inside the container have access to the device?

[nmvega]

Do you have a /dev/fuse inside the LXC container that is attempting to run Podman, and does the user inside the container have access to the device?

@mheon Wow that was fast. Let me check now. Note that I updated the question to mention that running it as root works, but that was just to try it.

[nmvega]

Do you have a /dev/fuse inside the LXC container that is attempting to run Podman, and does the user inside the container have access to the device?

@mheon As you suspected, there is no /dev/fuse:

user@ubuntu$ sudo ls -al /dev/fuse
ls: cannot access '/dev/fuse': No such file or directory

I did reboot the container for good measure previously.

[mheon]

You'll need to forward that from the host into the LXC container. I'm not really that familiar with LXC configurations, so I'm afraid I can't assist there.

[nmvega]

You'll need to forward that from the host into the LXC container. I'm not really that familiar with LXC configurations, so I'm afraid I can't assist there.

No worries. You already helped by pointing out the missing /dev/fuse. I'm searching around on that now, and perhaps community friends might chime on that part. Thank you again.

[nmvega]

You'll need to forward that from the host into the LXC container. I'm not really that familiar with LXC configurations, so I'm afraid I can't assist there.

No worries. You already helped by pointing out the missing /dev/fuse. I'm searching around on that now, and perhaps community friends might chime on that part. Thank you again.

@mheon Well I found this issue, and rummaging around it, found the following statement, which I appended to the Ubuntu GUEST LXC config file, rebooted that container, and it works (or seems to):

Added to config:

lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,rw,uid=165536,gid=165536 0 0

/dev/fuse now exists:

user@ubuntu$ ls -al /dev/fuse
crw-rw-rw- 1 root root 10, 229 Jul 14 01:03 /dev/fuse

The configuration entry seems straight forward enough, but I'll have to study it to see if it's optimal. LoL
Thanks again.

[nmvega]

I'll close this issue with a summary of the fix:

On the Fedora LXC HOST:

root@fedora# dnf install fuse-overlayfs
root@fedora# modprobe fuse
root@fedora# vi /var/lib/lxc/<Ubuntu-container-name>/config # On the Fedora LXC Host.

Add this line to the config file and save it:

lxc.mount.entry = /dev/fuse dev/fuse none bind,create=file,rw,uid=165536,gid=165536 0 0

Install this package on the Ubuntu LXC GUEST:

root@ubuntu# apt-get install fuse-overlayfs

and finally, reboot your Ubuntu container.

I hope this helps others.