Podman: ERRO[0328] 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay"

[edsantiago]

[ No idea what caused it, no reproducer. Filing as placeholder because an issue search finds no hits, and this seems worth putting on the radar.]

Long-running job. System load average was ~6 but I'm not sure why. podman commands were taking O(1 minute) to complete. For grins I ran a ps:

podman ps -a
ERRO[0328] 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay"                                          
error creating libpod runtime: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay": backing file system is unsupported for this graph driver                                                            

Right around that time, I saw the same error in the log of the long-running job; it was running podman logs CONTAINERNAME. I ^C'ed the job, waited for a prompt, can no longer reproduce. But on this system, podman ps -a is still taking O(minute), and something called exe seems to be hogging RAM.

I do not expect any action to be taken on this. I am filing in hopes that someone sees this in the future, finds this issue, and can add actually useful information.

podman-0.9.2-1536796590.gitfacab2ae.fc28.x86_64

[atlaskerr]

I'm also experiencing this problem on Arch Linux with v0.11.1

[giuseppe]

is the job running in a container and/var/lib/containers/storage is not a volume?

It looks like the issue is podman trying to use overlayfs on top of another overlayfs file system and that won't work

[rhatdan]

Is this issue still happening? Can we close this?

[edsantiago]

I haven't seen it since having filed it; @atlaskerr if you're still seeing it, could you respond to @giuseppe's question?

[rhatdan]

Reopen if you can get this issue to happen.

[seandjones92]

@rhatdan I am encountering this issue and can reproduce it on demand.

$ podman ps -a
ERRO[0000] 'overlay' is not supported over extfs at "/var/home/jonessean/.local/share/containers/storage/overlay" 
error creating libpod runtime: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/home/jonessean/.local/share/containers/storage/overlay": backing file system is unsupported for this graph driver

What else can I provide to help track this down?

[TomSweeneyRedHat]

@seandjones92 could you open a new issue and please supply the info requested in the issue report?

[debarshiray]

This might have been a duplicate of #2158.

[seandjones92]

@debarshiray after seeing your comment here I ran the following command

cp ~/.config/containers ~/.config/containers-BAK

After this I can run podman commands again without issue. It rebuilt the ~/.config/containers directory.

Here are the differences:

$ diff -aur containers containers-BAK/
diff -aur containers/libpod.conf containers-BAK/libpod.conf
--- containers/libpod.conf  2019-01-24 10:42:25.986409018 -0700
+++ containers-BAK/libpod.conf  2019-01-01 10:17:51.290997908 -0700
@@ -4,7 +4,6 @@
 conmon_path = ["/usr/libexec/podman/conmon", "/usr/libexec/crio/conmon", "/usr/local/lib/podman/conmon", "/usr/local/libexec/crio/conmon", "/usr/bin/conmon", "/usr/sbin/conmon", "/usr/lib/crio/bin/conmon"]
 conmon_env_vars = ["PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"]
 cgroup_manager = "cgroupfs"
-init_path = "/usr/libexec/podman/catatonit"
 static_dir = "/var/home/jonessean/.local/share/containers/storage/libpod"
 tmp_dir = "/run/user/1000/libpod/tmp"
 max_log_size = -1
diff -aur containers/storage.conf containers-BAK/storage.conf
--- containers/storage.conf 2019-01-24 10:42:25.949408857 -0700
+++ containers-BAK/storage.conf 2019-01-01 10:16:38.205998187 -0700
@@ -1,6 +1,4 @@
-[storage]
-  driver = "overlay"
-  runroot = "/run/user/1000"
-  graphroot = "/var/home/jonessean/.local/share/containers/storage"
-  [storage.options]
-    mount_program = "/usr/bin/fuse-overlayfs"
+RunRoot = "/run/user/1000"
+GraphRoot = "/var/home/jonessean/.local/share/containers/storage"
+GraphDriverName = "overlay"
+GraphDriverOptions = ["overlay.mount_program=/usr/bin/fuse-overlayfs"]

[mizdebsk]

I had the same problem - I had an old config generated by some previous version of libpod in ~/.config/containers/ - removing that directory and letting libpod recreate it fixed the problem for me.

[baude]

is this an issue anymore? can we close?

[mheon]

We might want to add the fuse-overlayfs mount option back to our defaults, the malformed configs are going to keep biting us otherwise

[rhatdan]

I think this is pretty much fixed now. Only old installs have this issue.

[agners]

I run into the same issue after installing fuse-overlayfs and simply changing the driver from vfs to overlay in ~/.config/containers/storage.conf. I guess ideally podman should add some hint how to properly migrate to overlay.

[rhatdan]

@mheon If we moved to my patch that allowed multiple storage drivers this would not be an issue, other then the wasted disk space.

[mheon]

@agners You need to set in your libpod.conf:

[storage.options]
     mount_program = "/usr/bin/fuse-overlayfs"

@rhatdan I think this is a mount options thing - I don't know if your multi-driver patch fixes it?

[rhatdan]

I am talking about people running with vfs then installing fuse-overlayfs, then adding the mount_program and changing driver. They still can not run until they rm -rf ~/.local/share/containers

With multiple drivers, you would not need to do the last step.

[simpleauthority]

FWIW happened to me on Arch after updating kernel. Things go wonky if you don't reboot after kernel replacement.

[srd424]

FWIW, just seen this when the overlay module was unavailable in a VM. Installing it and modprobing it didn't help until I removed /var/run/containers/storage/overlay/overlay-false .. presumably why @simpleauthority needed a reboot..

[mheon]

Oh, that's right, we added caching for that check... Should probably add documentation for that somewhere