Toolbox: `do_rm` fails to remove directories in upper layer (if whiteouts present?)

[mildred]

When I try to run a dnf command, I get the following error (example):

mildred@toolbox$ sudo dnf install git-subtree
Copr repo for stack owned by petersen                                           3.9 kB/s | 3.5 kB     00:00    
Fedora Modular 30 - x86_64                                                       48 kB/s |  24 kB     00:00    
Fedora Modular 30 - x86_64 - Updates                                            3.5 kB/s |  23 kB     00:06    
Fedora Modular 30 - x86_64 - Updates                                            216 kB/s | 310 kB     00:01    
Cannot rename directory "/var/cache/dnf/updates-modular-783da5de2e38c644/tmpdir.hYGi13/repodata" to "/var/cache/dnf/updates-modular-783da5de2e38c644/repodata": Directory not empty
Error: Cannot rename directory "/var/cache/dnf/updates-modular-783da5de2e38c644/tmpdir.hYGi13/repodata" to "/var/cache/dnf/updates-modular-783da5de2e38c644/repodata": Directory not empty

If I try to manually clean the cache (/var/cache/dnf), it fails with:

mildred@toolbox$ sudo rm -rf /var/cache/dnf/*
rm: cannot remove '/var/cache/dnf/updates-modular-783da5de2e38c644/repodata': Directory not empty

I believe this is a problem with podman filesystem, because if I inspect what's going on there:

mildred@toolbox$ tree /var/cache/dnf
/var/cache/dnf
└── updates-modular-783da5de2e38c644
    └── repodata
        └── 1a8871693a82cbcad89b11569f09d1a920fc545da3fbb350a732928803198b08-modules.yaml.gz

2 directories, 1 file

I can remove the *-modules.yaml.gz file all right, but it then reappears:

mildred@toolbox$ sudo rm /var/cache/dnf/updates-modular-783da5de2e38c644/repodata/1a8871693a82cbcad89b11569f09d1a920fc545da3fbb350a732928803198b08-modules.yaml.gz && tree /var/cache/dnf
/var/cache/dnf
└── updates-modular-783da5de2e38c644
    └── repodata
        └── 1a8871693a82cbcad89b11569f09d1a920fc545da3fbb350a732928803198b08-modules.yaml.gz

2 directories, 1 file

Stopping (with podman stop fedora-toolbox-mildred-30), making sure that there is no podman process left execing in the toolbox (ps faux | grep toolbox) and restarting the toolbox does not work. The file still reappears.

Now, outside the toolbox, if I search for the same file, I can find it in /home/mildred/.local/share/containers/storage/overlay/31fdb94449dc448a29016d5ac6b018251a1399a5946065a25171438f0d170dca/diff/var/cache/dnf/updates-modular-783da5de2e38c644/repodata/1a8871693a82cbcad89b11569f09d1a920fc545da3fbb350a732928803198b08-modules.yaml.gz. If I stop the toolbox container and remove that file, it solves the problem:

```mildred@faile$ rm /home/mildred/.local/share/containers/storage/overlay/31fdb94449dc448a29016d5ac6b018251a1399a5946065a25171438f0d170dca/diff/var/cache/dnf/updates-modular-783da5de2e38c644/repodata/1a8871693a82cbcad89b11569f09d1a920fc545da3fbb350a732928803198b08-modules.yaml.gz

mildred@faile$ toolbox enter
mildred@toolbox$ sudo rm -rf /var/cache/dnf/*
mildred@toolbox$ tree /var/cache/dnf
/var/cache/dnf

0 directories, 0 files
```

and dnf install works

[juhp]

Is it reproducible?

[mildred]

When the toolbox container is in that state, yes, always. But from a clean toolbox, not easily (although it's the second time I have thus issue I believe. The first time I committed the container and recreated a new toolbox from the saved image)

Le 2 juillet 2019 18:15:03 GMT+02:00, Jens Petersen [e-mail] a écrit :

Is it reproducible?

--
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/debarshiray/toolbox/issues/208#issuecomment-507745971

[juhp]

I see, thanks. Hmm okay haven't seen this - though I do occasionally get install failures for the odd package or two: then I have to install the failed packages again - but your problem sounds different and more severe.

(edit) For example just now I got:

  Installing       : ghc-vector-algorithms-devel-0.7.0.1-5.fc29.x86_64                                                        35/56 
Error unpacking rpm package ghc-vector-algorithms-devel-0.7.0.1-5.fc29.x86_64
error: unpacking of archive failed on file /usr/lib64/ghc-8.2.2/vector-algorithms-0.7.0.1/Data/Vector/Algorithms/Search.hi;5d1f1119: cpio: open

  Installing       : ghc-typed-process-devel-0.2.2.0-1.fc29.x86_64                                                            36/56 
Error unpacking rpm package ghc-typed-process-devel-0.2.2.0-1.fc29.x86_64
error: ghc-vector-algorithms-devel-0.7.0.1-5.fc29.x86_64: install failed
error: unpacking of archive failed on file /usr/share/doc/ghc/html/libraries/typed-process-0.2.2.0/index.html;5d1f1119: cpio: open

Maybe happens to me about once a week say - maybe some user container/overlay fs issue?

Anyway sorry this is OT...

[mildred]

For the records 9that's not the original issue described here) some installation issues are caused by selinux enforced on the host. You can always try running setenforce permissive outside the toolbox and retry dnf install in the container. It solved some issues for me (again, not the issue described here).

[debarshiray]

Thanks for the well-written bug report! :)

I can remove the *-modules.yaml.gz file all right, but it then reappears:

This reminds me of https://github.com/containers/fuse-overlayfs/pull/55 :)

More recently, and more likely for this bug, I see https://github.com/containers/fuse-overlayfs/issues/86

What version of fuse-overlayfs did you have?

[debarshiray]

@giuseppe confirmed in #podman on Freenode that it's likely a fuse-overlayfs problem. So knowing the fuse-overlayfs version is important.

[mildred]

What version of fuse-overlayfs did you have?

I have absolutely no idea. That was the version packaged by Fedora Silverblue, but since then I upgraded my OS. Looking as OS logs around the bug report time, I booted my OS on june 17th and shut it down on July 12th. I booted into /ostree/boot.0/fedora-workstation/34b4993cd5aee7abf50bbfa2870c158064ab6a0475322dc9dc6a8a958ca7a5ad/0 which was generated by rpm-ostree update on june 17th 14:31:29 UTC+2. If you know where to find changelogs for the silverblue builds, the version should be there.

[mildred]

What version of fuse-overlayfs did you have?

Most probably fuse-overlayfs-0.4.1-0.dev.git1ff7c64.fc30 https://koji.fedoraproject.org/koji/buildinfo?buildID=1286941 (running Fedora 30)

[mildred]

Occurred again today on Fedora 30 Silverblue, running:

• fuse-overlayfs-0.4.2-0.dev.git7bc2dd9.fc30.x86_64
• podman-1.4.4-4.fc30.x86_64

Error:

Cannot rename directory "/var/cache/dnf/updates-modular-783da5de2e38c644/tmpdir.Eb3neJ/repodata" to "/var/cache/dnf/updates-modular-783da5de2e38c644/repodata": Directory not empty
Error: Cannot rename directory "/var/cache/dnf/updates-modular-783da5de2e38c644/tmpdir.Eb3neJ/repodata" to "/var/cache/dnf/updates-modular-783da5de2e38c644/repodata": Directory not empty

[giuseppe]

any chance you could try https://bodhi.fedoraproject.org/updates/FEDORA-2019-035037bf44 ?

There are a lot of changes, also touching the "directory not empty" issue and performance improvements

[giuseppe]

more in details, the commit https://github.com/containers/fuse-overlayfs/commit/cecbcbb7ad588b8819dd6cfccb6a900c9a3a39c8 could be addressing the issue you've seen

[mildred]

Upgraded to the latest Fedora with fuse-overlayfs-0.5-4.fc30.x86_64 in it and the failing toolbox started working again. Thank you.

[debarshiray]

Thanks for confirming, @mildred Much appreciated!