Zfs: Unable to access snapshot files using /.zfs/snapshot/testsnapshot/
System information
Type | Version/Name
--- | ---
Distribution Name | Fedora
Distribution Version | 30
Linux Kernel | 5.2.18
Architecture | x86_64
ZFS Version | 0.8.2-1
SPL Version | 0.8.2-1
Describe the problem you're observing
Unable to access snapshot files using /.zfs/snapshot/test/
Verified on freebsd 11, this is working. Also ubuntu module 0.7.5 this is working.
I cant stress enough how serious this issue is.
Describe how to reproduce the problem
sudo bash
echo "should be working" > /opt/testsnapshot
zfs -r snapshot roottank@testsnapshot
rm /opt/testsnapshot
cd /.zfs/snapshot/testsnapshot/opt
bash: cd: opt: Object is remote
Include any warning/errors/backtraces from the system logs
There is no visible errors.
-->
roker
All 19 comments
I had the same problem with zfs 0.8.2, so I downgraded zfs to version 0.8.1 which allows accessing the root snapshots fine. BTW, snapshots of the other filesystems mounted under / such as /home/.zfs/snapshot/* can still be accessed with zfs 0.8.2.
jhyeon
on 15 Oct 2019
The same problem here. Debian testing, zfs-dkms 0.8.2-2. Some additional info:
x220i:~% ls /.zfs/snapshot
2019-10-16-2057/ zfs-auto-snap_daily4-2019-10-13-0412/ zfs-auto-snap_frequent-2019-10-17-1930/ zfs-auto-snap_hourly3-2019-10-17-1517/
zfs-auto-snap_daily-2019-10-12-0306/ zfs-auto-snap_daily8-2019-10-01-0311/ zfs-auto-snap_frequent-2019-10-17-1945/ zfs-auto-snap_hourly6-2019-10-17-0617/
zfs-auto-snap_daily-2019-10-14-0311/ zfs-auto-snap_daily8-2019-10-09-0257/ zfs-auto-snap_hourly-2019-10-17-1617/ zfs-auto-snap_hourly6-2019-10-17-1217/
zfs-auto-snap_daily-2019-10-15-0320/ zfs-auto-snap_daily8-2019-10-17-0318/ zfs-auto-snap_hourly-2019-10-17-1717/ zfs-auto-snap_hourly6-2019-10-17-1817/
zfs-auto-snap_daily-2019-10-16-0316/ zfs-auto-snap_frequent-2019-10-17-1900/ zfs-auto-snap_hourly-2019-10-17-1917/
zfs-auto-snap_daily4-2019-10-05-0258/ zfs-auto-snap_frequent-2019-10-17-1915/ zfs-auto-snap_hourly3-2019-10-17-0917/
x220i:~% ls /.zfs/snapshot/zfs-auto-snap_daily8-2019-10-09-0257/
x220i:~% ls /.zfs/snapshot/zfs-auto-snap_daily8-2019-10-09-0257/.
ls: cannot access '/.zfs/snapshot/zfs-auto-snap_daily8-2019-10-09-0257/.': Object is remote
zsh: exit 2 ls
x220i:~% zfs list -o space -rt all
NAME AVAIL USED USEDSNAP USEDDS USEDREFRESERV USEDCHILD
x220 159G 287G 0B 96K 0B 287G
x220/ROOT 159G 270G 0B 96K 0B 270G
x220/ROOT/debian 159G 270G 25.1G 245G 0B 0B
x220/ROOT/debian@zfs-auto-snap_daily8-2019-10-01-0311 - 1.47G - - - -
x220/ROOT/debian@zfs-auto-snap_daily4-2019-10-05-0258 - 1.21G - - - -
x220/ROOT/debian@zfs-auto-snap_daily8-2019-10-09-0257 - 2.15G - - - -
x220/ROOT/debian@zfs-auto-snap_daily-2019-10-12-0306 - 1.36G - - - -
x220/ROOT/debian@zfs-auto-snap_daily4-2019-10-13-0412 - 1009M - - - -
x220/ROOT/debian@zfs-auto-snap_daily-2019-10-14-0311 - 1.05G - - - -
x220/ROOT/debian@zfs-auto-snap_daily-2019-10-15-0320 - 2.01G - - - -
x220/ROOT/debian@zfs-auto-snap_daily-2019-10-16-0316 - 1.74G - - - -
x220/ROOT/debian@2019-10-16-2057 - 183M - - - -
x220/ROOT/debian@zfs-auto-snap_daily8-2019-10-17-0318 - 52.3M - - - -
x220/ROOT/debian@zfs-auto-snap_hourly6-2019-10-17-0617 - 42.8M - - - -
x220/ROOT/debian@zfs-auto-snap_hourly3-2019-10-17-0917 - 10.5M - - - -
x220/ROOT/debian@zfs-auto-snap_hourly6-2019-10-17-1217 - 17.3M - - - -
x220/ROOT/debian@zfs-auto-snap_hourly3-2019-10-17-1517 - 12.1M - - - -
x220/ROOT/debian@zfs-auto-snap_hourly-2019-10-17-1617 - 5.17M - - - -
x220/ROOT/debian@zfs-auto-snap_hourly-2019-10-17-1717 - 5.86M - - - -
x220/ROOT/debian@zfs-auto-snap_hourly6-2019-10-17-1817 - 37.6M - - - -
x220/ROOT/debian@zfs-auto-snap_frequent-2019-10-17-1915 - 5.95M - - - -
x220/ROOT/debian@zfs-auto-snap_hourly-2019-10-17-1917 - 6.00M - - - -
x220/ROOT/debian@zfs-auto-snap_frequent-2019-10-17-1930 - 11.8M - - - -
x220/ROOT/debian@zfs-auto-snap_frequent-2019-10-17-1945 - 13.8M - - - -
x220/ROOT/debian@zfs-auto-snap_frequent-2019-10-17-2000 - 6.68M - - - -
x220/home 159G 80.1M 0B 96K 0B 80.0M
x220/home/ark 159G 80.0M 0B 96K 0B 79.9M
x220/home/ark/tmp 159G 79.9M 0B 96K 0B 79.9M
x220/home/ark/tmp/android 159G 79.9M 0B 96K 0B 79.8M
x220/home/ark/tmp/android/music 159G 79.8M 0B 79.8M 0B 0B
x220/swap 162G 4.25G 0B 642M 3.63G 0B
x220/var 159G 12.4G 0B 96K 0B 12.4G
x220/var/tmp 159G 12.4G 0B 96K 0B 12.4G
x220/var/tmp/gobuild 159G 12.4G 0B 12.4G 0B 0B
I switced to 0.8.8-2 (from 0.7.12-2+deb10u1) at 2019-10-12. The snapshots were created automatically, before switch and after. All snapshots have the sensible size, I think, they're fully functional, except that I cannot access them.
arkoort
on 17 Oct 2019
PR #9384 which resolves #9381, still does not allow me to access snapshots under /.zfs/snapshot/. The reason might be that I am using initramfs which first mounts the root filesystem under /sysroot/ before doing switch_root. See my comments in #9381.
jhyeon
on 25 Oct 2019
Same here. Arch Linux, kernel 5.3.7, with package zfs-dkms 0.8.2-1 (archzfs-dkms)
Snapdirs of non-root-filesystems can be accessed without problems, e.g.
~
ls /home/.zfs/snapshot/20191030-2155
~
works. journalctl -b -g zfs then shows lines such as:
~
Oct 31 10:37:04 Celsius systemd[6199]: home-.zfs-snapshot-20191030\x2d2155.mount: Succeeded.
Oct 31 10:37:04 Celsius systemd[1]: home-.zfs-snapshot-20191030\x2d2155.mount: Succeeded.
~
And grep -F .zfs /proc/mounts shows the mount for a while.
However, trying ls /.zfs/snapshot/20191030-2155 yields no output and no error message.
Doing ls /.zfs/snapshot/20191030-2155/. (note the appended /.) yields the error:
~
ls: cannot access '/.zfs/snapshot/20191030-2155/.': Object is remote
~
And there are no associated entries in the output of journalctl -b -g zfs nor grep -F .zfs /proc/mounts.
Any hints on how to dig further?
ccorn
on 31 Oct 2019
I also get this error for any datasets that have mountpoint=legacy with 0.8.2.
ElvishJerricco
on 22 Nov 2019
I take that back. It's all my datasets that are mounted during initramfs that have the problem; my legacy mount that's mounted in stage 2 boot doesn't have this problem. Also, of course, if I create a new legacy mount, I can mount it and access its .zfs/snapshot/foo contents.
ElvishJerricco
on 22 Nov 2019
I have the same problem, it is a regression in 0.8.2 release (it worked fine in 0.8.1)
Bronek
on 2 Dec 2019
@Bronek you're likely seeing the issue resolved by PR #9384. We'll get the fix applied to 0.8.3.
behlendorf
on 2 Dec 2019
Posting this in here since #9384 was closed:
Looks like I found a different kind of bug related to this. I've applied the patch listed above. (#9353
My rpool looks like this:
rpool 1.27T 2.95T 176K none
rpool/ROOT 145G 2.95T 50.1G /
rpool/mysql 53.0G 2.95T 23.0G /var/lib/mysql
rpool/mysql-log 5.79G 2.95T 1.44G /var/lib/mysql-log
rpool/plexmediaserver 206G 2.95T 171G /var/lib/plexmediaserver
rpool/spool 115G 2.95T 35.5G /var/spool
rpool/virtual_machines 778G 2.95T 743G /var/lib/libvirt
I can see inside snapshots for rpool/ROOT:
windwalker:~# ls /.zfs/snapshot/zfs-auto-snap_daily-2019-12-07-1107/
bin boot.tar.xz dead.letter etc lib lost+found mnt opt proc run share sys tmp var
boot cdrom dev home lib64 media netshares path root sbin srv tftpboot usr
Looking inside snapshots for any other dataset in rpool results in "Too many levels of symbolic links" error:
ls /var/lib/libvirt/.zfs/snapshot/zfs-auto-snap_daily-2019-12-07-1107/
ls: cannot access '/var/lib/libvirt/.zfs/snapshot/zfs-auto-snap_daily-2019-12-07-1107/': Too many levels of symbolic links
Ryushin
on 8 Dec 2019
I just upgraded to 0.8.3, and the issue remains as described in my earlier post.
~ sh
$ ls /home/.zfs/snapshot/
20200121-1930 20200123-0040 20200124-2134 20200126-0146
20200122-1325 20200123-1559 20200125-1946 20200126-2030
$ ls /home/.zfs/snapshot/20200126-2030 # works
ccorn
$ ls /.zfs/snapshot/
20200121-1930 20200123-0040 20200124-2134 20200126-0146
20200122-1325 20200123-1559 20200125-1946 20200126-2030
$ ls /.zfs/snapshot/20200126-2030 # no output
$ ls /.zfs/snapshot/20200126-2030/.
ls: cannot access '/.zfs/snapshot/20200126-2030/.': Object is remote
~
ccorn
on 26 Jan 2020
I too am having this issue on 0.8.3 (though in my case it affects multiple filesystems under my zroot pool, which contains /). This issue doesn't occur in any of my other pools that do not contain /.
$ ls /home/.zfs/snapshot/
2020-0122-0020_weekly 2020-0128-0100_hourly 2020-0128-1400_hourly
2020-0109-0000_daily 2020-0122-1839_before-pacman 2020-0128-0200_hourly 2020-0128-1400_quarterhour
$ ls /home/.zfs/snapshot/2020-0128-1400_quarterhour/ #returns nothing
$ ls /.zfs/snapshot/
2020-0108-0834_weekly 2020-0122-0020_weekly 2020-0128-0100_hourly 2020-0128-1400_hourly
2020-0109-0000_daily 2020-0122-1839_before-pacman 2020-0128-0200_hourly 2020-0128-1400_quarterhour
$ ls /.zfs/snapshot/2020-0128-1400_quarterhour/ #returns nothing
(I've abbreviated the output of ls when listing the snapshots available)
seonwoolee
on 28 Jan 2020
Would just like to chime in... seeing the same behavior on 0.8.3 and 0.8.2 zfs on Archlinux
One example from the logs:
1583461577 zfs_ctldir.c:1086:zfsctl_snapshot_mount(): Unable to automount /new_root/var/lib/docker/.zfs/snapshot/autosnap_2020-02-29_00:00:08_daily error=512
marker5a
on 6 Mar 2020
I can also confirm that by creating a non-root pool, I can access everything fine in the snapshot folder, so seems to be similar to ElvishJerrico. Any root derived filesystem though does not work as expected
marker5a
on 8 Mar 2020
I suspect this error occurs on systems with initramfs which chroot(2)s to new root filesystem and starts init process by running switch_root(8).
jhyeon
on 11 Mar 2020
10128 indicates that reverting 5a1bf9e 093bb64 solves this particular issue. Indeed that works for me. But that reopens the issues those commits were meant to solve. What now?
ccorn
on 21 Mar 2020
10128 indicates that reverting 5a1bf9e 093bb64 solves this particular issue. Indeed that works for me. But that reopens the issues those commits were meant to solve. What now?
This was a good workaround for me.. yeah, as to what now, I'll leave that to the gurus :)
marker5a
on 23 Mar 2020
still happening on
zfs-0.8.4-r1-gentoo
zfs-kmod-0.8.4-r0-gentoo
non-root snapdirs are fine. but snapshots for root dataset shows up empty.
gyakovlev
on 13 Jul 2020
Filename: /lib/modules/5.5.7-200.fc31.x86_64/extra/zfs.ko.xz
version: 0.8.4-1
Slightly different behaviour:
-non root filesystems are okm (snapshot dirs visible)
-root filesystem is ok immediately after boot (napshot dirs visible) but problems occur aftersome heavy work (compiling kernel)
deeptho
on 22 Sep 2020
found 2 workarounds (still on 0.8.4)
I've explicitly added my root dataset to fstab like this
zroot/ROOT/default / zfs defaults,noatime 0 0
systemd (using generators) remounts / and/.zfs/snapshot/*content becomes accessible.another workaround, couple commands:
mkdir /sysroot
mount -o bind / /sysroot
after that snapshots will be accessible in /sysroot/.zfs/ just fine, but remain inaccessible via /.zfs/
/sysroot is the initial location dracut mounts root dataset to via zfsutil so somehow bind-mounting to this exact path or re-mounting to / after pivot_root restores expected behaviour.
gyakovlev
on 28 Sep 2020
Related issues
Greek64
路
3Comments
Baughn
路
4Comments
cinterloper
路
3Comments
kernelOfTruth
路
4Comments
geek-at
路
3Comments
Most helpful comment
10128 indicates that reverting 5a1bf9e 093bb64 solves this particular issue. Indeed that works for me. But that reopens the issues those commits were meant to solve. What now?