Zfs: Missing features w/ cachefilesd & zfs-0.8.4 on ubuntu/post 20.04, getting EBADF & EOPNOTSUPP

[satmandu]

System information

Type | Version/Name
--- | ---
Distribution Name | Ubuntu
Distribution Version | Groovy
Linux Kernel | 5.4.45 (also 5.4.0-34-lowlatency)
Architecture | arm64 & x86_64
ZFS Version | zfs-0.8.4-1ubuntu4

Describe the problem you're observing

cachefilesd aka fs-cache allows one to cache other filesystems (such as nfs)

It doesn't appear to be working with zfs, and that may be due to features that aren't supported on zfs. (It would be helpful to find out which features those are.)

cachefilesd appears to use xattr, which I already have enabled.

rpool/ROOT/ubuntu on / type zfs (rw,relatime,xattr,posixacl)

More info on cachefilesd here: https://www.0ink.net/2018/01/30/cache-nfs-files.html

Describe how to reproduce the problem

sudo strace /sbin/cachefilesd

Include any warning/errors/backtraces from the system logs

end of strace at https://paste.ubuntu.com/p/7xhH8Xsrnp/

close(1018)                             = -1 EBADF (Bad file descriptor)
close(1019)                             = -1 EBADF (Bad file descriptor)
close(1020)                             = -1 EBADF (Bad file descriptor)
close(1021)                             = -1 EBADF (Bad file descriptor)
close(1022)                             = -1 EBADF (Bad file descriptor)
close(1023)                             = -1 EBADF (Bad file descriptor)
openat(AT_FDCWD, "/etc/localtime", O_RDONLY|O_CLOEXEC) = 4
fstat(4, {st_mode=S_IFREG|0644, st_size=3536, ...}) = 0
fstat(4, {st_mode=S_IFREG|0644, st_size=3536, ...}) = 0
read(4, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\5\0\0\0\5\0\0\0\0"..., 3584) = 3536
lseek(4, -2252, SEEK_CUR)               = 1284
read(4, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\5\0\0\0\5\0\0\0\0"..., 3584) = 2252
close(4)                                = 0
getpid()                                = 23360
socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 4
connect(4, {sa_family=AF_UNIX, sun_path="/dev/log"}, 110) = 0
sendto(4, "<30>Jun 17 14:46:53 cachefilesd["..., 60, MSG_NOSIGNAL, NULL, 0) = 60
close(4)                                = 0
write(3, "bind", 4)                     = -1 EOPNOTSUPP (Operation not supported)

This is an strace run on 5.4.0-34-lowlatency/x86_64 giving the same results: https://paste.ubuntu.com/p/fCrfr3gW4D/

Starting the process normally through systemd I get this less descriptive error:

Jun 17 14:41:01 rpi4 systemd[1]: Starting LSB: CacheFiles daemon...
Jun 17 14:41:01 rpi4 cachefilesd[20374]:  * Starting FilesCache daemon  cachefilesd
Jun 17 14:41:03 rpi4 cachefilesd[20380]: About to bind cache
Jun 17 14:41:03 rpi4 cachefilesd[20380]: CacheFiles bind failed: errno 95 (Operation not supported)

[nivedita76]

I'm not at all familiar with this, but it looks like the cachefiles implementation in the kernel expects f_bsize to be <= PAGE_SIZE while zfs will return the recordsize property there. Try it on a dataset with recordsize = 4K and maybe it'll work.

[nivedita76]

Looks like it also needs the FS to implement bmap, which AFAICT ZFS implements only on FreeBSD.