Zfs: ACL operations result in “Invalid argument” on Linux 4.5

[fuzzykiller]

I’m running ZFS on Linux on my Arch machine. After upgrading to Kernel 4.5, I can no longer access ACLs:

[fuzzy@server ~]$ strace getfacl /
execve("/usr/bin/getfacl", ["getfacl", "/"], [/* 20 vars */]) = 0
brk(NULL)                               = 0xa98000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=93119, ...}) = 0
mmap(NULL, 93119, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc24abbd000
close(3)                                = 0
open("/usr/lib/libacl.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\300 \0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=35384, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc24abbc000
mmap(NULL, 2130592, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc24a7a8000
mprotect(0x7fc24a7b0000, 2093056, PROT_NONE) = 0
mmap(0x7fc24a9af000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x7000) = 0x7fc24a9af000
close(3)                                = 0
open("/usr/lib/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0000\10\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1960896, ...}) = 0
mmap(NULL, 3803536, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc24a407000
mprotect(0x7fc24a59f000, 2093056, PROT_NONE) = 0
mmap(0x7fc24a79e000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x197000) = 0x7fc24a79e000
mmap(0x7fc24a7a4000, 14736, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fc24a7a4000
close(3)                                = 0
open("/usr/lib/libattr.so.1", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\24\0\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=18736, ...}) = 0
mmap(NULL, 2113912, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fc24a202000
mprotect(0x7fc24a206000, 2093056, PROT_NONE) = 0
mmap(0x7fc24a405000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7fc24a405000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc24abbb000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc24abba000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fc24abb9000
arch_prctl(ARCH_SET_FS, 0x7fc24abba700) = 0
mprotect(0x7fc24a79e000, 16384, PROT_READ) = 0
mprotect(0x7fc24a405000, 4096, PROT_READ) = 0
mprotect(0x7fc24a9af000, 4096, PROT_READ) = 0
mprotect(0x604000, 4096, PROT_READ)     = 0
mprotect(0x7fc24abd4000, 4096, PROT_READ) = 0
munmap(0x7fc24abbd000, 93119)           = 0
brk(NULL)                               = 0xa98000
brk(0xab9000)                           = 0xab9000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=3531264, ...}) = 0
mmap(NULL, 3531264, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fc249ea3000
close(3)                                = 0
ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
getrlimit(RLIMIT_NOFILE, {rlim_cur=1024, rlim_max=4*1024}) = 0
lstat("/", {st_mode=S_IFDIR|0755, st_size=24, ...}) = 0
getxattr("/", "system.posix_acl_access", 0x7ffcc4be9be0, 132) = -1 EINVAL (Invalid argument)
open("/usr/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=2997, ...}) = 0
read(3, "# Locale name alias data base.\n#"..., 3072) = 2997
read(3, "", 3072)                       = 0
close(3)                                = 0
open("/usr/share/locale/en_US.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en_US/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.UTF-8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en.utf8/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale/en/LC_MESSAGES/libc.mo", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "getfacl: /: Invalid argument\n", 29getfacl: /: Invalid argument
) = 29
exit_group(1)                           = ?
+++ exited with 1 +++

setfacl similarly fails. This happens both on version 0.6.5.5 (gbb0eec6) and 0.6.5.6 (g5079f5b). I’m running the Arch stock kernel 4.5.0-1 x86_64. I have acltype set to posixacl.

The strange thing is, the ACLs are still applied (by the kernel?). I have a directory with default ACLs set so that new files are world-readable. It still works:

[fuzzy@server Public]$ umask
0027
[fuzzy@server Public]$ touch my-test-file
[fuzzy@server Public]$ ls -l my-test-file
-rw-r--r-- 1 fuzzy fuzzy 0 Apr 18 20:46 my-test-file
[fuzzy@server Public]$ cd
[fuzzy@server ~]$ rm /hd2/Public/my-test-file
[fuzzy@server ~]$ touch my-test-file
[fuzzy@server ~]$ ls -l my-test-file
-rw-r----- 1 fuzzy fuzzy 0 Apr 18 20:48 my-test-file

(I also included a regular directory for comparison.)

On other filesystems like ext4, ACLs work as expected. There may have been some API change.

If you need additional information, please don’t hesitate to ask!

[behlendorf]

@fuzzykiller an ACL regression accidentally made it in to the release branch and has been fixed in master. We'll get the fix in to the next point release. If you need the fix now you can cherry pick the following patch, please let us know if it resolves the issue as expected.

98f0369 Fix ZPL miswrite of default POSIX ACL

[fuzzykiller]

Thanks for the quick reply! I’m not sure this is the problem I’m facing. My ACLs were set months ago. Wouldn’t that mean they were written correctly?

I’ll try anyway, of course.

[kerberizer]

@behlendorf I'm not very experienced with ACLs, so I might be missing something important, but that commit doesn't seem to make any difference for me. On any existing or newly created dataset with acltype=posixacl and xattr=sa I still get "Invalid argument" in return to getfacl(1) or setfacl(1) on any file or directory I try. The system trace looks pretty much like what @fuzzykiller posted, with getxattr("<path>", "system.posix_acl_access", ...) returning EINVAL.

Edit: Just made a test with a freshly created pool, but the results are still the same. For the record, here are the SPL/ZFS module versions as well (I'm also using the default 4.5.0-1-ARCH kernel):
SPL: Loaded module v0.6.5-49_g224817e
ZFS: Loaded module v0.6.5-218_g8fc5674, ZFS pool version 5000, ZFS filesystem version 5

I'll be curious if @fuzzykiller reaches different results.

[fuzzykiller]

@behlendorf Nope, doesn’t work. I created a fresh VM with Arch Linux and tried a variety of things, including a master build (same versions as @kerberizer ), same results.

I then upgraded to Linux 4.5.1, once again using a master ZFS/SPL build (218/49, as above). However, results didn’t change: It still doesn’t work.

Unfortunately, I have no experience whatsoever with kernel debugging. But seeing how the bug is easily reproduced on an Arch install, you should be able to reproduce it, too.

[gedia]

The patches do not seem to work for me either. I'm on gentoo x86_64 with linux 4.5.1, spl, zfs and zfs-kmod built from master.

[behlendorf]

@kerberizer sorry, I answered this without looking as closely as I should have. This is definitely a different issue which needs to be investigated and it sounds like it's specific to 4.5 kernels. Can someone verify this doesn't occur with 4.4 and earlier kernels.

[Fabian-Gruenbichler]

@behlendorf acls work as expected here (zol 0.6.5.6, 4.4 kernel)

[kerberizer]

@behlendorf It was the logical thing to assume: I had myself expected that regression to be the culprit too.

Anyway, I can also confirm that the problem is solved by downgrading to Linux 4.4.x.

Linux version 4.4.5-1-ARCH (kerb@bluestorm)
SPL: Loaded module v0.6.5-49_g224817e
ZFS: Loaded module v0.6.5-218_g8fc5674, ZFS pool version 5000, ZFS filesystem version 5

The ACLs here work as expected. In particular, getxattr("<path>", "system.posix_acl_access", ...), which with Linux 4.5 was returning EINVAL, now correctly returns ENODATA on a file that has no explicitly set ACL, and the expected number of bytes -- on a file that does have such ACL. Setting ACLs also produces the expected results.

I'd be very curious to see what the actual problem is. I even tried to look through that 4.5 compat change in 4967a3e, but my knowledge unfortunately was too inadequate to notice anything amiss on casual observation. In any case, I'd be glad to help with any testing that might be needed.

[tuxoko]

@fuzzykiller @kerberizer
Please test https://github.com/zfsonlinux/zfs/pull/4549 with kernel 4.5

[kerberizer]

@tuxoko Thanks! All tests that I've done earlier now proceed as expected on 4.5. It'll be interesting to see what the other people will find as they seem to use the ACLs more extensively than I do.

[fuzzykiller]

@tuxoko It works. 😁

[gedia]

Works for me too...