Lizardfs: ACLs not honored

[BloodBlight]

I have a LizardFS folder that I am sharing out over SAMBA and having all sorts of issues. But the big show stopper is that ACLs are not being honored.

When I test on an EXT4 location:
setfacl -Rm g:USER :rwx /mnt/Test

And then as that user:
ls /mnt/Test
I get:
-rwxrwx---+ 1 OtherUSER OtherGROUP 9 Feb 15 09:13 /mnt/Test/Test.txt

However, when I run the same command on a lizardFS share:
setfacl -Rm g:USER :rwx /LizardFS/Test
I get:
ls: cannot access '/LizardFS/Test': Permission denied

I have the mfsacl option on the mount, but it doesn’t show up when looking at the mount report:
mfs#mfsmaster:9421 on /LizardFS type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)

And if I:
getfacl /LizardFS/Test/

I get:

# file: LizardFS/Test/
# owner: OtherUSER
# group: OtherGROUP
# flags: -s-
user::rwx
user:OtherUSER:rwx
user:USER:rwx
group::rwx
group:OtherGROUP:rwx
mask::rwx
other::---
default:user::rwx
default:group::rwx
default:mask::rwx
default:other::---
default:other::---

My client version is:
bionic,now 3.12.0+dfsg-1 amd64 [installed]

Running on Ubuntu 18.04:
Linux SERVERNAME 4.15.0-45-generic #48-Ubuntu SMP Tue Jan 29 16:28:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Ideas?

I am also having an issue with groups not being honored at all, only users. But that is not a LizardFS issue.

[pkonopelko]

Possibly related: https://github.com/moosefs/moosefs/issues/189

[BloodBlight]

Double checked, I already have ignoregid in my exports.

192.168.0.XXX /Shares/Test rw,alldirs,maproot=0,ignoregid

Not sure if I am missing something.

The master server is not domain joined, only the client.

[pbeza]

I have the mfsacl option on the mount, but it doesn’t show up when looking at the mount report:
mfs#mfsmaster:9421 on /LizardFS type fuse (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)

Note that mfsmount --help states that mfsacl mount option is deprecated:

-o mfsacl                   DEPRECATED, used to enable/disable ACL support, ignored now

That's why it doesn't show up in the mount report. ACL is always enabled since commit d6511fd77e9f4397f20c9f2bb15f56b03d370405.

commit d6511fd77e9f4397f20c9f2bb15f56b03d370405
Author: Piotr Sarna <[email protected]>
Date:   Fri Sep 8 11:31:56 2017 +0200

    all: Deprecate enable_acl flag

    This commit makes mount and client lib ignore enable_acl/mfsacl flag,
    as ACL support is always enabled. Setting the -o mfsacl parameter
    in mount is still possible in order to provide backwards compatibility.

    Change-Id: I550d34e2e87d83c6ed895523766d98c56c59861a

I'm not able to reproduce your problem:

I also tried to run setfacl on the files mounted with mount.cifs //172.20.0.5/test /mnt/test/ (instead of running it directly on LizardFS share) but it still works for me without any problem.

You can try to remove ignoregid mount option. You can also run mfsmount with --nostdopts – this will not add standard LizardFS mount options – and use only those mount options that works for me (i.e. rw,relatime,user_id=0,group_id=0,default_permissions,allow_other – note that you have additional nosuid mount option but it shouldn't be a problem). Please write back if it helped. If not, please share your thoughts about setting up environment so that I was able to reproduce the bug.

[BloodBlight]

It was not that the ACLs where not applied and shown, it was that they where not honored. So granting a user permission would not actually work even though the file showed that the user had access.

1. Create an unprivileged user and a group for that user.
2. Create a folder and set the owner to root:root.
3. Create a file with some text in the folder and confirm it is also root:root.
4. Set the perms on that folder and file to something like 770 (or what ever you prefer, just so that "other" has no permissions).
5. Then add that users group (not the user) to the folder by ACL.
6. Test accessing the folder as that user.

[pbeza]

OK, now I get it. I confirm the problem.

Test live preview:

Test log:

root@lizardfs-client:~# ls -la /mnt
total 16
drwxr-xr-x 1 root root 4096 Apr 26 08:27 .
drwxr-xr-x 1 root root 4096 Apr 26 07:22 ..
drwxr-xr-x 2 root root 4096 Apr 26 07:22 lizardfs
drwxr-xr-x 2 root root 4096 Apr 26 08:27 lizardfs-test
root@lizardfs-client:~# mount | grep -i lizardfs
root@lizardfs-client:~# cat /etc/fstab
lizardfs-master: /mnt/lizardfs lzfs defaults 0 0
root@lizardfs-client:~# mount /mnt/lizardfs
root@lizardfs-client:~# mount | grep -i lizard
lizardfs-master:9421 on /mnt/lizardfs type fuse.mfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other)
root@lizardfs-client:~# ls -la /mnt
total 12
drwxr-xr-x  1 root root 4096 Apr 26 08:27 .
drwxr-xr-x  1 root root 4096 Apr 26 07:22 ..
drwxrwxrwx+ 3 root root    0 May  2 07:04 lizardfs
drwxr-xr-x  2 root root 4096 Apr 26 08:27 lizardfs-test
root@lizardfs-client:~# mkdir /mnt/lizardfs/test
root@lizardfs-client:~# for i in {1,2,3}; do echo "Test 0${i}" > /mnt/lizardfs/test/test_0${i}.txt ; done
root@lizardfs-client:~# chmod -R 770 /mnt/lizardfs/test
root@lizardfs-client:~# setfacl -m g:test:rwx /mnt/lizardfs/test/
root@lizardfs-client:~# ls -la /mnt
total 12
drwxr-xr-x  1 root root 4096 Apr 26 08:27 .
drwxr-xr-x  1 root root 4096 Apr 26 07:22 ..
drwxrwxrwx+ 4 root root    0 May  2 07:05 lizardfs
drwxr-xr-x  2 root root 4096 Apr 26 08:27 lizardfs-test
root@lizardfs-client:~# ls -la /mnt/lizardfs
total 4
drwxrwxrwx+ 4 root root    0 May  2 07:05 .
drwxr-xr-x  1 root root 4096 Apr 26 08:27 ..
drwxr-xr-x  8 root root    0 May  2 06:20 duplicates
drwxrwx---+ 2 root root    0 May  2 07:05 test
root@lizardfs-client:~# ls -la /mnt/lizardfs/test
total 2
drwxrwx---+ 2 root root 0 May  2 07:05 .
drwxrwxrwx+ 4 root root 0 May  2 07:05 ..
-rwxrwx---  1 root root 8 May  2 07:05 test_01.txt
-rwxrwx---  1 root root 8 May  2 07:05 test_02.txt
-rwxrwx---  1 root root 8 May  2 07:05 test_03.txt
root@lizardfs-client:~# getfacl -p /mnt/
# file: /mnt/
# owner: root
# group: root
user::rwx
group::r-x
other::r-x

root@lizardfs-client:~# getfacl -p /mnt/lizardfs
# file: /mnt/lizardfs
# owner: root
# group: root
user::rwx
group::rwx
other::rwx

root@lizardfs-client:~# getfacl -p /mnt/lizardfs/test/
# file: /mnt/lizardfs/test/
# owner: root
# group: root
user::rwx
group::rwx
group:test:rwx
mask::rwx
other::---

root@lizardfs-client:~# su test
test@lizardfs-client:/root$ ls -la /mnt/lizardfs
total 4
drwxrwxrwx+ 4 root root    0 May  2 07:05 .
drwxr-xr-x  1 root root 4096 Apr 26 08:27 ..
drwxr-xr-x  8 root root    0 May  2 06:20 duplicates
drwxrwx---+ 2 root root    0 May  2 07:05 test
test@lizardfs-client:/root$ ls -la /mnt/lizardfs/test
ls: cannot open directory '/mnt/lizardfs/test': Permission denied
test@lizardfs-client:/root$ getfacl -p /mnt/lizardfs
# file: /mnt/lizardfs
# owner: root
# group: root
user::rwx
group::rwx
other::rwx

test@lizardfs-client:/root$ getfacl -p /mnt/lizardfs/test
# file: /mnt/lizardfs/test
# owner: root
# group: root
user::rwx
group::rwx
group:test:rwx
mask::rwx
other::---

For sake of completeness the same (working) test on localhost:

It seems that it has nothing to do with Samba.

BTW: it seems that LizardFS' ACL sanity tests check validity of the ls and getfacl outputs only, therefore this bug was not detected during tests. Here is more comprehensive test that passes.

[pbeza]

Here is hotfix for mfsmount3 (FUSE_CAP_POSIX_ACL flag added which is handled by kernel here).

To make it work with mfsmount you need to mfsmount with -o allow_other and -n option (--nostdopts) – this also works for mfsmount3 without applying FUSE_CAP_POSIX_ACL. I'm not sure if it's workaround or proper solution of the problem. Work in progress...

BTW: related stuff just for reference for developers interested in helping out to solve the problem:

• [fuse-devel] [PATCH] fs/fuse: Add POSIX ACL support and Support for posix acls in fuse – seem to be most relevant; note that glusterfs disables default_permissions if ACLs need to be supported
• [fuse-devel] ACL support
• Add support for FUSE_POSIX_ACL
• Improve support for ACLs
• Fix support for FUSE_POSIX_ACL
• Without default_permissions, cached permissions are only checked on first access
• Add capability flag for default_permissions
• [PATCH v7 0/7] fuse: mounts from non-init user namespaces
• ACL support
• Warning: unresolved security issue
• Linux namespaces

Details for those who are willing to debug:

• File permissions are checked in FUSE module in fuse_permission() function which calls generic_permission()
• There were some modifications in FUSE kernel module last year (Feb 21, 2018), specifically in acl.c – this may be be the source of the problem

Working mfsmount3 example after applying FUSE_CAP_POSIX_ACL flag:

[pbeza]

For now problem was solved for mfsmount3 only (see: proof). I'm not sure whether it is still possible to support ACL in FUSE 2.9.9 (which is provided by libfuse2 package) after changes in kernel 4.9. For details refer what Miklos Szeredi answered in this fuse-devel mailing list thread.

If you have any suggestions how this issue can be fixed using FUSE 2.9.9, please share your thoughts. ;-)

[trapexit]

It's not possible to enable newer features in 2.9.x as there is no access to the INIT instruction's outarg flags variable. There's no way to just define the flag yourself and include it.

https://github.com/libfuse/libfuse/blob/fuse_2_9_bugfix/lib/fuse_lowlevel.c#L1846

It might be possible if you rewrote the whole low level init function and tried to short circuit it but I've not looked close enough at the code to tell if that's plausible.

[pbeza]

Thanks @trapexit for your comment – it was important for me to make sure that it's not possible to support ACL using libfuse 2.9.x without modifying Linux kernel.

BTW: I've just found 2 patches that modify both kernel and libfuse 2.9.x to support ACLs. Here they are:

• https://github.com/lizardfs/lizardfs/blob/master/utils/libfuse_acl.patch
• https://github.com/lizardfs/lizardfs/blob/master/utils/linux_fuse_acl.patch

To sum up: if anyone want to use libfuse 2.9.x with ACLs, then sorry – you need to patch your kernel and libfuse or upgrade to libfuse3.

[trapexit]

Not sure I follow. The kernel isn't the problem when talking about libfuse2. The problem is that libfuse2 itself provides no way to ask the kernel to enable that feature as it provides hard coded set of supported FUSE features and gives no raw access to the value returned back to the kernel during the handshake.

[BloodBlight]

How is MooseFS handling this? It doesn't seem to be affected.

On Wed, May 29, 2019, 6:44 AM trapexit [e-mail] wrote:

Not sure I follow. The kernel isn't the problem when talking about
libfuse2. The problem is that libfuse2 itself provides no way to ask the
kernel to enable that feature as it provides hard coded set of supported
FUSE features and gives no raw access to the value returned back to the
kernel during the handshake.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/lizardfs/lizardfs/issues/816?email_source=notifications&email_token=AA6FDTT5XIURCWIUKOTYVUTPX2CCRA5CNFSM4GYOVJ4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWPL3DA#issuecomment-496942476,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA6FDTVPOMABSS3PQDBQRMDPX2CCRANCNFSM4GYOVJ4A
.

[trapexit]

I'm not familiar with MooseFS or LizardFS but it appears to me they are managing posix acls entirely manually. https://github.com/moosefs/moosefs/blob/6bd69b6fa45806c49f130212c914150e6c9c17d6/mfsclient/mfs_fuse.c#L5277

[pbeza]

Not sure I follow. The kernel isn't the problem when talking about libfuse2. The problem is that libfuse2 itself provides no way to ask the kernel to enable that feature as it provides hard coded set of supported FUSE features and gives no raw access to the value returned back to the kernel during the handshake.

ACL support for FUSE3 was added in kernel 4.9 – that's why I mentioned that you need to upgrade the kernel to be able to use libfuse3 with ACLs.

There are 2 scenarios if you have libfuse 2.9.x:

• If you have kernel older than 4.9, you need those two patches to patch both the kernel and libfuse 2.9.x. Those patches add .get_acl handler to the kernel's inode_operations structure (this patch is no longer needed for kernels newer than 4.9 because .get_acl was added to the kernel 4.9) and provide a way to enable ACL in userspace while mounting a filesystem.
• If you have kernel at least in version 4.9, you need libfuse3 which provides a way to ask the kernel to use ACLs by setting FUSE_CAP_POSIX_ACL flag.

I'm not familiar with MooseFS or LizardFS but it appears to me they are managing posix acls entirely manually. https://github.com/moosefs/moosefs/blob/6bd69b6fa45806c49f130212c914150e6c9c17d6/mfsclient/mfs_fuse.c#L5277

That is actually the answer for my question if it is possible to support ACL using libfuse 2.9.x without modifying Linux kernel. So the answer is _yes, there is such an option_ however IMO leaving ACL handling to the kernel is more reasonable than handling it manually in userspace.

[trapexit]

This is not the most popular course of action but with my project I ended up vendoring / embedding libfuse into my project because some of my users had old version of libfuse2 which were buggy, updating libfuse2 on said platforms was non-trivial for my users, libfuse3 wasn't available popularly at the time, and even when libfuse3 was available it didn't always have updates to support the newest features (at least the version included in the distro.)