Lizardfs: [Feature] FS notify support (fsnotify/fanotify/inotify/dnotify)

[mcassaniti]

What would be required to support fsnotify events to reach the client? I can see some commented out definitions in protocol/MFSCommunication.h that suggest this may have been started. See https://github.com/libfuse/libfuse/wiki/Fsnotify-and-FUSE for a general idea. I believe that the mfsmount daemon would need to inject the events into the kernel.

Happy to send through a PR with some direction from a developer.

[mwaeckerlin]

@mcassaniti, do you mean that inotifywait does not work on a LizardFS (fuse) client?

→ Yes, +1, that's exact the bug I just wanted to report…!

Reproduce the problem:

1. LizardFS mount a volume, e.g. /config on two clients
2. in client A run: inotifywait /config/reverse-proxy.conf
3. In client B edit /config/reverse-proxy.conf

Expected result: inotifywait terminates with message

Observed result. Nothing happens.

My temporary solution:

• find out, what client the process with inotifywait → client Å
• in that client (A), manually touch file /config/reverse-proxy.conf
• inotifywait in client A creates an event

So locally, it works.

I need this in a docker swarm environment to update an nginx reverse proxy when the configuration changed, without the need of redeploying and therefore without down time of my services.

[onlyjob]

Unable to reproduce: inotifywait works perfectly on LizardFS mount for me.
I'm on Debian, using _official_ (not vendor) packages...

[onlyjob]

I've just realised that @mcassaniti wants _inotify_ to work between clients...
I doubt it would be feasible (or even possible) to implement...

[mwaeckerlin]

Yes, @onlyjob, it does not work, if access is on another client. But that's an absolutely normal use case, e.g. you have a docker swarm (or a openshift cloud) with LizardFS as backend for the volumes. Then one instance running on one worker node writes something and another instance on another worker node should be triggered.

BTW: Same with file locks, I didn't test them, but I hope, they work across clients? Otherwise data corruption would be quite probable…

[mcassaniti]

@onlyjob That was the use case I was after. I understand it requires both the kernel side of FUSE to support the notifications, along with the LizardFS FUSE mount to be generating them. I'm under the impression that there is no code support within LizardFS for any form of notifications?

[onlyjob]

Since distributed notifications are not implemented, as workaround, I think of using something like Nomad to pin POD (several related containers or processes) to one host.
I'm not sure but It might be possible to do so with Swarm as well...

[mwaeckerlin]

@onlyjob, pinning all processes to one host contradicts the basic idea of a cloud — if I would like to pin everything to one host, then I would not even have need for a LizardFS!

Of course, I could implement a new way of triggering, at least in my containers, by sending signals over tcp/ip, but that requires code changes in all systems and is far away from the elegance of file system notifications.

I strongly suggest to put notifications on the LizardFS-todo-list.

What about file locking? If I lock a file on one host, is it locked on all other hosts too? That would also be a necessary feature for distributed concurrent access and it would allow more types of redundant process distribution. So if not yet supported, I'd suggest to put locking on the todo list too.