Runtime: How to implement System.IO.FileSystem.Watcher on FreeBSD

[ghuntley]

This issue is to discuss how we can implement System.IO.FileSystem.Watcher on FreeBSD which does not have inotify:

• [ ] Determine/discuss if this be a source of trouble?
• [ ] We could stub it out at the moment and throw a PlatformNotsupportedException as suggested in by Stephen - https://github.com/dotnet/corefx/pull/2021#issuecomment-111602342
• [ ] Implement proper solution using kqueue/libev? Any recommendations here are very much welcome.

Reference Material

• https://www.freebsd.org/cgi/man.cgi?query=kqueue
• http://software.schmorp.de/pkg/libev.html

• http://libevent.org/

  Related Issues

• https://github.com/dotnet/corefx/issues/1626#issuecomment-111975933

• https://github.com/dotnet/corefx/issues/1626#issuecomment-111975140

[stephentoub]

Hopefully some OS-provided solution will work well, but worst case you could also consider a polling-based solution, e.g. https://github.com/dotnet/corefxlab/tree/master/src/System.IO.FileSystem.Watcher.Polling
cc: @KrzysztofCwalina

[jonmill]

I need to look again, but I believe the kqueue system works very similarly to the inotify system; if this is the case, we could potentially re-use the Linux implementation for FreeBSD. I'll dig in and find out more

[jonmill]

It looks like the kqueue system requires opening a handle per file whereas Linux requires per directory. The Linux implementation won't work here so there are a couple options:

1. Make a FreeBSD implementation using kqueues and error out if (read as when) the process hits the maximum handle limit
2. Have the FreeBSD implementation have an external dependency on other software (an inotify port, other 3rd party FSW implementations, etc)
3. Throw PlatformNotSupported for the FSW and kick the can down the road a bit

If we want a robust implementation, I'd vote for looking into option 2 since my initial investigation showed that there are a few third-party libraries (runtime and compile time) that help with this problem...it would require customers install packages to use the FSW, however, but we already have precedence for that (OpenSSL for example).

Thoughts?

[saper]

kevent/kqueue is the best way and it's pretty easy in my opinion. So I'd go for option dotnet/corefx#1.

[jonmill]

We have this throwing PlatformNotSupported right now but since FreeBSD work is mainly being done by the community, adding the Up-For-Grabs label

[janhenke]

Fair enough. I would still like to see the FreeBSD port completed. Is there any strong opnion about the three possibilities you mentioned above?

[jonmill]

@janhenke I think the kqueue implementation would be the best bet; it would be functionally similar to the FileSystemWatcher.Linux implementation but with FreeBSD nuances; it feels like the best option here unless FreeBSD has some special APIs (like OSX), but I don't know enough about FreeBSD to know that :)

[janhenke]

I would then grab this one and try to come up with a kqueue based implementation.

[krytarowski]

http://dmitrymatveev.co.uk/gsoc11 inotify emulation for BSD

[krytarowski]

@janhenke can you please try to reuse libinotify with CoreFX on FreeBSD? It's kevent(2)/kqueue(2) based and MIT licensed.

[janhenke]

@krytarowski I'll look at it.

[janhenke]

Sorry, I have to give this one back. My private life does not leave enough time for seriously getting into this.

[krytarowski]

I should be there soon with NetBSD

[sec]

@krytarowski What do you mean "soon" ? :)

[krytarowski]

@sec, it was blocked by lack of functional process plugin in LLDB on NetBSD - I need it to debug crashes in managed code.

[josteink]

As far as I can tell, the LLDB SOS plugin should now be functional (based on superficial testing on FreeBSD). Hopefully that will assist further progress.

[RussellHaley]

Filewatchers should definitely use the native kqueues. You can check the mono implementation for details, but there are bugs mono that the guys on the FreeBSD-mono@ mailing list have tried to patch before. The issue is that the mono implementation targets OSX, not FreeBSD and there are some implementation differences. This is third hand information though.

[ghost]

@cschuber, this one.
It would be nice to implement it using kevent/kqueue, point#1 from https://github.com/dotnet/corefx/issues/2046#issuecomment-146043297, instead of introducing dependency on inotify. Now that corefx native sources are all converted to C to share code with mono, so it might be possible to even port the filewatcher implementation from mono: https://github.com/mono/mono/blob/eec6abca721a189ee4f0b1646e73248e24319a95/mono/metadata/filewatcher.c (i think bi-direcitonal code sharing story is on, probably @marek-safar can confirm). This way, kqueue implementation of filewatcher can thrive on its own, and multiple OS like freebsd, netbsd and in future dragonfly and solaris can take advantage of it.

[ghost]

some of the issues with os-freebsd tag can also have os-netbsd as it has the same feature parity afaict.

[wfurt]

related to dotnet/corefx#30600 FileSystemWatcher.OSX consumes way too many threads

[Duffyx]

For your information into choosing kqueue:
https://forums.freebsd.org/threads/inotify-for-freebsd.38162/page-2#post-317478

Although, a developer of Intel is doing some work on FreeBSD for a more scalable implementation.
This is on his todo list: Scalability | inotify/fsevents equivalent. | Kqueue doesn't scale well.

I'm not saying you should stop the implementation using kqueue altough in the future it might make more sense to use the new inotify/fsevents equivalent.

[josteink]

If the future looks like inotify, why not just use libinotify for now?

Surely the FreeBSD code there will be improved to support native inotify-like behavior once landed?

Or am I missing something? Is depending on another third party library such a big problem?

[krytarowski]

What is future? NetBSD supports kevent(2), the same like all modern BSDs.

[Duffyx]

If the future looks like inotify, why not just use libinotify for now? -> Good point.
I don't recall that depending on a third party library is a problem at all.
Just sharing some information I picked up on mailinglist/forums. (I'm not a dev :-))

[krytarowski]

This is a biased linux-centric point of view. BSD use kevent(2)/kqueue(2) and it's not going to go away.

[krytarowski]

Performance concerns are questionable. Also this thread is from 2016 (but someone picked it recently).

[josteink]

This is a biased linux-centric point of view.

OK. Fair enough. You can call it that. If it meant we could borrow code from mono, or the Linux-codebase I'd rather call it pragmatism though.

As you may or may not know, when I (as one of total 3-4 individuals) set out back in 2015 to try to port .NET Core to FreeBSD the ambition was never for the port to be perfect on first take. The goal was to get something running, so that we had a base to build and improve on.

An example of this was pthread-performance in CoreCLR on FreeBSD. It wasn't like on other platforms. But rather than making it a show-stopper, we left it as something to look into and improve later.

I think we got to around 95-97% complete in pretty short time before a turn of events in my life meant I could no longer keep contributing, and from there on it never really caught the same pace again.

That still saddens me somewhat.

BSD use kevent(2)/kqueue(2) and it's not going to go away.

My point was more that if we could land this faster by using libinotitfy now, we would improve on it later, when we had something working to ship.

Also this thread is from 2016

Oh believe me. I know. This was one of the few remaining bits to have (at the time) complete support for both CoreCLR and CoreFX.

[krytarowski]

I can tell that I've ported all of existing CoreCLR/CoreRT and almost all of CoreFX code to NetBSD. At some point in time it was superior to all other BSDs together... however I'm still preempted by toolchain work (LLDB-centric, sanitizers, kernel sanitizers etc).

Beware a fight of things like kevent vs epoll.

And quoting 'intel' a linux-centric company is ridiculous, ask them to share CPU errata for their features with BSDs.

[saper]

The issue is about the number of file descriptors required to watch a large number of files/directories.

libinotify will not fix the problem, because it uses kqueue(2) internally and fakes inotify call.

[wfurt]

I agree with @josteink about basic functionality and improvements. Choosing obviously wrong path would probably be waste of effort but I think performance and scalability could come after having functional implementation. I updated build so it will create stub returning PNSP exception.
Think about it as first buggy implementation.

BTW Latest FreeBSD port effort is not mostly at https://github.com/dotnet/coreclr/issues/18067
There are still some test failures in corefx but @mateusrodrigues got PowerShell running on FreeBSD.
That seem better than hello world.
We have functional msbuild and we can build almost all components on FreeBSD.
My goal is to get daily CI builds.

[wfurt]

BTW we are looking for contributors and alpha users. If anybody has cycles to help, please let me know.

[saper]

Is Microsoft contracting freelance corefx developers?

[krytarowski]

At least the NetBSD porting one is one-man-show, but I would get some help. We have already ported NetBSD to Azure (but still not merged with the mainline), I got LLDB functional enough etc.. I still don't have a Linux-KVM equivalent in order to bootstrap from a Linux guest the toolchain.. but it's on my roadmap before returning to .NET.

[wfurt]

You may be able to use FreeBSD toolchain + rebuild native parts for NetBSD @krytarowski
Linux managed code heavily depends on /proc and other Linux specific functions.

[radovanovic]

It might be useful to someone - this is how I implemented it long time ago for mono https://github.com/radovanovic/monobsd/blob/freebsd/mcs/class/System/System.IO/FreeBSDWatcher.cs (it was quite difficult to push it upstream since they don't care about *BSD).

Here is also simple test I wrote when working on it https://github.com/radovanovic/monobsd/blob/freebsd/mono/tests/fs_watcher.cs (I used this test to discover how MS implementation on Windows behaved and I was using that as guide how to implement it on FreeBSD, apparently mono implementation on all OS deviated from it in some respects).

[wolfspider]

I think kevent/kqueue is the way to go also. FoundationDB for FreeBSD uses kevent/kqueue as the alternative to the Linux inotify and it's mostly a Linux software. The Darwin version is almost identical to the FreeBSD version in ports right now. I have somewhat of an unofficial dev branch of FDB with Python and NodeJS bindings. I guess CoreFX interop for Darwin is using FSEventStream it would be cool if it used kqueue and kevents as well.

[RussellHaley]

@wfurt - @radovanovic's copy looked good to me. I compared it to others I had seen in bugzilla (not that I'm an expert). I patched and built it with mono but never ran the tests.

I'd run this and move on until there's an issue. The FreeBSD community is expecting you to use kqueues; I don't think that should be overlooked in this discussion. It's not Linux.

Thanks for putting this forward yet again @radovanovic.

[radovanovic]

If somebody pings me, or sends notification on mailing list when we can build CoreFX (without watcher) entirely on FreeBSD, I will give my best to find free day to make watcher work on CoreFX - I think it should be relatively straight forward.

[krytarowski]

You may be able to use FreeBSD toolchain + rebuild native parts for NetBSD @krytarowski
Linux managed code heavily depends on /proc and other Linux specific functions.

I've finally managed to fill this gap and I got Linux/Windows to run on NetBSD with hardware-accelerated emulation and I wrote a quick summary on it:

http://blog.netbsd.org/tnf/entry/the_hardware_assisted_virtualization_challenge

Now I need to get LLDB fully functional in order to investigate future crashes of .NET on NetBSD... working on it.

[davidchisnall]

We've had a number of discussions on adding this kind of functionality to FreeBSD. The audit framework does provide a mechanism for monitoring file creation / modification, but it's not a best fit for this because audit events must not be lost and so there are a lot of constraints on the things that are allowed to monitor them (i.e. backpressure from the things watching audit events can handle this). Apple's FSEvents framework has a fallback mechanism that says 'sorry, you dropped messages, you need to go and re-scan things for changes - go and see if anything in the tree that you care about has a modification time after you started dropping messages', which lets the kernel stream events as fast as they happen and makes it the monitor's job to handle the case when they're not scheduled fast enough to handle them (for example, if a high-priority process goes and touches every file on your filesystem and preempts the low-priority monitor process).

This API is further complicated on any POSIX system by the fact that a directory is not the canonical location of a file. On NTFS and HPFS+, there is a canonical path for any file and this provides a location for hooking event delivery. In a traditional UNIX system, any file has N hard links (where N can be zero - it's valid to create a file and remove it from the file system, while still keeping it open), and there isn't a canonical root. This is further complicated by the existence of chroot / jails (where the path used to open the file may not be the location of the file in its current filesystem) or capsicum (where the process opening the file may not have access to the global namespace at all). Designing a good API that can be implemented in the kernel with these constraints is very difficult.

FreeBSD includes the filemon interface for tracing filesystem events by child processes, but this is not really a general-purpose solution (it's intended for a build system to be able to accurately track all of the dependencies of each build step).

If anyone has a good design for an implementable API that the kernel can expose, there's interest in the FreeBSD community to provide something...

[am11]

Would it make sense to tackle this old issue pragmatically for now? For example, port of KeventWatcher implementation from mono over to CoreFX, where the following options are available at https://github.com/mono/mono/blob/master/mcs/class/System/System.IO/, followed by CoreFX style adaptations.

• DefaultWatcher.cs
• FAMWatcher.cs
• FileSystemWatcher.cs
• KeventWatcher.cs
• NullFileWatcher.cs

To be able to build CoreFX recent master on FreeBSD, I tweaked some scripts and versions in CoreFX to ultimately use FreeBSD v3.x preview listed at https://github.com/dotnet/core-sdk/blob/master/README.md. However, CoreFX is currently using v2.2 of the SDK, and due to some API differences, I could not make much progress. Is there a patch available that we can apply on top of CoreFX master to build with SDK v3.x (any OS)?

[wolfspider]

@am11 If you move the 2.2 SDK folder into v3 does it not find it or is it not building? What happens when you try to build it?

[am11]

@wolfspider, with minimal patch: https://github.com/dotnet/corefx/compare/master...am11:freebsd-ci, it downloads the SDK for FreeBSD, but subsequently fails while executing FindBestConfigurations MSBuild task:

excerpt from: https://cirrus-ci.com/task/5635235873030144

expand..

  Tool 'dotnet-reportgenerator-globaltool' (version '4.0.5') was successfully installed.
/root/.nuget/packages/microsoft.dotnet.build.tasks.configuration/1.0.0-beta.19205.6/build/Microsoft.DotNet.Build.Tasks.Configuration.targets(106,5): error MSB4018: The "FindBestConfigurations" task failed unexpectedly. [/tmp/cirrus-ci-build/external/dir.proj]
/root/.nuget/packages/microsoft.dotnet.build.tasks.configuration/1.0.0-beta.19205.6/build/Microsoft.DotNet.Build.Tasks.Configuration.targets(106,5): error MSB4018: System.ArgumentException: Unknown value 'freebsd' found in configuration 'netcoreapp-freebsd-Debug-x64'.  Expected property 'OSGroup' with one of values Windows_NT, Unix, Linux, OSX, FreeBSD, NetBSD, WebAssembly, AnyOS. [/tmp/cirrus-ci-build/external/dir.proj]
/root/.nuget/packages/microsoft.dotnet.build.tasks.configuration/1.0.0-beta.19205.6/build/Microsoft.DotNet.Build.Tasks.Configuration.targets(106,5): error MSB4018:    at Microsoft.DotNet.Build.Tasks.Configuration.ConfigurationFactory.ParseConfiguration(String configurationString, Boolean permitUnknownValues, Configuration baseConfiguration) in /_/src/Microsoft.DotNet.Build.Tasks.Configuration/src/Configuration/ConfigurationFactory.cs:line 230 [/tmp/cirrus-ci-build/external/dir.proj]
/root/.nuget/packages/microsoft.dotnet.build.tasks.configuration/1.0.0-beta.19205.6/build/Microsoft.DotNet.Build.Tasks.Configuration.targets(106,5): error MSB4018:    at Microsoft.DotNet.Build.Tasks.Configuration.FindBestConfigurations.Execute() in /_/src/Microsoft.DotNet.Build.Tasks.Configuration/src/FindBestConfigurations.cs:line 40 [/tmp/cirrus-ci-build/external/dir.proj]
/root/.nuget/packages/microsoft.dotnet.build.tasks.configuration/1.0.0-beta.19205.6/build/Microsoft.DotNet.Build.Tasks.Configuration.targets(106,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [/tmp/cirrus-ci-build/external/dir.proj]
/root/.nuget/packages/microsoft.dotnet.build.tasks.configuration/1.0.0-beta.19205.6/build/Microsoft.DotNet.Build.Tasks.Configuration.targets(106,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/tmp/cirrus-ci-build/external/dir.proj]
Build FAILED.
/root/.nuget/packages/microsoft.dotnet.build.tasks.configuration/1.0.0-beta.19205.6/build/Microsoft.DotNet.Build.Tasks.Configuration.targets(106,5): error MSB4018: The "FindBestConfigurations" task failed unexpectedly. [/tmp/cirrus-ci-build/external/dir.proj]
/root/.nuget/packages/microsoft.dotnet.build.tasks.configuration/1.0.0-beta.19205.6/build/Microsoft.DotNet.Build.Tasks.Configuration.targets(106,5): error MSB4018: System.ArgumentException: Unknown value 'freebsd' found in configuration 'netcoreapp-freebsd-Debug-x64'.  Expected property 'OSGroup' with one of values Windows_NT, Unix, Linux, OSX, FreeBSD, NetBSD, WebAssembly, AnyOS. [/tmp/cirrus-ci-build/external/dir.proj]
/root/.nuget/packages/microsoft.dotnet.build.tasks.configuration/1.0.0-beta.19205.6/build/Microsoft.DotNet.Build.Tasks.Configuration.targets(106,5): error MSB4018:    at Microsoft.DotNet.Build.Tasks.Configuration.ConfigurationFactory.ParseConfiguration(String configurationString, Boolean permitUnknownValues, Configuration baseConfiguration) in /_/src/Microsoft.DotNet.Build.Tasks.Configuration/src/Configuration/ConfigurationFactory.cs:line 230 [/tmp/cirrus-ci-build/external/dir.proj]
/root/.nuget/packages/microsoft.dotnet.build.tasks.configuration/1.0.0-beta.19205.6/build/Microsoft.DotNet.Build.Tasks.Configuration.targets(106,5): error MSB4018:    at Microsoft.DotNet.Build.Tasks.Configuration.FindBestConfigurations.Execute() in /_/src/Microsoft.DotNet.Build.Tasks.Configuration/src/FindBestConfigurations.cs:line 40 [/tmp/cirrus-ci-build/external/dir.proj]
/root/.nuget/packages/microsoft.dotnet.build.tasks.configuration/1.0.0-beta.19205.6/build/Microsoft.DotNet.Build.Tasks.Configuration.targets(106,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() [/tmp/cirrus-ci-build/external/dir.proj]
/root/.nuget/packages/microsoft.dotnet.build.tasks.configuration/1.0.0-beta.19205.6/build/Microsoft.DotNet.Build.Tasks.Configuration.targets(106,5): error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) [/tmp/cirrus-ci-build/external/dir.proj]
    0 Warning(s)
    1 Error(s)
Time Elapsed 00:00:30.48

This was tested on FreeBSD 11.2. If we switch to FreeBSD 12.0 in CirrusCI YAML, we need to apply @wfurt's LD_PRELOAD workaround from https://github.com/dotnet/corefx/issues/1626#issuecomment-459842620. The reason why we still need to apply this workaround is because 3.0.100-preview-009812 is the last release for FreeBSD (from Jan).

expand..

--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -1,5 +1,5 @@
freebsd_instance:
-  image: freebsd-11-2-release-amd64
+  image: freebsd-12-0-release-amd64

freebsd_build_task:

@@ -10,4 +10,4 @@ freebsd_build_task:

build_script:
    - ./src/Native/build-native.sh   # this is redundant; just for testing native assets build beforehand
-    - ./build.sh
+    - LD_PRELOAD=/usr/lib/libpthread.so ./build.sh

and it gives a different error about missing openssl (although there is no change in the package install step in .cirrus.yml):

expand..

Welcome to .NET Core!
---------------------
GitHub: https://github.com/dotnet/core
Docs: https://aka.ms/dotnet-docs
HTTPS: https://aka.ms/aspnet-core-https
Telemetry statement: https://aka.ms/dotnet-cli-telemetry
System.TypeInitializationException: The type initializer for 'Crypto' threw an exception. ---> System.DllNotFoundException: Unable to load shared library 'System.Security.Cryptography.Native.OpenSsl' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: Shared object "libSystem.Security.Cryptography.Native.OpenSsl" not found, required by "dotnet"
at Interop.Crypto.GetMaxMdSize()
at Interop.Crypto..cctor()
--- End of inner exception stack trace ---
at Interop.Crypto.EvpSha256()
at Internal.Cryptography.HashProviderDispenser.CreateHashProvider(String hashAlgorithmId)
at System.Security.Cryptography.SHA256.Implementation..ctor()
at System.Security.Cryptography.SHA256.Create()
at Microsoft.DotNet.Cli.Telemetry.Sha256Hasher.Hash(String text)
at Microsoft.DotNet.Cli.Telemetry.Sha256Hasher.HashWithNormalizedCasing(String text)
at Microsoft.DotNet.Cli.Utils.ApplicationInsightsEntryFormat.<>c__DisplayClass10_0.<WithAppliedToPropertiesValue>b__1(KeyValuePair`2 p)
at System.Linq.Enumerable.ToDictionary[TSource,TKey,TElement](IEnumerable`1 source, Func`2 keySelector, Func`2 elementSelector, IEqualityComparer`1 comparer)

[wfurt]

I think minimal implementation would be good. Having same behavior as Mono is also good as we are heading to convergence in general between corefx and Mono.

As far as the build: I need to update build instructions.
While back when I tested it, I could not build corefx with 3.0 even of supported platforms. That may have changed. As 3.0 looms closer I was unable to find enough time to keep everything up to date.

[josteink]

This was tested on FreeBSD 11.2. If we switch to FreeBSD 12.0 in CirrusCI YAML, we need to apply @wfurt's LD_PRELOAD workaround from dotnet/runtime#14537 (comment).

I did submit a fix for this. Has there been no new “official” SDK build since then, or was the fix insufficient?

[am11]

Has there been no new “official” SDK build since then

Seems like this is the case; FreeBSD was disabled in official SDK build matrix on January 8 due to some problems: https://github.com/dotnet/core-sdk/issues/248.

---

I made a slight progress by applying https://github.com/dotnet/corefx/pull/36766. The next errors are related to missing (preview5) runtime packages, ILAsm, CoreCLR and friends:

error NU1102: Unable to find package runtime.freebsd-x64.Microsoft.NETCore.ILAsm with version (>= 3.0.0-preview5-27608-73)
..
error NU1102: Unable to find package runtime.freebsd-x64.Microsoft.NETCore.Runtime.CoreCLR with version (>= 3.0.0-preview5-27608-73)

To get unblocked, I think there are at least two possibilities:

1. bootstrap new OS using source-build: https://github.com/dotnet/source-build/blob/master/Documentation/boostrap-new-os.md (defaults seem to be Linux-centric: https://cirrus-ci.com/task/6306235459567616)
2. lobby dotnet/core-setup#5083 for internal build leg, based on @dagood's comment: (https://github.com/dotnet/core-sdk/issues/248#issuecomment-481728506). 8-)

[wfurt]

This goes all the way to https://github.com/Microsoft/azure-pipelines-agent/pull/1906
We need the agent to run the builds and without it it hard to push on infrastructure.

[danmosemsft]

We can keep tracking this but it's not aligned to work on the 3.0 schedule

[josteink]

So we finally went with inotify instead of kqueue, just 4-5 years after I suggested so, because nobody wanted to implement the kqueue-version.

Oh well. Better late than never. 😄

Thanks @wfurt ! Good job!

[wfurt]

This is mostly because @rootwyrm made me realized how easy it is. So we get some functionality now and we can improve if somebody bumps to issues.
There is now wave of new effort to stabilize around 3.x and get source-build working for freebsd. I'll probably post update on main freebsd issue as it would be great to get help from anybody who has any time left. Even if that is to follow steps and provide feedback.

[rootwyrm]

Ha, I will gladly take the blame here, and also deliver the very important caveats.

The reason kqueue should remain preferred as a long term goal is that it's several orders of magnitude faster. libinotify is just a shim and may not be maintained. There's also the caveat of still missing O_SYMLINK. So IN_ATTRIB, IN_MODIFY, IN_MOVE_SELF, and IN_DELETE_SELF don't work. If you all would consider these important, I can see about making noises about finally accepting the kernel patches that fix this.