Runtime: FileSystemWatcher tests fail intermittently on Unix due to too many open inotify instances

[tmat]

http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/centos7.1_release_tst_prtest/1094/console

19:32:38    FileSystemWatcherTests.FileSystemWatcher_EnableRaisingEvents [FAIL]
19:32:38       System.IO.IOException : The configured user limit (128) on the number of inotify instances has been reached.
19:32:38       Stack Trace:
19:32:38             at System.IO.FileSystemWatcher.StartRaisingEvents()
19:32:38             at FileSystemWatcherTests.FileSystemWatcher_EnableRaisingEvents()
19:32:38 Finished:    System.IO.FileSystem.Watcher.Tests

[stephentoub]

By default the number of inotify instances allowed is relatively low (128), and we use one to back FileSystemWatcher. If the machine on which the tests are running happens to be doing other things at the same time that creates inotify instances, we could bump up against this.

@mmitche, can we up the default limits on the machines used in CI? I believe this can be done by adding a line like the following to the end of the /etc/sysctl.conf file:

fs.inotify.max_user_instances=1024

[mmitche]

@stephentoub Yeah....does that need to be updated on all of the *nix machines?

[stephentoub]

Thanks, Matt. Only the Linux ones.

[mmitche]

@stephentoub This is being done on the new dynamically allocated image

[stephentoub]

Thanks.

[justinvp]

Ran in to this again here:

http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/ubuntu_debug_tst_prtest/2902/consoleFull#-21371504001f1a4601-6aec-4fd5-b678-78d4389fd5e8

02:18:22    FileSystemWatcherTests.FileSystemWatcher_WatchingAliasedFolderResolvesToRealPathWhenWatching [FAIL]
02:18:22       System.IO.IOException : The configured user limit (1024) on the number of inotify instances has been reached.
02:18:22       Stack Trace:
02:18:22             at System.IO.FileSystemWatcher.StartRaisingEvents()
02:18:22             at System.IO.FileSystemWatcher.StartRaisingEventsIfNotDisposed()
02:18:22             at System.IO.FileSystemWatcher.set_EnableRaisingEvents(Boolean value)
02:18:22             at FileSystemWatcherTests.FileSystemWatcher_WatchingAliasedFolderResolvesToRealPathWhenWatching()

And here:

http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/ubuntu_release_tst_prtest/2920/consoleFull#-21371504001f1a4601-6aec-4fd5-b678-78d4389fd5e8

01:52:13    FileSystemWatcherTests.FileSystemWatcher_StopCalledOnBackgroundThreadDoesNotDeadlock [FAIL]
01:52:13       System.IO.IOException : The configured user limit (1024) on the number of inotify instances has been reached.
01:52:13       Stack Trace:
01:52:13             at System.IO.FileSystemWatcher.StartRaisingEvents()
01:52:13             at FileSystemWatcherTests.FileSystemWatcher_StopCalledOnBackgroundThreadDoesNotDeadlock()

[stephentoub]

@mmitche, is it possible that multiple corefx test runs might be running on the same CI machine at the same time? The error message highlights that the larger limit is being used, but we're still hitting it, which implies to me either a) we're not properly disposing some of these, b) multiple jobs are running concurrently on the same machine and some of our tests that push these values to the limit are causing problems for other test processes, or c) something else on the machine is using a ton of inotify instances.

cc: @sokket

[mmitche]

@stephentoub Except for OSX, no.

[stephentoub]

Again here: http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/ubuntu_debug_tst_prtest/3099/console

[stephentoub]

Again here: http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/ubuntu_debug_tst_prtest/3111/console

[stephentoub]

Again here: http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/opensuse13.2_debug_tst/936/console

[joshfree]

@ianhays can you take a look since you're already working on FSW related test issues?

[ianhays]

In https://github.com/dotnet/corefx/pull/8231 I added code to guard against failing when a test hits the inotify limit, so the FSW tests should be fine. It's still possible to hit the limit but it's significantly less likely now, especially in an innerloop run.

I'm closing this as (presumably) resolved for now.

[ericeil]

This this again here: http://dotnet-ci.cloudapp.net/job/dotnet_corefx/job/master/job/outerloop_ubuntu14.04_debug_prtest/22/

[ianhays]

It makes sense that we're hitting this limit again on Outerloop runs. If we're hitting it on innerloop then something else is wrong.

We're hitting it on Outerloop runs because the CreateManyConcurrent test is creating new watchers up to the inotify limit to verify the exception that we throw in that case. The problem is that we run tests in parallel and if another test tries to make a new FSW during or shortly after the CreateManyConcurrent test then it will unexpectedly fail by no fault of its own.

This will be resolved by https://github.com/dotnet/corefx/pull/11721

[ericeil]

Yes, this was an Outerloop run.