Resque: Resque fork + exit! incompatible with default Rails 5 development config on OSX

[jtokoph]

This week I've been noticing a lot (hundreds) of seemingly orphaned fsevent_watch processes accumulating on my system while resque workers were running. Each time a job was processed about 30 more processes would accumulate. (Note that it's normal for there to be about 30 processes (one per directory being watched), just not multiples of the 30 processes)

Notice the second batch of processes have a parent of 1 (launchd) and process group id of the resque master process

  PID  PPID  PGID COMM
76554 63443 76554 resque-1.26.0: Waiting for play_arena,test,tick
76557 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76558 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76559 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76560 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76561 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76562 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76563 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76564 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76565 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76566 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76567 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76568 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76569 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76570 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76571 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76572 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76573 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76574 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76575 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76576 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76577 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76578 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76579 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76580 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76581 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76582 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76583 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76584 76554 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76585 76554 76554 (sysctl)
76589     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76590     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76591     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76592     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76593     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76594     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76595     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76596     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76597     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76598     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76599     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76600     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76601     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76602     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76603     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76604     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76605     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76606     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76607     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76608     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76609     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76610     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76611     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76612     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76613     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76614     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76615     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch
76616     1 76554 /Users/jtokoph/.rvm/gems/ruby-2.3.1@arenas5/gems/rb-fsevent-0.9.7/bin/fsevent_watch

I've narrowed down the issue to the combination of the new default file watcher in Rails 5 development environments (ActiveSupport::EventedFileUpdateChecker) and that resque exits from worker forks with exit! instead of exit.

The result of this issue is that I'm unable to spawn any new userland processes until all of these orphaned processes are cleaned up due to the numproc limits being hit (fork: Resource temporarily unavailable)

For reference, the new evented file watcher can be found here.

It uses the Listen gem to use the correct change watching system per platform. On OSX, the listen gem spawns a utility called fsevent_watch to watch each app directory. It ultimately spawns these in each fork so that every fork can be updated when files change.

The issue arises because each worker exits with exit! instead of exit which seems to disable the proper cleanup of any spawned processes. So every job fork causes a bunch of fsevent_watch processes to spawn, and those processes don't get properly cleaned up upon exit!.

I've been able to solve this for myself by setting the RUN_AT_EXIT_HOOKS environment variable when running resque:work.

RUN_AT_EXIT_HOOKS=yes rake environment resque:work QUEUE=*

While this may or may not be a bug, I think this is going to be an issue that tons of people are going to start running into now that Rails 5 has been officially released. I would suggest that a fix somewhere between resque/rails/listen be developed but I'm not sure what the proper fix would be.

In the meantime I think the RUN_AT_EXIT_HOOKS env variable should be better documented in the readme if it's the proper solution for now.

Related materials I've stumbled upon in my investigation:
rails/rails#22312
rails/rails@a75496b
resque/resque@2d4f8e1
resque/resque#862
resque/resque#878
resque/resque#1167

[msquitieri]

I'm having the same issue as well. Running resque on OS X 10.11.6 with a new Rails 5.0.0.1 project causes my entire machine to come to a crawl. I can't even open a new terminal window. Is there a fix pending?

[goalaleo]

EDIT: not a fix but a workaround

If you have gem listen in the project here's a simple fix that worked for me (running Rails 5 on OS X):

in development.rb change config.file_watcher = ActiveSupport::EventedFileUpdateChecker to config.file_watcher = ActiveSupport::FileUpdateChecker, and remove following gems:
listen and spring-watcher-listen <- if using. From other issues it seems like the culprit might be rb-fsevent gem which listen depends on.

[koenpunt]

@goalaleo that's a workaround, not a fix, because FileUpdateChecker uses polling, which is less responsive than EventedFileUpdateChecker

[goalaleo]

@koenpunt you're correct, wrong choice of words

[yaauie]

There appears to be an upstream fix in the rb-fsevent library that is being used by listen: https://github.com/thibaudgg/rb-fsevent/releases/tag/v0.9.8

[chrisccerami]

Is this issue resolved?

[sblackstone]

I think this issue is having several rails processes running in development mode at the same time - each one boots up its own set of processes until we reach an OS-level limit.

[marcoribi]

Is a fix available?

[jeremy]

Fixed by https://github.com/thibaudgg/rb-fsevent/pull/62 which ensures zombie processes self-terminate even when their parent is killed.