Ember-cli: Watchman problem after recent update (0.2.4)

[jtsom]

After updating to ember-cli 0.2.4, when I start an app I'm seeing watchman print out a bunch of warnings / errors before the app boots up:

version: 0.2.5
Livereload server on port 35729
Serving on http://localhost:4200/
Recrawled this watch 1 times, most recently because:
/Users/john/Devel/ember/borrowers: kFSEventStreamEventFlagUserDropped
To resolve, please review the information on
https://facebook.github.io/watchman/docs/troubleshooting.html#recrawl

These lines repeat many times before the app starts. The app seems to work ok, as far as I can tell. This didn't happen before 0.2.4, and is still happening with 0.2.5.

I can get it to stop, at least temporarily, by stopping the watchman process with:
watchman shutdown-server

Thanks

[stefanpenner]

This should also be opened as a watchman issue

Cc @wez thoughts?

[cigoe]

I am getting this as well this morning after a 0.2.1 to 0.2.3 update

version: 0.2.3

A new version of ember-cli is available (0.2.5). To install it, type ember update.
Livereload server on port 40529
Serving on http://localhost:4200/app/
Recrawled this watch 1 times, most recently because:
/Users/chris/devel/web/test: kFSEventStreamEventFlagUserDropped
To resolve, please review the information on
https://facebook.github.io/watchman/docs/troubleshooting.html#recrawl

[jtsom]

I had posted it here, because I hadn't seen it until I did an ember-cli update. Watchman hasn't been updated, as far as I know.

[quiddle]

I ran into the recrawling issue as well. I ran watchman shutdown-server and then ember-cli went back to being awesome.

[wez]

That message is telling you that your system is having trouble keeping up with the filesystem notifications.

Can you provide more details on the size of the tree, number of watches and how frequently you are updating them? Can you also give us an idea of how powerful the system is (CPU, RAM, whether the disk is a spinning laptop disk, SSD or a high rpm spinning disk)

The output from:

watchman version
watchman watch-list

and making a gist of your watchman log file would also be useful. (If you installed via homebrew it will be something like /usr/local/Cellar/watchman/3.1/var/run/watchman/$USER.log)

@stefanpenner: is 0.2.4 and later using watch-project? Might be a good idea to suggest that users delete all watches as part of the upgrade, assuming that this is another case where excessive number of watches is causing a system performance problem.

[stefanpenner]

@wez yes it is. One thing to note, is that users may still have some projects using old versions.

I will make a note in the release notes. Does the latest watchman have the delete command or should I offer my existing she'll script?

[calderas]

I'm having the same issues on 0.2.3 and 0.2.5
Here's my log file: https://gist.github.com/calderas/87f7c36dc6d151cb67ac

Running watchman shutdown-server does the trick.

[runspired]

I'm not seeing these particular errors, but I do strongly suspect watchman is leading to kernel panics on my mac. I'd love to know if anyone else is bumping into that.

[wez]

@calderas: how many files and dirs do you have in that tree? Are there large portions that change often?

@stefanpenner @calderas' log file only shows a single watch, so I think we should tackle that separately. The watch-purge command isn't in 3.1 and we haven't rolled 3.2 yet, so it would be awkward to direct folks to use it, and may not be effective. Let's try to get some more data on this.

[jtsom]

It's a fairly small app - it's actually here:

https://github.com/jtsom/borrowers

it's from the Ember-cli 101 book.

As far as changes, the messages display as soon as the app is launched, so I haven't even edited any files yet.

And again, after doing the watchman shutdown-server, it doesn't seem to happen again.

I did see it happening after updating the app's ember-cli

This is on a Mid 2013 Macbook Air, with SSD, 1.7Ghz,, 8gb memory

watchman version
{
    "version": "3.1.0"
}

watchman watch-list
{
    "version": "3.1.0",
    "roots": [
        "/Users/john/Devel/ember/borrowers"
    ]
}

[calderas]

@wez: here's the file tree and files per dir.
Also, I only see this error when running ember server which updates the tmp folder

./app - 633
./bower_components - 1584
./config - 1
./dist - 132
./docs - 1
./node_modules - 40112
./public - 117
./server - 5
./tests - 134
./tmp - 3401

[wez]

Interesting note about ember server. If you watch activity monitor while it runs, is the CPU and/or disk heavily loaded?

@stefanpenner: does ember server create and delete a lot of files in a short time in the tmp folder?
Are there deep trees with many files being created and deleted under tmp?

[stefanpenner]

@stefanpenner: does ember server create and delete a lot of files in a short time in the tmp folder?
Are there deep trees with many files being created and deleted under tmp?

yes.

We had discussed this in IRC some time ago, and the answer to my rapidly changing tmp dir concern was watchman will handle it fine. As it handles large rebases on the facebook code-base without trouble.

I suspect this may actually not be the case. Although testing on my a mega-sized code-base at work didn't yield these same issues. Maybe my workstation has more power...

@wez would adding an explicit ignore for tmp via the .watchmanconfig be of any benefit here?
It should not be explicitly watched, rather it just happens to be in the watch-project root.

I believe I could instead make multiple watch-project roots instead, which is still better then our current situation although not as ideal as a single root.

• bower_components
• vendor
• node_modules
• app
• tests

Also note, we do have plans to move the tmp dir out of the project directory, but are pending some somewhat non-trivial broccoli work.

@rwjblue / @joliss can you share the status of this effort?

[mikegrassotti]

FWIW I'm seeing same issue when running ember server. Project is about a year old, started seeing this a few days ago. Happens on multiple machines, both new MB Pros with SSD drives. Build times are pretty fast (about 12s first pass, 5s after changes) and CPU utilization seems normal. Project tree looks like this:

app: 278
config: 1
bower_components: 970
dist: 361
node_modules: 29479
tests: 100
vendor: 4

$ ember server
version: 0.2.3
Serving on http://localhost:4200/
Recrawled this watch 45 times, most recently because:
/Users/mpg/code/lf/lift: kFSEventStreamEventFlagUserDropped

$ watchman version
{
    "version": "3.1.0"
}

$ watchman watch-list
{
    "version": "3.1.0",
    "roots": [
        "/Users/mpg/code/lf/lift"
    ]
}

@stefanpenner happy to dig in further, lmk what would help...

[stefanpenner]

@mikegrassotti if you could try a .watchmanconfig that explicitly ignores tmp / node_modules and various combinations and permutation that would be handy

[mikegrassotti]

Done. Hasn't recrawled yet...

watchman get-config .
{
    "version": "3.1.0",
    "config": {
        "ignore_dirs": [
            "node_modules",
            "tmp"
        ]
    }
}

[stefanpenner]

Done. Hasn't recrawled yet...

awesome, then the fix should be relatively simple. We should add .watchmanconfig with that in the base blueprint. @wez does that make sense? I would assume watch-project would essentially ignore, non-watched child-roots by default. But maybe i mis-understood the goal of that command.

Got time for a pr ?

[calderas]

adding .watchmanconfig solved the issue for me as well :+1:

[mikegrassotti]

@calderas great! @stefanpenner gotta run will try for a PR this evening

[stefanpenner]

@calderas great! @stefanpenner gotta run will try for a PR this evening

thank you kindly sir :)

[wez]

Sorry, got caught up at work today. Can you turn up the log level while the tmp dir manipulations are happening? I'd like to understand more about what is actually happening on the filesystem; these systems all sound strong enough that we shouldn't see this.

The easiest way to capture logging information is to run this in one window:

watchman -p log-level debug > /tmp/some-file

then trigger the problem in another window. You'll want to CTRL-C the above once you're done, as it will tick over several times a second and the log file will get rather large.

Make sure that the log file is not inside any of your watched roots, or you'll have a bad time.

I think it's fine to ignore the tmp dir in .watchmanconfig in the meantime. It should not be necessary to ignore node_modules.

[stefanpenner]

I'd like to understand more about what is actually happening on the filesystem; these systems all sound strong enough that we shouldn't see this.

a brief TL;DR is: On build/rebuild broccoli does a massive amount of rapid symlinking (and sometimes, but ideally less writes if something changes)

[wez]

I couldn't reproduce this with https://github.com/jtsom/borrowers but I took a look over our fsevents code.
We're considering this change:
https://reviews.facebook.net/D38715
which may help reduce the chances of triggering this overflow condition.

[stefanpenner]

@wez neither can I :(

We're considering this change:
https://reviews.facebook.net/D38715
which may help reduce the chances of triggering this overflow condition.

interesting.

[jtsom]

As I may have mentioned, I only saw it right after I did an update to ember-cli of this app and another test one I had. Once I shut down watchman, after that the messages did not return. As a test I removed node_modules and bower_components, and reinstalled, but the messages did not return.

[ilkkao]

FWIW I still experienced repeating file added xxx (mostly my own app files) messages even after adding node_modules, bower_components, and tmp to the ignore list in watchman config.

Real fix for me was the watchman commit from @wez that is visible above

[stefanpenner]

@ilkkao ya, @wez's fix seems great for us :)

[mike-post]

I started seeing this on a brand new system with a new ember application. It was totally weird and unexpected, given that everything is new. I'm using watchman 3.3.0.

The fix was was to run watchman watch-del <project root> and to rerun ember serve. I confirmed that the watch was successfully recreated by ember-cli via watchman watch-list. No more warnings about recrawling due to kFSEventStreamEventFlagUserDropped after this.

[shaunc]

We have same symptoms and work-around that @mike-post reports.

[jpaas]

Started seeing this as soon as I upgraded to watchman 3.3.0 on OSX/Brew even with a valid .watchmanconfig file. @mike-post's suggestion hasn't worked. If I add node_modules to the ignored directories, the recrawl errors go away, but it continuously adds all the files in my project and rebuilds the app. I am back to NodeWatcher because I haven't been able to get this to work.

[stefanpenner]

If I add node_modules to the ignored directories, the recrawl errors go away, but it continuously adds all the files in my project and rebuilds the app. I am back to NodeWatcher because I haven't been able to get this to work.

this is strange, @wez any thoughts come to mind?

[jpaas]

Update: after a while, the continuous rebuilding stopped. But the next time I ran my ember server, the recrawl messages came back. Let me know if there's anything I can provide to help diagnose.

[rlivsey]

I'm no longer seeing kFSEventStreamEventFlagUserDropped issues, but if I have addons installed with isDeveloping: true set then changes in them aren't picked up by watchman, but they work fine with node watcher.

Shall I open a different issue for that or might it be related to discussion on ignoring node_modules etc…?

[jpaas]

I also have an addon with isDeveloping: true. I'm also using ember-intl 2.0.0-beta.14 which I noticed adds a bunch of folders to be watched. That's why I tried adding node_modules to my ignore_dirs.

[jtsom]

This comes back after I do an update of ember-cli. Run the update ember-cli steps and the next time starting the server I get a bunch of:

Recrawled this watch 4 times, most recently because:
/Users/john/Devel/ember/borrowers: kFSEventStreamEventFlagUserDropped
To resolve, please review the information on
https://facebook.github.io/watchman/docs/troubleshooting.html#recrawl

Doing the watchman watch-del command clears the messages... until the next ember-cli update.

[jasonmit]

@jpaas

I'm also using ember-intl 2.0.0-beta.14 which I noticed adds a bunch of folders to be watched. That's why I tried adding node_modules to my ignore_dirs.

Upgraded ember-cli on these addons recently (the folders you are likely see intl-* ember addons under node_modules which each have their own .watchmanconfig as per the ember-cli addon blueprint). None of my addons explicitly set isDeveloping: true though.

[wez]

If you're seeing kFSEventStreamEventFlagUserDropped often after deleting the watch, it is an indicator that your system is unable to keep up with the rate of changing files. You can try increasing the fsevents latency for your repo. You can do this by editing your .watchmanconfig file and setting the field:

"fsevents_latency": 0.02

This doubles the default latency; you're trading timeliness for efficiency. Keep increasing it until you find a value that works for you.

It would be great if you can share the results of your tuning along with the size (number of files and directories) of your repo, and whether you are using a spinning disk or a flash based storage device.

[wez]

For the other comments above, I don't really know anything about ember beyond it being a valued consumer of Watchman, and would appreciate it if you can give me enough context about the nature of the files in the watched tree for me to give some more useful contributions :-)

One important aspect of Watchman is that it does not resolve symlinks (see https://github.com/facebook/watchman/issues/105 for context). If the isDeveloping flag you mentioned results in an optimization that creates symlinks inside the watched tree, you won't get notified of changes to the targets of those symlinks.

If that is happening, is it possible to have ember resolve the symlink and initiate a watch-project on the dirname of the target and have that feed into its build trigger?

[stefanpenner]

For the other comments above, I don't really know anything about ember beyond it being a valued consumer of Watchman, and would appreciate it if you can give me enough context about the nature of the files in the watched tree for me to give some more useful contributions :-)

Thanks I really appreciate the support here :) If you have some free time in the next week, we can briefly sync up to provide context. I'll pop by the irc channel.

If that is happening, is it possible to have ember resolve the symlink and initiate a watch-project on the dirname of the target and have that feed into its build trigger?

Yup it is, and is actually my plan to resolve that issue. Thanks for confirming its the prescribed course of action.

I believe we should just ship with node_modules ignored by default, but @wez quick question: can i ignore node_modules, and then explicitly re-watch something within while reusing the same watch-project from the root? Or would should i watch project the select node_modules children I want to watch?

something like:

ignore: node_modules
watch: node_modules/foo/bar/baz

[wez]

Ignores are a relatively blunt instrument in watchman. There is some platform specific stuff that comes into play; on every OS except OS X the watch APIs require that we individually establish a watch on every directory in the tree. The ignore_dirs configuration tells watchman to pretend that that directory doesn't exist and avoid both recursively crawling and recursively watching that dir.

On OS X, fsevents is already recursively watching the tree and we have to manually check every path against the ignore list (typically case insensitively, depending on the filesystem) which drives up the CPU cost in the watcher thread. This is a performance hot spot; if we are too slow to collect these notifications and enqueue them to the crawling thread we risk triggering kFSEventStreamEventFlagUserDropped.

I'm not 100% sure of the ramifications of allowing "gaps" in the tree if we ignored a parent but watched a child, but I do wonder what problem we're trying to solve and whether ignores are really the right solution.

What specifically is the problem that gets solved by ignoring the node_modules dir? What are the symptoms? Is it kFSEventStreamEventFlagUserDropped, and only in some specific circumstances?

Rather than by default ignoring node_modules, I think ember may be better off using a larger fsevents_latency. This allows fsevents to batch up notifications before passing them to watchman and makes things more efficient when a large volume of files is changed in a very short time period. I suspect that you can tolerate something closer to 0.2 seconds latency which is an order of magnitude larger than the default and still small enough that it should appear virtually instantaneous to human eyes.

[wez]

Regarding symlinks, I'm thinking that watchman needs a better answer for its various clients/consumers. I'll collect some thoughts on that on https://github.com/facebook/watchman/issues/105

[mike-post]

So I'm hardly an expert (just a user), but I don't know that a lot of files are being modified at once when this occurs. I've seen it when I modify a single file. Perhaps there's some ember/broccoli stuff going on in the tmp die that I don't understand though.

[jpaas]

Adding on to what @mike-post says, I get the recrawl warning when I start the ember server at which point any actual changes I make to any files aren't recognized.

And in terms of the hardware I'm running, I have a Macbook pro with an i7 processor and SSD drive.

[wez]

@jpaas I'd like to see your watchman log file and the output from watchman watch-list if you can get that to trigger again

[jpaas]

I found something interesting. I use npm link to load a private npm module locally, but the link had been broken. The first time I started my project, I didn't get any recrawl warnings. When I realized the link was gone, I readded it, and the recrawl warnings came back.

watch-list (ember-cli-imd is the symlinked node module):

{
    "version": "3.3.0",
    "roots": [
        "/Users/jpaas/Documents/osaaru/imdhealth/admin-webapp/node_modules/ember-intl",
        "/Users/jpaas/Documents/osaaru/imdhealth/ember-cli-imd",
        "/Users/jpaas/Documents/osaaru/imdhealth/admin-webapp/node_modules/ember-intl/node_modules/ember-intl-messageformat/node_modules/ember-intl-messageformat-parser",
        "/Users/jpaas/Documents/osaaru/imdhealth/admin-webapp/node_modules/ember-intl/node_modules/ember-intl-messageformat",
        "/Users/jpaas/Documents/osaaru/imdhealth/admin-webapp/node_modules/ember-intl/node_modules/ember-intl-relativeformat",
        "/Users/jpaas/Documents/osaaru/imdhealth/admin-webapp",
        "/Users/jpaas/Documents/osaaru/imdhealth/admin-webapp/node_modules/ember-intl/node_modules/ember-intl-format-cache"
    ]
}

Log file:

2015-08-01T13:04:01,689: [listener] Using watcher mechanism fsevents
2015-08-01T13:04:01,689: [listener] file limit is 256 kern.maxfilesperproc=10240
2015-08-01T13:04:01,689: [listener] raised file limit to 10240
2015-08-01T13:04:01,689: [listener] Using socket from launchd as listening socket
2015-08-01T13:04:01,689: [listener] failed to parse json from /usr/local/Cellar/watchman/3.3.0/var/run/watchman/jpaas.state: unable to open /usr/local/Cellar/watchman/3.3.0/var/run/watchman/jpaas.state: No such file or directory
2015-08-01T13:04:07,001: [client:fd=4] path /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp is on filesystem type hfs
2015-08-01T13:04:07,388: [client:fd=13] path /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp/node_modules/ember-intl/node_modules/ember-intl-relativeformat is on filesystem type hfs
2015-08-01T13:04:07,388: [client:fd=19] path /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp/node_modules/ember-intl/node_modules/ember-intl-messageformat is on filesystem type hfs
2015-08-01T13:04:07,388: [client:fd=26] path /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp/node_modules/ember-intl/node_modules/ember-intl-format-cache is on filesystem type hfs
2015-08-01T13:04:07,388: [client:fd=22] path /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp/node_modules/ember-intl/node_modules/ember-intl-messageformat/node_modules/ember-intl-messageformat-parser is on filesystem type hfs
2015-08-01T13:04:16,383: [client:fd=183] path /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp/node_modules/ember-intl is on filesystem type hfs
2015-08-01T13:04:16,383: [client:fd=181] path /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp/node_modules/ember-intl is on filesystem type hfs
2015-08-01T13:05:37,918: [notify /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp] /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp: kFSEventStreamEventFlagUserDropped: scheduling a tree recrawl
2015-08-01T13:05:43,715: [notify /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp] /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp: kFSEventStreamEventFlagUserDropped: scheduling a tree recrawl
2015-08-01T13:06:17,314: [notify /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp] /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp: kFSEventStreamEventFlagUserDropped: scheduling a tree recrawl
2015-08-01T13:06:19,746: [notify /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp] /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp: kFSEventStreamEventFlagUserDropped: scheduling a tree recrawl
2015-08-01T13:06:24,711: [notify /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp] /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp: kFSEventStreamEventFlagUserDropped: scheduling a tree recrawl
2015-08-01T13:07:36,363: [notify /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp] /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp: kFSEventStreamEventFlagUserDropped: scheduling a tree recrawl
2015-08-01T13:07:40,181: [notify /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp] /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp: kFSEventStreamEventFlagUserDropped: scheduling a tree recrawl
2015-08-01T13:07:43,046: [notify /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp] /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp: kFSEventStreamEventFlagUserDropped: scheduling a tree recrawl
2015-08-01T13:08:38,257: [client:fd=37] path /Users/jpaas/Documents/osaaru/imdhealth/ember-cli-imd is on filesystem type hfs
2015-08-01T13:08:46,166: [notify /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp] /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp: kFSEventStreamEventFlagUserDropped: scheduling a tree recrawl
2015-08-01T13:08:51,378: [notify /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp] /Users/jpaas/Documents/osaaru/imdhealth/admin-webapp: kFSEventStreamEventFlagUserDropped: scheduling a tree recrawl

[wez]

@jpaas interesting. You have seemingly have multiple overlapping watches. Is ember-cli-imd the only one of those that is symlinked? Can you run:

cd /Users/jpaas/Documents/osaaru/imdhealth
find . -name .git -o -name .watchmanconfig
find . -type l -ls 

[wez]

I think https://reviews.facebook.net/D40653 will help with these overlapping watches

[jpaas]

find . -name .git -o -name .watchmanconfig:

./admin-webapp/.git
./admin-webapp/.watchmanconfig
./admin-webapp/node_modules/ember-can/.watchmanconfig
./admin-webapp/node_modules/ember-cli/blueprints/app/files/.watchmanconfig
./admin-webapp/node_modules/ember-cli-app-version/.watchmanconfig
./admin-webapp/node_modules/ember-cli-bootstrap-datepicker/.watchmanconfig
./admin-webapp/node_modules/ember-cli-deploy/.watchmanconfig
./admin-webapp/node_modules/ember-data-route/.watchmanconfig
./admin-webapp/node_modules/ember-export-application-global/.watchmanconfig
./admin-webapp/node_modules/ember-intl/.watchmanconfig
./admin-webapp/node_modules/ember-intl/node_modules/ember-intl-format-cache/.watchmanconfig
./admin-webapp/node_modules/ember-intl/node_modules/ember-intl-messageformat/.watchmanconfig
./admin-webapp/node_modules/ember-intl/node_modules/ember-intl-messageformat/node_modules/ember-intl-messageformat-parser/.watchmanconfig
./admin-webapp/node_modules/ember-intl/node_modules/ember-intl-relativeformat/.watchmanconfig
./ember-cli-imd/.git
./ember-cli-imd/node_modules/ember-cli/blueprints/app/files/.watchmanconfig

find . -type l -ls generates a list too long to post here I think

[runspired]

I think we may need a way to detect an addon is linked vs installed instead of relying on isDevelopingAddon.

[stefanpenner]

You have seemingly have multiple overlapping watches. Is ember-cli-imd the only one of those that is symlinked?

Overlaps are possible, although likely the result of some overzealous watch. I could detect and warn when it happens...

[jpaas]

My addon does have isDevelopingAddon set to true. Are you saying that's causing the overlapping watches? i.e. set it to true when using NodeWatcher, and false when using watchman?

[wez]

@jpaas I pushed https://github.com/facebook/watchman/commit/c9956ecb606062674db627f5dcbc3fd22f00860d this morning which should help with the overlapping watches. Would you mind trying this?

watchman watch-del-all
brew uninstall watchman
brew install --HEAD watchman

and then whatever ember specific steps are needed to re-establish the watches, and then share the output from watchman watch-list

[wez]

For clarity, there are two separate issues we're discussing here; one is recrawls (due either to mis-tuned fsevents_latency and/or overlapping watches) and the other is lack of notifications (likely due to symlinks created by npm link or similar).

I believe that the isDevelopingAddon thing is related more to symlinks than overlapping watches.

[jpaas]

Hey @wez it seems to be working and my watches look much cleaner. Thanks!

{
    "version": "3.6.0",
    "roots": [
        "/Users/jpaas/Documents/osaaru/imdhealth/ember-cli-imd",
        "/Users/jpaas/Documents/osaaru/imdhealth/admin-webapp"
    ]
}

[wez]

@jpaas great! Please let us know if you see any recrawl / kFSEventStreamEventFlagUserDropped warnings with this build

[jasonmit]

@stefanpenner should we npmignore .watchmanconfig?

@wez thanks for the fix :+1:

[stefanpenner]

@stefanpenner should we npmignore .watchmanconfig?

I hope not, I believe @wez's fix will resolve the issue.

[stefanpenner]

I think we may need a way to detect an addon is linked vs installed instead of relying on isDevelopingAddon.

yup this is the plan, the tricky bit is detecting links created which ember s is running. Lets see if we can utilize more of watchman here first.

@wez is it possible for watchman to peek into the symlinked dir, and see if a .watchmanconfig exists? If so, let watchman do its thing, otherwise do nothing?

It would seem if watchman is already watching both sides (the linked and the linker), the overhead and gotchas are potentially quite reasonable. Although I could be missing something important.

[cleverlemming]

Had same problem w/kFSEventStreamEventFlagUserDropped on OS X. Tried changing fsevents_latency.
3-step solution @wez posted fixed it.

[jsalva]

my /path/to/ember-cli/project/.watchmanconfig:

{
      "fsevents_latency": 0.2,
      "ignore_dirs": [
            "node_modules",
            "bower_components",
            "tmp",
            "dist"
      ]
}

watchman version:

> watchman watch-list
{
    "version": "3.8.0",
    "roots": [
        "/path/to/ember-cli/project/"
    ]
}

when I run

watchman shutdown-server
ember build --watch

I'm still getting

version: 1.13.8
Building.Recrawled this watch 20 times, most recently because:
/path/to/ember-cli/project/: kFSEventStreamEventFlagKernelDropped
To resolve, please review the information on
https://facebook.github.io/watchman/docs/troubleshooting.html#recrawl
To clear this warning, run:
`watchman watch-del /path/to/ember-cli/project/ ; watchman watch-project /path/to/ember-cli/project/`

along with a bunch of recrawl warnings, and finally the app builds, but does not actually rebuild when I make changes.

Build successful - 5864ms.

Slowest Trees                                 | Total               
----------------------------------------------+---------------------
AssetRewrite                                  | 2956ms              
Funnel: App JS Files                          | 558ms               

Slowest Trees (cumulative)                    | Total (avg)         
----------------------------------------------+---------------------
AssetRewrite (1)                              | 2956ms              
Funnel: App JS Files (1)                      | 558ms               
Babel (14)                                    | 367ms (26 ms)     

This is a very vexing issue I've been trying to troubleshoot for the past 2 days now. Any thoughts?

[stefanpenner]

This is a very vexing issue I've been trying to troubleshoot for the past 2 days now. Any thoughts?

several things conflating:

1. OSX is getting confused
2. watchman (and our usage of it) causing multiple warnings to spew, when only 1 is needed
3. watchman handling re-attaching differently from first run (it suppresses that warning on first run)
4. some broccoli plugins doing more IO then they should be (causing OSX to get confused)

I'm working on 4. which will help, in the interim sometimes increasing your fsevents_latency can help.

[mike-post]

For an extreme case of 4, try adding the ember-intl addon to your project. (Scroll way back to the beginning of this issue for a link issue.) I seem to recall the IO went nuts when I tried adding that to a project.

[stefanpenner]

@mike-post ya, there are several add-ons that can easily cause this symptom. But largely the route cause is:

• largely: broccoli-plugins needs to be improved: initial WIP -> https://github.com/hjdivad/broccoli-funnel/pull/2 (this also is leading to some obvious mega performance boosts)
• a small amount: watchman appears to handle some scenarios differently then others (it masks the issue in some scenarios)

[jsalva]

Is there a chance ember build --watch is ignoring my .watchmanconfig ? I set my fsevents_latency from 0.1 to 1000000.0 (incrementally by orders of magnitude in base 10), and I'm not detecting any noticeable differences in build time (and obviously no change in the appearance of kFSEventStreamEventFlagKernelDropped)

-- edit --

doesn't seem to be the case; I put every directory in the ignore_dirs which stopped all "file added" messages, but had no effect on the recrawls and kFSEventStreamEventFlagKernelDropped warnings.

[jpaas]

Although @wez 's beta watchman seemed to work for me for a bit, I moved up to 3.7.0 when it was released and now it I seem to have a gamut of problems.

1. A bunch of these when I run ember s

Recrawled this watch 17 times, most recently because:
../admin-webapp: kFSEventStreamEventFlagUserDropped

1. Spontaneously it will re-add all of the files in the project and reload. Sometimes several times a minute.
2. It won't pick up actual changes to an npm linked add-on. Sometimes.

[ilkkao]

FWIW I switched to watchman 3.8.0 two days ago on el capitan and for the first time kFSEventStreamEventFlagUserDropped errors have completely disappeared.

[stefanpenner]

FWIW I switched to watchman 3.8.0 two days ago on el capitan and for the first time kFSEventStreamEventFlagUserDropped errors have completely disappeared.

Awesome, that is a good sign. OSX may have fixed there FSMonitoring...!!:party:

[wez]

I've been a bit snowed under and haven't had a chance to update the release notes for watchman 3.8 yet, but 3.8 includes a change that should make most recrawl scenarios much much less likely to occur.

Regarding changes in .watchmanconfig, if you make changes there, you need to watchman watch-del DIR ; watchman watch-project DIR to have them take effect.

We don't auto-reload .watchmanconfig when it changes because this can have some bad side effects if you have a rebase heavy workflow and keep moving either side of an important configuration change.

[bgentry]

FWIW I switched to watchman 3.8.0 two days ago on el capitan and for the first time kFSEventStreamEventFlagUserDropped errors have completely disappeared.

I was just experiencing this on a ~brand new ember app. Found out I was running watchman 3.7.0. Upgraded to watchman 3.8.0 via homebrew and the errors disappeared. I am still running OS X Yosemite 10.10.5.

[jpaas]

Same here. I'm still on Yosemite 10.10.5 and moving up to homebrewed watchman 3.8.0 made the problems I was seeing go away.

[jpaas]

I may have spoken to soon. Its no longer recognizing changes to my locally npm linked node module even though the node module dir is in the watchlist. Changes to the parent ember project are picked up.

OS X 10.10.5
Watchman 3.8.0

[jsalva]

upgrading to 3.8.0 didn't solve my issue on 10.9.5

[kurtisnelson]

If anyone is landing on this issue when trying to figure out what's going on with El Capitan/OS X 10.11, this solved it for me https://github.com/ember-cli/ember-cli/issues/2703#issuecomment-75886720

[stefanpenner]

watchman is adding support for informing it of phases of rapid changing, once that lands we can use it. It will likely help with this issue: https://github.com/facebook/watchman/pull/187

[stefanpenner]

Watchman folks are working on this, we will have to be patient.