Watchman: [Feature Request] Symlink support

[JohnyDays]

Is there any performance / complexity reason for not supporting symlinks?

With the release of react-native, and the use of watchman in its packager, it's troubling that npm's (react-native's chosen package manager) endorsed method of developing local interdependent libraries with npm link is not supported, and having to commit to a remote repo and download on every change is obviously not ideal.

[wez]

Watchman recursively watches files that are contained within a filesystem tree.
Symlinks can point to an arbitrary location on the filesystem.
Changes to symlink targets that are outside the tree are not observable.

Why not simply resolve the symlink and watch its target? Because it isn't simple. Here are a handful of reasons that make this a difficult prospect. This is not an exhaustive list, it's just a few reasons I can rattle off the top of my head:

• The target of the symlink may not exist at the time we want to establish the watch, and neither may any of the parent directories in the symlink path. The operating system will not allow us to watch a path that does not exist
• Any of the directory components in the target symlink path may themselves be a symlink and may change at an arbitrary time in their lifecycle in a way that is not detectable unless we watch every single directory component in the symlink target path
• The target of the symlink, or any of the directory components of the symlink target, may be a remote filesystem that doesn't support change notifications
• Each additional watched directory consumes more filesystem watch resources. There are finite limits on these, and not every system allows raising these limits.
• Because the hypothetical watches that we'd establish for each symlink are distinct from the watch used for the root of the real tree, we lose certain ordering guarantees when processing change notifications from the kernel. We wouldn't be able to tell whether a change to the symlink target happened logically before or after changes that we observe in the main tree. This important property is used to ensure that filesystem changes have been observed up to the logical point in time of a query. More details on this can be found at https://facebook.github.io/watchman/docs/cookies.html
• Filesystem changes are fraught with TOCTOU (time of check, time of use) issues already. Adding user-space resolution of symlinks increases the chances of more TOCTOU bugs dramatically; any component of the symlink path may change while we're processing information about them.

Correctly handling all of the above would be tremendously complex and still be error prone. As a result, it is unlikely that we'll ever add support for resolving and tracking symlink targets.

[wez]

Taking a step back from this, can you describe what you're trying to do and why symlinks are important? I'm a casual and occasional node and npm user and am not familiar with npm link

[JohnyDays]

I understand why this would be difficult, and I'll close the ticket and resign to using the above solution for developing local modules.

I'll attempt to explain the process behind npm link, with my use case as an example.

I am developing a module called react-native-waterfall, it implements a generic waterfall view.
I am at the same time developing, in a sister folder, a module called react-native-social-waterfall, and this module depends on the above module.
I am at the same time developing utility modules, which both of the above modules depend on.

npm requires modules from either:

1. a globally qualified name e.g react-native-waterfall
2. a path e.g ../react-native-waterfall
3. an url e.g https://github.com/facebook/watchman/

The second solution has 2 problems:

1. Every time you change any file in a dependent module, you have to run npm install ../react-native-waterfall to reinstall the module.
2. You are checking in code that doesn't mirror the final product, where you will be using globally qualified namespaces hosted on npm.

The third solution has 2 problems as well(both are slow)

1. Every time you change any file in a dependent module, you have to commit it and push it to origin.
2. You must then go back to your dependent module and do npm update to grab the latest commit

npm link works the following way:

1. You go into the depended upon module and write npm link, this will register react-native-waterfall as a globally qualified name
2. You go into the dependent module and write npm link react-native-waterfall, this will register a symlink to the globally qualified name, using the folder where you wrote npm link.

This allows the following benefits:

1. Your checked in code mirrors the final product.
2. You do not have to do anything to update the local dependency files, meaning you can quickly test code without committing / running commands.
3. You can even edit the depended upon module in your dependent folder, and the changes will be applied to the original module.

[wez]

Thanks. How does watchman fit into this?

[JohnyDays]

I am using watchman with the react-native packager, which automatically parses dependencies, transforms code and reloads the application when you change a file. However I have to continuously restart the packager because it doesn't detect the symlinked dependencies's changes.

[JohnyDays]

Idea:
I understand why the added complexity of implicitly following symlinks would cause too many troubles, both in implementation and in maintenance. So would it be possible to instead implement an explicit/manual symlink helper in the form of:

/dir1/
/dir2/

watch dir1
watch dir2 --link-to dir1

Any change in a file at dir2 would trigger an event for an analogous file in dir1
This could easily be automated into a npm link workflow, along with some kind of unlink function

[wez]

Sorry to disappoint; it's too complex to build something to handle this that will work according to expectations.
It sounds like this is something that the react package manager should be handling, since it is the component that has special knowledge of the situation.

[wez]

I've been giving this some thought. I think full-blown symlink handling is too fraught with problems to be something we can commit to, but it doesn't mean that we can't help out in some of the easier or more common cases.

I'll collect some thoughts into a wiki page through the weekend and link to it from here

[JohnyDays]

Sounds good! Would be a great productivity boost for any npm-related consumers developing modules simultaneously, which seems to be a great deal of them nowadays

[amasad]

This came up a few times. I think we can handle it in the RN packager by detecting symlinks and starting a new watch on these symlinks.

There is also reports that the initial file system crawl (which queries watchman) is wrong if there is any linked directories. @wez: even if watchman doesn't follow the symlink shouldn't it report them like any other file?

[wez]

@amasad I've been brainstorming with @bhamiltoncx and we have a plan for this. I'd say hold off from implementing anything in RN for the moment; we have some diffs in progress.

[erickreutz]

Any updates on this? Really makes developing modules a pain.

[cgarvis]

:+1:

[d-vine]

+1

[ajwhite]

+1, unfortunately, as you know, RN is in a bit of a bind without symlinks, as it's a common practice to npm link local modules for development. I understand the challenges you describe in https://github.com/facebook/watchman/issues/105#issuecomment-105270042, but I wanted to chime in with a +1 for the pain point I'm currently experiencing.

For now I've thrown in a bash script to recursively copy all my local dependencies into my project's node_modules directory. It isn't a perfect solution, but it works. So hopefully others could find this workaround useful.

[Ehesp]

+1

[jbpin]

+1

[chetstone]

What are module developers doing in general to work around this? @ajwhite, your solution sounds reasonable. Do you run your script by hand or have you wired up watchman to run it for you?

[ajwhite]

I'm running them by hand currently. I'm sure there's a better way

[Ehesp]

I'm just opening a new project from the node modules directory and working on it that way... Sucks but it's the easiest way I'm finding without messing about with scripts and stuff.

[ajwhite]

Yeahhh I was doing that too @Ehesp, but then the module was growing a bit so I had to get it into a safer place.

[chetstone]

@Ehesp can you manage such a project with git without messing around with git submodules? I guess since node_modules is in .gitignore at the top level that might be doable.

[Ehesp]

It's a bit stinky but it does work like you say, since you also need to install any dependancies at root project level (so npm i without saving it).

[chetstone]

Yeah watchman-make is pretty easy to set up do these simple copies.

[ajwhite]

@chetstone thanks for the link! I've never really gotten my hands directly on watchman, it's always been just part of my tool chain boilerplate. This is helpful, appreciated.

[jbpin]

I wonder if a sinopia instance loaded as docker container on a vagrant provisioned virtual machine will solve this.. It's a bit too heavy to manage. Npm link is a way better direction.

[corbt]

@wez is this still on the roadmap for watchman? If not it would be good to know so that we can add support to the react-native packager.

[wez]

Sorry for letting this sit. At this time, we have no one looking at this issue because it is not a priority in our main use cases. I'd be happy to give guidance on any of the components below if someone would like to contribute pull requests to help bring us closer to having this done.

What's we'd discussed as an implementation strategy was this:

1. Add the target of a symlink as a metadata field in the state that watchman tracks per file
2. Whenever a file of type symlink is changed, record readlink() in that slot
3. Add link target as an available field for queries (https://facebook.github.io/watchman/docs/cmd/query.html#available-fields)
4. Add a watch_symlinks: true option to .watchmanconfig. When set, each observed symlink in that watch will be implicitly watch-project'd (See #347)
5. Add follow_symlinks: true option to the query engine. When set, the query engine behavior will change to be aware of symlinks. (See #349)
6. We'd need to come up with some concept of global clock to meaningfully understand and reason about changes being made across multiple projects. (See #350)
7. Subscriptions are tricky. (See #351)

"Aware of symlinks" is rather difficult

1. The initial query, while the lock is held on the initial root, collects the non-symlink results and separately records the set of symlink results.
2. Before releasing the lock, any of the symlink results that point into the current root are translated such that they report the symlink name but the result of the target.
3. After releasing the lock, any of the symlink results that point into different watches are then processed in a similar fashion, but with an awareness of the effective path of the symlink.

Global clock is something that needs some thought

Right now we track an abstract clock per watched project. There are two components to the clock; the _root number_ and the _tick counter_. Each time a change is observed, the _tick counter_ is incremented. This is fundamentally how we perform time-based queries. The _root number_ is bumped each time we recrawl a root and allows us to disambiguate certain classes of stale clock values when deciding which events a client has missed.

Since each watch ticks independently, and symlinks can point to many different projects, there are multiple different clocks. We'd need some basis for synchronizing these watches to be able to meaningfully and easily query across them.

One possibility is adding a global _root number_ and _tick counter_ that are bumped whenever there is a recrawl and whenever a change is observed, respectively. We'd then add a global clock file to the state that we track per file, and update this at the same time that we update the existing oclock field. We'd then need to add a global_since generator to the query engine to indicate that this is the query strategy that should be used.

See #350

Subscriptions

A given watch may be the target of multiple symlinks from multiple projects. It is desirable that a change be propagated out to subscribers watching all references to that symlink. That means that when assessing subscriptions:

1. We'd need to track the graph of subscription source and target
2. When a file node changes, we'd need to walk the edges of that graph to compute the set of visible aliases
3. For each of the aliased watches, for each subscription that is using 'follow_symlinks: true', we need to inject a change event with the aliased path. This requires coordination across multiple locks with multiple connected clients.

See #351

Caveats

1. Not all of the symlink targets in a project will be watchable.
   The filesystems in which the targets reside may not support change notifications at all, or may be restricted from being watched by local policy (https://facebook.github.io/watchman/docs/config.html#enforce_root_files, https://facebook.github.io/watchman/docs/config.html#illegal_fstypes)
2. There's plenty of scope for TOCTOU attacks, care needs to be taken to ensure that we can't be DoS'd into looping forever while trying to resolve symlinks.

Partial Support

All-singing, all-dancing support is a lot of work, but isn't strictly needed if you can contrive to have all of the files residing physically under the same watched root:

/path/to/myproject/.watchmanconfig    (defines the project root)
/path/to/myproject/node_modules/      (contains dirs that have symlinks that all point
                                      to paths under  /path/to/myproject/node_modules/

In this case, all of the changes are taking place under the same root, so we wouldn't need to do any of the implicit watch-project stuff, wouldn't need a _global clock_ and wouldn't need to coordinate subscriptions across multiple watches.

The scope of work would be reduced to managing the aliasing within the same root at query execution time.

[samhunta]

You can create module aliases like:

/**
 * @providesModule mylibraryname
 */

at the top of the module directory index and include it like you would any other npm package:

import { SuperBadAssComponent } from 'mylibraryname';

[dutzi]

@wez, the partial support you mentioned would be very helpful!

It would help cases in which a team is working on multiple packages: pack1, pack2 and pack3, where pack1 is dependent on pack2 and pack3. _And_ pack2 is dependent on pack3.

In a situation like this the ultimate workflow would be:

~/pack1
├── node_modules/pack2 -> ~/pack2
└── node_modules/pack3 -> ~/pack3

~/pack2
└── node_modules/pack3 -> ~/pack3

~/pack3

But this would also be sufficient:

~/pack1
├── node_modules/pack2 (git repo)
│   └── node_modules/pack3 -> ~/pack1/node_modules/pack3
└── node_modules/pack3 (git repo)

And definitley much better than the only stucture we can use right now, having multiple clones of the same repo inside node_modules:

~/pack1
├── node_modules/pack2 (pack2 git repo)
│   └── node_modules/pack3 -> (pack3 git repo)
└── node_modules/pack3 (pack3 git repo)

[jsierles]

Is this latest commit related to the 'partial support' mentioned above? Just curious what more needs to be done.

[wez]

I created some internal tasks to work towards implementing 'partial support' as outlined above for new hires at FB to pick up during bootcamp so that we can semi-passively make progress towards implementing this. If you are looking in from outside FB and want to see this move faster by submitting some code, please don't be afraid to reach out; I'd be happy to give some guidance.

[jsierles]

Yeah, I'm interested! But don't have a ton of experience with C. I work with React Native and having this would really improve the dev experience for those of us writing modules for it, as it would allow testing them inside real apps while we develop.

[dutzi]

Hi all, being tired of how screwed up this situation is for us I wrote a simple tool that listens to changes in one folder and copies them to another folder. It's called WML (Watchman-Links, since it's based on watchman). Check it out here.

It's very simple to use:

# add the link to wml using `wml add <src> <dest>`
wml add ~/my-package ~/main-project/node_modules/my-package
# start watching for changes
wml start

[ekryski]

@wez after doing a ton of debugging and tracing this afternoon. I actually think that allowing watchman to support symlinks (in some fashion) or copy-pasting (wml and custom scripts) are really the only options for React Native.

Would you be able to elaborate on what needs to be done to support:

• following symlinks to initially discover the correct paths
• watching symlinked paths

I'm not very sharp with C but am motivated to see this work, so I'm willing to help so that this works out of the box with RN. Watchman is the only thing that is fast enough or can support the large number of files in an RN project.

[wez]

@ekryski I put a pretty detailed dump of what is needed in the long comment above. In terms of breaking this down into tasks, #347 is almost done and I just opened #349, #350 and #351.

349 is in the FB Bootcamp onboarding pool of tasks, which is to say that they have no explicit priority and no guarantee of making progress--they're waiting for an appropriately interested person to come along and pick them up. I'll look at adding the others into the pool a bit later.

347 corresponds to item (4) in my list above, and #349 to (5), #350 to (6) and #351 to (7).

I think RN really wants subscriptions to trigger pushing things down to the mobile device/simulator, which means that you need everything that we've spec'd so far.

[vongohren]

So has this been picked up? Now we are trying to share some code between web and native react. Like redux reducers and stuff, business logic. But we thought the easiest way would be to make a "commons" folder outside web and native, to have this symlinked inside web and native.

But watchman doesnt find any of the files in the symlinked folder.

This way we could share busniess logic between web and native.

[ThaJay]

@Snorlock we use git submodule for that. Shared (smart, no view) components and logic are in a 'core' repo which is a submodule of both apps.

[wez]

@Snorlock: @farnz is working towards this, but it's not our highest priority.

[vongohren]

Ok, sounds good, we decided to use a wrapper for watchman and copying our files inn when we hit npm run start, modified the start script. So watchman for commons folder and watchman for the native project runs the same time

[gyurobenjamin]

Symlink is working for me. All you need to do, is create a package.json in your shared directory with a same name what you want to use with your alias.
eg.:

+-- apps
|   +-- _core
|       +-- bar.js
|       +-- package.json // {"name": "@core"}
|   +-- app1
|       +-- index.js
|       +-- package.json
|   +-- app2
|       +-- index.js
|       +-- package.json
+-- node_modules
+-- package.json

Run this in your app1 and app2 directory:

ln -s ../../_core ./node_modules/@core

Make sure you will restart your packager etc, and remove caches.

And from this point you can use in your code like this in your app1/index.js:
Import bar from '@core/bar'

[zubko]

I've used the way by @gyurobenjamin, thanks, it works for now,
in my case I've also added a post install script to package.json, because node_modules isn't checked in to git repo.

"scripts": {
    ....
    "postinstall": "... the same ln command ..."
}

[aleclarson]

Might be obvious, but hard-linking works great if your symlink isn't pointing at a directory.

[faisalil]

@farnz are you still looking into this?

[wez]

No one is actively working on this issue at the moment