Node: fs.lstatSync doesn't handle specific Windows REPARSE POINTS

[paulbouwer]

• Version: 7.9.0
• Platform: Windows 10 (16179)
• Subsystem: fs (lstat)

Referenced issues:

• hyper #1776 - Gaze is adding all folders in HOME to watch list
• gaze #239 - fs.lstatSync doesn't handle specific Windows REPARSE POINTS

Issue

While tracking down an issue affecting hyper, it lead me to the use by gaze of fs.lstatSync.

It seems that on Windows, there is something different about the reparse point on OneDrive local sync folders, that results in a EINVAL error from fs.lstatSync. The fs.lstatSync correctly handles other junctions, folders and files in HOME with no error.

Error example - OneDrive

C:\> node
> fs.lstatSync('C:/Users/pabouwer/OneDrive')
Error: EINVAL: invalid argument, lstat 'C:\Users\pabouwer\OneDrive'
    at Object.fs.lstatSync (fs.js:961:11)
    at repl:1:4
    at ContextifyScript.Script.runInThisContext (vm.js:23:33)
    at REPLServer.defaultEval (repl.js:339:29)
    at bound (domain.js:280:14)
    at REPLServer.runBound [as eval] (domain.js:293:12)
    at REPLServer.onLine (repl.js:536:10)
    at emitOne (events.js:101:20)
    at REPLServer.emit (events.js:191:7)
    at REPLServer.Interface._onLine (readline.js:241:10)

C:\> junction -nobanner "C:/Users/pabouwer/OneDrive"
C:\Users\pabouwer\OneDrive: UNKNOWN MICROSOFT REPARSE POINT

Working example - My Documents

C:\> node
> fs.lstatSync('C:/Users/pabouwer/My Documents')
Stats {
  dev: 2348871125,
  mode: 41398,
  nlink: 1,
  uid: 0,
  gid: 0,
  rdev: 0,
  blksize: undefined,
  ino: 8444249301777451,
  size: 27,
  blocks: undefined,
  atime: 2017-04-23T23:24:58.150Z,
  mtime: 2017-04-23T23:24:58.150Z,
  ctime: 2017-04-23T23:24:58.185Z,
  birthtime: 2017-04-23T23:24:58.150Z }

C:\> junction -nobanner "C:/Users/pabouwer/My Documents"
C:\Users\pabouwer\My Documents: JUNCTION
   Print Name     : C:\Users\pabouwer\Documents
   Substitute Name: C:\Users\pabouwer\Documents

[Trott]

@nodejs/platform-windows @nodejs/fs

[refack]

So windows likes to create these "Broken" junctions all over the place that have the equivalent of chmod 000, and I think they are only used internally for WIN32 to resolve the actual folder they link to. You can't navigate into them even native windows tools bork when interacting with them as @paulbouwer demonstrated

C:\> junction -nobanner "C:/Users/pabouwer/OneDrive"
C:\Users\pabouwer\OneDrive: UNKNOWN MICROSOFT REPARSE POINT

Does anyone think we should handle them in a special manner?

P.S. Just the first answer I found in Google https://goo.gl/Daj2vu (the jist: don't mess with them)

P.P.S Every user home has these and none of them work.

 <JUNCTION>     Application Data [C:\Users\refael\AppData\Roaming]
 <JUNCTION>     Cookies [C:\Users\refael\AppData\Roaming\Microsoft\Windows\Cookies]
 <JUNCTION>     Local Settings [C:\Users\refael\AppData\Local]
 <JUNCTION>     My Documents [C:\Users\refael\Documents]
 <JUNCTION>     NetHood [C:\Users\refael\AppData\Roaming\Microsoft\Windows\Network Shortcuts]
 <JUNCTION>     PrintHood [C:\Users\refael\AppData\Roaming\Microsoft\Windows\Printer Shortcuts]
 <JUNCTION>     Recent [C:\Users\refael\AppData\Roaming\Microsoft\Windows\Recent]
 <JUNCTION>     SendTo [C:\Users\refael\AppData\Roaming\Microsoft\Windows\SendTo]
 <JUNCTION>     Start Menu [C:\Users\refael\AppData\Roaming\Microsoft\Windows\Start Menu]
 <JUNCTION>     Templates [C:\Users\refael\AppData\Roaming\Microsoft\Windows\Templates]

[bzoz]

Sorry, I can't reproduce this. I've tried different machines, a fresh Win10 install, with OneDrive uninstalled, etc. Each time lstatSync returned without error.

[refack]

Sorry, I can't reproduce this. I've tried different machines, a fresh Win10 install, with OneDrive uninstalled, etc. Each time lstatSync returned without error.

Someone is behaving better either the OS or libuv.

[bzoz]

@paulbouwer can you provide some more info about your setup?

[refack]

I was reading today about "OneDrive Files On-Demand", maybe that is the case, I'll try repro.

[UsmanMohammad]

Having the same issue, I am using insider build 16215 and have OneDrive Files On-Demand enabled

[refack]

@UsmanMohammad thank you for the info.

[paulbouwer]

@bzoz - I have this issue on Windows 10 that is pre the OneDrive Files On-Demand feature. I'll check what other additional info I can find.

[cerebrate]

@paulbouwer Unfortunately, there're quite a lot of Windows features build on reparse points and have been since W2K, any of which can cause this error - mounting a volume on a folder, remote storage, distributed file system, junction points/symbolic links (which I believe lstat understands as equivalent to Unix-style symlinks, or at least it doesn't fail on them), single-instance storage, and that's not even counting the non-Microsoft stuff. Any of those other than junction points, I believe, will produce the same lstat()/lstatSync() error.

(Edited for thinko.)

[bzoz]

I've opened a PR to fix this in libuv: https://github.com/libuv/libuv/pull/1419

[MikeKovarik]

So... uhhh... this is awkward. 4 months later and I still can't use OneDrive. :(

[bzoz]

The fix was added to libuv 1.14.0. A PR for updating libuv in Node.js is already open: https://github.com/nodejs/node/pull/14866

[refack]

Also for context, this is a pre-release feature (OneDrive Files On-Demand). If all goes smoothly @bzoz's fix will be in node before this feature goes RTM.

[MikeKovarik]

It's not only affecting the on demand feature but it has become a problem for the whole OneDrive as of couple of last Win Insider builds. One of my PCs I have not been using on-demand on became affected recently and giving me a hard time working. So I'm glad to see this finally come to an end.

[wadetb]

I somehow missed this issue and fix last week, so I independently developed and submitted a new PR to libuv - https://github.com/libuv/libuv/pull/1522. Compared to e5024c5, my PR solves this issue and other issues, so it may still be worth integrating.

The issue with the accepted change is that by preventing fs__stat_handle from failing, the retry logic in fs__stat_impl has become impossible to activate. The retry logic re-opens the file without FILE_FLAG_OPEN_REPARSE_POINT when symlink detection fails, and is the correct way to deal with non-symlink reparse points.