Deno: fs events ops

[kitsonk]

There are several use cases where the ability to watch a file(s) for changes is an important API. For example, invalidating cache headers for serving static file based content, or processing logs or files, etc.

Many OSes provide a mechanism for this without having to "poll" the stats for the file. Node.js offers fs.watch().

Instead of an event listener though, it feels some sort of async generator would be more idiomatic to the rest of Deno.

[ry]

Yes I would have in core. This looks pretty nice: https://github.com/passcod/notify

[ry]

Here's a potential API, but I leave the exact spelling of these methods to the implementor

  const watcher = await Deno.watch("/home/test/notify", { recursive: true });
  while (true) {
    const event = await watcher.getEvent();
    console.log("event", event)
  }

Here's a brief sketch of how I would do this

1. Add new ops to cli/msg.fbs
   1A. One to create a new watcher, maybe FSEventsOpen. This would return a new resource id (rid).
   1B. One to poll for a new event, FSEventsPoll. It would take an rid and return a new event.
   1C. Closing the rid does not need a new op. The existing Close op can be used.
2. Add new handlers for the two new ops in ops.rs
   2A. Add a new Resource type in resources.rs, you can probably follow the example of tokio_process::Child
   2B. Add a new typescript frontend to the op, by adding js/fs_events.ts, follow the example in js/process.ts.
3. Write some tests.

[balupton]

Having written watchr, I would encourage no built in watching apis, as watching is such a difficult problem. I think in nearly all cases people would be best served by facebook's watchman instead. Which does not use the node apis at all.

[sholladay]

Personally, I've frequently been annoyed at how file watching was a mess in Node, a situation that is still unresolved today. Yes, it's hard to implement correctly, but that just means we need more eyes on it. Also, a lot of projects need it as a very low-level building block. For context, chokidar has almost 11 million weekly downloads on npm, about as much as React and Express combined (i.e. it is among the most downloaded packages).

I would love to see this in core and exposed as an async iterable.

const watcher = Deno.watch('/home/test/notify', { recursive: true });
for await (const event of watcher) {
    // ...
}

The snippet above is using for-await-of. Node recently started to use this for streams and I've fallen in love with it. It's standard syntax, which aligns nicely with Deno's philosophy.

[jcao219]

I can try to see if I can make a prototype of @sholladay 's API into reality.

--

Quick update -- I haven't forgotten about this effort :) I've just been busy with moving to a new continent recently.

--

Getting back to work on this right away!

[Fenzland]

I'm waiting for this feature too. In some using cases, we need to listen to events with different types. Nesting several if or switches in the for is not so good. I have a plan to combine both event listener and for-await-of together:

const watcher = Deno.watch('/path', { recursive: true });
watcher instanceof Watcher;
watcher.addEventListener('add', onAdd);
watcher.addEventListener('modify', ()=> {});
watcher.addEventListener('chmod', ()=> {});
watcher.removeEventListener('add', onAdd);

watcher.close();

What's more, A close() API is necessary.

I'm become agree with zekth at denoland/deno_std#386.
Using for-await-of is not a good idea. It runs serially. Your handler cannot handle the second event until the process of first event has done.

[MarkTiedemann]

What's more, A close() API is necessary.

I think if we use async generators, then we don't need a close API, but can use the built-in return API (https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Generator/return).

let watcher = Deno.watch(/*...*/);
// ...

// some time later
await watcher.return(); // wait for watcher to close

EDIT: Here's an example:

async function* watch() {
  yield { type: "add", path: "/a" };
  yield { type: "delete", path: "/a" };
  yield { type: "add", path: "/b" };
}

async function main() {
  let watcher = watch();
  for await (let event of watcher) {
    switch (event.type) {
      case "add":
        console.log(`+ File '${event.path}' was added.`);
        break;
      case "delete":
        console.error(`- Error: File '${event.path}' was deleted. Aborting!`);
        await watcher.return();
        break;
    }
  }
}

main();

$ deno run watch.ts
+ File '/a' was added.
- Error: File '/a' was deleted. Aborting!

[Fenzland]

I think if we use async generators, then we don't need a close API, but can use the built-in return API

That's right. But now, I change my mind and not support for-await-of any more.

[ry]

Added to 1.0 blockers

[bartlomieju]

@jcao219 I remember you were working on an implementation, any update?

[jcao219]

@bartlomieju Hi all, sorry for the long delay.

I have had a working implementation, but it's been a while since I've synced it with master branch, so I'll need to update it for the new JSON message passing changes. Let me do that ASAP and get right back to y'all.

[Dan503]

It would be nice if the watcher could tell the difference between adding/removing a file and simply renaming it.

Chockidar can't tell the difference between renaming and adding/removing so when you rename something it will fire both an "add" event and a "remove" event. This can lead to tasks being run twice when you rename a file.

[bartlomieju]

Ref #3452

[bartlomieju]

This feature is almost complete (#3452), only common interface must be established. I'm marking this issue as "help wanted" - if anyone wants to pick up my PR feel free.

[ry]

Fixed in #3452