Video-hub-app: Automatic Importing with `inotify`

[cal2195]

🎉 Feature Request! 🎉

More of a personal requirement, but I could see it useful for others too:

I'm running VHA in a docker container, and it would be awesome if it used inotify to watch all the video folders, and automatically imported new videos after an IN_CLOSE_WRITE event! 💡

This would prevent the need for me to rescan the entire directory of videos when adding a single video, and also prevent the need to manually trigger it.

This would be under a feature toggle of course! 👍

I'll look into this! 💯

[whyboris]

Thanks Cal!
Seems like a simple "watch folder" button would do the trick on the UI front.
Main question to think about is if it should be a per-hub setting or a app-wide 🤔
I'm unsure how to answer this question -- perhaps you have an intuition about which is better for users 🤷‍♂

[whyboris]

Also it seems like inotify is a _Linux_-specific thing.
There's surely a way to do it on Windows and Mac too (other apps have features like that) -- I hope it doesn't require re-scanning the whole directory (imagine a hub pointing at a full hard drive) 🤔

Currently initial part of the scan locks up the UI with a green progress bar ...

The feature you describe sounds like it's related for your long-ago discussion about a _Daemon_ mode #161 😁

Perhaps if all a background process does is create screenshots with correct hash file names, the app re-scan would be super quick (only requiring the meta-data extraction) 🤔

All thoughts welcome 👍

[cal2195]

This is available on other systems, node.js even has a built in function for this:
https://nodejs.org/docs/latest/api/fs.html#fs_fs_watch_filename_options_listener

But I don't think this will be suitable for my purposes, but I can test it now - I need to know when a file is closed for writing, to avoid importing a copied file before it's finished copying! 💡

This is related to #161 - As I have it in docker, this might be a simpler solution for now, and handy for many people! 😄

Getting CLOSE_WRITE will be my priority for now, but so I'll start with Linux, and port it to the other systems after each with their respective file-watching-ways. 💯

[cal2195]

Some thoughts:

As expected, fs.watch() doesn't tell you anything about the event, just that something happened to the file. Also, adding inotify as a dependency will through an error on other platforms. 😢

Two options:

1. Implement each platform independently, using they're respective native way.

2. Use fs.watch() combined with something else. For things like rsync (my usecase), it would still work as rsync writes to a temp file, and then renames it to the correct name when finished. Using system copy functions doesn't however (how I assume a lot of people would do it). For this, we could either wait X seconds of no updates for a file and assume it's completed, or use something like opened (https://github.com/ronomon/opened) to see if the file is still open by a process (this could mess up with video players, etc.)

Thoughts? 💭

[cal2195]

More developments:

Currently looking at https://www.npmjs.com/package/chokidar and it looks pretty sweet! 🎉

Multi-platform, and has some nifty features that could even help with #369 💡

Along with allowing file change events, it also stats the files to work out better events, which can be returned for your own use (scanning the hdd), and it also can output all files as it initially scans the hard drive - oh, and it's also async! 😄

We could rethink how the importing works, and scan and import files async, and then react and import to file changes too - what do you think? 💭

[cal2195]

My thought would be:

1. Create new hub.

2. Kick off chokidar, which scans the video folder, outputting files async, along with adding file watchers.

3. As files come in, begin importing them in the background and update the final array (seems like the app is quite happy with updating the final array on the fly).

4. Once scan is complete, new files get treated just the same way, import and update the array.

5. Any files already imported are just skipped.

On a second open, if a toggle is selected, it will do 2-5 again, skipping any known files and importing any new ones in the meantime!

This would have a much better UX experience in that the app wouldn't block on initial or subsequent imports! 🎉

[cal2195]

It even has awaitWriteFinish! 🎉 🎉

[cal2195]

I have a plan of how this can work, and using what I've learned from coding image-viewer! 😄 I've started working on it, and I'll post a review as soon as I have something working! 📝

[cal2195]

Current update:

• Importing new videos on open working, along with new videos appearing while the app is running.

• Manual thumbnail generation working.

• Moved quite a few more methods to be async with Promises (much improving the responsiveness of the app).

My current plan is, if this feature is enabled, new videos will be imported and thumbs generated automatically in the background. Removed files will be automatically removed too.

I think I will also move the current system over too, but in such a way that it will only scan when requested, and won't continue looking for updates after a full scan. This should allow for simpler code, better app responsiveness as it will use the new async code, and also, the app won't need to be locked anymore while scanning! 🎉

Thoughts @whyboris? 💭

[cal2195]

• Automatic thumbnail extraction working!

• Parallel fs requests! (Checking 8 files at a time)

• Resume scanning/extraction on reopen!

• Using new scanning method for new hubs working!

🚀

[whyboris]

Exciting! I think this is your code so far:
https://github.com/whyboris/Video-Hub-App/compare/master...cal2195:file-watch

I'm unsure if running video extraction in parallel is a good idea. I suspect most people are still using mechanical hard drives and random access is slow (making the needle jump between 8 files isn't great I think), and the processor might not be able to handle 8 videos in parallel (thus having to switch tasks, incurring further overhead of task switching). I am pretty sure sequential extraction isn't going to benefit from going parallel (though it would be great if I am wrong).

What do you think is the bottleneck in extracting screenshots? I'm guessing decoding a 4K video makes for a hard-enough job for the HD and the CPU (GPU?) at the same time. I don't have a guess which is slower. But I'm unsure 🤷‍♂

I ran the app extraction with my _Task Manager_ showing -- the RAM jumped up to ~700MB ... I'm worried if we do things in parallel the RAM might run out and the OS will start dumping things to HD for temporary memory, compounding the trouble 🤔

I don't have much experience with any of this -- so all this is from theoretical understanding of things and may be wrong.

I just realized you may be talking about fs requests not for extracting but only for initial info ... so that may be different, though I'm unsure 😅

All else sounds like exciting improvements! Thank you 🙇

[cal2195]

You're correct at the end there - I'm only talking about metadata requests. 😄 I'm running multiple mechanical drives here, and most of the CPU time is taken with iowait, so multiple requests could make sense.

Luckily, it's a single parameter! 🎉 So we can test out which gives the most videos-per-second, and even make it user configurable!

Thumbnail extraction is still sequential, but could also be parallel in the same manner - just one param change! 👍

We could have a simple output when testing to see the exact videos/time extracted, or just go with what feels snappiest! 😄

Working on fixing up bugs now, and getting it all together! One of the best features - it's 100% backwards compatible with .vha2 files currently! 🎉

[whyboris]

@cal2195 thank you so much for your #383 🙇 I believe this issue has been solved 🎉

Please let me know if that's not the case 😅