Video-hub-app: Proposal: Change storage path for all hashed based files (clips/thumbnails/filmstrips)

[TheTacoScott]

Currently all files that are generated (clips/thumbnails/filmstrips) get put into 3 bucketed folders under their vha location.

for example:

<root>/clips/00c97592786e278a9591aeea96ae36c4.mp4

I think there is wisdom to migrate this to

<root>/(clips|thumbnails|filmstrips)/<first_char_of_hash>/<second_char_of_hash>/00c97592786e278a9591aeea96ae36c4.(jpg|mp4)

This can help circumvent limitations and performance implications of large amount of files in 1 directory. Older linux file systems had the 10k+ performance issues thing. Plus it justs makes debuggin and manual browsing easy. Lays the foundation for 100k+ video hubs

[whyboris]

If the issue is with only with Linux, I'll respectfully decline a major refactor to make it work slightly faster there 🤔

I glanced now and for Windows 10, the maximum number of files seems to be 4,294,967,295 and for Mac it is likely similar. And given that the user isn't opening any of these folders (_Node_ is), I don't imagine there are performance issues.

Have you experienced slowdowns that you think exist because of the folder structure?

[TheTacoScott]

Slowdowns occur when debugging and while basically any new filesystem can support 1M+ files with ease there are certain performance considerations as you have to make underlying system calls X times to retrieve all the files. I'm not what syscall nodejs makes for "file open" and "file exists" events and if those involve dir listing in part or in full but it's worth considering regardless.

This "improvement" ,if you want to call it that, I don't think this falls into the bucket of major refactor in that it the exist code can check both places in whatever order is most prudent, migrate files that exist in the old place to the new place or simply leave them there for any old hub.

From a "benefits" perspective I think in the short term this is small/trivial/worthless but it raises the ceiling in the medium term+

[whyboris]

I have a customer who said his collection is over 100,000 video files (and the initial import process occasionally crashed for him), so VHA 3.0.0 will have the better way to import and handle files.

I'll reach out to the individual and ask for feedback about how well everything works with version 3.0.0

I'm unsure if I'd be able to amass such a large video collection so as to test things out 😅

[TheTacoScott]

My "video" collection is 12M+ plus. but 99% of them are 64x64 (and typically about 90 frames) moving perceptual hashes. Based on how vha works I like the idea of being able to swap in/swap out which files are considered a part of a hub at any given moment by updating the vha2 file based on an outside-of-the-program search that can project into the vha2 the result.

Having all 12M files set in their respective locations just makes this quicker for me specifically (thus the hash prefix directories would be useful) if all I have to do is update the vha2 with the set I care about, but it for the general user at large I'm not sure this is a big value add.

Anywho, digging that app.

EDIT: these files are 8x8 videos, the 64x64 was making be crazy as it's 64 pixels total. This doesn't matter and no one cares, but it was bothering me.

[cal2195]

I've considered this before, and it could be quite a good option for very little cost. A collection of 100k videos means 300k files in total, which is not very nice to ever list in any sort of file manager! 😉

Assuming permissions allow (which they should as new videos need write permissions anyway) a simple "update" routine could easily move all the files into correct folders. An easy fix by just incrementing the "save file version number" already present when it's completed! 👍

[cal2195]

Along with this, I think we should come up with a clean solution for upgrading any (at least somewhat recent) file version to a new version. I added the file version for this purpose:
https://github.com/whyboris/Video-Hub-App/blob/857924bf96bbfdc8c27ea86beec2e6364451a5f9/interfaces/final-object.interface.ts#L16

It should be pretty easy - just have a new function, that takes a file of version X, and runs each upgrade step to get it from X to the current file version! 👍 These steps can just be added to the in future, allowing any old file to be upgraded! Just like applying any patches you've missed along the way one at a time! 😄

[TheTacoScott]

Good ol' database evolutions coming to a vha2 file near you. =)

[whyboris]

Thanks again @cal2195 for implementing the vha file version number -- it makes the upgrade to version 3 nice and easy (though in theory I could have checked for presence of a particular field - its absence would have indicated older version). But it's nice and clean - and you're the one that made it that way 🙇 🤝 😁 Cheers!

[whyboris]

As I'm trying to finalize 3.0.0 I'm trying to close out issues.

This one feels like a bunch of extra work (every bit of code is a liability for bugs) and would require an upgrade script to go from version 2 to 3.

I would prefer for users of VHA, when upgrading, to just open their old file and have everything run smoothly. Additionally, it's a nice bonus that they could also go back to version 2 without losing anything.

This was _not_ a bad suggestion - I just think it's a lot of work with minuscule benefit (and even at that, one that only a few users would ever notice). Thank you all for the discussion 🙇