Video-hub-app: Proposal: Alter/improve/split file format for .vha2 files

[TheTacoScott]

Having a vha2 file be json might come with serialization/deserialization implications as it grows in size so there might be wisdom in migrating to sqlite or some similar local database variant.

In the absence of a full file format migration or split, there may be wisdom in refactoring the schema to better account for the predominant access pattern (which forgive my ignorance) is:

• Doing a lookup by vha custom hash (seems unlikely).
• Doing a lookup by partialPath and fileName (most likely).

Assuming that is the most common access pattern it would make some sense to refactor this "images" array to be a map with partialPath and fileName being the compound key.

I don't see a strong use case for the ordering benefit of an array here given the probable access patterns

[whyboris]

Thank you for the proposal! There are likely benefits to going to a database.

I am strongly in favor of a simple JSON because it makes is _SUPER_ easy for even non-technical people to open & edit the file safely. I guess what you might be suggesting is to simply use SQLite under the hood - even if we don't change the saved file format.

While size _is_ a concern (so many duplicate keys!), I've not seen any problems with a 10,000 video hub (no problem with storing, opening, saving, browsing, searching, or sorting).

A rewrite of this magnitude is a serious undertaking and I'm not yet sold on the benefits.

Where do you see the biggest improvements for the users would be? Are you witnessing slow search times for your hub?

[TheTacoScott]

Well... you're certainly asking the right question: Where do you see the biggest improvements for the users would be?

That basically shuts down down my argument, but here it is anyway:

Immediate benefit to the user: 0.00000000001.
But this foundation raises the ceiling of of what one can do, but I think I agree it doesn't provide enough value for consideration.

[whyboris]

😅 I've been itching to get back to _Face Detection_ work in VHA and I have 2 other projects I've been meaning to start since about 2 years ago -- never find enough time 😓

I've been trying to say 'no' to most things that take time :trollface:

I'm thinking after version 3.0.0 release (and possibly face detection work) I'll finally feel comfortable to work on the other two ideas. Video Hub App is already 99% of what I wanted it to be and I don't see much need to add many more (major) features at this time.

I appreciate the suggestion and I always welcome a discussion 🙇

[cal2195]

I agree this would be a massive overhaul of the app - currently everything is loaded into RAM and modified in RAM, and then written back to disk when it's finished.

With a database, every lookup gets sent off individually, and you get a result returned. For smaller hubs (I'm thinking <50k videos), this would probably actually be slower. Pipes in VHA are incredibly fast given just how much data they sift through! 🎉

On the other hand - this would open up the app to the possibility of having the option of a remote database, allowing for multiple instances to operate at the same time, which could realise my idea of having a "server" importing videos and doing all the work, and a client that just adds tags, and interacts with the videos! 🤔

[TheTacoScott]

Sqlite, or sqlite variants have in memory version of their db for small dbs which I imagine would be just as fast. BUT, ultimately this isn't about speed it is about raising the ceiling of "max videos" or for allowing "db on other host" as you mentioned.

A db accessed/updated at runtime also allows for more 3rd party interactions against said db, outside the runtime.

The server that does the work, and a client that interacts with videos is kind of the "plex" route, which the project can choose to pursue or not.

But making the vha2 file a database makes it so people can do this type of thing on their own with the project not being on the hook for it.

[whyboris]

From my brief look into it, it seems like the theoretical limit for arrays is 4.2 billion items. @TheTacoScott -- you said you have about 1.4 million videos -- have you tried running them (or a subset of them) through the app? I'm curious if you've witnessed some sluggishness.

I was never planning on taking the app in the _Plex_ direction -- hosting anything non-locally. I think currently you could have the vha file and its associated folder on a network drive - when the app opens the file it will fetch all the images from the server as needed (though likely less 'instantly' as it would on a local hard drive).

I'm aware of how Plex can be used but I didn't mean to compete with it.

A bit of history: I decided to code VHA as a toy project to see if Angular could handle things. Everything worked out and it was a cool-enough app that I thought "I just need to polish up and release". Thankfully enough people loved it (and @cal2195 encouraged me to open source it). Since I put up the code online, Cal contributed great improvements and the app became better. But for about a year now all the features I've been adding are because customers email praises and requests for more features -- so I keep thinking "may as well".

I'm not opposed to getting VHA into a server-ready form. A refactor to get us there could be done somewhat gradually: move all the interaction with the finalArray away from home.component into a dedicated service. After that, just swap out all the pipes for a custom sql query builder and it's mostly ready.

I feel like hosting files locally on the PC is faster (especially with an SSD) than having them over a network. Opening the app up to further interactions by having a server can be cool, but I guess I'm currently just focused on getting VHA 3 released without bugs first.

I love the excitement over this feature ... I'm not saying 'no' -- just want to manage expectations -- it might take some time 😅

[TheTacoScott]

I am in the process of creating a vha2 file that has everything it needs and will report back on the performance. It'll end up being about 90M total tags, so I can report on search perf as well.

[TheTacoScott]

Also: I totally get it. I'm not a pushy guy about anything here. I can learn and code this all myself and PR it if I had the time. =)

We're both fighting the same battle of not having enough time to tackle the lower priority things. =)