Gun: GunDB Limitations

[brettneese]

Hey @amark!

I listened to you JS Jabber episode the other day and got thinking about how I might use Gun in a project I'm working on.

Just a practical question, though: assuming am S3 storage adapter, what kind of concerns are there about using Gun across a large dataset?

I'm working on a project that involves scraping a lot of JSON data - not quite "big data" levels, but enough to think carefully about database limitations. Currently at around 20GB of json right now, and growing by 2-3GB per month, and will eventually want to expand the dataset quite a bit (multiple times that).

I know Gun operates primarily in memory, so I'm curious what walls I might run into, and if there's anything special I should be thinking about before diving into Gun.

[amark]

hey @brettneese great questions! This should get into docs somewhere.

The S3 adapter stores things on a per-object (1 level deep) basis. Each S3 file can store up to 5TB. However, because the 0.3 version of GUN by default parses things on a per-object (1 level deep) level, each per-object (1 level deep) would need to be small enough to fit in RAM.

So for instance, if you had a user list that was 1GB in length (1 level deep, not counting the user objects themselves, just the list itself) then that should fit within NodeJS's 1GB heap limit. If however you had a 9GB user list (1 level deep, not counting the user objects themselves), then that would probably crash your process from being out of memory.

Also, currently 0.3 sucks at importing data or doing anything procedural or performance based. I recommend you try using 0.5 which should be snappy :D (although only has 60% coverage but should work for most use cases). Let me know if you have any problems!

Did this answer your questions? How are you doing, BTW?

[brettneese]

@amark I'm doing great, thanks!

So to clarify, 0.5 doesn't have this limitation? How would I query, for instancs, a user object in a list of users without pulling the whole top level object into memory first?

[jeffbski]

@amark just to add to the conversation, I found out recently that you can increase the amount of memory Node.js can use by simply including a command line argument.

node --max-old-space-size=4096 script.js

The above command increases the old-space size to 4GB. I've read about people using up to 28GB with successful results.

[amark]

0.3 and 0.5 have these limitations because the limitations are more the limit of RAM (nice note on increasing NodeJS RAM size!) and S3.

0.5 however has the ability to process only one property at a time (even from a large list object, like billions of users). This is particularly great for browser<-->server, however the S3 storage driver still saves things on a per-object level which needs to be optimized in the future.

@brettneese your users should be keyed individually so you can load them directly. This is the power of graphs, every object anywhere in the database can be loaded directly if you know its key:

var alice = gun.get("people/alice").put({name: "alice"});

var users = gun.get("users");

users.set(alice);

users.map(cb); // map over each user in the user list!

Now you can load an individual user directly, which is very bandwidth/memory efficient!

[brettneese]

@amark hmm. the problem with this is that my data is indexed by time, and I need to query by timestamp, which is imprecise -- what I _really_ want when I load an object by timestamp is the closest result to that timestamp (ie the 1 result that is greater than x timestamp.)

i suppose this is what indexes are for, I just wanted to avoid going that route if possible ;-)

[amark]

Ah yeah :( that isn't baked in yet. Although I knkw a radix tree would be a pretty efficient algorithm to start with to get this (you'd only go down to the resolution you want to load).

I'm not familiar off the top of my head with any other approach that doesn't require scanning a table though. Do you? So this would be an inherently slow query on any system. But things are in memory (gun and other dbs) so it should be pretty fast most of the time.

[amark]

Oh, FYI I'm in Italy this last week so sorry for slow responses. My internet gets worse tomorrow, just as a warning.

[brettneese]

@amark No worries. Still thinking this through, it's definitely a side project at the bottom of my barrel.

I'm actually not terribly familiar with data structures as I come from a non-CS background. Radix trees sound perfect though, just not sure how I'd implement them.

[amark]

@brettneese thank goodness we don't need to be CS students ;) it looks like https://www.npmjs.com/package/radix-tree maybe has already implemented this for you? Is this helpful or no?

[brettneese]

@amark That's very useful!

Basically instead of just storing the timestamp, I should do something like:

Year/Month/Day/(Hour?)

And from the timestamp I can parse and do a flat out query, rather than scanning the DB every time. This would be perfect.

I suppose I don't even particularly need to use a formal radix tree for that, I could store everything I need under a YearMonthDay object which sounds like it'd be pretty bandwidth/RAM/S3-friendly?

[amark]

Well, a UTC timestamp is already going to be organized in yearmonthdayhoursecondmillisecond so you don't need to manually parse it yourself (that would be a pain!) but will still match a nice radix structure.

Not sure about your second comment though.

[amark]

@brettneese gonna close this, however I'm still interested in the radix tree maybe going in core, which I've left open: https://github.com/amark/gun/issues/136 .

[brettneese]

@amark Oh yeah, I meant to close this much earlier. Sorry to keep it hanging for so long. I might PM you at some point to talk about this more.

[amark]

great :D [email protected]

[beebase]

@brettneese I have no clue how I ended up reading this thread. Anyway, if you're looking for a time tree graph structure, this approach might help https://graphaware.com/neo4j/2014/08/20/graphaware-neo4j-timetree.html

[amark]

Update: Radix storage engine got built, is now the default in GUN for NodeJS.

Timegraph got built, now available lib/time :)

Tons of other cool things too.

[exside]

@amark could you provide some basic links/info about radix (how it works, why it's good etc.) for us frontend devs? =) I'll happily improve the docs if I understand better!

EDIT: read https://en.wikipedia.org/wiki/Radix_tree and added it to the docs for others like me... got the idea!

[Dletta]

@exside Hallo und merci fuer den Link in den Docs! Benutzt du GUN im Moment? (Der Gitter Chat is eine der besten Quellen fuer Hilfe in diesem Projekt)

[amark]

@exside any other info you want to know? @Dletta also worked on RAD so he can answer quite a bit.

Also wanted to drop a visualization I made, that I think is fun/useful:

https://gun.eco/see/radix.gif