Gun: Handling large JSON file

[terrybleger]

Hi,

I just knew this library from londonjs on youtube and I think you guys have done a great work. Congrats and thanks for making this project open source.

Since I haven't use it yet, pardon me for not reading through all the docs first, and also pardon me for not looking the source codes first also. (You know, the thing about haven't had the time to sleep well ...)

I'd like to ask some quick question:

1. I'm assuming gunDB is reading/loading from json file directly either local/s3. So if I'm correct, can gundb read a very large json file?
2. How do you parse the json file? Streaming / JSON.parse()?

And also all I know that currently V8 can't even handle/store too large object. So those might be very stupid questions too.

Thanks.

[amark]

Terry, thanks!

A) Don't use data.json file module in production or with large files. It is intended entirely for just doing local testing in a convenient way. It will likely crash, and for good reasons.

B) Using S3 is the preferred method for deployed apps. It is cheap and can handle a ton of data. Because of GUN's graph data structure, large data is automatically broken down into smaller pieces (each object in a document is stored as its own partial). This also mean GUN automatically streams the data as well.

C) Exceptions. In the current version of GUN, if you have a single shallow object ({a1: 'foo', a2: 'bar', ... , ... , .... , ZZZZZZ999999999999999999999999: 'lol'}), then this might bog (bandwidth) things down. However, give it a try yourself and let us know. We're improving this in future versions as well.

D) I've heard V8 has a 1GB heap and you can't go beyond that, however I've tested this a few times and it doesn't seem to be true (memory usage was over 1GB). But I'm not very good at understanding that stuff, to be honest. The good news is GUN is fault-tolerant, so even if your process crashes, when it restarts GUN will pick back up where it was at before, but now with fresh memory.

TLDR; Basically, "large JSON" is fine because GUN uses partials and streams on those partials ( https://github.com/amark/gun/wiki/Partials-and-Circular-References-(v0.2.x) ) . However if by "large JSON" you mean 1 object that has NO DEPTH and millions of fields/properties on it, then bandwidth usage will be heavier/slower and memory allocation will be large/useless.

Does that answer your question?

[terrybleger]

Thanks so much for the reply. Yes! Basically that answer my question.

a) OK, noted.

Yeah. Actually this is the first time I knew that JS can't handle too large object/array. So, I have a data that is more than 1 million records but only about 155 MB in size. I tried to store it all in memory in JS. I thought that was nothing since peope already using db like redis which can handle large GB of memories.

Previously I used Java as the client, it can handle large object/array without problem. But it just too slow when parsing the data from JSON even using lib like Jackson or Boon. Because what surprises me is that JS is a lot way faster than Java when parsing JSON.

b) Wow, that is great to hear. Thanks.

c) OK. This one is noted too. Thanks.

d) Actually I already tried using thing like node --max-old-space-size=8192. But that won't help either.

Hearing your answers makes me wanna use gunDB more. Since I already had too much frustation using many db out there. And most graph db is just too much complicated to use. I wanna have a graph that is just like a plain JS object and then finally found gunDB. The thing about the way gunDB can handle circular reference I think it's just genius, since I never thought that before. Mind blowing.

Thanks.

[amark]

Thanks!

Just as a warning (now hearing more about your project) that the current version of GUN is not very good at importing data. You might run into this https://github.com/amark/gun/issues/152 problem. The reason why is because GUN is functional/reactive, so when you do procedural things (like importing data) it starts triggering too many event listeners.

But we've fixed that in the upcoming version. GUN is a lot smarter about handling procedural operations, and doing things like data imports should work well. But this version is not out yet. So just heads up that if things crash to not get discouraged!

Please tune in and make sure to report anything so we can use it as a test case for the next version. Also don't be shy about hitting up the https://gitter.im/amark/gun . Thanks for closing.

[matthiasg]

@amark what exactly is gun capable of storing ? how much data ? is it all in ram ? trying to compare yo orientdb :)

[amark]

@matthiasg GUN persists to a variety of storage layers (S3, Level, etc.). That of course is only limited by the amount of storage of the hardware.

GUN does cache things in memory, and NodeJS with V8 supposedly only has a 1GB heap - I don't know how true this is or not. This is a NodeJS constraint not a GUN constraint.

How much? Comparison to Orient? We'll know soon, we're developing a bunch of benchmark test suites to stress test GUN with a new distributed testing framework we're building called Panic ( https://github.com/gundb/panic-server ). But we don't know yet - that is the honest answer, but we're engineering things to measure and test that.