Orbit-db: Does db.load() put all the data in memory ?

[maroodb]

I want to know if db.load() put all data in memory or not, if yes how can we deal with a huge amount of data?

[fazo96]

The oplog is not kept all in memory if I understand correctly, but all of the Stores that are available by default keep the Index entirely in memory.

The Load operation is O(n) unfortunately and has to be done at every application startup. This means that the load time of your application will grow linearly with the amount of data (ouch). There is no quick fix to this yet that I know of.

Then, you have the Store/Index problem where the Index is kept all in memory. You can solve this by writing your own custom Store that uses an external database, localstorage, IndexedDB or some other not-in-memory solution to keep the data.

I suggest you check out how the kvstore is written (it's pretty simple really) so that you undestand how you can write your own Store that could either be a more efficient implementation of an existing store or something new.

[mistakia]

@fazo96 I'm not sure if I'm looking at the right thing - but it looks like the oplog is kept in memory.

My understanding is that upon calling load(), the entire oplog (from ipfs-log constructor) and a store specific index are kept in memory.

This is not ideal (especially on mobile). It appears you can pass an amount to load() but I'm not sure what thats for and this will affect the completeness of the index?

For replication to work properly, only the heads and multihashes of entries are needed to exchange entries properly. However, I'm not sure what ipfs-log needs to be able to join logs (after exchanging entries) and append new entries.

I'm going to do a bit more digging and see if I can limit it to just keeping the heads and entry hashes in memory and read the rest from disk.

[its-VSP]

Hi @mistakia @fazo96 @maroodb
I got into a context where i could able to create db but running into below error when i am trying to load it

TypeError: Cannot read property 'getPublic' of null

where i am running local ipfs daemon without '--enable-pubsub-experiment' option
Can you help me in solving it and explain why that option is required if that is causing above error

[aphelionz]

Hi @its-VSP. Sorry this got buried. "--enable-pubsub-option" is simply required for OrbitDB, there's no way around it.

For more information about how data is marshalled between a storage layer, memory, and replication, please check out the new OrbitDB Field Manual and feel free to open issues there if you have more questions!