Orbit-db: db.load() sometimes finishes before db is actually synced.

[shd101wyy]

Hi there,

I have the following code:

const db_addresses = ['db address1', 'db address2', ...]
for (let i = 0; i < db_addresses.length; i++) {
      const db = await this.orbitdb.keyvalue(db_addresses[i])
      await db.load()     

      // This sometimes returns `undefined` 
      // but not the real value stored in db.  
      const contentHash = db.get('name') 
}

However, if I manually delay a few seconds like this:

const db_addresses = ['db address1', 'db address2', ...]
for (let i = 0; i < db_addresses.length; i++) {
      const db = await this.orbitdb.keyvalue(db_addresses[i])
      await db.load()     

      await delay(4000) // delay 4s

      // The real value stored in db will be returned.
      // No `undefined` returned. 
      const contentHash = db.get('name') 
}

Thank you ;)

[shd101wyy]

Might be related to https://github.com/orbitdb/orbit-db/issues/258 .

ps: I feel .load() and .events.on("replicated", ()=> ...) are very confusing. Maybe .load() should finish when replicated is triggered?

[shd101wyy]

Again, thank you very much for creating this awesome project. ;)

[shd101wyy]

nvm, I think I find the solution. I need to use events.on('read', ()=> ...).

[haadcode]

@shd101wyy I know what's happening here: for loop is not asynchronous, meaning await won't wait. With await, an asynchronous version of the for loop should be used, ie. for ... of:

for (let address of db_addresses) {
      const db = await this.orbitdb.keyvalue(address)
      await db.load()     

      // This sometimes returns `undefined` 
      // but not the real value stored in db.  
      const contentHash = db.get('name') 
}

That will wait for the load to finish before starting the next loop. Let me know if this didn't work for you!

[haadcode]

And thank you for your kind word! ❤️ 😄

[shd101wyy]

@haadcode Thank you very much! Nice job!

[shd101wyy]

@haadcode Sorry I just tried for ... of ., it still returns undefined sometimes when I try to get a value from db.

Besides, I was wrong about .load() and .events.on('ready', ()=> ...).
.load() actually finishes after .events.on('ready', ()=> ...).

Thanks ;)

[shd101wyy]

Seems that db will replicate, and await db.load() finishes before replicated. Maybe that's why I get undefined because I haven't finished replication.

So my question is how should I know if the db will be replicated or not? When I run the code on my local server, there will be no replications. But if I run them remotely, replications will happen.

What's more, replicated seems to be called multiple times in one iteration of the for loop (I didn't update db).

Please let me know if you need more further information form me. Thanks.

[haadcode]

Oh ok, I get it, the database you load() is a remote database.

Let's recap: load() is for loading the database that is already replicated (can be partly or fully) on that peer, ie. locally. If the database hasn't been replicated before, load() doesn't have anything to load and returns an empty database, but as soon as it's connected to a peer that has the database, it'll replicate as the peer who has the database will send their latest entries/heads to the peer that doesn't have the database.

So, when opening a remote database, that doesn't exist locally, replicated event should be listened to as it'll notify when something was updated in the database. You got it all correct, but when to query, should be:

const db_addresses = ['db address1', 'db address2', ...]
for (let i = 0; i < db_addresses.length; i++) {
      const db = await this.orbitdb.keyvalue(db_addresses[i])
      await db.load()     

      //await delay(4000) // delay 4s
      // Wait until we have received something from the network
      db.events.on('replicated', () => {
          const contentHash = db.get('name') 
      })

      // When the database was updated *locally*, ie. an entry was added with db.put(),
      // we can use the 'write' event to get a notification the database was updated
      db.events.on('replicated', () => {
          const contentHash = db.get('name') 
      })
}

I hope this helps! Thank you for reporting these problems and asking questions. I now realize the replicated/write distinction is not clear and I'm under the impression that as a user, you would expect load() to load the whole database whether remote or local, is that correct? Any suggestions as to how to make it more clear?

[haadcode]

Sorry, missed your previous comment on suggestions how to improve it:

ps: I feel .load() and .events.on("replicated", ()=> ...) are very confusing. Maybe .load() should finish when replicated is triggered?

Thanks for that feedback! The reason it's working as it now, is that we wnt to be able to load the locally persisted state of the database into memory as quickly as possible and then "start listening for more updates" via the replicated event. Furthermore, replicated event gets fired for every update to the database, so for example, when there's a database that has 4 entries and it gets replicated from a remote peer, replicated will fire 4 times (1 for each update). We also don't make a distinction whether the database is fully replicated or just partly when the replicated event fires.

Let me think about how we could change this (I don't have a proposal yet). If you have any thoughts, would be happy to hear.

Perhaps we should have just one update event for whenever a change happens, locally or remotely (ie. combine replicated+write to one event). That way there's only one event to listen to to know when something was updated in the database and it should be re-queried.

Maybe we should also have an event for "the database is fully replicated and all updates are now there" instead of current replicated event meaning "something was updated".

What do you think?

[shd101wyy]

Hi @haadcode , thank you very much for your response.

I now realize the replicated/write distinction is not clear and I'm under the impression that as a user, you would expect load() to load the whole database whether remote or local, is that correct?

Yes, as a user I hope await db.load() finishes when the database finishes loading (syncing) either locally or remotely and the database is ready to be queried. So when I query a value from the database after calling await .load(), it won't return me undefined.

---

Perhaps we should have just one update event for whenever a change happens, locally or remotely (ie. combine replicated+write to one event). That way there's only one event to listen to to know when something was updated in the database and it should be re-queried.

Yes it would be awesome. I think combining both replicated and write into one event is a very nice idea. Maybe you just need to pass a argument like isModifiedLocally to the callback function so that user can know if that update happens locally or remotely?

---

Maybe we should also have an event for "the database is fully replicated and all updates are now there" instead of current replicated event meaning "something was updated".

Yes it would be very nice.

---

Below are my immature idea:

const db = await orbitdb.keyvalue('dbname or remote address');
await db.load()      // db is ready to be used (fully loaded and synced).  

db.get('name')       // this won't return `undefined` because db is fully loaded(synced) now.  

// one entry is updated in the database
db.events.on('update', (address, local)=> {
    console.log('db is modified locally: ',  local)
})

// all entries are updated
// but I think this might be hard or unnecessary to implement? I am not very sure
db.events.on('updated', (address, locall)=> {
    console.log('db is modified locally: ', local)
})

Thank you!

[cristiano-belloni]

Hi @haadcode, just a quick one regarding something you wrote here: if a database is completely remote, loading it has no effect, am I right? In other words, there's no automatic "pinning" of remote dbs and any remote db must be replicated from peers, is that correct?

(In case I'm correct, do you plan to add a pinning functionality? I'm at the point where I can show remote user data and feeds, but I'd like for the user to be able to pin a feed or post they like)