Orbit-db: How to create a database from a known address

[AlessandroChecco]

In all your examples, you create a new IPFS and OrbitDB instance.
I would like to replicate your Live Demo 1, but I am not sure how to refer to an existing address when a client want to access a known database.

I guess I need to call new OrbitDB (or IPFS) with an optional parameters that indicate the address? I couldn't find it anywhere...

Thanks!

[AlessandroChecco]

Ok I think it's here:

const db = await orbitdb.eventlog('site.visitors')
// Or: const db = await orbitdb.eventlog(anotherlogdb.address)

but I am not sure: will eventlog function understand that the first case ('site.visitors') is not an orbitdb address and will create one? While putting a valid /orbitdb/multihas/db address in anotherlogdb.address will access to an existing one?

[maxkerp]

@AlessandroChecco Yes you are correct. Giving the stores only a string lets orbit-db create a new store. If you give a store a full orbitAddress it will try replicating the store.
Since stores are persistent what you could do is the following:

// Expecting to have the title 'peer1' and 'peer2' for different html pages.
<script>
currentPeer = document.getElementsByTagName('title')[0].textContent

if ( currentPeer === 'peer1') {
// You only need to run this block once, after that comment it out

  const store = await orbitdb.eventlog('site.visitors')
  await store.load()

  const fullAddress = store.address
  console.log(fullAddress)

  const hash = await store.set("Some event happened")
  console.log(hash)
} else {

  const fullAddress = // address from console of first browser window from first run
  const hash = // hash from console of first browser window from first run

  const store = await orbitdb.eventlog(fullAddress)
  await store.load()

  console.log(store.get(hash))
}
</script>

Remember that your ipfs instance has to be configured to swarm

const config = {
  EXPERIMENTAL: {
    pubsub: true
  },
  config: {
    Addresses: {
      Swarm: [
        // Use IPFS dev signal server
        // '/dns4/star-signal.cloud.ipfs.team/wss/p2p-webrtc-star',
        '/dns4/ws-star.discovery.libp2p.io/tcp/443/wss/p2p-websocket-star',
        // Use local signal server
        // '/ip4/0.0.0.0/tcp/9090/wss/p2p-webrtc-star',
      ]
    }
  }
}

const ipfs = new Ipfs(config)
const orbitdb = new OrbitDB(ipfs)

[maxkerp]

Maybe I can extend the questions a little bit, since I have a related problem at the moment and I think it makes sense to document the findings all in one issue. Although I studied the code from orbit-core and orbit-db on Saturday, I couldn't find an answer to this.

In orbit-db/examples/browser/ you have to manually copy/paste the full orbitAddress (/orbit/Qm../storeName) to the second (different) browser window to replicate the database among peers, so both peers can read/write to the same one (, instead of having each their own with the same name). Getting the address from one peer to another is what I call the "Address Handshake" (for a lack of a better term :sweat_smile:). Within orbit-db it doesn't tell how to do the "handshake" automatically.

In orbit-core/examples/browser/ two (different) browsers connect to each other automatically, without having to tell each other the full orbitAddress. How is this done? It doesn't really say how orbit-core achieves this in the repo either. I have some ideas which I will look into today, but any help pointing in the right direction is much appreciated. I know you guys a busy, but even two or three sentences about what code to look at will be a huge help.

What I'll be looking at
I think it has to do with the orbit-db-pubsub module and how peers in ipfs form a swarm. Theres is a "hack" in orbit-core which causes every peer to host the same ipfs object ({app: "orbit.chat"}). So this is how you know, that all your peers are your peers for the same reason.

What I don't understand is, that looking at the stores in the developer tools (orbit-core/examples/browser/) shows that they don't have the same full address, they sync up nonetheless. Which is how I know I'm missing something important. I thought reading/writing to the same store is done by having replicated the same (meaning identity) exact address.

Hopefully I can share some insights this evening :smile:

---

Btw, I think orbit-core should be mentioned in the orbit-db README.md. It's the best source of an example app utilizing orbit-db, but somehow I managed to overlook it, although I've been studying orbitdb for my bachelor's thesis for quite some time now. Judging from a conversation on the irc channel and seeing it only has 37 stargazers compared to ~ 1000 for orbitdb/orbit and orbitdb/orbit-db I might not be the only one (, I can create a pull request for this if wanted).

[fazo96]

@maxkerp when two nodes create a database with the same parameters, they end up with the same address because the db creation is deterministic

This means that if you want to open (for example) the same key value store from anywhere without ever knowing the address you can do this:

const db = orbitdb.kvstore('mystore', {
  write: ['*']
})

this will create a key value store named mystore where anyone is allowed to write. Anywhere you run this code will generate the same address (you can check with db.address.toString()) because it was created with the same parameters. In OrbitDB, creating a database just means opening one without knowing the address yet.

By default, if you don't specify the write permissions, the db is created so that only you (the orbitDb instance that's creating it) can write. This is why by default you couldn't get them to talk, each one was opening a different db because it had different permissions.

Let me know if this was clear enough :+1:

Edit: to respond to @AlessandroChecco's original question:

const db = orbitdb.kvstore('mystore')
const db2 = orbitdb.open(db.address.toString())

This works. db.address.toString() is just a string (looks like /orbitdb/Qm..../dbname and you can send it or hardcode it easily.

If you put a string like mystore it will just create a db, but by default it will use write permissions so that only you can write, so two different instances doing orbitdb.kvstore('mystore') will end up with different db addresses and different dbs.

[maxkerp]

Hi @fazo96,

thanks a lot for your comment, this is actually a key point I was missing the whole time. Somehow I managed to create only read-only stores and thus thought that any kind of store is always namespaced to the node creating it.

From a conversation on the irc channel with @haadcode I took away that orbit-core is just not up-to-date with orbit-db (orbit-core is missing the accessController) and I'm on the right track. I might not have made myself clear enough. I really need to work on my question asking skills :sweat_smile:.

This isn't really clear in the guides and could have saved me some days of work knowing it, I might create a pull request adding a few sentences about that public stores are also globally unique. (If not in the next weeks, I definitely want to contribute to the documentation after my thesis is done and share everything I learned.) So an application using orbit-db with public stores should definitely prefix its stores to avoid name clashing with other apps doing something like const quizes = await orbitdb.docs('myAppIdentifier.quizes', allWriteTrueOptions)

---

I don't know if this is part of this issue, but since we're here:

Opening a store in orbit-db should replicate a store with all its contents right away, right?

When I open a store in a different browser (B) it gets the latest values from the other browser (A) only when A writes to the store. So B has no remote data until someone in A decides to update or add data. Why isn't it getting all the values right away? What am I missing concerning replication?

These are the options given to the store and I tried it with a maxHistory: 1000:

const defaultOptions = {
  indexBy: 'key',
  maxHistory: -1,
  sync: true
}

Big thank you to both of you, you're helping me a lot :+1:

[fazo96]

@maxkerp that sounds wrong, node B should get all of the data that A has immediately after establishing a connection to A. Keep in mind that if nobody reachable has a copy of the data pieces you are missing then you won't be able to get them.

Here's some reasons why that might be failing:

• are you calling await db.load() right after opening the DB? This is needed to load the data "remembered" from previous sessions that is stored in the browser's IndexedDB or in the file system if you are on node.js. __Always__ do this except if you have some specific reason for not doing it. If A doesn't do this, when you restart it it won't load data written previously, and if B doesn't do this, it will need to replicate every time instead of just using what he got previously
• are you sure you are opening the DB right away? Maybe A is opening the DB only when it needs to write. A and B should both open the DB as soon as possible so that they can replicate the data with each other. If A opens only when it needs to write, then B won't be able to get anything from A until A writes
• are you using the latest stable orbitdb release? I think there might be replication issues in master. Also check that you are using the latest stable js-ipfs
• try putting some logging to these DB events: replicated, replicate, replicate.progress, load, load.progress. In the API.md file there is some more info on these

This is a very good discussion because it helps understanding which parts of how orbit-db works are less clear to people new to the project :)

[coyotespike]

Thanks all, this is a helpful discussion!

@fazo96, suppose one ran const db2 = await orbitdb.kvstore(db.address.toString()) - would that not work as well? I haven't found orbitdb.open() in the documentation, curious where you found that.

[maxkerp]

@coyotespike https://github.com/orbitdb/orbit-db/blob/a516d6d38a38140bda3325cc1281f343620d8a75/src/OrbitDB.js#L279
https://github.com/orbitdb/orbit-db/blob/a516d6d38a38140bda3325cc1281f343620d8a75/src/OrbitDB.js#L59-L66

Its part of the public API :wink: orbitdb.<store_type> uses orbitdb.open under the hood. So when using full addresses you are always on the safe side, because orbitdb wouldn't create a new store if given a valid full address.

[maxkerp]

@fazo96 You are completely right. Everything works just as you described. I think I messed stuff up by using to many different local .html files. Since the caching does happen per path. So tab_1.html, tab_2.html, and diffrent_browser.html all have their own cache. The tabs don't share a cache.

I have one new question:
Within one browser tab, does one store always have exactly one pubsub topic per store? If yes, I might have found a bug. I managed to create 4 topics for the same store.

# After closing the only store opened
2018-05-11T13:31:21.236Z [DEBUG] orbit-db.ipfs-pubsub: Topics open: 3

[coyotespike]

Thanks, @maxkerp, I knew that at one point! :wink: Been awhile since I looked at that part of the code.

I am still seeing that the database will not return. db1 is on one machine. On another machine, no matter which way I write it, the promise does not resolve:

const db2 = await orbitdb.open(<db1Address>);
---
const db2 = await orbitdb.docstore(<db1Address>);

[maxkerp]

Hmm.. Try looking at the debug output, add a window.LOG = "debug" to the .html page.
Is the ipfs node configured to swarm? The example in https://github.com/orbitdb/orbit-db/issues/366#issuecomment-387018956 should work just fine, no matter what store type.

[maxkerp]

One thing that just came to my mind:

Do the options to open() have to be the same as the ones to create()?
I realized, that a public store (write: ["*"]) can be opened with any kind of indexedBy: '<something>'-option, which lets you actually use different identity-keys in different tabs, meaning you can end up with a store like:

[
  { key: 'someKey', foo: 'bar' },
  { _id: 'someID', prop: 'value' },
]

Shouldn't this be part of the store itself?

Related to the above, do I have to know, if a store is public, when opening it by a full address, since I would need to pass in the write: ["*"]-options? I might have time to check for myself.

[coyotespike]

Finally figured it out: an issue with the IPFS node.

node2.swarm.connect(<node1Multihash>) did not work to enable communication. Both nodes had Swarm enabled, but only one was connected to the signaling server. After connecting both, swarm.connect worked (which may be irrelevant), and the database synced. Thanks for your help Max!

[fazo96]

@maxkerp it's important to understand that to replicate data two nodes don't have to use the same store. The name of the store is part of the address but maybe some of the nodes are running a different implementation and indexing data differently.

Under the "Index", which is the way you see the data when you use the document store for example, there is a oplog which is an instance of https://github.com/orbitdb/ipfs-log

Using the Log, the Store computes the current Index which is the documents as you see them when you query. This means that you can change how this Index is computed by the Store without influencing the rest of the nodes

Here is the "base" store that you can use to create your own: https://github.com/orbitdb/orbit-db-store

And here is the docstore if you want to check out the code. It's very simple, just check out the base store first https://github.com/orbitdb/orbit-db-docstore

[aphelionz]

This is a great discussion and it's awesome to see the community helping each other out like this! I'm gonna close this issue now and direct everybody over to the OrbitDB Field Manual repo. Feel free to open issues, and start new and awesome discussions there.