Orbit-db: Can't replicate db from another node

[Zorinik]

Good morning,
I'm trying to spin up OrbitDb for the first time in Node.js.
First I tried IPFS, installed as npm module in two computers: in one computer I added a file (via a .js file), and in the other one I retrieved it, it worked pretty well and suprisingly fast, gotta admit.

So I assumed that the 2 computers are able to find each other pretty quickly.

Then, I added OrbitDb, created a simple keyvalue, but I can't manage to replicate it on the other side!
This is the code I used in the "write" node:

const IPFS = require('ipfs')
const OrbitDB = require('orbit-db')

async function main() {
    const ipfs = await IPFS.create()
    const orbitdb = await OrbitDB.createInstance(ipfs)

    const db = await orbitdb.keyvalue('first-database');
    console.log(db.address.toString());
    await db.load();

    await db.put('key1', 'test');
}

main()

And this is the code in the other PC:

const IPFS = require('ipfs')
const OrbitDB = require('orbit-db')

async function main() {
    const ipfs = await IPFS.create()
    const orbitdb = await OrbitDB.createInstance(ipfs)

    const db = await orbitdb.keyvalue('/orbitdb/zdpuAoxx2Ag97s2GGq9EocJF6iucXivpeGzKTvFdJB4p6RskH/first-database')
    console.log(db.address.toString())
    await db.load()

    db.events.on('replicate', (address) => {
        console.log('Begin to replicate...')
        console.log(address)
    });

    db.events.on('replicate.progress', (address, hash, entry, progress, have) => {
        console.log('Progress: ' + progress)
    });

    db.events.on('replicated', (address) => {
        console.log('Replicated!')
        console.log(db.get('key1'))
    });
}

main()

The events "replicate", "replicate.progress" and "replicated" are NEVER fired, so it doesn't even try to replicate the other address at all.
What am I doing wrong, what am I missing?

Thank you!

[tabcat]

Hi Zorinik,
replicating stores has always been somewhat difficult (at least for me).

It's good that you know the nodes can transfer files/talk to each other.
I think the issue with this is that discovering pubsub peers, in my experience, can be pretty iffy compared to discovering content on the network.

Some things that can help is adding some more discovery methods to your ipfs node. A common one is websocket-star. You can see partly how to do this in this custom libp2p example.

I'd also be interested in any results using pubsub peer discovery or stardust.

[ohager]

I'm struggling with the same problem...I created a quick'n dirty Proof of Concept to try the replication feature of Orbitdb (after I struggled with tools like the control-center).

That PoC has a producer who creates the orbitdb and adding periodically some dummy records. Furthermore, there's a consumer that just wants to listen to the added records. Each of them runs its own IPFS node. Running both on my local machine works like a charm. Then we tried it remotely on different computers in different global regions (no cloud hosting)...and it didn't work. I'm no network expert, and I know that is something about the configuration, but I really don't know what do now. Any help is highly appreciated.

@Zorinik, could you solve your problem? @tabcat, it seems you passed already through these issues...maybe you could help too

[phillmac]

On your consumer, take the Id output here by the producer: https://github.com/ohager/orbitdb-remote-poc/blob/master/producer.js#L34
and run this on the consumer: await ipfs.swarm.connect('/ipfs/<producer id here>')
I see you are using js-ipfs, possibly try using go-ipfs via ipfsd-ctl and go-ipfs-dep
WARNING
I'm no longer recomending upgrading to > 0.4.23 because of broken p2p-circuit relay functionality
In fact I actively advise sticking with 0.4.23 as the last working version. see https://github.com/ipfs/go-ipfs/issues/7433

Bad advice don't follow
The dht efficiency has improved beyond measure in the latest stable release of go-ipfs (v0.5.1 as of writing); most times the ipfs nodes will be able to find each other and peer right out of the gate

[ohager]

thanks for fast and concise response. I'll try ASAP. :pray:

Just one more doubt: I tried using the Demo from orbitdb.org (https://ipfs.io/ipfs/QmeESXh9wPib8Xz7hdRzHuYLDuEUgkYTSuujZ2phQfvznQ/) and added the database address there... but without luck. Any ideas on that?

[phillmac]

Try this version of the demo. I think it's slightly more up to date: https://ipfs.io/ipfs/Qma4jzYSZkzUt4Tz4Pxr7KGupHCViaFNZh13bsEhovPkZp/

[ohager]

Hi @phillmac. I just tried what you suggested...running on my local machine works fine, but we got the following error:

Connecting to swarm using /ipfs/QmdkhnnKfGD9dWteVP9so1Bfi36FZXLbueDwduMHNiquNy
(node:6620) UnhandledPromiseRejectionWarning: AggregateError:
    Error: No transport available for address /p2p/QmdkhnnKfGD9dWteVP9so1Bfi36FZXLbueDwduMHNiquNy

[phillmac]

WARNING
I'm no longer recomending upgrading to > 0.4.23 because of broken p2p-circuit relay functionality
In fact I recommend sticking with 0.4.23 as the last working version. see https://github.com/ipfs/go-ipfs/issues/7433

[Zorinik]

I'm still unable to solve the issue; I tried the last provided solution (ipfs.swarm.connect), but I get the same error as of @ohager :
(node:11344) UnhandledPromiseRejectionWarning: AggregateError: Error: No transport available for address /p2p/QmRzzfyk9Ckwg6pwTTiyywa7fFx2icU9QtHNXDYpCutadg at TransportManager.dial (C:/Users/zorin/Desktop/Siti/healthcare-app/node_modules/libp2p/src/transport-manager.js:83:21) at DialRequest.dialAction (C:/Users/zorin/Desktop/Siti/healthcare-app/node_modules/libp2p/src/dialer/index.js:140:36) at C:/Users/zorin/Desktop/Siti/healthcare-app/node_modules/libp2p/src/dialer/dial-request.js:58:29 at async C:/Users/zorin/Desktop/Siti/healthcare-app/node_modules/p-some/index.js:53:19 at maybeSettle (C:\Users\zorin\Desktop\Siti\healthcare-app\node_modules\p-some\index.js:31:11) at C:\Users\zorin\Desktop\Siti\healthcare-app\node_modules\p-some\index.js:69:23

[phillmac]

@Zorinik For nodejs, you need to enable listening on TCP ports in your config. e.g.

"Addresses": {
    "Swarm": [
      "/ip4/0.0.0.0/tcp/4001"
    ]
  }

This will enable the TCP transport and fix the no transports error

[Zorinik]

Still no luck, same error; this is the code I use; what am I getting wrong?

const ipfs = await IPFS.create({ "config": { "Addresses": { "Swarm": [ "/ip4/0.0.0.0/tcp/4001" ] } }, relay: {enabled: true, hop: {enabled: true, active: true}}, pubsub: {enabled: true}, EXPERIMENTAL: {pubsub: true, ipnsPubsub: true} }) await ipfs.swarm.connect('/ipfs/QmRzzfyk9Ckwg6pwTTiyywa7fFx2icU9QtHNXDYpCutadg') const orbitdb = await OrbitDB.createInstance(ipfs)

Both IPFS and Orbit are updated to their last version.

[ohager]

@phillmac I saw your discussion in https://github.com/ipfs/go-ipfs/issues/7433 - I still had no luck to make it work. I understood that go-ipfs nodes take advantage of DHT, but it was mentioned that DHT does not work in Browsers...does it mean it might work using nodejs, or is it not implemented in js-ipfs at all?
Is there some way to get in contact with you like discord? Such, you could help out directly. It's all pretty confusing to me. And we are building a product which relies on that feature, otherwise I need to look out for working alternatives.

[aphelionz]

@ohager please join us on Gitter, which is bridged to Discord. It would be great if we could both solve the problem and come to come sort of consensus on the proper updated solution

[ohager]

@phillmac @Zorinik @tabcat
I had some time today for further stuff and made it work, such that I can go on using it in my project.
For me it's sufficient to set the Bootstrap addresses accordingly. Even the issue with connection loss on each side was solved.... it's all a question of correct configuration. For non-local communication port forwarding may be necessary.
I created a demo repo (https://github.com/ohager/orbitdb-remote-poc) to demonstrate the replication and pubsub stuff. Works fine now!
From my perspective this issue can be closed, but the final decision remains to @Zorinik . Thanks for your attention and help.

[am2222]

@ohager Hello,
I tried your git and it worked but is it possible to do that in the browser version? Since I tried and it seems in browser we cannot make socket server.

[ohager]

@am2222 tbh, I don't know. :shrug: