Go-ipfs: jsipfs compatibility blockers

[whyrusleeping]

Currently js-ipfs and go-ipfs cant quite communicate properly. This will be a log of existing issues that need to be resolved/worked around.

• [x] js-ipfs does not implement secio
  
  
  
  • can workaround this in testing, cannot interop with main network without this
  
  
• [x] js-ipfs does not have 'plaintext' handler for unencrypted comms
  
  
  
  • @diasdavid pushed a fix for this
  
  
• [ ] with encryption disabled, go-ipfs doesnt properly resolve the remote sides peer ID
  
  
  
  • we assume this knowledge is gained during the secio handshake
  
  
• [x] with encryption disabled, go-ipfs never seems to call IdentifyConn on the connection when js-ipfs dials in. When go-ipfs dials in (also encryption disabled) it _is_ called.
  
  
  
  • CORRECTION: IdentifyConn gets called, but it appears to hang while opening a new stream to the js-ipfs node
  
  
• [x] go-ipfs does not have websockets merged into master
  
  
  
  • have experimental support working, deploying to the main network tricky due to multiaddr bugs (failure to parse addresses)
  
  

[jbenet]

For crypto auth, @diasdavid is there a good TLS implementation on the browser? if so:

• [ ] add vanilla TLS support to go-ipfs
• [ ] add vanilla TLS support to js-ipfs

For websockets, there's:

• [x] websockets listen/dial support in go-ipfs
• [x] websockets selective address advertising (> 0.4.2)

[whyrusleeping]

using TLS means we have to add an extra step to verify identity. We currently use the secio handshake to ensure that we're talking to the person we think we're talking to.

[daviddias]

TLS 1.3 is still a draft and it doesn't have any implementation in JavaScript. (Nor it has been widely battle tested as stated by the RTC)

TLS 1.2 and 1.1 are available in Node.js in very close ties with the HTTP module through the OpenSSL library and there is an implementation in JavaScript on the node-forge library, more details on:

• http://digitalbazaar.com/2010/07/20/javascript-tls-1/
• http://digitalbazaar.com/2010/07/20/javascript-tls-2/

Note that these libraries are very aligned with their expected work mode, that is, the client (browser) making HTTP requests to a Server (that has a certificate signed by a certificate authority), there isn't something as easy as a 'connection upgrade' to upgrade a channel to become crypto channel with TLS.

[whyrusleeping]

Things we need to do for js-ipfs to communicate with a separate go-ipfs network with encryption disabled:

• [x] set up custom build of go-ipfs with websockets and multistream protocol name fixes.
• [ ] deploy go-ipfs bootstrappers with:
  
  
  
  • --disable-transport-encryption
  
  
  • websocket listeners
  
  
  • bootstrapper configs changed
  
  
• [ ] fix issue in go-ipfs where peer ID is not properly discovered without secio
• [x] debug identify handshake not working between go-ipfs and js-ipfs

[confiks]

You might also get away with using the gopherjs transpiled version of Go's crypto package. Can make interop with go-ipfs more likely as well. You'd probably not have any problems finding equivalent libaries for the hashes, but RSA and especially the curve implementation may bite you with interop.

[dignifiedquire]

I've started looking into putting secio into js-ipfs and I think it will be the most straightforward solution for our current problems. If and when we settle on some version of TLS we can add that to both implementations at the same time.

[whyrusleeping]

A couple more things david and I just found out:

• js-libp2p uses /ipfs/identify/1.0.0 while go-ipfs uses /ipfs/id/1.0.0
• go-ipfs uses varint delimited protobufs in all of its protocols. js-libp2p currently does not

[whyrusleeping]

note for @dignifiedquire on secio, we use msgio to send length prefixed protobuf messages for the handshake. Hopefully this saves you some time there

[dignifiedquire]

@whyrusleeping thanks ❤️

[daviddias]

@dignifiedquire also, we can't use protocol-buffers-stream cause it adds a 'message Id', unfortunately :(

[dignifiedquire]

@diasdavid moved this issue to https://github.com/diasdavid/js-libp2p-swarm/issues/62

[dignifiedquire]

Added tracking issue for integrating secio: https://github.com/diasdavid/js-libp2p-swarm/issues/63

[daviddias]

We've managed to get interop between go-ipfs and js-ipfs at the bitswap level \o/! :D

However, it was required to perform some hacks in go-ipfs to make it work, on the secio level, due to assumptions that are made on go-libp2p. @whyrusleeping could you document those here?

[daviddias]

As per js-ipfs ms2, we need:

https://github.com/ipfs/js-ipfs/blob/master/ROADMAP.md#dependencies

@whyrusleeping do you have an ETA for these?

[whyrusleeping]

@diasdavid The first one you don't need. You guys are implementing secio anyways.

The second one is coming, I'm hoping to get it in within the next couple of weeks.

The third one is going to require refactoring multistream with the changes we talked about the other day (passing the selected protocol up on the stream object). I'm going to try and have that done by the end of august.

[daviddias]

1) Indeed, but it would make testing so much easier, because as we finish secio, knowing that everything works without it, would be perfect.

2) sweet!

3) Right, we didn't agree in making a network breaking change, but to support both. Would you mind rebasing master onto https://github.com/ipfs/go-ipfs/pull/2704, so that we use the latest ipfs for the tests and not that build from a month and half ago.

Thank you! ❤️

[whyrusleeping]

The problem with 1 is that its really hard to do 'correctly'. I have it working on that test branch but its a gnarly hack.

[whyrusleeping]

Dude, js and go can interop now, its super great.

[daviddias]

woooooot :D 🎉🎉🎉🎉🎉