Js-ipfs: ⚡️ v0.27.0 RELEASE 🚀

[haadcode]

@diasdavid, I ran tests from master against fresh version of ipfs-log and soon-to-be-released new version or OrbitDB and the tests are running all fine! 👍 🎉 There are a bunch of replication and pubsub tests as well as file and object API tests, so it looks solid. There's couple of API breaking changes that I found that I think should be highlighted:

• ipfs.cat() --> ipfs.catReadableStream() (when previously used with readable-stream)
• pubsub message topicCIDs --> topicIDs

Feel free to use the above PRs for testing the before final release and let me know if you need any help.

[pgte]

@diasdavid ran the tests for ipfs-pubsub-room, and released version v0.6 of it that supports the new pubsub message format. All tests pass.

Updated y-ipfs-connector (released version 1.6.0 of that) and tested peerpad with it. Also, all looking good! 👍

[daviddias]

@pgte @haadcode mind testing again with latest master?

[daviddias]

Question: Shall we disable WebRTC by default?

I'm leaning towards yes. Seems that we are not yet ready to use WebRTC without causing confusion to the users (see: https://github.com/ipfs/js-ipfs/issues/1089). https://github.com/ipfs/js-ipfs/issues/1088 is one of our top priorities and I would be more comfortable enabling WebRTC-Star or WebSockets-Star once we make more progress.

This doesn't mean that WebRTC won't work, you just need to add the multiaddr manually so that it gets activated.

Thoughts?

[Beanow]

Question: Shall we disable WebRTC by default?

Yes for me as well.

Regardless of how you use WebRTC there's always the risk of using one too many resources and crashing. Though the primary reasons to be using it is having out-of-the-box peer discovery. Which you either want to go "true P2P" the easy way, or to have pubsub.

For both I think we should add clear configuration examples. As I would expect that to top the FAQs when WebRTC is not available by default.

[interfect]

Without WebRTC enabled by default, how will browser nodes be able to communicate with each other? Will WebSockets allow that?

Will turning on WebRTC manually just bring back the browser crashes?

[daviddias]

@interfect, it will prevent the crashes and avoid causing confusion to users that do not require to connect to every browser node.

The proper solution for the problem will be #1088

[daviddias]

Update: PR here https://github.com/ipfs/js-ipfs/pull/1111

[daviddias]

I'm working towards https://github.com/libp2p/js-libp2p/pull/122 to give another option for Browser connectivity which in theory and from tests run, will be way more stable than the current WebRTC situation.

[daviddias]

I believe that this PR -- https://github.com/ipfs/js-ipfs/pull/1111 -- will make a lot of people happy :)

[Beanow]

@diasdavid yay! :tada:

As a sidenote this may be a good moment to review how we should warn users about the unsafe rendezvous. As this is the first time it should work out-of-the-box. The WebRTC crashes should have been a big enough deterrent for most production use until now.

Ideally it would be clear during development and runtime, but at the same time allow for upgrading to requiring safe variants without config changes later (provided they didn't manually configure an unsafe rendezvous server).

[daviddias]

@Beanow I realized today that we can actually have their _pseudo safe_ version :) (I wouldn't call anything safe or secure until properly audited). See https://github.com/ipfs/js-ipfs/blob/master/src/core/runtime/libp2p-browser.js#L15

[Beanow]

Ah that's right. It's only if you want to add the module yourself it becomes an issue.
Perhaps then we should add something like: new WebSocketStar({allowUnsafe: true}) as an explicit requirement before attempting to connect without ID challenge. For those that wish to pass the module.

[daviddias]

@Beanow The way it is built right now is that the crypto puzzle validator is the rendezvous point, if someone wants to "pass in the module" they will have to use another rendezvous point.

[Beanow]

@diasdavid which would be OK if there is a distinction in addresses between the two modes and the client can validate the expected behavior of the rendezvous. Like http vs https.

In other words I'm suggesting WebSocketStar should be the https example and downgrading security properties should be explicit. Similar to how you would need to allow self-signed certs explicitly in an https client.

[daviddias]

Understood and agreed, @Beanow, although I believe it is not clear what the "unsafe" wording being used in these rendezvous points means.

To clarify, libp2p and therefore IPFS as well, will __always__ validate that they are trying to dial to the right node through the SECIO handshake.

That the crypto puzzle that the rendezvous does is to enable that point to also be able to check that it is being dialed by the right now, avoiding someone to pollute the routing table.

This is a known issue and our proposal to fix goes in two stages:

• a) enable the security guard that @mkg20001 implemented in both webrtc-star and websocket-star (already done).
• b) Upgrade libp2p module loading structure so that transports can use other components already loaded (as per initial design but then it didn't get into the implementation) so that instead of having to use rendezvous points, they can dial to any IPFS node and make any IPFS node a rendezvous point. I've discussed this proposal in length with multiple people and now @pgte is taking the lead for it, track here https://github.com/libp2p/js-libp2p/issues/130

[mkg20001]

implemented in both webrtc-star and websocket-star

@diasdavid webrtc-star only has type checking to prevent server crashes. crypto challenge isn't added.

[daviddias]

@mkg20001 opened a issue to track that https://github.com/libp2p/js-libp2p-webrtc-star/issues/128 :)

[Beanow]

@diasdavid true, safe / unsafe is pretty polarizing for a subject like security in general, we'd need an unambiguous term for the flag.

While DI and SECIO are great, those depend on the transport client, it doesn't make directly instantiating the transport client more explicit in it's behavior.

Though in spite of SECIO, being able to trivially proxy traffic still lets you collect metadata and gives you handshakes and cyphertext to store, replay or attempt any other kind of attack on. A situation that proved pretty detrimental for WEP and WPA.

[daviddias]

@Beanow absolutely correct :) Just so you know, we are indeed working towards getting IPFS audited. Both known and unknown issues will get a priority tag so that we can work on patching them.

[mkg20001]

A good solution would be to add an "allowJoinWithDisabledChallenge" (let's make the name long and annoying to type so nobody uses it :D ) parameter and throw listening errors if a user tries to listen on an insecure server and the flag is false.

[daviddias]

Let's make sure to continue those discussions, especially in the issues for js-libp2p and the transports.

In another news, just went through the items on the Release checklist:

• [x] docs are updated
• [x] tests are passing
• [x] pubsub-room and peerpad work with new js-ipfs and also with the new transport!

It is time to hit the drums and announce that the release is ready and invite everyone to try it out before publishing to npm! 🚀

[Beanow]

@diasdavid hate to be nitting over this buuuut. I like @mkg20001's solution, but it's the very definition of an API breaking change.

Which means: libp2p-websocket-star goes from ^0.5.1 to ^0.6.0 in the direct dependencies and doesn't automatically end up in this release. Which is why I'm bringing it up now and on this issue.

[daviddias]

@Beanow I'm confused by your statement. libp2p-websocket-star latest is v0.5.1, it has the crypto puzzle and there is nothing in the pipeline for that one now. What were you trying to convey?

Update: If you were trying to say that adding the Crypto Puzzle to _webrtc-star_ will be a breaking change, I do not necessarily agree or disagree because it depends on the migration path we set up for users. We could:

• Start by allowing both to work (as it is now) and in a future release of js-ipfs disabling the handshake without the Crypto Puzzle. This way we mitigate the effects and leave space for smooth migration.
• Just ship it right now and encourage all users to upgrade. This would result in delaying this release or just shipping another version once it is done.

I'm more inclined to the smooth transition path.

[mkg20001]

@diasdavid @Beanow Added the requested changes to libp2p-websocket-star

Should I ship it as 0.5.2 or 0.6.0?

[Beanow]

@diasdavid adding a constructor argument such as allowJoinWithDisabledChallenge is an API change. And if we release [email protected] with ^0.5.1 for the dependency, it won't satisfy this API change when it lands.

Anyone using the built-in [email protected] from [email protected] would be vulnerable to silent downgrades. For example by changing the rendezvous server address.

[daviddias]

We do not need to set up this flag allowJoinWithDisabledChallenge at all, the transport has access to the PeerId, it can always execute the Crypto Puzzle. See https://github.com/ipfs/js-ipfs/blob/master/src/core/runtime/libp2p-nodejs.js#L16

Also, thanks for catching the ^0.5.1, that is a typo. We don't release pre 1.0.0 deps with ^, only ~. That said:

If in the future a peer agrees to join without the Crypto Puzzle (because its users set it up like that), a previous node that doesn't support that mode won't _get downgraded_ as it will just try to do the Puzzle with the other peer and then fail.

I might be missing something really obvious here. My apologies for making this convo long and possibly frustrating. I'm sure we can get to the bottom of it :)

[mkg20001]

@diasdavid No if the id is set it can still join to a server without the puzzle! (Fixed in 0.6.0)

[daviddias]

@mkg20001 ah, it was a bug on js-libp2p-websocket-star then, got it. Glad to you already fixed it. Let's make sure to track these things in issues when they pop up.

[ghost]

The /p2p-websocket-star addresses need fixing, they lack the /tcp part which is essential for TCP-based addresses, see libp2p/js-libp2p-websocket-star#35

[daviddias]

@lgierth the transport actually supports multiple variations https://github.com/whyrusleeping/js-mafmt/blob/master/test/index.spec.js#L87-L92. We have been doing the same for p2p-webrtc-star for while too https://github.com/whyrusleeping/js-mafmt/blob/master/test/index.spec.js#L75-L80

Are you proposing that we should fail on a multiaddr that doesn't explicitly call the tcp+443+wss?

[ghost]

Are you proposing that we should fail on a multiaddr that doesn't explicitly call the tcp+443+wss?

Yes that would be good -- they're incorrect multiaddrs: /p2p-websocket-star requires a stream underneath, and /ip4 and /ip6 don't provide that capability. (In the case of /p2p-webrtc-star it's okay since there's backward-compat concerns, but the old-style addresses should be removed at some point too.)

It's not critical to remove the support right now, but at least the docs and tests should be updated before release, so that nobody starts to use these addresses in the first place.

[mitra42]

is pubsub intended to replace YJS, or is YJS using ipfs-connector running over it, or is the best practice to configure YJS differently to use pubsub ?

Obviously there are parts of YJS crucial to pubsub type applications - which dont appear to be in the API - especially the ability to get the history of a channel in order for a newly joining peer to build the current state. Is it intended that YJS should be the way to do this going forward ?

[haadcode]

@diasdavid tests in orbit-db running fine with the latest master 👍

[daviddias]

Hi all, I created an issue for us to track the path moving out of the centralized rendezvous points which also brings a lot of context to why they exist and why they still exist -- https://github.com/libp2p/js-libp2p/issues/134.

It's not critical to remove the support right now, but at least the docs and tests should be updated before release, so that nobody starts to use these addresses in the first place.

Alright, I'll update it everywhere! Thanks for reviewing, this why I like to create an announcement first and wait a bit to release :)

Update: Track: https://github.com/ipfs/js-ipfs/pull/1117

@diasdavid tests in orbit-db running fine with the latest master 👍

Woot! Thanks @haadcode! Does that mean that there will be a new orbit.chat soon with latest js-ipfs?

is pubsub intended to replace YJS, or is YJS using ipfs-connector running over it, or is the best practice to configure YJS differently to use pubsub ?

Good question. y.js using it's ipfs-connector uses PubSub already to propagate it's events (see the code: https://github.com/ipfs-shipyard/y-ipfs-connector/blob/master/src/index.js#L33).

One is not designed to replace the other, in fact, one enables the other. What you see happening is:

• PubSub is _innovation_ that has made its way into the IPFS Core Protocol
• a sync'able KeyValue Store such as the one powered by y.js is still in IPFS Userland. That is why you have to _attach it_

We do have the intention to have a KeyValue Store inside IPFS but that is still at its early stage -- https://github.com/ipfs/interface-ipfs-core/issues/157 --. The first step to that will be getting the API and a KeyValue store that works only locally (like MFS) and then design its syncing layer. Again, that is still very early stage, until then continue to check out things like y.js.

[mitra42]

@diasdavid - thanks for the clarification. To be clear, we don't use YJS for KeyValue store, we use it for Append-Only-Logs, a critical building block for a lot of apps, and very hard to do in IPFS by any other means.

[haadcode]

Woot! Thanks @haadcode! Does that mean that there will be a new orbit.chat soon with latest js-ipfs?

@diasdavid I won't have time to update orbit.chat anytime soon, but it is on my TODO list (we have API breaking changes in the new orbitdb, so orbit.chat needs a bit of work too).

we don't use YJS for KeyValue store, we use it for Append-Only-Logs

@mitra42 you might find https://github.com/orbitdb/orbit-db and its Log Database helpful for out-of-the-box append-only logs "natively" on IPFS. We just released a major new version that fixes many of the problems you might've had with previous versions.

[mitra42]

@haadcode - the semantics of orbit-db append only logs, dont quite match, but I'll send you an email since its off-topic for release v0.27.0

[daviddias]

Thanks @alexsicart for trying out this pre-release \o/ Quoting: https://twitter.com/alexsicart/status/937305396263473152

"Thanks! I've tried and the performance improvement is hugee!! Also less memory usage."

[daviddias]

Notes on upgrading from js-ipfs 0.26 to 0.27

• The Files API got an upgrade, make sure to check it out at https://github.com/ipfs/interface-ipfs-core/blob/master/SPEC/FILES.md (and see update on the first post of this issue)
• The PubSub API got a small update that changes the key topicCID to topicID to match go-ipfs API
• WebRTC is now not enabled by default, please check the FAQ https://github.com/ipfs/js-ipfs#faq

[daviddias]

js-ipfs 0.27 released!

Tweet: https://twitter.com/daviddias/status/937565486912606208