Js-ipfs: track: `bitswap`

[RichardLitt]

@diasdavid Bitswap doesn't have a repository associated with it. (Neither does Peer routing, but I'm not sure if that is top-level and if it needs one yet or not).

Has work been started on it yet? @mafintosh is keen to help now, if so, or to just start cracking.

[daviddias]

Thanks for adding this @RichardLitt :)

I would love to be able to give you something like bitswap, even if a rough implementation, but what happens is that we are currently under the development of libp2p in Node.js and during this process, we learned how to improve IPFS' network stack. What this means is that now we also have to upgrade go-ipfs network stack to be full libp2p and until then, even a rough impl of bitswap in Node.js won't work with any of the IPFS nodes that currently exist.

@mafintosh I'll keep you posted though :) Thank you for being available to help!

[RichardLitt]

@diasdavid I don't think I'd be able to do this, either way. But @mafintosh could probably make it work without needing to improve IPFS, he's hoping to use this module for something else, too.

@mafintosh What do you think?

[daviddias]

Bitswap is fun! So fun that we even made it the first episode of "Jeromy coffee talks" link

We can start hacking on bitswap, specially if we are able to define its interface beforehand, which would enable us to have multiple bitswaps working for us, some using libp2p, others using other transports or even disks that are mounted on the network, enabling more and more strategies to be defined and studied with its different implementations.

Thoughts @whyrusleeping ? Having a clear interface would help start shaping it on JavaScript land :)

[mafintosh]

This is great! I was actually reading up on this yesterday. Thanks for making the video.

[whyrusleeping]

@diasdavid yeah, bitswap actually has a really simple interface.

The idea is that bitswap implements the same interface that a blockstore does, so you have:

Get(key) (Block, error)
Put(key, Block) error

[daviddias]

TASKS:

• [x] write down the spec (how it works)
• calls
  
  
  
  • [x] jsipfs bitswap stat - Show some diagnostic information on the bitswap agent.
  
  
  • [ ] jsipfs bitswap unwant ... - Remove a given block from your wantlist.
  
  
  • [x] jsipfs bitswap wantlist - Show blocks currently on the wantlist.
  
  

[dignifiedquire]

work on this is happening here: https://github.com/ipfs/js-ipfs-bitswap

[daviddias]

Note that the branch with bitswap has been merged :)

[dignifiedquire]

Bitswap is implemented for the most part,

• js-ipfs commands
  
  
  
  • [x] ipfs bitswap stat
  
  
  • [ ] ipfs bitswap unwant
  
  
  • [x] ipfs bitswap wantlist
  
  
• block-service integration
  
  
  
  • [x] offline integration
  
  
  • [x] online integration
  
  

[dignifiedquire]

The main thing about unwanting and cancellation which is missing from this milestone depends on using an async primitive that understands cancellation.

[daviddias]

@dignifiedquire we had a conversation of how to do that with the current state of how modules are coupled, here are some notes, but might be good to jump in a call an clarify any loose ends and bring the clarifications back to the issue, also, nothing is set in stone, this is a hard and tricky part :)

Cancelation can happen in two ways:

a) The ipfs daemon (http-api) is running, a request is made and then 'canceled' by stopping the process, as in:

> jsipfs daemon &
> jsipfs cat /ipfs/QmHash
...
^c # kill the process

b) We are using the js-ipfs core (node embedded in an app) and we want to cancel a block request

ipfs.get(hash, callback) 
ipfs.unwant(hash, callback2)

A third option would be if a user does jsipfs cat and then jsipfs bitswap unwant through the cli, leaving the jsipfs cat hanging. But that is up to the user to own the execution of the jsipfs cat and it is an explicit call to unwant.

In the first case, a)

The http request would blow up, and the http-api handler would call unwant on bitswap, then the logic of case b) is applied.

In the second case, b)

ipfs.get(hash, function (err, streamOfThings) {
  // if the refcount goes to 0 in the unwant, we will get an `err` here, i there was another call looking for the same hash, we will still get the stream of things and this is where a dev would have to make a selfguard, a thing similar to the `context` in go-ipfs. However, I expect that there will be almost no cases of having a process doing several .get to the same context and just trying to cancel one, that is why it is left for the dev, for their specific case.
}) 
ipfs.unwant(hash, function (err) {
  // decrements the refcount of `hash` from the want list and all the hashes that were added under the `hash`
})

[daviddias]

@dignifiedquire can we get a brain dump of your thoughts on what to take bitswap next?

[dignifiedquire]

I will write up something later this week

[daviddias]

@dignifiedquire still waiting on that braindump.

Currently bitswap is not fully interopable with Go, we need to make sure we can exchange blocks, whole files without having any issue. In order to test this, the go-ipfs https://github.com/ipfs/go-ipfs/tree/deps/libp2p/3.5.1 branch must be used. @dignifiedquire can you take a look at this?

[dignifiedquire]

@diasdavid yeahh, not sure about it myself right now, will need to get back into bitswap. Will debug and fix the interop this week.

What I know right now we are missing

• Integration with the dht to request finding peers which have a block we want
• integration from the js-ipfs side with bitswap to unwant blocks (the low level part of this is done in js-ipfs)

[dignifiedquire]

@diasdavid why is this milestone2, we can not complete this until we have some way of finding peers with blocks we need.

[daviddias]

@dignifiedquire interop is ms2 :)

[dignifiedquire]

interop is there :P

[daviddias]

Now bitswap is at least 236% more documented at https://github.com/ipfs/js-ipfs-bitswap#usage

Closing this issue as we have both interop and a state of where the project is, further development will happen in a different thread.