Webtorrent: WebRTC DHT

[piedshag]

Discussion for any developments with the webrtc dht

[astro]

https://www.bountysource.com/issues/9787231-dht?utm_campaign=plugin&utm_content=tracker%2F298985&utm_medium=issues&utm_source=github

[feross]

This is something to look at for anyone looking to implement this: https://software.intel.com/en-us/blogs/2015/03/18/meshcentral-experimental-webrtc-mesh

cc @jhiesey

[lmatteis]

perhaps this is related https://github.com/diasdavid/webrtc-explorer

[smarts]

I was looking for a [Google] Chrome Browser extension for handling BitTorrent metainfo files and magnet URNs and I stumbled across this project, which seems really cool! If I understand correctly DHT is required for the handling of magnet links that don't include tracker URLs, correct? If so does that mean webtorrent doesn't work in the browser for magnet URNs w/o webrtc-dht?

[feross]

@smarts When we detect a magnet link without a tracker on https://instant.io, we just add the tracker wss://tracker.webtorrent.io so that there's a place to get peers from. You can do the same in your code. :-)

[feross]

https://github.com/hallettj/bitable

[smarts]

Thanks @feross!

[fsteff]

Any progress on this? I really need this feature (in combination with mutable keys, but one after the other...)
I´ll take a look at what @fbodz has already done and try to support him.

[feross]

No progress. If someone wants to work on this, the WebTorrent project would greatly benefit!

[iaaaan]

Same here, I'm very excited about this. I added $35 to the bounty... it's very few but I hope it might help in some way.

[moshest]

I'm doing research about it a couple of hours and I think that DHT algorithm isn't fit right for WebRTC:

1. Each connection channel is only one way so each DHT request will cost us two channels.
2. Each channel should remain open for future requests, otherwise we will need to build the n-buckets database all over again.
3. Each connection require offer-answer handshake via singling server or third peer.

It's seems that WebRTC is very expensive for DHT requests.
Any suggestions for maybe other algorithm?

[jhiesey]

I actually spent some time thinking about this problem a few weeks ago and I think a WebRTC DHT may be feasible. I'll add my thoughts here soon.

[daviddias]

@feross what would be necessary requisites to have a DHT interesting enough to be used for the WebTorrent project. I'm asking this with the following thoughts in mind:

• Implementing Kademlia DHT for browsers would be quite expensive, even unfeasible if we keep open connections to the older peers (browsers would go quickly out of resources)
• BitTorrent performs DHT searches using a iterative method (we ask peer A, and then peer B), this would mean that adding to the WebRTC DataChannels opened with our KBucket, we would be also be signalling on the fly more DataChannels to the peers we learn along the way to find the destination. Another way to perform the search is "Store and forward", which is how webrtc-explorer works (also IP, or DNS in recursive mode).
• Do we have to use Kademlia at all? After all if we keep the same ID namespace and deterministic properties of a Structured DHT, we could have on the browser side a different algorithm that is more considered of browser capabilities/limitations.
• Is it expected for a bittorrent client to be able to stream a file being served by a browser? Well, only with the hybrid client in the middle to do the "providing" step, correct?

Happy to work of these things for the WebTorrent project :) Please free to push any notes, ideas or any other info you have thought about this.

[moshest]

After some more thinking, I think that the signaling server is the best place for doing DHT queries. Webtorrent should have a few open signaling servers exactly for that case.

Please take a look at PeerJS. We can change the server implementation and add support for DHT queries between those servers.

[feross]

@diasdavid

I was thinking we could adapt the Kademila algorithm. So, normally the bucket size is K=8 or K=16. But we only need that level of redundancy in each bucket because there are no guarantees when you add a DHT node to the routing table that they'll be online when you need to contact them later. With WebRTC, we have to keep the connection active. This means we can reduce the number of nodes in each bucket to something lower, like K=2.

In traditional Kademlia (K=8), for an network of 1M nodes (evenly distributed across the id space), each node will have 144 entries in routing table (check my math).

For K=2 and a network of 1M nodes, we only need 40 entries in the routing table, a much more reasonable number of data channels. :)

Another difference caused by WebRTC is that when other people have you _in their routing table_, you have to keep an open connection to them so they can contact you. These are "wasted" connections for you, since they aren't necessary for your own routing table. Nonetheless, they need to be kept open for the other node's sake.

If a node is equally likely to be in every other node's routing table, then that would add 40 additional connections. But, over time Kademlia actually prefers long-running nodes since the client never evicts a responsive node from a bucket. This means that over time long-running nodes will end up in lots of client's routing tables. With the WebRTC model, that means that the long-running nodes will end up needing to keep tons of connections open to support other people's routing tables.

To solve this, I think we could change the part of Kademlia where we keep longer-running nodes in the table, so every node becomes equally likely to be added to the routing table.

Of course, we could ditch Kademlia entirely and come up with something better from scratch, but then it will be harder for existing torrent clients to add support. So something that's basically Kademlia with a few changes would be easier for them to support.

For WebTorrent, it doesn't matter. We can add anything that works. If there was a reliable general purpose WebRTC DHT not based on Kademlia, WebTorrent would start using it immediately. Even if desktop torrent clients don't support it, it's still useful for web peers to find each other. And later, if a WebRTC DHT that is closer to Kademlia came along, we could always switch to it later :)

"Store and forward" might be a good change for WebRTC since it eliminates the STUN/ICE connection overhead that would happen for each connection.

It'd be awesome to hear your thoughts on these ideas, @diasdavid!

[feross]

@moshest Putting that logic into a central server would mean that it's no longer _distributed_, the _D_ in DHT. At that point, it's basically a tracker server, and we already have that part working :) What we really want is a DHT so we can eliminate the central point of failure that is a tracker server.

[daviddias]

Thanks @feross , lots of insights! That was exactly what I was looking for, to kick things off :)

I think a K bucket of 2 is a good bet on a WebRTC DHT, or I would even say that a K bucket of 1 should suffice. If I'm not missing anything, K buckets offer mainly redundancy for greater availability, having more than one peer at a specific distance means that we always have more than one shot to contact someone on that branch/bucket, which is great for non browser DHTs, since peer representations are pairs IP:Port. In a browser DHT, since we have to keep Data Channels open to avoid all of the STUN/ICE dance each time, we know which is the precise moment a peer goes away, giving us the chance to react and quickly do the handshake with a new peer.

Since we need to limit the number of "xor-metric distances" (going with "xmd" for short) used, so that we avoid having too many DataChannels open, we can consider some strategies for finger distribution, for e.g:

• Inspired on CHORD/PASTRY, giving privilege to closer distances and ensuring that at least we have some fingers to make long jumps.
• Uniform gap, simpler, for e.g with a cap of 20 xmd and for a DHT with 160 bit, we would have entries for 8 bit xmd, 16 bit xmd, 24 bit xmd, 32, 40, 48 ... 160 and one kbucket for each..

The first case is interesting for things like PAST (or DynamoDB), because peers store data ( and not pointers), so enables data replication to closer peers. But since this is WebTorrent and we only need to do is know who is 'providing' what, the second might make more sense. Dunno, ideas :D

One thing to have in mind though is that by having a xmd cap of 20 and K=2, we only set the minimum data channels needed, there still might be some cases where a peer is far enough to not be picked by another peer k bucket , but since it has to know some peer on that branch, it will open a connection to the peer that didn't picked him, creating kind of a one direction channel.

For WebTorrent, it doesn't matter. We can add anything that works. If there was a reliable general purpose WebRTC DHT not based on Kademlia, WebTorrent would start using it immediately

Independently of which the DHT is based on, there will always have to be something to do the Signalling. but for WebTorrent case, you already have that. Would it be interesting that instead of starting two signalling servers, we could have a way to mux between signalling for peers that will do file exchange and for peers that will do handshake for DHT? Like a protocol muxer on top of a WebSockets connection (or even WebRTC DataChannel directly with the server)? Please do tell me how could I expose better the webrtc-explorer DHT in a way that we could make that experiment :)

[maxogden]

A couple random thoughts:

1. QUIC/UDP DataChannels (https://www.youtube.com/watch?v=mIvyOFu1c1Q) will make a WebRTC DHT a lot more efficient, as it has a 0RTT protocol for p2p connections, vs the 4RTT one used by SCTP/DTLS/UDP Data Channels today
2. Perhaps we can store-and-forward peers external ip:port from the initial STUN and have some mechanism for attempting to have these peers plus new peers hole punch each other without having to coordinate through a STUN server.

The idea here is that once you've holepunched once (see 3.4 here for background), and you have a cone NAT, your public ip:port should stay the same for all future connections.

Let's say you have peers A, B, C. A and B successfully holepunch to each other through a STUN server. A and B now know each others external ip:port. Lets say C comes along and connects to B, also through STUN. B and C now know each others external ip:port.

A <--> B <--> C

To get A and C connected, we could do STUN again, but thats expensive. We can use B to swap SDPs between A and C and have them initiate a holepunching dance without STUN. This would bypass the ICE candidate collection phase (since we are effectively caching it here) and also avoid extra roundtrips to the STUN server.

[maxogden]

According to someone on twitter, if you explicitly set ICE/STUN servers to empty arrays you can skip most external ICE checks (no way to skip internal ones though).

[maxogden]

Update: you can't skip internal ICE, because you need to find out the internal port that was created for your new SCTP session, so you can relay this port to the other peer. Since peerconnections will always have different ports (e.g. they wont do port multiplexing since its an entirely encrypted protocol), my scheme proposed above where you reuse the SDP from an earlier peer won't work because you won't know their new port without directly asking them, which ends up being the same number of round trips as the normal ICE candidate + signaling flow :(

[allouis]

@maxogden Although that flow is the same number of round trips, it seems like a better option because we end up with "signaling peers" rather than servers, keeping the overlay network decentralised.

@feross Do you think having a WebRTC backed dgram module would be a good start to the bittorrent-dht being usable in the browser, or we should be starting this from scratch?

I've been working on some small prototypes for this and would be interested to hear thoughts.

[substack]

I think a dgram overlay network is the most exciting way to tackle this problem because it means we can use bittorrent-dht as-is and we can also implement all sorts of other distributed protocols that were implemented under an assumption of udp/tcp primitives.

Unfortunately, an overlay network with forwarding means we'll need to implement routing algorithms. From what I gather, mesh routing algorithms don't scale very well (perhaps up to hundreds of nodes or maybe low thousands of nodes), so internet-scale routing systems are split up into interconnected autonomous systems (AS) with bridge nodes that advertise the autonomous system prefixes on their local network. I haven't found any research papers so far that bring together all of these ideas into a comprehensive technique, so we might need to experiment with how to glue these ideas together. Here's a good overview of how these routing systems interconnect: http://www.cc.gatech.edu/~traynor/cs3251/f13/slides/lecture13-routing.pdf

As for routing protocols, babel seems like a good candidate. I've already started to implement babel with some simulations to check the results.

Each AS could maintain a minimum number of connections to other AS networks so that it's very unlikely to get a disconnected graph.

Which AS you join on start can be a product of the node ID and then an AS could split when it gets too big.

Here's what I have so far for the babel routing protocol implementation: https://github.com/substack/babel-routing-protocol

[substack]

It looks like IPFS might have a similar approach separating peer routing and content routing to cope with NAT traversal: https://github.com/ipfs/specs/issues/1

[allouis]

we can also implement all sorts of other distributed protocols that were implemented under an assumption of udp/tcp primitives.

^ This would be a great place to get to.

mesh routing algorithms don't scale very well (perhaps up to hundreds of nodes or maybe low thousands of nodes)

This is an issue, but maybe not one we need to address immediately, if I understand Kademlia correctly the number of open connections we'd need to support the dht would be 160 (or whatever number of bits we use for nodeIds), the browsers maximum peer connections is 256 so this seems pretty like it could be an option

What concerns me about a dgram overlay network is how we bootstrap it? Ideally it would function as the other browserify alternatives to core node modules, a drop in replacement with the same api, would we have browser only methods for doing this?

[maxogden]

I have heard the practical number of data channels you can currently open in Chrome today is around 30 before it starts melting your computer. Would be nice if someone wanted to try and optimize/benchmark bulk data channel usage to see how many they can squeeze out of modern browsers.

[jhiesey]

I've actually been working on a somewhat different DHT architecture here: https://github.com/jhiesey/webdht At this point the code is pretty messy and mostly undocumented. I hope to get it cleaned up this weekend.

Instead of a dgram overlay network, my architecture is connection-based. Suppose a node A calls Kademlia's FIND_NODE rpc on node B, and gets back the id of nodes C, which node B is also connected to. Node A can then get a connection to node C by creating a virtual connection routed through B that is multiplexed over the existing connection between nodes A the connection between nodes B and C. This virtual connection can then be used as-is, or the webrtc signaling data can be sent over it to upgrade it to a data channel.

My feeling about this is that trying to reuse bittorrent-dht as-is with a datagram overlay will be more limiting than building something custom. I can elaborate on this more soon.

@maxogden I too would by very interested in benchmarks of how many data channels can be opened at once.

[jtremback]

@substack you may be interested to look into Scalable Source Routing. I don't know of any available open source implementations, but it theoretically scales with memory = log n, where n is the number of nodes. This allows networks of almost any size, where nodes cache only 256 routes each. However, it's a lot more complex than Babel, and probably slower.

[allouis]

@jhiesey I think an architecture like the one you described will be essential if the limit on open data channels is as low as suggested by Max.

My feeling about this is that trying to reuse bittorrent-dht as-is with a datagram overlay will be more limiting than building something custom.

Maybe I'm missing something, but couldn't the dgram overlay implement this node multiplexing?

Have you thought about when a connection would be upgraded, would it be based on the volume of traffic? bandwidth possibly? (can we get that information from WebRTC connections?)

[1p6]

I just found a few modules that could work as a dht over webrtc:
https://github.com/gordonwritescode/kad
https://github.com/omphalos/kad-webrtc
https://github.com/omphalos/kad-localstorage

[lmatteis]

Found this paper from last year's P2P computing conference: http://www.tsn.hhu.de/fileadmin/redaktion/Oeffentliche_Medien/Fakultaeten/Mathematisch-Naturwissenschaftliche_Fakultaet/Informatik/Soziale_Netzwerke/pubs/Andreas.Disterhoeft_2015_IEEE.P2P.2015.ShortPaper_Protected.Chords.in.the.web.Secure.p2p.framework.for.dosn.pdf

[substack]

I wonder how feasible it would be to push some of this discussion upstream to webrtc implementors. It seems like a few modest extensions would make implementing a DHT so much more feasible, for example:

• making the SDP expiry time configurable, up to very high values
• making SDPs reusable, perhaps up to a limit or unlimited

I think those few changes would be enough, possibly only the first one. These changes would give us a stable-enough addressing system that we can begin to use all the well-known distributed systems that are built around the assumption of stable addressing and udp-style listening where anyone can send us a message we can decide to respond to.

[maxogden]

@substack SDP expiry is determined by the router, right? routers vary in how long they allow hole punching, I think from around 20 seconds on the low end to 5-10 minutes on the long end.

[lmatteis]

One main problem that I've noticed studying this is that with regular DHT you save IP addresses in a routing table, so when you go offline, and then go back online, you can query them and bootstrap off of previously found nodes. With WebRTC you can't really _save_ the information you've acquired about peers because each handshake is unique. So when you go offline and then back online, you have to start from scratch from bootstrap servers.

[maxogden]

@lmatteis Yes exactly. Also WebRTC connections are much more expensive (probably like 10 round trips if you include STUN, signalling, TCP, TLS, SCTP) than BItTorrent DHT UDP messages (1 round trip)

[lmatteis]

@maxogden I see. Perhaps one way to implement it would be to have a separate table be populated with signaling servers which is used for bootstraps across disconnects. Peers (WebRTC tabs) can introduce new signaling servers to the network manually, and can get propagated across the network. Or to make it automatic there can be a social network of signaling servers: signaling server A can introduce signaling server B to the network and all the peers using A will add B to their tables. Then peers using B will automatically obtain A.

EDIT

Actually, I'm wondering about what would happen if we treat signaling servers as Nodes in the DHT network, each with their own node ID and stuff is found based on XOR distance. Then, there could be a similar protocol on top of that as bittorrent (get_peers/announce), but, rather than sharing IP addresses (as bittorrent's DHT does), they share WebRTC contact info.

[frenchfaso]

@lmatteis sounds like a distributed signaling server..
Would it be possibile to build a distributed signaling server for WebRTC on something like the ethereum blockchain? Does it make any sense?

[smarts]

I'm not trying to shut down the great conversation here, but I also don't want perfect to be the enemy of good. Has anyone looked at what @1p6 suggested? @jhiesey have you made any progress?

[mikeal]

Wondering if this thread should be updated, I know that @diasdavid has done a lot of work that may overlap with this.

[xuset]

I really like all the good ideas that have been suggested in this conversation. In a perfect world, we should be able to reuse the bittorrent-dht package within the browser by swapping out which socket it uses. So in my spare time, I've been working on a project that may be of helpful , peer-relay. It relays messages between peers (WebSockets and WebRTC), and includes a dram like Socket that uses the relay behind the scenes. Doing this, I've gotten the bittorrent-dht package running in the browser which was pretty exciting. I'm skeptical of whether relaying messages is the _best_ way of getting a dht working in the browser, but I think it has potential.

[daviddias]

Absolutely! I believe a lot of the energy being put on libp2p can be reused for the DHT to work with WebTorrent, actually, we even should be able to be part of the same DHT network and have WebTorrent and libp2p/IPFS records passing around, leveraging the connected graph created.

One of the things we are working towards is to have relay on libp2p, so that we can enable nodes that only speak X transport to talk with nodes that only support a Y transport, through a node that speaks X and Y (WebRTC -> WebSockets -> TCP for example). //cc @whyrusleeping

\o/ yaay for collaboration 🌟

[ScriptSmith]

That's fantastic @xuset! I'd love to see your current in-browser bittorrent-dht implementation. Do you have any source code or instructions available?

Thanks heaps!

[xuset]

@ScriptSmith this took a bit longer than I anticipated, but I finally got the web-dht demo up on github. It uses the bittorrent-dht and peer-relay packages to get a dht running in the web browser. It does this by creating a Socket like interface that is backed by peer-relay, then passes this to bittorrent-dht's constructor so it uses peer-relay instead of udp.

I forgot to mention in my previous post that peer-relay is more of a proof of concept than anything. I've tested the web dht with ~10 peers and it works, but whether this approach is feasible for a large number of peers is beyond me at the moment. I plan on doing some tests with more peers soon.

[lmatteis]

@xuset from a high-level, how does this work? WebRTC connections need to be bootstrapped from zero every time you close the tab. Therefore your routing table cannot be saved across sessions, which really destroys the point of using a DHT at all: it puts too much control/load on the bootstrap nodes.

[perguth]

... your routing table cannot be saved across sessions,

Yes, we can! We can eg. create identities out of (some-peer-id, signalhub-url) and then go out and reconnect.

[lmatteis]

@pguth right, but that would put too much control on the signaling servers. To disrupt the network you'd just need to shut down a bunch of signaling servers.

[perguth]

@lmatteis also a problem we can deal with: signaling server are small, we do not need to trust them (see eg. secure-webrtc-swarm) and we can even reduce load by just using them for bootstraping.

[xuset]

@lmatteis

from a high-level, how does this work?

When peer-relay bootstraps, it connects directly to a handful of peers either over websocket or webrtc (nodejs peers have a built in websocket server). It forms it's routing table with k-bucket so the goal is for the peer to know about peers whose IDs are close to it's own and also to know about a few peers that are further away (based on the IDs XOR distance). So when a peer wants to send a message to another peer, it looks at it's routing table and finds the peer with the closest ID to the target peer and sends the message. If the receiving peer is not the target, the receiving peer looks at it's routing table and repeats the process. So in theory, with every hop the distance between the message and it's target is cut in half until it reaches the target.

The amount of hops required for message to reach it's target is log(n) which seems excessive to me for the approach to be feasible. I think this can be mitigated with more intelligent routing though.

One of the benefits to this approach is that no signaling servers are needed since the signalling information can be relayed using the above technique.

... your routing table cannot be saved across sessions

That's definitely a good point. The routing table does need to be persisted across sessions. I can't think of any benefit to persisting peers that only support incoming webrtc connections because you can't bootstrap to them. But peers that support incoming websocket connections (with a websocket server) are definitely good peers to save. So if those websocket peers were saved locally, then it's still possible to connect to the network even if all the bootstrap peers are down.

[lmatteis]

@xuset you've explained how Kademlia works and that I know of course already. You have not described how your "idea" deals with the fact that webrtc-nodes don't have "contact information" that can be shared with others, as the contact changes for each peer connection one makes.

So when you're recursively looking up nodes to find your data (as per Kadmelia's lookup algorithm), you can't ask peers for "give me contacts of nodes closest to this node id", because there's not such thing as a "webrtc contact". The only thing you can do is assign a (peer-id, signalhub) tuple for each node in your routing table, as @pguth mentioned, but that's putting an awful lot of burden on the signaling servers.

[xuset]

Ooops, I guess I didn't make that part clear. peer-relay isn't a dht, kademlia, or anything like that. All it does is send messages from point A to B by relaying them. As you noticed, the routing is largely inspired by kademlia.

bittorrent-dht uses socket.address() to get the contact info so when using the socket that peer-relay provides, .address() returns an (id, port) instead of an (ip, port). This id comes from peer-relay and is used to identify that peer for sending messages to.

Each peer-relay instance has it's own routing table that only includes peers that it is currently connected to, either over ws or webrtc. So when socket.send(msg,...,id,port) is called, peer-relay will looks at it's routing table and sends it to the closest peer that it is connected to. @lmatteis Did that clear things up?

[lmatteis]

How do you connect to (id, port)? You need a signaling server that both of you connect to.

[xuset]

A signaling server isn't needed with peer-relay; every peer-relay instance acts _like_ a signaling server. The signaling information is relayed through peers you're already connected to.

For example: A is connected to B and B is connected to C like this A <-> B <-> C. If A wanted to connect to C over webrtc, it would pass the signaling info through B. Then A can connect to C.

There is a special case where the peer-relay instance is not connected to any peers. It first has to connect to websocket peer, then webrtc peers can be connected to by relaying the signaling info through other peers, either over websocket or webrtc.

[xuset]

@lmatteis I realize we've had a bit of confusion on how everything fits together. This may be due to me not explaining things well, but I've updated the readme for peer-relay and web-dht to include a 'how it works' section that should hopefully clear up any confusion! Let me know what you think 👍

[feross]

@xuset This sounds like the approach that @jhiesey was taking in jhiesey/webdht. Basically, you use a signaling server to bootstrap yourself onto the network (like the current DHT bootstrap servers), then you use those first peers to get introduced to further peers. The peers you're connected to act as signaling servers to introduce you to other peers that they are connected to. I think it's a good approach.

Did I understand correctly?

[lmatteis]

@feross question is, how will a webrtc tab routing table look like? Will it only contain URLs of WebSockets signaling servers?

[xuset]

@feross You understood correctly. The two approaches are very similar, and I think the main difference is that peer-relay is just an overlay with a dgram like interface while jhiesey's webdht has the overlay with a dht on it. I think both are good ways of going about a webrtc dht.

[ghost]

Subscribing as I'm super-interested in a datagram overlay network. I'm coming from the cjdns-world and am working on a packet switching transport for ipfs at the moment. Haven't read the complete issue yet, but so much good stuff <3

[oxyflour]

Hey! I implemented decentralized signaling in my research project simple-peer-chord and I guess sharing my approach will be helpful for the discussion. In my approach

• Chord protocol is implemented because the connection numbers can be updated dynamically by adjusting the finger table length.
• A new node will first connect to a bootstrap node with a known URL (via websocket), then the bootstrap node will act as a proxy between the new node and others to transfer signaling data. No signaling server is required in the process.
• The bootstrap nodes (running in node.js) will exchange known URLs in a certain channel periodically. They will also try to rejoin each other with the URLs to keep the network state consistent.

The Chord seems working well with small networks containing a few tens of nodes. After all, I guess it is not suitable for large networks. Too many hops will be required if each node only has a few connections to others.

[jtremback]

@oxyflour Generally, networks rely on the "small world effect", where a few nodes have a very high number of connections, resulting a small number of hops between all the nodes in the network. Can you have some form of supernode connected to a much larger number of other nodes in Chord?

[oxyflour]

@jtremback Yes. Just enable a large amount of connections and set the maximum finger length like this. But I haven't got a chance to test more than 20 connections on one single node as connection numbers increase very slow with the increment of node numbers in Chord.

[georgeaf99]

Hey y'all, @xuset and I spent some time thinking about ways to adapt the Kademlia algorithm to the needs of a WebRTC DHT. We did not end up fully pursuing our ideas, so we thought it would be a good idea to share what we came up with.

_Considerations:_

Ideally, a DHT that works over WebRTC would be a superset of Kademlia, allowing for easier performance analysis and implementation. Kademlia in particular is a good place to start, because its properties are formally proven.

As far as we could tell, the main compatibility issue for WebRTC and KAD is that persistent connections between browsers are very resource intensive. BitTorrent's implementation simply stores the IP address of other nodes, not a persistent connection. Thus, our main goal was to reduce the number of connections any given peer must have at one time. Although persistent connections may appear to be a disadvantage, they do have the nice property of notifying nodes when their connections have been severed, allowing a node to preemptively refresh it's k-bucket instead of waiting until another node is discovered.

_Approach:_

Our main insight was that it would be possible to limit the number of persistent connections by only connecting to a subset of nodes in each k-bucket. We chose to model the number of connections in each k-bucket with a new variable _c_{b_ which is determined by the function _C(b)_.}

Since knowledge about a node's minimum subtree, aka highest non-empty k-bucket, is the most important routing information, _C(b)_ can be optimized to connect to a minimum number of nodes in lower k-buckets. If the connection to a node in a lower k-bucket was to disconnect and there were no longer any connections in that k-bucket, a node from a higher k-bucket could be asked to find a proper substitute. This node is guaranteed to share the same k-bucket and likely to know of a node which could be connected to. Thus, the lower the k-bucket the higher the probability that a substitute node could be found from a higher k-bucket. The highest k-bucket does not have this property, so we thought that _c_max(b) = k_.

Routing would work in basically the same way, where _alpha_ nodes are sent _find_ _node_ or _find_ _value_ messages, but using the relays as a layer of indirection instead of directly sending the message to their IP address using TCP/UDP.

_Analysis:_

Given a network of size _N_, each node would have to have _k*log(N)_ nodes in it's k-bucket. Lets say the network is of size 1 million and _k_ is 8, then each node will have to have 160 connections. That is far too many WebRTC connections to have open at one time. Now, lets consider a function _C(b)_ s.t. _C(b) = 2_ for all _b < k - 2_, _C(b) = 4_ for _b = k - 1_, and _C(b) = k_ for _b = k_. The network which uses this connection function will have 48 connections.

_Further Work:_

We started working on this problem because we were very interested in building distributed websites, but left it here, because we don't have a great deal of expertise in matters like this and we started Planktos, a tool that helps website owners load their static content from a torrent. In order to vet this idea further, the KAD prime algorithm needs a concrete specification, an optimization strategy for _C(b)_, and a probabilistic analysis (likely using a simulation). Additionally, we have discussed the likelihood that turnover on the web will be much higher than it is for BitTorrent, making optimization of a WebRTC DHT more difficult and novel.

[maxogden]

@georgeaf99 interesting analysis, thanks for sharing.

As far as we could tell, the main compatibility issue for WebRTC and KAD is that persistent connections between browsers are very resource intensive.

I would add to this that the time it takes to establish new connections is also a major prohibiting factor with WebRTC as it's implemented in browsers today. Unfortunately it looks like in the 1.5 years since tracking this issue WebRTC has made no visible (to me at least) movement towards either of these issues.

[nazar-pc]

After reading this thread and looking at multiple linked sources I see 2 fundamental problems:

• browser nodes are typically relatively short-lived (also https://github.com/w3c/webrtc-pc/issues/230 causes difficulties)
• WebRTC connections are expensive

It is impossible to resolve mentioned issues in general because of the current web platform limitations, but I believe we can get pretty close.

What if we have 2 types of web platform-related nodes:

• browsers (these can only support WebRTC and connect to existing WebSockets servers)
• servers/hybrid clients (these can do WebRTC, WebSockets and even UDP to connect to browser nodes and the rest of mainline DHT)

When composing routing table in browser we can do a few tricks:

• use fexible buckets:
  
  
  
  • K=1 for WebRTC nodes
  
  
  • K=2+ for WebSockets nodes (I expect these to be more lightweight, not tested though)
  
  
• replace WebRTC nodes in bucket with WebSockets nodes when possible (maybe sending message back to WebRTC node with suggestion to re-connect to this new WebSockets node?)
• upon restart only reconnect to WebSockets nodes

We will rely more on servers/hybrid clients than on browsers and we will possibly have more hops in browsers that we'd like to have otherwise, but it should be pretty good anyway, since regular BitTorrent clients also apply some rules in this area, just our rules are a bit more specific.

[nazar-pc]

I've being thinking and working on this lately and got something that looks like at least partially working (can put some data on one node and grab it back on another node).

My work in based on customized components currently used as following:

• webtorrent-dht inherits bittorrent-dht
• k-rpc-webrtc inherits k-rpc
• k-rpc-socket-webrtc inherits k-rpc-socket
• webrtc-socket (superset of the subset of dgram interface implemented from scratch)

User-facing interface is almost exactly the same, most changes are done closer to the transport layer.

I will publish it after more testing (both in terms of functionality and scalability), also I didn't yet try to wire it with webtorrent itself.

For now I will be thankful for review on following PRs that are necessary for my superset libraries to work:

• https://github.com/mafintosh/k-rpc/pull/8
• https://github.com/webtorrent/bittorrent-dht/pull/163

Also in order to reduce browser build size by ~3x (down to 40kB min+gz) following PRs need to be merged too:

• https://github.com/mafintosh/k-rpc/pull/9
• https://github.com/webtorrent/bittorrent-dht/pull/165

For those who know C++ I'd be thankful for any help regarding crash that happens when webtorrent-dht is running as client (like browser, without WebSocket server) under Node:

• https://github.com/js-platform/node-webrtc/issues/323

For now I'm pretty optimistic about implementation, will see how it scales soon:)

[nazar-pc]

Initial version of potential WebTorrent DHT implementation as well as potential WebTorrent DHT protocol extension BEP draft alongside with some basic documentation and simplest examples have been published: https://github.com/nazar-pc/webtorrent-dht

It doesn't rely on WebSocket for anything more than bootstrapping and by extending BitTorrent's DHT protocol regular nodes are also acting as relays similarly to @xuset's peer-relay.

I'm not sure it scales very well yet, since it is hard to create (not even establish) more than 150 connections (https://bugzilla.mozilla.org/show_bug.cgi?id=1375818, https://bugs.chromium.org/p/chromium/issues/detail?id=736306) and number of possible established connections is even more limited.

However, having reference implementation it is now much easier to identify actual needs and push browsers vendors to fix existing issues.

[RangerMauve]

What about having a DHT client exist within a browser extension so that it could always be active for users that would want that?

[ghost]

Having looked at nazar-pc code, i have two criticism:

1, Why it's not written in typescript - when i'm looking at the webtorrent code and the dht - lifescript one,
i have a feeling reading a c++ code.
2, WebSocket signaling server has no benefit in the case of DHT.
Websocket is expensive stuff, you cannot imagine that some milions of node would connect to the same webserver, if my assumption is correct looking at the code. (i'm not familiar with lifescript, and to be honest it's very hard to read without classes, single responsibility etc.)

The signaling should be done through the K-RPC protocol, sending the RTCIceCandidates, and storing it in the K-Bucket instead of the IP:port addresses.

I have investigated the technologies in the last few days, and that's what i learnt, or got as an idea, if i understood the whole thing correctly, which is to be honest very hard, processing thousands of javascript code. (webtorrent depends on many)

[nazar-pc]

1. I wrote it in dialect I like, it works like any other JavaScript after compilation, but you need to be familiar with LiveScript in order to read it freely
2. Why no benefit? It seems like a natural fit, bi-directional connection in contrast to HTTP.

you cannot imagine that some milions of node would connect to the same webserver

There is no need to have millions of connections at the same time and even then you can just throw load balancer on it and scale as much as you need. I'd expect WebRTC to be a bottleneck here rather than WebSocket.

I think it would be more appropriate to discuss implementation details in the mentioned repo to avoid distracting lots of people subscribed to this issue. Feel free to open issues there. I'll be using that implementation for one of my future projects that is not WebTorrent-related, so any feedback is definitely appreciated.

[RangerMauve]

sending the RTCIceCandidates, and storing it in the K-Bucket instead of the IP:port addresses.

Sadly you can't reuse SDP or ICE, you can only use it once per PeerConnections, so storing it for later won't make sense.

I think that signaling over k-rpc when possible would be better than always using websockets

[nazar-pc]

I think that signaling over k-rpc when possible would be better than always using websockets

In my implementation WebSocket is only used briefly during bootstrap (as we can't connect other nodes without signaling using WebRTC). As soon as WebRTC connection is established, the rest of signaling is sent using established WebRTC connections with other peers.

https://github.com/nazar-pc/webtorrent-dht/blob/master/bep.rst should explain how this works in details.

Storing SDP for future connections is not possible unfortunately.

[jimmywarting]

Have someone ever thought about using web push?

• You get a token and a endpoint that you can share around to other ppl so they can message you back
• it's persistent like a ip:port (so you can reconnect to same node after you restart or get some network failure)
• you don't need to have a open socket connection to each and everyone all the time
• sdp signaling can be exchanged throught push payload to start establish a new webrtc connection

The bad parts

• Requires service worker
  
  
  
  • can only be installed on whitelisted hosts (https or http://localhost) but i think this is good that everyone is pushing for https
  
  
• Must ask for notification permission to enable it (even if you only sending background push)

navigator.serviceWorker.register('sw.js').then(function(reg) {
  return reg.pushManager.subscribe({userVisibleOnly: false})
})
.catch(function(err) {
  // Chrome currently only supports the Push API for subscriptions that
  // will result in user-visible messages. You can indicate this by calling 
  // pushManager.subscribe({userVisibleOnly: true}) instead. 
  // See https://goo.gl/yqv4Q4 for more details.
});

[RangerMauve]

I was actually thinking about this a while ago.

I think the problem I was seeing was that you can't send a push notification to another user from within a browser due to CORS restrictions. Unless the push notification service has the proper headers, the browser won't allow sending POSt requests to it.

[jimmywarting]

due to CORS restrictions

Yea, notice that too. I wonder why they didn't enable CORS. forced to use a backend/proxy server
if i want to build a peer to peer chat between two people that have friended eachother and exchanged token then i think they should be able to ping eachother without having to go throught a backend

Edit: mozilla's endpoint responds with CORS headers

[codebudo]

Some sort of back end is required to tell Peer A where Peer B is (and visa versa). This is the function of a STUN/TURN server for NAT traversal in P2P networks and can be very light weight. Would something else be required to meet CORS requirements?

On Nov 30, 2018, at 8:05 AM, Jimmy Wärting [e-mail] wrote:

due to CORS restrictions

Yea, notice that too. I wonder why they didn't enable CORS. forced to use a backend/proxy server
if i want to build a peer to peer chat between two people that have friended eachother and exchanged token then i think they should be able to ping eachother without having to go throught a backend

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

[jimmywarting]

A STUN/TURN server don't have to do any CORS preflight stuff as a client you never send any ajax request to those servers - it's pretty much handled for you in the background

[mikeal]

I should have updated this a long time ago.

I had an informal conversation about 4 months ago with some standards/browser folks that work in this area. From their point of view, a bunch of p2p people have been asking for raw socket access and the ability to open ports in order to resolve this, which is just never going to happen in the browser.

Once we talked through it, I landed on language that was really helpful: What we need is a re-usable signal for WebRTC connections.

For a variety of security reasons the browser can't scrap the signal exchange flow but we could potentially create a signal that is reusable and could be added to a DHT and used by other peers for a longer period of time.

[jimmywarting]

re-usable signal for WebRTC connections.

Are you talking about ORTC? have heard about ORTC but never investigated it.
Is ORTC something diffrent then WebRTC?
How can we have re-usable sdp?
which browser support it?
can the same sdp be used by others?

[RangerMauve]

Edit: Disregard that, I wasn't being constructive.

[mikeal]

I haven't spent enough time with ORTC to know if they solved this or not.

In general, it's best to be a little less specific with browser vendors on feature asks like this. Describing the ask in existing WebRTC terms that don't change the security model except in one specific way which you actually need is the best way to work through a variety of possible solutions.

The performance of data channels makes it very costly to have many connections and doesn't scale well

This isn't a given. I have my own thoughts on why the current performance is terrible but performance is a solvable problem. Nothing about WebRTC makes it inherently less performant and my current view is that the implementations are just old and nobody is really touching them, the browser vendors are just binding to a bunch of RTP libraries.

People complain about the performance a lot but, according to Mozilla, nobody has ever logged a usable issue on the subject. What aspect of performance is bad? Where's a reusable test case?

For the most part, people who write browsers are not also writing web applications, so you can't assume they are aware of things we are aware of unless someone has done the work of properly communicating it to them. Yes, they don't make it easy, they all have obtuse bug tracking processes that are used by literally no other project, but that's the situation we're in.

There's no way to accept incoming connections without signalling servers

If you had a re-usable signal that lasted 24 hours you'd be fine.

[mappum]

What we need is a re-usable signal for WebRTC connections.

Seems like a worthy goal, but to implement re-usable signals wouldn't you need to build a DHT or central routing service anyway? We'd have to assume users are roaming around to different networks and would need to locate them as their address changes.

[nazar-pc]

WebRTC-based DHT is definitely possible and while performance is a concern, potential number of users in this case I think is several magnitudes higher.

I've actually studied DHTs and various approaches to their construction for a while and created another WebRTC-based DHT called Detox DHT (and ES-DHT that is used as generic independent framework under the hood) that I use instead of previously mentioned version.

It is built from scratch using alternative design that takes some major ideas from Kademlia and other papers, but doesn't replicate Mainline DHT with its inherent incompatibilities with WebRTC.

It has somewhat different focus, but still might be of interest to others.

Detox DHT repository contains source code and tests. By following link to ES-DHT it is based on, you can find framework's source code, tests, design document, specification and references to papers.

[mikeal]

Seems like a worthy goal, but to implement re-usable signals wouldn't you need to build a DHT or central routing service anyway? We'd have to assume users are roaming around to different networks and would need to locate them as their address changes.

You have this problem no matter what. DHT's that store IP addresses suffer from the exact same issue.

[mappum]

@mikeal Right, just trying to point out that getting browsers to allow re-usable SDP just pushes the problem of implementing a DHT onto them (which maybe is what we want).

[RangerMauve]

Assuming you had reusable signals, how would connections work?

Would it be something like:

Receiver side:

• Create something similar to a RTCPeerConnection
• Create SDP offer or something similar
• Publish somewhere public
• Listen on incoming connections
• Create RTCPeerConnection per connection
• Somehow configure it with your offer?
• Somehow get their SDP answer?
• Profit!

Initiator side:

• Find the published SDP
• Create an RTCPeerConnection
• Set the SDP offer
• Create SDP Answer
• Somehow get it to the person? This will need centralization in the form of signalling servers
• Profit!

You could potentially have a DHT that bootstrapped by talking to an initial centralized node, and then using this reusable SDP from then on, but you're still constrained by having to rely on centralized signalling servers.

getting browsers to allow re-usable SDP just pushes the problem of implementing a DHT onto them (which maybe is what we want).

I'm 100% into this. A few months ago I was trying to convince the folks behind the Beaker Browser to provide an interface to the DHT used for Dat in order to enable different types of p2p applications. Since it'd be cheaper to have a single DHT set up per browser rather than per browser tab or origin, having a DHT API provided at the browser level helps performance, and having a standard high level API simplifies p2p application development.

This is currently being replaced by a higher-level PeerSocket API that goes a step further by providing an interface that opens sockets to peers for a given channel and abstracts away the DHT or MDNS systems that are behind the scenes.

[klueq]

What could make some progress on this is an elaborate 10-page proposal about what needs to be changed in WebRTC to make this P2P scenario possible. The suggested changes need to be simple and they should also bring some business value. The proposal could make a point about the emerging market of P2P apps or maybe eliminating the ICE step from video calls and thus making them connect faster. Then this proposal need to be presented to the decision makers and if they are convinced, we'll likely see in a year how the WebRTC team rolls out the new API. I would push for an option where a previously established RTCPeerConnection could be saved and then restored by id: WebRTC would send a UDP message to the saved ipv4:port and if the response is correct, we assume that it's safe to restore the connection and skip all the SDP handshakes.

Another option is to let the desktop nodes of webtorrent act as signalling servers.

[Slender1808]

any news ?

[perguth]

Go ahead, create a PR 😊🛅🌌🌟

On Tue, Jul 28, 2020, 12:52 AM Slender1808 [e-mail] wrote:

any news ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/webtorrent/webtorrent/issues/288#issuecomment-664678192,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAU2C74YO6WTF7GDM654SDR5YAKXANCNFSM4A6HJ4VQ
.

[Slender1808]

Does WebRTC use compression in video or audio transmission?
if not, would you be able to convert data as text into images using native image transmission?

[perguth]

That a WebRTC question and probably yes 👍

On Tue, Aug 4, 2020, 3:25 AM Slender1808 [e-mail] wrote:

Does WebRTC use compression in video or audio transmission?
if not, would you be able to convert data as text into images using native
image transmission?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/webtorrent/webtorrent/issues/288#issuecomment-668323140,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAAU2C4XIIHLVLHZFI4CSO3R65PRVANCNFSM4A6HJ4VQ
.