Libtorrent: Support WebTorrent Protocol

[kevincox]

WebTorrent is a new implementation of the bittorrent protocol designed to work over WebRTC. This allows browsers to download torrents without plugins. However, due to the limitations of WebRTC it isn't possible to make the protocol compatible with bittorrent (or uTP).

If libtorrent supported WebTorrent it would be able to communicate with both the existing swarm as well as the browser swarm that is restricted to WebTorrent only.

[DiegoRBaquero]

+1

[B00tLdr]

+1! This would make it possible to have something like popcorntime natively in the browser :D
Maybe use something like: https://github.com/xhs/librtcdc for this. @feross could maybe add some info if this is possible/easy.

[RobinHoutevelts]

Having support for WebTorrent would be huge

[kevincox]

I would also like to reiterate that while having a good swarm of bittorrent, and a good swarm of webtorrent are both really important, libtorrent would be in the critical situation of bridging the two which is critical to the success of webtorrent.

[cathalgarvey]

Huge +1, I think Webtorrent integration is part of the future of Bittorrent; casual swarms and torrent streaming in-browser are killer apps.

_edit: I always mix up the names for Bitcoin and Bittorrent, sorry!_

[ngosang]

+1

[gubatron]

WebTorrent should be the one to submit a patch with the WebRTC transport implementation.

AFAIK they had to change a few things on their side bittorrent-protocol wise to make it work on browsers, who better than them to do this.

As of now, there's simply two separate networks, without libtorrent having this transport the WebRTC will never be able to access the seeds on the TCP/UDP side.

However, from a practical standpoint, will people keep a browser tab open to seed? possibly creating memleaks on their browser, slowing down their day to day work? or will this just add a massive number of seeds that will ultimately be a huge performance hit for the existing bittorrent network?

[kevincox]

However, from a practical standpoint, will people keep a browser tab open to seed?

This is hard to say, however with web technologies quickly growing there is no reason why some people wont run their torrents in the background, just like the do with native apps (no tab required). I feel the line between desktop and web is growing thin and not providing features to web apps because they are web apps is becoming an antiquated view point. You may be right, but I suspect that the playing field will change very quickly.

[transitive-bullshit]

@arvidn I'm a webtorrent contributor interested in adding support to libtorrent. Do you have any advice on where to start or potential gotchas to be aware of at the outset?

[aldenml]

@fisch0920 My suggestion is start digging peer_connection and peer_connection::start for clues of how to create a possible "webrtc_peer_connection". Do you have a WebRTC library candidate, or you want to recreate the protocol?

[yipperr]

@kevincox i agree the line between apps and browsers are going thin there is also the issue where webtorrent might negatively impact the swarm by allowing sequential file transfer of media files to be selected the relative ease for media files to be streamed into the video tag of the browser might make it appealing for people to opt for sequential downloads more often that the rarest first and also since lot of people noticed the synergies and obvious usage as this popcorn time in a browser this kind of compounds the facts and might really cause concern on swarm stability

[arvidn]

@fisch0920 aldenml's advice is right btw. the peer_connection is the base class connecting peers to the rest of the bittorrent engine. bt_peer_connection derives from it and implements the bittorrent protocol, there's also web_peer_connection and http_seed_connection for the two kinds of web seeding protocols. deriving from peer_connection to implement a new peer protocol is how it's intended to work. now, I wouldn't be surprised if there will be some rough edges, as its interface has been designed around bt_peer_connection

[obv-mikhail]

Might be possible to use this https://github.com/mozilla/webrtc-standalone for webtorrent support.

[aldenml]

Do you see this in the Makefile?

LIBS = \
$(GECKO_OBJ)/mfbt/libmfbt.a.desc \
$(GECKO_OBJ)/layout/media/webrtc/libwebrtc.a.desc \
$(GECKO_OBJ)/media/libopus/libmedia_libopus.a.desc \
$(GECKO_OBJ)/media/libvpx/libmedia_libvpx.a.desc \
$(GECKO_OBJ)/media/libjpeg/libmedia_libjpeg.a.desc \
$(GECKO_OBJ)/media/libspeex_resampler/src/libspeex.a.desc \
$(GECKO_OBJ)/netwerk/srtp/src/libnksrtp_s.a.desc \
$(GECKO_OBJ)/media/mtransport/standalone/libmtransport_standalone.a.desc \
$(GECKO_OBJ)/media/webrtc/signalingstandalone/signaling_ecc/libecc.a.desc \
$(GECKO_OBJ)/xpcom/libxpcomrt/libxpcomrt.a.desc \
$(GECKO_OBJ)/dom/media/standalone/libmedia_standalone.a.desc \
$(GECKO_OBJ)/netwerk/standalone/libnecko_standalone.a.desc \
$(GECKO_OBJ)/intl/unicharutil/util/standalone/libunicharutil_standalone.a.desc \
$(GECKO_OBJ)/security/nss/lib/pk11wrap/static/libpk11wrap_s.a.desc \
$(GECKO_OBJ)/security/nss/lib/freebl/static/libfreebl_s.a.desc \
$(GECKO_OBJ)/db/sqlite3/src/libdb_sqlite3_src.a.desc \
$(GECKO_OBJ)/memory/fallible/libfallible.a.desc \
$(GECKO_OBJ)/media/libyuv/libyuv_libyuv/libyuv.a.desc

how many MB is this standalone library?

I think a good starting point is find a minimal WebRTC communication stack, or figure out a clear interface to plug inside libtorrent as another protocol.

[jscinoz]

@aldenml https://github.com/xhs/librtcdc suggested by @B00tLdr should be a good fit - it's certainly much more lightweight than webrtc-standalone, since librtcdc it implements data channels only, whereas webrtc-standalone implements all the media related functionality of WebRTC too, which isn't necessary for WebTorrent support.

[aldenml]

yes, I saw the link right after I left my comment, thanks

[arvidn]

Just for the record. The receive buffer in peer_connection right now is a complete mess. Mostly because it provides "convenience" functions to advance read pointers and store the current message size being received. Things that are mostly useful for the bittorrent peer, but not really for anyone else. So, if anyone will be looking at integration with webRTC, it would probably make sense to simplify the receive buffer too.

[dessalines]

I'd like to help out with webRTC support, where would be a good place to start?

EDIT: Yo @aldenml , I'd like to start working on this. I have libtorrent building correctly on my machine, but I suppose I need a good way to test out changes that I've made, IE, how to run the client_test.

Also, I'm familiarizing myself with peer_connection.cpp, but I'd definitely need some help figuring out the architecture, mainly where the best place to put the new connection type should be, and how best to include the suggested webRTC library, https://github.com/xhs/librtcdc .

[dessalines]

@aldenml Sorry if this pinged you twice, I edited my comment above to ping you, but I'm not sure if github edits will re-ping someone. If there's some other client like IRC or gitter that would be better for possibly bringing me up to speed on this, let me know.

[aldenml]

not a problem (and thanks). I don't have experience with any example in libtorrent, only the tests.

[adamvoss]

@dessalines Thanks working on this! I have dug into your questions.

Running client_test

client_test is included in the repo under the /examples folder. Executing make client_test from within that directory compiles it and puts the result in the same directory; alternatively, there is an entry in 'examples/Jamfile' if your are using bjam rather than make. Then a simple ./client_test <filename.torrent> in that directory to run it.

Where to get started

arvin has advice in this comment. Paraphrasing:

peer_connection is the base. There are 3 implementations: bt_peer_connection, http_seed_connection, and web_peer_connection

You would start with a new implementation, webtorrent_peer_connection (or similar), deriving from peer_connection

by convention, it would appear your ".hpp" file would go in /include/libtorrent and your ".cpp" file in /src

Visual representation of hierarchy:

└─ bt_peer_connection
└─ web_connection_base
│  └─ http_seed_connection
│  └─ web_peer_connection
└─ webtorrent_peer_connection    ( <-- you add this)

How to include librtcdc

I am not familiar enough with the repository to say definitively. However, you can always get started and move the files later. For the sake of development, what I would do is just use git submodule to pull down the code at the root of the project.

git submodule add https://github.com/xhs/librtcdc.git

(I am guessing the root because I see ed25519 has its own folder at the root). Later it can be decided whether a submodule is okay or if a copy of the source will be included and where the final location should be.

[arvidn]

I think pulling in librtcdc as a submodule would probably be best. one challenge I would expect is that the peer_connection class has a pretty rigid "framework". It owns the socket, it implements rate limits, SOCKs proxies, some send- and receive- buffer management and some other high level features. The receive buffer part is not great unfortunately, it's fairly tied to BitTorrent protocol. You specify how many bytes you expect to receive and then you get called every time more bytes are received.

If librtcdc itself owns sockets, that might make it difficult (at least if you want to integrate with the main message pump/asio.io_service in libtorrent). It the library let you pass byte in and it passes bytes back out again, it should make it easier to integrate on top of libtorrent peer connections.

About the headers, I'm hoping to move towards having only public headers live in include/libtorrent and have private headers live in include/libtorrent/aux_ (and in the libtorrent::aux namespace). Some internal classes do that, but not all yet.

[ducdigital]

+1 any progress so far?

[nkosi23]

@gubatron @yipperr

or will this just add a massive number of seeds that will ultimately be a huge performance hit for the existing bittorrent network?

i agree the line between apps and browsers are going thin there is also the issue where webtorrent might negatively impact the swarm by allowing sequential file transfer of media files to be selected the relative ease for media files to be streamed into the video tag of the browser might make it appealing for people to opt for sequential download

I read the bittorent white paper a long time ago so my knowledge of the protocol is not very deep, so you really triggered my curiosity here. Why would a large number of seeds affect the performance of the network ? And what's the deal with sequential file transfers ? Could you please introduce me to the situation, or give me links me to relevant documentation (whichever is the easiest for you).

Many thanks !

[arvidn]

@reddyBell The concerns I believe were attempted to be conveyed are:

1. in-order downloading defeating rarest-first (and even piece distribution). You can google "bittorrent rareset first" for in-depth studies and descriptions. I have some old slides about this, starting on page 10: http://libtorrent.org/bittorrent.pdf

2. not accepting incoming connections. An increasing portion of peers that cannot accept incoming connections, the worse the swarm performance and robustness will be. Specifically, it may be increasingly difficult to find a peer to connect to (when flooded by peers that you can't connect to).

[kevincox]

@arvidn As far as I understand these problems aren't part of the protocol, but assumptions on how the protocol will be used. Am I understanding this correctly or are there intrsinic problems with the webtorrent protocol?

[arvidn]

I don't know enough about webtorrent or WebRTC to know if any of those are intrinsic or just general concerns.

[transitive-bullshit]

@arvidn great summary of the concerns

1. agreed that most webtorrent usage will practically be sequential downloads via media playback, but this may be offset by the fact that all webtorrent playback seeds as well as downloads for the entirety of playback as opposed to normal torrent engines where many users typically stop seeding after download completes. this is really just a hypothesis on both sides of the argument, and i honestly feel that we won't know how overall swarm health is affected until experimenting with it in practice.

2. definitely a valid concern -- the DHT scales very well, however, in terms of num peers and as long as we have some way of explicitly differentiating between webtorrent-only peers and webtorrent+bittorrent peers, then i don't seeing this being an issue in practice.

[adamvoss]

It sound like there are not insignificant changes to libtorrent to make this happen, and perhaps librtcdc would be a good placeholder implementation while making those changes. However, looking into it further, librtcdc does not seem like would be a good fit for bridging with browser-based clients at this time. Based on its issue tracker, it appears to not be very compatible with web browsers (https://github.com/xhs/librtcdc/issues/9, https://github.com/xhs/librtcdc/pull/11#issuecomment-159529822). The README indicates it is being rewritten (as of ~Feb 2017), so if enough time passes it may be the right solution after all.

[rllola]

+1

Anyone actively working on this ? I would be interested in contributing.

[arvidn]

not that I know of

[graingert]

@rllola ^

[rllola]

@graingert Yes ?

[lgrahl]

Hey guys. I'm throwing in RAWRTC here as an alternative to librtcdc. Feel free to ping me if you have any questions about it.

[zodman]

+1

[leethax666]

With Peertube getting more popular, this would be awesome

[Donokami]

+1

[abomm]

Yes, this would be very useful!

[alexandre-janniaux]

Hi, is there someone working on the issue currently ? I'd love solve this one otherwise.

[backkem]

I highly doubt it. In addition, the WebTorrent BEP (webtorrent/webtorrent#168) still isn't finished. You would have to build against the current JS WebTorrent implementation or try to get the BEP finished first.

[dessalines]

There are already several torrent clients which support webtorrent : https://github.com/webtorrent/webtorrent/issues/369

[alexandre-janniaux]

@dessalines: actually, I don't want "a torrent client which support webtorrent", but I want "libtorrent to support webtorrent" as I already have some code written with it aiming at being used this way too.

I've taken a look at librtcdc, librtcdcpp and rawrtc, and have the following feelings for now:

• rawrtc and openwebrtc should not be used : they are far too big for just supporting data channels
• librtcdcpp isn't very maintained nor has a lot of support, and has some conception flaws according to the main fork's commits
• librtcdc isn't very maintained too, except from the same guy as the librtcdcpp fork, but still is very small and should work. Maybe adding tests and build system for cross compilation would be a good idea.
• datachannel (c++) is dead and never worked.

Thus, I might go on the librtcdc option and check the BEP later.

[backkem]

@alexandre-janniaux Maybe check out this issue as well. You may be able to judge if this makes using the chromium WebRTC implementation a viable option.

[lgrahl]

@alexandre-janniaux I'm wondering why you believe RAWRTC is too big. In what way is it bigger than librtcdc? Both require usrsctp, some ICE implementation and OpenSSL for DTLS. OpenWebRTC and Google's libwebrtc are not even in the same league regarding their size and dependencies.

Edit: I explained differences in detail in https://github.com/rawrtc/rawrtc/issues/8#issuecomment-285772026.

Edit 2: In terms of LOC and complexity, you have to consider the dependencies as well. libnice is very high level whereas the ICE implementation of re/rew is pretty low-level, so you need to do a lot of stuff yourselves such as interface gathering, parsing and generating SDP, ... which is fine because more control is good. However, don't forget this kind of code is part of libnice, so it's not less complexity just because librtcdc needs to do less on its own.
But you're correct that RAWRTC's data channel implementation has more lines of code. All of that code is needed and may simply not be handled within librtcdc (e.g. EOR, sending and receiving large amount of data). I don't consider having those features (actually, not handling EOR is a bug) a bad thing.

I'm really puzzled by your estimation and I'm of course happy to answer any further questions.

[alexandre-janniaux]

@lgrahl Hi, thank you for your clues, I hope you don't take my message personnally. I believe I wasn't aware enough of what was behind. I mainly focused on which library would be the easiest to use for a first implementation, libtorrent integration being the first difficult part.

But I believe that whatever the choice, the library will have to be maintained and improved so as to match webtorrent current and future needs.

I will move to your gitter so as to ask you some questions in a few days, if you agree to answer them.

[ExE-Boss]

@lgrahl Actually, Firefox has fixed EOR handling over a year ago.

Also, for latest information, see documentation on MDN.

[lgrahl]

@alexandre-janniaux No worries. And yes, please do contact me. Regarding maintaining the lib, etc., I can explain the situation in detail on Gitter as well.

@ExE-Boss Yes, I'm aware (I did that and I mostly copied RAWRTC's code for that purpose :wink:). But Firefox's data channel implementation is not available as a library. librtcdc & co however (AFAIK) have not fixed EOR.

[leethax666]

I'd also like to add that archive.org just started using webtorrent with their dweb.me decentralized web mirror, so it's catching on

[bepvte]

any result from the gitter conversation? im really hyped up about this feature and cant wait for it to happen

[lgrahl]

@redg3ar I can only speak for myself but I have not been contacted by anyone regarding this issue so far.

[zero77]

@arvidn
I think webtorrent-hybrid would be a better choice than just webtorrent.

Because, it connects to both TCP/UDP and WebRTC peers, meaning there will be more peers for each torrent.

Also, i may be rong but i think it uses TCP/UDP which may be easier to implement into libtorrent than WebRTC.

https://github.com/webtorrent/webtorrent-hybrid

[nstephenh]

@zero77 Correct me if I'm wrong, but since libtorrent is a traditional (TCP/UDP) torrent client already, adding webtorrent (webRTC) support would make libtorrent a hybrid client. There is no "hybird" protocol, simply clients that can support both types of torrent. The part of webtorrent that is is new/different/exciting/useful is its use of webRTC.

[zero77]

@nstephenh
yes, that makes sense if webRTC was added to libtorrent it would make it a hybrid client.

But, what i was wondering is if webtorrent-hybrid would be easier to implement.

Because, it may be closer to a traditional (TCP/UDP) client instead of a webtorrent which is also as you said webRTC, and may be closer to home for developers.

[nstephenh]

@zero77
There's no difference between webtorrent hybrid and webtorrent from a protocol standpoint, or anything different about how the libtorrent developers would have to implement it. As far as I can tell, the webtorrent-hybrid repo is a node hack to get the webtorrent cli client to be able to connect to webRTC peers because the library they used needs a browser window open.

[jimmywarting]

As far as I can tell, the webtorrent-hybrid repo is a node hack to get the webtorrent cli client to be able to connect to webRTC peers

That's correct.

If webtorrent would be included in this project it would almost double all the logic. could as well throw away everything and just use webtorrent instead. doe it would run slower since it's javascript and you would need to install other dependencies like NodeJS then also

no what we need is to be able to connect to WebRTC peers by using any of librtcdc, librtcdcpp or rawrtc

[aldenml]

I find this development https://bugs.chromium.org/p/webrtc/issues/detail?id=7613#c9 very interesting. I think a more c++11 and libtorrent friendly API layer could be made in a different project, in order to provide precompiled binaries.

[lgrahl]

The webrtc.org (Chromium's WebRTC) implementation is quite good when it comes to ICE stack maturity but the data channel implementation is just barely acceptable. For example, it still suffers from message integrity violation leading to the well-known common limit of 64 KiB messages due to its popularity. Here's another one regarding the lack of Blob support (which is the default setting) since 2013 and the most starred issue in that project. That should tell you how much Google is interested in driving data channels further. Draw your own conclusion based on that. :slightly_smiling_face:

[aldenml]

@lgrahl I just read the issues and are indeed problematic, having said that, it is my impression that the majority of WebTorrent clients are running with some variant of Chromium's WebRTC, either via a real browser, or electron hack in hybrid clients. Two questions:

• How these issues are impacting WebTorrent today?
• To integrate Data Channel in libtorrent, do you still need ICE/STUN/TURN?

[arvidn]

without really knowing anything about the webtorrent protocol (is it documented anywhere?) I would be surprised if the issue with packets > 64kiB would be an issue. In bittorrent, the largest packets are about 16 kiB. I don't know about blob, but the ticket seems to suggest that there's some distinction between a blob of data an an array of bytes. I wouldn't expect anything more than an array of bytes to be necessary.

[lgrahl]

@aldenml

it is my impression that the majority of WebTorrent clients are running with some variant of Chromium's WebRTC

True but in the browser you don't have any choice and in Electron it's either a) use the data channel implementation shipped with Electron or b) integrate another data channel library. The latter is likely a lot of work since you need to create bindings, etc.
The difference is that for this library, it will not make a lot of difference in terms of effort because this is C/C++ and there are no bindings required for the majority of implementations. :slightly_smiling_face:

Regarding your questions:

Edit: Disclaimer: I'm trying my best to stay objective but be aware, as a dev of an alternative data channel implementation, I might be biased.

1. I'll pass that on to @feross since I haven't looked at the webtorrent implementation. However, at the very least it requires chunking of messages into a maximum of 64 KiB (previously 16 KiB, if you're interested, here's an explanation) sized messages. Which the webtorrent protocol probably already does because of this well-known limitation. From a user's perspective, we also encountered some annoying deadlocks in webrtc.org's native implementation (e.g. sending a message in the receive handler) but I'm starting to nit-pick... I just want to hint that, as someone who spent a lot of time reading and hacking on data channel implementations, I don't consider the webrtc.org data channel implementation any better (in terms of maturity, future prospect, etc.) than many other implementations. For the sole purpose of using a data channel implementation it might also be a bit bloated.
2. Yes, you need ICE (which requires STUN) at the very least. For this use case, TURN will be irrelevant. (Obviously, data channels also require DTLS, SCTP and a data channel implementation on top of those.)

[lgrahl]

I don't know about blob, but the ticket seems to suggest that there's some distinction between a blob of data an an array of bytes.

That's indeed only relevant in the world of JavaScript. It was meant as an example to point out that (surprisingly) Google pays little attention to their data channel implementation.

[ngosang]

without really knowing anything about the webtorrent protocol (is it documented anywhere?

@arvidn The protocol is almost the same as Bittorrent (I think it has lees features), just use WebSockets instead of TCP sockets and JSON instead of bencode. Should be very easy to implement.

Official code in JS (folder lib) => https://github.com/webtorrent/webtorrent

Code example:

    ws = create_websocket_connection(tracker)
    params_json = '{"action":"scrape","info_hash":"\u0008\u00ad\u00a5\u00a7\u00a6\u0018\u003a\u00ae\u001e\u0009\u00d8\u0031\u00df\u0067\u0048\u00d5\u0066\u0009\u005a\u0010"}'
    ws.send(params_json)
    result =  ws.recv()
    ws.close()

Update:
WebTorrent trackers => https://github.com/ngosang/trackerslist/blob/master/trackers_all_ws.txt

Online Webtorrent streaming + .torrent file => https://webtorrent.io/

[zero77]

@arvidn

without really knowing anything about the webtorrent protocol (is it documented anywhere?)

Its not much, and i am not sure if this is what you are looking for but, here is all the WebTorrent documentation i could find.

https://webtorrent.io/docs
Possibly the same:
https://github.com/webtorrent/webtorrent/blob/master/docs/api.md

[gubatron]

I think he's speaking more of the wire protocol, not the JS api.

On Wed, Dec 12, 2018 at 12:02 PM zero77 [e-mail] wrote:

@arvidn https://github.com/arvidn

without really knowing anything about the webtorrent protocol (is it
documented anywhere?)

Its not much, and i am not sure if this is what you are looking for but,
here is all the WebTorrent documentation i could find.

https://webtorrent.io/docs
Possibly the same:
https://github.com/webtorrent/webtorrent/blob/master/docs/api.md

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/arvidn/libtorrent/issues/223#issuecomment-446704094,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAKAif_IdVVdsvgpmji-EW9KQRrd0Ncfks5u4VKpgaJpZM4GOuA9
.

[backkem]

Please don't underestimating how hard this would be. It's the tracker that uses JSON over WebSockets to exchange peer information. Therefore, you need to support WebSockets (likely a new dependency) and implement this new type of tracker. Next, the wire protocol itself is identical to normal torrents but it's transported over WebRTC Data Channels. Peers in a normal torrent swarm can connect to each other via their IP address. However, WebRTC requires you to exchange an offer and receive an answer. To do this you need to adapt the way peer information is stored & handled plus you have to implement this offer/answer ceremony. This would likely require extensive refactoring to fit into the existing peer & connection logic. After all this, you also need to be able to setup a WebRTC connection itself. Therefore you need a c++ WebRTC library. WebRTC is a massive networking stack (STUN, TURN, ICE, DTLS, SCTP, DataChannels, SDP, ...) and thus a massive dependency to add. In addition, it isn't obvious which existing WebRTC library to use since they all have disadvantages, as extensively discussed in this thread already. This choice is therefore not an easy one to make. Lastly, you need someone who's willing/able to make this all happen.

[jimmywarting]

WebRTC requires you to exchange an offer and answers so we can break it into smaller steps and start easy.
Lets start of with making WebRTC work and worry about how WebSocket trackers and peer exchange work latter.

• [ ] Find some WebRTC library that supports DataChannels
• [ ] I only know how javascript and how the browser's webrtc api works so maybe try making this code work in your codebase first?
  
  
  
  • [ ] After that you can try to and see if you can manage to connect to a browser. I created a other fiddle that will initiate a offer that you have to answer to and reply back. When the hole offer/answer is exchanged and the datachannel is open i will send a hello message that you can hopefully dump to your console log
  
  

When you have a datachannel connection open then you got your "socket"
it's then over to discovering other peers over a websocket tracker. i didn't find any BEP explaining how this works. so the best i can give you is the source - websocket-tracker.js

A websocket-tracker have some similarities (i think) to a normal tracker but extends a little further to include the WebRTC part Since it's not as easy as connecting to a ip:port you have to create multiple RTCpeerConnection beforehand and generate offers and creating 1 datachannel that other peers will find and they will try to make a connection with you.

what will follow is going to be a couple of "answer" message from the websocket where you have to set the remoteDescription in order to establish the connection between two RtcPeerConnections.

If you want to know more how webtorrent works i can assist you more over a chat

[d3fault]

Just a thought: Supporting BitTorrent and WebTorrent simultaneously definitely sounds feasible and desirable for the short-term, but longer term wouldn't it be simpler (design/code-wise) to just deprecate BitTorrent and move everything to WebTorrent? All "desktop" clients could migrate to WebTorrent, but it's impossible for "browser" clients to ever support BitTorrent. WebTorrent might be less efficient than BitTorrent (I'm just guessing), but the difference is probably negligible.

Of course, convincing all the torrent app makers to move to WebTorrent would be the hard part.

[arvidn]

@d3fault documenting the protocol would be a good start, otherwise there's nothing to solicit opinions about (with regards to complexity and performance for instance)

[oskvadd]

@arvidn Is there any known working implementation for WebRTC and LibTorrent yet, or is it just to start code a new one? Whould be nice to get this thing started! :)

[arvidn]

@oskvadd I'm not aware of any attempt to start programming up WebRTC support in libtorrent, I would be happy to help (and I've made some comments earlier in this thread with where to start).

In my mind, I think ideally the smallest possible subset of WebRTC should be used. e.g. just data channel. And if it can be made small enough, it may even make sense to build a custom data channel on top of boost.asio directly. But my impression so far is that there's quite a lot of complexity involved that probably makes that impractical.

[dbrgn]

If you want a small but correct datachannel-only implementation, then RawRTC is probably what you'll want to use. Writing your own implementation will probably increase complexity instead of reducing it.

Using Google's libwebrtc would give you a huge amount of build system and code complexity, and their DC implementation isn't even correct in some cases. (@lgrahl knows more about that, I'm only a DC user, not implementor.) And the implementation by Mozilla isn't standalone/portable, as far as I know.

[aldenml]

Hi @lgrahl, out of curiosity, I was reading a little more about ICE, and digging a little bit in rawrtc, and I fail to see why a ICE/STUN stack is needed in the libtorrent side. There is some capability already to get your "external ip/port". Do you think it is possible to assemble that information as an ICE candidate and offer it directly to the other WebRTC peer?

[backkem]

Interesting idea. It's likely possible to skip the STUN binding request which is used to determine your public IP address. It should also be fine to establish an ICE connection using a single manually crafted (server reflexive) candidate (assuming you're not behind a super restrictive firewall). However, the other peer will still expect to complete the ICE process by sending connectivity checks to nominate the final pair (ref). So you can simplify somewhat but you can't get rid of ICE entirely.

[lgrahl]

@aldenml An ICE agent does a lot more than just fetching the external address. It also provides a (perhaps slightly complicated) candidate pairing process and checking procedure to determine available paths.

Gathering candidates is the easy part of ICE. :slightly_smiling_face:

[dessalines]

Questions for anyone who might know:

Ignoring the work it would take to build this, how heavy would it be? For those of us who seed thousands of torrents, would the additional webRTC connections (DHT and tracker announces), be a considerable burden on libtorrent, in memory and cpu?

And if that's the case, wouldn't it maybe be easier to build browser addons for firefox and chrome, that run a torrent client that uses libtorrent? Basically the trade-off for that would be the installation process vs adding all this additional connection infrastructure to libtorrent. I found project like moztorrent that seem to be working on this.

[graingert]

I imagine that they'll take up about the same resources per connection

On Thu, 27 Dec 2018, 19:04 Dessalines <[email protected] wrote:

Questions for anyone who might know:

Ignoring the work it would take to build this, how heavy would it be? For
those of us who seed thousands of torrents, would the additional webRTC
connections (DHT and tracker announces), be a considerable burden on
libtorrent, in memory and cpu?

And if that's the case, wouldn't it maybe be easier to build browser
addons for firefox and chrome, that run a torrent client that uses
libtorrent? Basically the trade-off for that would be the installation
process vs adding all this additional connection infrastructure to
libtorrent. I found project like moztorrent
http://moztorrent.mozdev.org/ that seem to be working on this.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/arvidn/libtorrent/issues/223#issuecomment-450211276,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAZQTHAEMmpke0S4PFPijAUTs_KL32efks5u9RmpgaJpZM4GOuA9
.

[backkem]

I don't have any numbers but I do think it would consume more resources. The interaction with the tracker uses WebSockets and needs to communicate more. In addition, in contrast to plain UDP/µTP WebRTC data channels requires a rather large stack of different protocols to transfer the data.

WebTorrent just allows different use-cases. Like plugin-less torrents in the browser and arguably obfuscating traffic as normal web traffic. You may end up turning it off for heavy-duty use-cases. You may even want to have it disabled by default. That doesn't mean it isn't worth exploring.

[ExE-Boss]

@dessalines

MozTorrent hasn’t been updated in 13 years and is most certainly abandoned.

[aldenml]

Well, I think that having another browser addon, webtorrent-hybrid and similar efforts are always good, but not nullifies this one. Whatever the result of this thread, in order to be included probably it will need to meet the main libtorrent goals.

@lgrahl, does rawrtc contain an ICE agent that can be separated (in terms of logic and code) from the STUN part in order to have that non-trivial functionality?

[lgrahl]

@aldenml One would need to replace the ICE gathering part of RAWRTC for that purpose. It will require some work but it's fairly modular.

[graingert]

Here's an issue tracking the feature in aria2 https://github.com/aria2/aria2/issues/595

[aldenml]

After more reading, I came to the conclusion that in order to get ICE/STUN/DataChannel right, it's better to use a well stablished library/framework, it's a complex stack and it's probably the only way to keep a stable connection with browsers.

Then I looked at platform support, and only WebRTC give you support for Windows, Linux, macOS, iOS and Android. I coded a minimal program with two local peer connection and opened data channels for them and I managed to run it on macOS and Linux, I just assume that other platforms are doable. The negatives of WebRTC:

• Very big source code: For macOS It's ~8GB just the download per build instructions (it's ~4GB without the .git dirs)
• Very big binaries: After some stripping, I get a ~7MB binary for my minimal demo program.
• _Three_ separate threads for his inner working and his own low level async/event and socket framework.

My questions now are: Given that any asio integration will be minimal at best, and (per last point) that you will end up running two separated "P2P" stacks, can WebRTC really be considered? What happens if all other libraries have similar integration limitations?

[dbrgn]

@aldenml the webrtc.org stack is huge. Since you only need data channels, I'd recomment to take another look at RawRTC, which doesn't contain multiple codecs and multiple implementations of echo cancellation and adaptive data rate that you don't need in WebTorrent. Also, building the webrtc.org stack is a pain.

I'm quite sure that it should already work on Linux, macOS and Android. As far as I know, @lgrahl also plans that RawRTC should work on Windows, it's just a matter of doing/finishing the port. Both usrsctp and libre (RawRTC dependencies) support both Windows and iOS.

[aldenml]

@dbrgn I want to do that, take a closer look at rawrtc, let's see if I can find the time.

I would like to have some feedback from you @arvidn, since I think the problem of the other framework providing his own event system and using internal working threads remain. Is that something you would consider acceptable? Is passing around packets from one stack to another possible to abstract in the asio stack?

[arvidn]

I would be hesitant to include webtorrent support unconditionally. I would prefer it either be implemented as:

(1). a plugin (which would require some changes to the current plugin customisation points to allow arbitrary peers, which would probably make the http seed code a bit better separated too). or

(2). be an optional build flag, similar to i2p, DHT and encryption support. In this case it would still be beneficial to not have to spread out support all over the place, but rather generalise the peer_connection abstraction. (the current design is in 90's style OO, which is quite rigid, and not very flexible) (I was young and didn't know better)

I don't think it's necessarily a show-stopper for the webRTC stack to have its own threads, sockets and message loops. But the current peer abstraction doesn't support that well. I still want libtorrent to be practical on embedded systems, that's why I would prefer there to be a build option to disable the additional 7 MiB for webtorrent support still.

[aldenml]

Good to have your feedback, I think that if size is not an issue, WebRTC is my personal preference, build can be streamlined to something workable. I took a quick second look at rawrtc, and I just find WebRTC more enjoyable to work with.

[jimmywarting]

7mb don't sound like much in my ears

[ExE-Boss]

It's still quite a bit for an embedded system.

RAWRTC is much slimmer and modular as it doesn’t come with its own codecs and media stack. This is because RAWRTC is just the ICE, STUN, TURN and DataChannel implementation.

[kevincox]

The compile option is probably the best option. It allows good integration into the library but it is easy to disable for the targets that need it.

[lgrahl]

I took a quick second look at rawrtc, and I just find WebRTC more enjoyable to work with.

Do you mind me asking to elaborate? I'm willing to improve it in order to fit the needs of this and potential other use cases.

[gubatron]

7mb don't sound like much in my ears

7mb would more than double our 6.5Mb linux installer, it's all relative I guess.

For those bloated NodeJS/Electron projects of these days it sounds small

[ExE-Boss]

For those over‑bloated NodeJS/Electron projects of these days it sounds small

Tell me about it…

In all seriousness, I uninstalled 2GiB worth of a single Electron application roughly a week ago.

---

P.S.: @WebReflection wrote these articles about JS code bloat:

• https://medium.com/@WebReflection/avoiding-babels-production-bloat-d53eea2e1cbf (also applies to Electron)
• https://itnext.io/a-bloatless-web-d4f811c7991b

[kevincox]

Hi everyone. We get that 7mb is negligible for a lot of use cases such as phone and desktop apps. However there are still other use cases such as embedded where 7mb is fairly significant. Can we please stop discussing the former and notifying everyone.

The conclusion is that if this is implemented there will need to be a way to have minimal size impact. This may mean making the feature optional.

[ArtemGr]

We get that 7mb is negligible for a lot of use cases such as phone and desktop apps.

7mb isn't neglible, phones have limited memory, Google Instant Apps must be less than 10mb large total, Flutter's entire runtime is 5mb.

Can we please stop discussing the former and notifying everyone.

Of course!

[aldenml]

@lgrahl Sure I can quickly elaborate:

1. Documentation and examples, it's not only that the WebRTC code base is fairly well documented in the parts I looked at, but that every single example I found in the internet looking for this was done using it.
2. On the personal side, the Java-ish API is more familiar to me, and I can follow their OOP style (again only in the parts that I was looking at).
3. The emphasis in the memory ownership clarifications and the use of smart pointers, something that's very important to me.

As you can see, there is nothing against rawrtc, it's just me picking a side. Now, for what I see, the more challenging part will be the libtorrent peer re-architecture, which in turn should easy the task of integrations like this, to the point that switching (or adding) a datachannel library shouldn't be a problem.

[lgrahl]

@aldenml The first one is a good point regarding documentation of RAWRTC which is very sparse. I absolutely agree that this needs to be improved. I could nit-pick and go into detail if you want me to but I'll keep it brief for now: I don't understand where you get the impression from that webrtc.org's native API does any better in regards to documentation and examples.

2 and 3: Well, RAWRTC is C after all. However, memory allocation and ownership in RAWRTC is consistent and clear. create functions allocate reference-counted memory and it's the application's job to mem_deref it at some point. Whenever something is being referenced or copied, the function's documentation will say so (albeit that those comments should rather be added to the docstring of the function).

Now, for what I see, the more challenging part will be the libtorrent peer re-architecture, which in turn should easy the task of integrations like this, to the point that switching (or adding) a datachannel library shouldn't be a problem.

I think we can agree on that.

[aldenml]

For what is worth, this is the result of my experiments building WebRTC, https://github.com/aldenml/webrtc-datachannel

At this moment I lost the enthusiasm with the WebRTC path. It's a very capable piece of software and I learned a lot dealing with it, but to the specific point of integrating it with libtorrent, I just think it's the wrong approach. Size and complexity are among the biggest drawbacks.

I'm adding a link to another project https://github.com/aiortc/aiortc, it's in python, but maybe simple enough to replicate the logic using asio, which for me is the ideal solution.

[ibsusu]

Hopefully this isn't a dead topic and hopefully my question isn't too naive.

Would there be an issue if we just set up a websocket server on a different port that eschewed the webrtc nonsense and just provided a websocket interface that would be run for every libtorrent client? My rationale is that webrtc is difficult and websockets are simple, uwebsockets seems to be a good project and the hello world binary in the examples is 56008 bytes which seems to be a reasonable size increase (probably not all of it) for the gained functionality.

I'm currently working on implementing an xhr and websocket interface on the webtorrent side of this. I was hoping that this idea would make sense. Again maybe I'm dumb or missed something obvious but webrtc seems pointless if we can set up a fast and lightweight ws server.

edit: I just saw the other post about it above. woops.

[cchudant]

That is a good idea, but we still need to refractor a huge part of this project.
Once we have this refactored it won't be that hard to implement the webrtc transport as well since rawrtc looks like a good lib - but we might want to disable it with a compile flag

[WebReflection]

Not sure if useful at all, but since websockets have been mentioned, pocket.io is a lightweight drop-in replacement for socket.io 👋

[ibsusu]

Just an update. A lot of people are saying that WebRTC won't work and want to try a websocket approach. Websockets are not the correct answer, WebRTC is. After internalizing the protocol I think it's the only good way forward and will give us a future proof way to include browsers into existing torrent swarms. Websockets shouldn't be considered in the future and it's my feeling that unless websockets get some sort of QUIC support then anyone mentioning using it outside of signaling has not done enough research. WebRTC not that complex, just sit on your hands and read the spec.

I'm working on integrating it now.

[ibsusu]

I'm trying to figure out the best way to integrate webtorrent stuff with libtorrent. Webtorrent trackers currently require all clients that use webrtc to be connected via websockets for signalling purposes. I was wondering whether this would be feasible for tracker operators to support the additional load on the same servers doing the announce or whether the signalling should be kept separate. Right now if they're kept separate I'm imagining 3 servers running for tracker operators: the tracker, the signalling server and the stun/turn server. Not to mention the additional storage or connectivity requirements between the signalling and tracker servers (though I might have this part wrong I imagine redis or something would be used)

I've taken a cursory look at Ocelot and Opentracker and fitting websockets for signalling is doable but there is no spec for this. Do you think it would be alright to follow webtorrent's design with respect for this? More specifically for libtorrent, is an always on websocket connection acceptable to the tracker and clients for a peer_connection? Not having it mean the tracker will have to push browser webrtc offers another way, which might be preferable in comparison to the overhead of all of the connections but I'm not sure yet. I think I've figured out where the negotiation should happen but the peer_connection_args might need to include whether to use webrtc or not.

This is sort of a brain dump but any input would really be appreciated. If I don't get any I'm going to wing it and see what happens.

[aldenml]

@ibsusu I didn't get to the part of the tracker, but here it is what understood. The Webtorrent tracker is the central service where WebRTC peers candidates are stored (for peer exchange). The communication with it is via websocket, but once you get those candidates, it's up to you to feed them in your WebRTC framework for actual P2P communication (in this case, using DataChannel, not websocket).

The above means that you don't need to modify existing trackers, just add the capability to the client to connect to Webtorrent trackers. At this point, no actual WebRTC is involved, so it should be the "less difficult" part.

[ibsusu]

@aldenml so we should just accept that libtorrent would be required to have an always on websocket connection to them? k, I'll build it that way then if too many people have issue with it we can just pick another method.

[arvidn]

@ibsusu we can consider the mechanism to find peers independent of the mechanism of talking to peers, right?

So, I don't understand why existing tracker operators would have to do anything. If they run a tracker, that is one mechanism to find peers that works.

To talk to peers you can either use WebRTC or normal sockets to talk to normal bittorrent peers.

Could you clarify what the websocket connection would be necessary for? It's not clear what you mean by "signalling purposes". Why isn't WebRTC used for "signalling"?

[lgrahl]

@arvidn WebRTC requires to signal data prior to being able to establish a peer-to-peer connection. This signalling data includes for example the IP addresses (ICE candidates) and the peer's certificate fingerprint. Browsers effectively have only two further transport options: HTTP and WebSocket. That is why @feross wrote a WebSocket tracker: https://www.npmjs.com/package/bittorrent-tracker

[ibsusu]

@arvidn Please correct anywhere I'm wrong, I'm still learning how the torrent system works at a lower level.

First, there's two parts to WebRTC.

One is signaling. Signaling is the sending, relaying and receiving of the SDP offer and answer which includes the ip address and port the the client should be contacted through. It is also used for the ICE candidate exchange. This can happen via any method we can use to communicate between the tracker and client or even between clients:

1. Websockets (client to tracker)
2. XMLHttpRequests (client to tracker)
3. Already existing WebRTC connections (tracker to client if we set it up or client to client)
4. Sockets (native client to native client or native client to tracker)

Two is the actual peer to peer connection which is set up by the browser WebRTC subsystem and the client WebRTC subsystem (I'm going to be using rawRTC).

My question comes from how peers find each other, if they all find each other from the tracker that supports the current signalling method that Webtorrent supports, websockets, then all of the clients require a constant connection to it. But if they find each other through DHT then it seems like the peers need to become a signaling relay to provide a way to for offers and answers to travel if one of them is a browser client.

So the websocket connection is for support of the current tracker that Webtorrent uses. If all clients would support that specific tracker feature they would need to use websockets. I wanted to find a way to update all trackers so that browser clients would be able to use them too but maybe that is a bit ambitious.

Am I wrong somewhere? In a lot of places? Let me know please.

Also, here are two images which are super helpful in understanding how signaling works. It's a stupid, convoluted process but it's the best we got.
https://mdn.mozillademos.org/files/12363/WebRTC%20-%20Signaling%20Diagram.svg
https://mdn.mozillademos.org/files/12365/WebRTC%20-%20ICE%20Candidate%20Exchange.svg

[ibsusu]

If all clients would support that specific tracker feature they would need to use websockets.

This is one big part that I'm not sure about. I'm looking into it but if someone could let me know that'd be great.

edit:
Yeah, they need to be connected by websocket for it to work. For starters I'll just make sure libtorrent clients can just connect via webrtc and websockets then figure out the rest from there??

[jimmywarting]

But if they find each other through DHT then it seems like the peers need to become a signaling relay to provide a way to for offers and answers to travel if one of them is a browser client.

Finding other peers throught DHT in browser isn't possible at this moment. Since peer can't connect to each others without exchanging offer/answer.

so you are right

But if they find each other through DHT then it seems like the peers need to become a signaling relay to provide a way to for offers and answers to travel if one of them is a browser client.

To make something like DHT possible then they also have to be connected to eachother so they can relay messages. Joining a DHT isn't as easy as it seems either cuz you need a bootstrapping server every time you want to join.

There have been a few attempts at making DHT possible with peer-relay, simple-peer-chord and webrtc-explorer and more in webtorrent/webtorrent#288 but as of right now there is no spec or BEP for a WebRTC DHT

For starters I'll just make sure libtorrent clients can just connect via webrtc and websockets then figure out the rest from there??

Good starting point!

• Start by connecting to websocket trackers.
• Send out (non-trickled) sdp offers when you announce and both offer and answers will be sent to you

[lgrahl]

Throwing in https://github.com/w3c/webrtc-nv-use-cases/issues/15 as background to what we think is needed to be able to support DHTs in WebRTC NV (Next Version). Expressing your support (for example in form of :+1:'s) is very welcome.

[arvidn]

I see. so webRTC isn't quite as peer-to-peer as I thought. Everyone still needs to have a connection to some central "introducer". Is that different than ICE? or are you talking about ICE specifically?

Such solution won't scale. Today a tracker can handle millions of peers cheaply. To have millions of websocket connections open is not cheap.

Is there no way in WebRTC/datachannel to connect to an arbitrary (IP, port)? Do you have to go through a relay for the connection setup?

If ICE is mandatory, it adds another complication since the "tracker" is intimately connected to the peer-transport then. With normal bittorrent, how you find peers is independent from how you talk to them. (DHT, PEX, LSD, HTTP trackers, UDP trackers are all independent from the peer connections themselves).

[lgrahl]

Such solution won't scale.

Which is why I'm trying to bring these points into NV (whenever that will happen and hoping that the rest of the WebRTC working group agrees).

Is there no way in WebRTC/datachannel to connect to an arbitrary (IP, port)?

None for browser-to-browser communication. Browser to native, perhaps. I can elaborate if you wish but it's a hack at best and it won't scale for this use case.

If ICE is mandatory, it adds another complication since the "tracker" is intimately connected to the peer-transport then. With normal bittorrent, how you find peers is independent from how you talk to them. (DHT, PEX, LSD, HTTP trackers, UDP trackers are all independent from the peer connections themselves).

I can't quite follow you. Dumbed down, ICE is just there to try all possible combinations of IP addresses between A and B. And those IP addresses have to be signalled as they are arbitrary, temporary and can only be used for a single peer-to-peer connection.

[ibsusu]

@jimmywarting Thanks for the info. It feels like I'm getting caught up on the last three or so years of ideas. It seems like we need some sort of DHT mid-way point bootstrap node every x amount of jumps in order to reduce the pain of bootstrapping into the network for browsers. Even then the amount of round trips that WebRTC causes is a bit of a problem. Relaying messages can also cause problems.

Until some standards committee gives us something else we can use I think midpoints will be ok. anyways, work now.

[ibsusu]

None for browser-to-browser communication. Browser to native, perhaps. I can elaborate if you wish but it's a hack at best and it won't scale for this use case.

Elaborate for me please, ideas are helpful.

[ExE-Boss]

Since libtorrent is native, it could probably do that native‑to‑native and native‑to‑browser WebRTC workaround.

[ibsusu]

I see. so webRTC isn't quite as peer-to-peer as I thought. Everyone still needs to have a connection to some central "introducer". Is that different than ICE? or are you talking about ICE specifically?

Such solution won't scale. Today a tracker can handle millions of peers cheaply. To have millions of websocket connections open is not cheap.

Is there no way in WebRTC/datachannel to connect to an arbitrary (IP, port)? Do you _have_ to go through a relay for the connection setup?

If ICE is mandatory, it adds another complication since the "tracker" is intimately connected to the peer-transport then. With normal bittorrent, _how_ you find peers is independent from how you talk to them. (DHT, PEX, LSD, HTTP trackers, UDP trackers are all independent from the peer connections themselves).

@arvidn I feel like I have to clarify here that while the current Webtorrent tracker requires websockets for signaling to every client including native ones there is no requirement that all trackers relay WebRTC offers to native clients in the same manner. Since they can use any method then only the webtorrent clients would need to have a constant connection. I don't know how the system would look over time considering this.

@lgrahl Is it at all possible to reconnect using the same RTCPeerConnection/session maybe just keep it alive or are we just stuck? I can't seem to find anything to modify the RTCPeerConnection.connectionState in the browser and the docs all seem to point towards having to renegotiate every time a peer connection is closed or disconnected for 5 seconds or more in the browser.

[jimmywarting]

I can't seem to find anything to modify the RTCPeerConnection.connectionState in the browser and the docs all seem to point towards having to renegotiate every time a peer connection is closed or disconnected for 5 seconds or more in the browser.

Nope, Only when the connectionState is new will you be able to reuse it. If nobody replied to your offer or answer can you send it to someone else. Something like a WebRTC store could maybe be useful (for browsers at least) if we it keeps usable RTCPeerConnection that have already gather all ice candidates.

webtorrent/webtorrent#189
https://bugs.chromium.org/p/chromium/issues/detail?id=825576

[lgrahl]

Elaborate for me please, ideas are helpful.

The idea is to let the native side listen on a specific address (IP and port), reuse a certificate for each incoming connection and also reuse ICE username fragment and password. This information would have to be published once and could then be used by multiple clients. The native implementation would act as a DTLS server and accept any incoming connections, not validating their certificate fingerprints (opens up the possibility of a MITM attack which may or may not be relevant for this use case). It may be technically feasible to reuse the same UDP port for multiple connections. I don't know of anyone who has done this so far but I want to try it out eventually. So, this is more theoretical than anything else.

Is it at all possible to reconnect using the same RTCPeerConnection/session maybe just keep it alive or are we just stuck?

An ICE restart is possible but requires further signalling. But it cannot be reused for a different peer.

[ngosang]

I read the whole discussion and I think we are mixing too many things. For me, the issue is simpler.

Real use case:

1. when libtorrent starts, it listen in different ports, one for each protocol. http, udp (and, if you enable webtorrent you need to open one more port).

1. You have a .torrent file with the following trackers:
   Pure webtorrent clients (e.g. browser) will connect only to the wss tracker. wss tracker only returns clients that speak webtorrent. Native clients will connect to http and upd trackers (and wss if we implement it). This .torrent works in all clients and you are not breaking the Bittorrent Specs.

http://open.acgnxtracker.com:80/announce
udp://tracker.coppersurfer.tk:6969/announce
wss://tracker.openwebtorrent.com:443/announce

1. libtorrent send the announce request to every tracker. http and upd are already supported. We have to implement support for ws & wss protocols (I think it's plain or ssl websocket) For wss, we need to send our ip, open webtorrent port and fingerprint. The tracker returns the list of webtorrent peers with enough info to establish the connection with the peers. The peers could support more protocols, but we don't know. I prefer to assume they are only webtorrent peers. We need to keep the list of peers in memory, could be the same list for http and upd trackers but include one "special" flag meaning webtorrent peer.

2. libtorrent will try to connect to each peer. by now, there are regular peers and peers with support for uTP (this is a flag). If the peer is webtorrent, we have to stablish the connection with websocket or webrtc (i don't know the protocol, but i'm sure the wss tracker gave us enough info to connect). I don't know how libtorrent chooses which peers are better. I assume it prefer to contact with uTP because is more efficient. If webtorrent has more overhead, we can try to connect to regular peers first...

[ibsusu]

@lgrahl, do you mean that the native client would just use a 'torrent specific' cert and creds shared between all native clients and have the tracker or already connected clients negotiate for them? I hope I'm not misunderstanding but doing it this way reduces our network calls by half and reduces the pain of being a relay. There are already banning and probation mechanisms in libtorrent so it should be ok.

[lgrahl]

@ibsusu That information doesn't need to be shared (and probably shouldn't) but it needs to be published. Basically, you skip the signalling with this as long as you have a well-known location where you can retrieve the necessary information (i.e the tracker). No idea if torrent trackers are flexible enough to allow to contain this information. But the existing browser-focused WebTorrent implementations would need to add support for that shortcut.

@ngosang The big picture highly depends on how WebTorrent currently operates with trackers and other peers in detail. Someone should look into this and update the big picture accordingly. :slightly_smiling_face:

[ibsusu]

@ngosang There's no spec saying native clients have to connect to the webrtc enabled trackers with websockets; Sharing or publishing webrtc credentials to trackers or webrtc connected peers could be enough for every type discovery method. I think the published cert & creds idea is more efficient on the network than every webrtc enabled client being constantly connected to a wss tracker so I'm trying both and I'll make a PR for the webtorrent tracker as well, it seems too good to not at least try.

[arvidn]

Here's a crazy thought, possibly a bit off-topic.

It seems a lot simpler to add support for libtorrent to accept incoming websocket connections, and just use normal bittorrent wire protocol inside. This would allow a web app to announce with (HTTP) trackers, get peers and connect to libtorrent peers, at arbitrary IP addresses.

It wouldn't allow web apps to connect to each other though, but that seems quite complicated. Perhaps so complicated that it will never happen.

[ibsusu]

@arvidn That would be nice except for the security requirements the browser imposes on us.

see 3b. https://github.com/webtorrent/webtorrent/issues/1492

[ibsusu]

There will be two implementations:

1. An unsecured websocket only p2p implementation with trackers and every other protocol (DHT, PEX).
2. A websocket signaling WebRTC p2p implementation with trackers. This is limited because certificate authority crap. I'm upset.

Dealing with SSL requirements and certificate authorities makes a free form swarm impossible. See you in a month or three if no one takes care of this before then. I need to go into my cave.

[ArtemGr]

Is there no way in WebRTC/datachannel to connect to an arbitrary (IP, port)? Do you have to go through a relay for the connection setup?

Last time I checked (about a year ago?) it was possible. The information about the ICE candidates (IP:port pairs) is exchanged via JSON. Instead of deserializing a piece of JSON received via a signal server one can generate the RTCIceCandidate​Init manually.

So getting the peers from a normal torrent tracker and then trying to reach them through the WebRTC is, IMHO, a viable option as long as the ports match.

[lgrahl]

A websocket signaling WebRTC p2p implementation with trackers. This is limited because certificate authority crap. I'm upset.

Don't rant on the certificate being arbitrary... I think I made it fairly clear that the certificate is not the cause of the problem. There are other limitations requiring you to go through a signalling process for each peer-to-peer connection in a browser-to-browser scenario.
And I think I also made it clear somewhere above that browser-to-native may not be as limited.
Please, don't kill the messenger. It's not that I like the status quo of WebRTC but you're welcome to participate. Spec work is tedious but someone has to do it... and it's better than just ranting about it. That is IMO the number one issue we have in Web APIs and often the reason why they suck.

Other than that, good luck and come back if you have any questions.

Last time I checked (about a year ago?) it was possible. The information about the ICE candidates (IP:port pairs) is exchanged via JSON. Instead of deserializing a piece of JSON received via a signal server one can generate the RTCIceCandidate​Init manually.

You still need to know all the variables which are random for browsers. Explanation is here: https://github.com/w3c/webrtc-pc/issues/2179

TL;DR: Right now, browser-to-browser will not work without individual signalling and negotiation (meaning both browser peers have to communicate with each other prior to establishing a connection). Browser-to-native may work with published signalling data (meaning a browser peer can pick up the published data of the native implementation and connect, but not the other way around).

[mascondante]

Vuze has a way they got it done. Not sure if that will be helpful here. What they did seems to be make a little server in java and bake in a small chromium install. It pops open a little chromium window that tells you to keep it open and somehow handles things that way. If that code would be of any interest to you just install their program and follow the instructions here to download an example torrent. It will download their plugin to your user directory. For me it was "AppData\Roaming\Azureus\plugins\azwebtorrent". I would just upload it but I don't want anyone suspecting I slipped something naughty into the archive. Hope there's something useful in that code for you guys.

[graingert]

That would be totally unacceptable in libtorrent

On Fri, 3 May 2019, 03:48 mascondante, [e-mail] wrote:

Vuze has a way they got it done. Not sure if that will be helpful here.
What they did seems to be make a little server in java and bake in a small
chromium install. It pops open a little chromium window that tells you to
keep it open and somehow handles things that way. If that code would be of
any interest to you just install their program and follow the instructions here
https://wiki.vuze.com/w/WebTorrent to download an example torrent. It
will download their plugin to your user directory. For me it was
"AppData\Roaming\Azureus\plugins\azwebtorrent". I would just upload it but
I don't want anyone suspecting I slipped something naughty in the archive.
Hope there's something useful in that code for you guys.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/arvidn/libtorrent/issues/223#issuecomment-488903489,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADFATHJWTWJQKVVKKQQNUTPTORXZANCNFSM4BR24A6Q
.

[jimmywarting]

I have tried vuze's webrtc and that popup bridge is super annoying.

[andreekeberg]

Yeah, I can't believe Vuze actually shipped that feature..

Also, sorry if I'm ignorant here but wouldn't it be possible to port the webtorrent/webtorrent-desktop implementation from Node.js to C++?

[ExE-Boss]

Well, webtorrent‑desktop is implemented using Electron, so it’s a Chromium window.

And webtorrent‑cli is actually just a Bittorrent CLI client.

You’d need webtorrent‑hybrid to get a WebTorrent CLI client, which works by running a hidden Chromium process that handles the WebRTC connection.

[andreekeberg]

Well, webtorrent‑desktop is implemented using Electron, so it’s a Chromium window.

And webtorrent‑cli is actually just a Bittorrent CLI client.

You’d need webtorrent‑hybrid to get a WebTorrent CLI client, which works by running a hidden Chromium process that handles the WebRTC connection.

Ah, that's right I was thinking of webtorrent‑hybrid!

Either way, running an actually hidden Chromium (if that's an option at all for libtorrent) must be better than a visible popup asking the user to keep it open, that's just horrible UX..

[disarticulate]

another library doing webrtc outside a browser
https://github.com/aiortc/aiortc

[graingert]

I think embedding python might be just as concerning as embedding chromium

On Fri, 21 Jun 2019, 04:18 Eric Xanderson, [e-mail] wrote:

another library doing webrtc outside a browser
https://github.com/aiortc/aiortc

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/arvidn/libtorrent/issues/223?email_source=notifications&email_token=AADFATGN7J5UAZB4N6R6ZL3P3RB65A5CNFSM4BR24A62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYHJVCI#issuecomment-504273545,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADFATBPZAHPKNN24OLIVA3P3RB65ANCNFSM4BR24A6Q
.

[disarticulate]

Yes, my intent wasn't to say embed python, but for a option on how to build
out.

I have no c++ experience.

On Fri, Jun 21, 2019, 03:18 Thomas Grainger [e-mail]
wrote:

I think embedding python might be just as concerning as embedding chromium

On Fri, 21 Jun 2019, 04:18 Eric Xanderson, [e-mail]
wrote:

another library doing webrtc outside a browser
https://github.com/aiortc/aiortc

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<
https://github.com/arvidn/libtorrent/issues/223?email_source=notifications&email_token=AADFATGN7J5UAZB4N6R6ZL3P3RB65A5CNFSM4BR24A62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYHJVCI#issuecomment-504273545
,
or mute the thread
<
https://github.com/notifications/unsubscribe-auth/AADFATBPZAHPKNN24OLIVA3P3RB65ANCNFSM4BR24A6Q

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/arvidn/libtorrent/issues/223?email_source=notifications&email_token=AEFHWPI6IBAIETC5NUSSIILP3SFERA5CNFSM4BR24A62YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYHZHYA#issuecomment-504337376,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEFHWPIBDNQIZLEZJCW2IQDP3SFERANCNFSM4BR24A6Q
.

[bobpoekert]

There will be two implementations:

1. An unsecured websocket only p2p implementation with trackers and every other protocol (DHT, PEX).
2. A websocket signaling WebRTC p2p implementation with trackers. This is limited because certificate authority crap. I'm upset.

Dealing with SSL requirements and certificate authorities makes a free form swarm impossible. See you in a month or three if no one takes care of this before then. I need to go into my cave.

Even if it weren't for ssl websocket is based on tcp so you can't do hole punching

[backkem]

There will be two implementations:

1. An unsecured websocket only p2p implementation with trackers and every other protocol (DHT, PEX).
2. A websocket signaling WebRTC p2p implementation with trackers. This is limited because certificate authority crap. I'm upset.

Dealing with SSL requirements and certificate authorities makes a free form swarm impossible. See you in a month or three if no one takes care of this before then. I need to go into my cave.

Even if it weren't for ssl websocket is based on tcp so you can't do hole punching

The websocket transport is only used to connect to trackers. There you don't need hole punching. For the p2p connection WebRTC takes care of hole punching.

[paullouisageneau]

I recently wrote a light library implementing WebRTC Data Channels in C++ to power another of my projects: https://github.com/paullouisageneau/libdatachannel
It is still young but it should be fully functional already (more examples and tests are coming). It's under LGPLv2, and I'd be glad if it can be useful to someone else.

[paullouisageneau]

I opened a draft PR with a WebTorrent implementation based on the libdatachannel library: https://github.com/arvidn/libtorrent/pull/4123
It works, but still needs testing!

[transitive-bullshit]

This is absolutely amazing work @paullouisageneau. Really excited to see where this work leads! 😄

[shirishag75]

This is absolutely amazing work @paullouisageneau. Really excited to see where this work leads! smile

The great part about this is that @arvidn is actually testing and asking him questions on the draft PR and @paullouisageneau is making the changes based on the queries or suggestions made by arvidn . This means there is a high possibility that after modifications and any cleaning up required, it may get integrated in libtorrent in a future release which furthers clients such as qbittorrent and other apps. which use libtorrent. I, for one am looking for this update, as well as I'm sure others are.

[paullouisageneau]

Thanks @transitive-bullshit !
There is still work to be done on it for sure, since the modifications are quite extensive :sweat_smile:

[shirishag75]

Thanks @transitive-bullshit !
There is still work to be done on it for sure, since the modifications are quite extensive sweat_smile

@paullouisageneau I only hope you keep us in the loop. Would all the modifications be back-ported upstream or your or arvidn end up carrying patches ? How do you see it happening ?

Edit - I do see that @arvidn seems to have raised lot of queries and provided hints of how some things could be better in src/rtc_stream.cpp . Hope we get it all worked out and imbubed in libtorrent. I did notice arvidn talking about C++17 change and hopefully we see that happening after libtorrent 12 comes out.

[backkem]

FYI, There is a new C WebRTC implementation from AWS: https://github.com/awslabs/amazon-kinesis-video-streams-webrtc-sdk-c/tree/master/src/source
Fun fact: It happens to be built by the creator of pion/webrtc, the Go WebRTC implementation.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[filips123]

It's still work in progress (#4123) so I don't think it should already be closed.

[disarticulate]

It's still work in progress (#4123) so I don't think it should already be closed.
Agreed.

[arvidn]

I thought I had configured the bot to leave PRs alone, but I guess not oh, right. This is the ticket

[FranciscoPombal]

@arvidn you can create a new label named "no-stale", or "important" or something like that, and add it to stale bot's exempt labels.

[Mikaela]

Judging by the current stale.yml, this should be labeled as pinned to not get staled, alternatively security but it seems a bit inappropiate to me.

https://github.com/arvidn/libtorrent/blob/9f3192f69184f9d684ea3841ad2f1738772d72ca/.github/stale.yml#L6-L8

[mmathys]

Is this WIP?

[paullouisageneau]

Yes, this is still WIP. We're getting there!

[paullouisageneau]

Implementation-wise it's finished, it could be merged after reviews and bump to C++17.

[paullouisageneau]

I think this can be closed now #4123 is merged!