Tribler: Our DHT-based torrent health check is unreliable

[devos50]

To check the health of a torrent from the DHT, we download the torrent in upload mode. When we receive the metadata, we get the list of peers we are connected to and consider them as leechers (or seeders if they have everything). This mechanism does not accurately represent the real swarm size. To get proper popularity estimations, we need something better.

One solution is to rely on the get_peers method (which is exposed in the Python bindings since libtorrent 1.1).

Related to #4256.

[devos50]

I managed to get some peers using the get_peers method. Possible idea: exploit the PEX extension to perform a crawl of peers in the swarm? Not sure whether this would work and what the accuracy of this approach would be.

[devos50]

Opened an issue on the libtorrent repository.

[devos50]

Very related scientific work: https://www.cs.cornell.edu/courses/cs6464/2009sp/projects/yc336/final_doc.pdf

Unfortunately, they do not provide an implementation of their tool.

[devos50]

Current status:

• I'm able to retrieve peers from the DHT using the dht_get_peers method (which has been exposed since libtorrent 1.1)
• The current idea is to perform a crawl through the swarm using PEX by manually sending a handshake, extended message, listening for the ut_pex message and connect to these new peers. However, it might take a considerable amount of time before I get accurate numbers on the swarm size (since ut_pex messages are sent periodically, every minute). Also, I'm running into the connectivity issues as described in this SO post.
• Arvid suggested to use BEP33. However, from my understanding the libtorrent Python bindings do not expose enough functionality to send BEP33 messages to other peers. Arvid noted that peers in libtorrent still respond to them though.
• Another idea is to simply try a few popular trackers (e.g. coppersurfer) and try to fetch swarm healths from them. This is less reliable since peers might not advertise the health to our pre-defined trackers. It could be used as fallback option however.
• As a last resort, we could use the exit nodes to download a piece of a torrent and have this exit node estimate the swarm size.

[devos50]

With some custom code added to libtorrent, I am able to send BEP33 messages to others. I also get responses that include the bloom filters, by listening to alerts in lt.alert.category_t.dht_log_notification and printing them. However, I'm unable to fetch the actual bloomfilter from these messages since I cannot get the raw packets from alerts posted to Python code. It seems the limited exposure of libtorrent methods in Python is preventing me from doing the stuff I want once again 👎

[devos50]

Progress! After some modifications to libtorrent, I'm able to retrieve the required BEP33 bloom filters. We should now merge the received bloom filters (by XORing the bits of them with each other) and determine the number of seeders and leechers. Hopefully I can finish this tomorrow and run some initial tests on our academic torrent dataset.

My actual code changes are documented on this libtorrent issue: https://github.com/arvidn/libtorrent/issues/3701#issuecomment-472938036

[devos50]

I finished the implementation and I am now able to estimate the health of arbitrary info hashes, without relying on any centralised tracker 🎉 . I made the application available as a command line tool. The implementation can be found here. My libtorrent fork can be found here.