Libtorrent: Using libtorrent for DHT crawler

[fonic]

Now that libtorrent implements BEP 51 (#1652), it seems feasible to use it as a basis for a DHT crawler. Which is what I'm currently attempting. However, a DHT crawler is not supposed to download torrents, only metadata. For now, I solved this by modifying torrent handling to never transition states from 'downloading metadata' to 'downloading'.

Would you be willing to implement something that blocks downloading per user request, e.g. an additional flag for add_torrent_params?

I'm aware that DHT crawling is not the focus of this library, but a flag like this is and code to handle it is actually all that's missing to support crawling as far as I can tell.

[arvidn]

you can set a torrent to upload_only mode in add_torrent_params. You can also set file_priorities to all 0 in add_torrent_params. I believe those would have the desired effect

[arvidn]

I'm interested in how it goes implementing a crawler on top of libtorrent, please let me know!

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[fonic]

Well, it's definitely possible. I have a working proof-of-concept, I'm currently doing a clean rewrite. However, libtorrent is not suitable out of the box, I applied several small patches to make it happen.

[arvidn]

if you have any you think would be suitable for up-streaming, please feel free to submit PRs!

[fonic]

I'd be happy to. When I'm done, I think I'll have a pretty good impression of what makes sense and what does not.

[fonic]

I wonder if you could help me with a few things. I'm currently migrating from libtorrent 1.1 to 1.2 with deprecated functions being disabled.

I need a list of known DHT nodes. I'm using session.dht_live_nodes() to request it. However, this method requires passing a node id. Before, I used session.dht_state() to determine it. But with 1.2, there seems to be no equivalent.

EDIT: session.save_state() provides the node id, so that's the replacement?

[fonic]

@arvidn:

you can set a torrent to upload_only mode in add_torrent_params

Are you supposed to supply this when adding the torrent, like this?

handle = self.session.add_torrent({"info_hash": info_hash, "upload_only": True})

If so, it is currently not included in the Python bindings as the command fails.

You can also set file_priorities to all 0 in add_torrent_params. I believe those would have the desired effect

They would, but the manual states:

In order to download just the metadata (.torrent file) from a magnet link, set file priorities to 0 in add_torrent_params::file_priorities. It's OK to set the priority for more files than what is in the torrent. It may not be trivial to know how many files a torrent has before the metadata has been downloaded. Additional file priorities will be ignored. By setting a large number of files to priority 0, chances are that they will all be set to 0 once the metadata is received (and we know how many files there are).

So this is not very reliable as you don't know the number of files in advance. One could of course supply 1000+ file priorities set to 0 to be on the safe side, but this would be very bad coding... It would be much better if one could supply something like 'default_file_priority': 0 which is applied to all files by libtorrent once metadata has been retrieved.

[arvidn]

Are you supposed to supply this when adding the torrent, like this?

it's part of the flags field (of add_torrent_params). These are the possible flags. I think in python this would be something like:

handle = self.session.add_torrent({"info_hash": info_hash, "flags":lt.torrent_flags.upload_mode})

So this is not very reliable as you don't know the number of files in advance. One could of course supply 1000+ file priorities set to 0 to be on the safe side, but this would be very bad coding... It would be much better if one could supply something like 'default_file_priority': 0 which is applied to all files by libtorrent once metadata has been retrieved.

iirc, there was some suggestion that all subsequent file priorities should be copied from the last one in the list. That seems like a simple approach.

[fonic]

Thanks, I'll give it a try!

[arvidn]

there was some suggestion that all subsequent file priorities should be copied from the last one in the list. That seems like a simple approach.

This is not currently implemented, and thinking some more about it, the only real benefit of this approach over your suggestion is that it can be implemented without breaking ABI, since it doesn't need to introduce any new fields to add_torrent_params. However, it does change behavior, in some fairly significant ways. Existing code might break if this behavior is introduced.

I think yor suggestion is better. It is backwards compatible and is more explicit about what is requested. The fact that it would need an additional field in add_torrent_params means it would have to go into master though.

If you're interested in adding this feature, I think it would be pretty straight forward to do so, in torrent::init() in torrent.cpp. It has access to the add_torrent_params object as m_add_torrent_params. If the requested default differs from the default priority, the m_file_priority vector could be resized near the top of that function.

[fonic]

I'll be happy to look into it and submit another PR. So far, contributing to your project has been a delight in contrast to other projects I contributed to in the past. Lots of projects out there make an extra effort in promoting that contributions are highly welcome, but when you actually do put hours of work into something and try to submit it, people are often like '_we don't like it, but thanks for playing_'.

By the way, the crawler turns out to be working much better than I ever anticipated. Would you care to test it at some point?

[arvidn]

By the way, the crawler turns out to be working much better than I ever anticipated. Would you care to test it at some point?

sure

[fonic]

By the way, the crawler turns out to be working much better than I ever anticipated. Would you care to test it at some point?

sure

You've got mail.

[cannadayr]

any update on this? I would be interested if you need any end-user testing. email is on my profile.