Libtorrent: get metadata info without downloading the complete file

[fastesol]

As I read the different posts and libtorrent documentation, I know (as documented), I have to download the torrent file in order to get the metadata. but how the uTorrent App works, when I just start downloading, I get the metadata within a second then after getting the metadata, I can pause downloading. So, it doesn't restrict me to download a complete file in order to return metadata.

[arvidn]

you can set all file priorities to 0 when you start the torrent. That way it will download the metadata and then be done, and disconnect all peers.

[fastesol]

I tried the following code - where I set the file priorities right after I get the metadata. (as only metadata have the info about total # of files)

ses = lt.session()
ses.listen_on(6881, 6891)
params = {'save_path': '/tmp/torrent_files/'}

ses.add_dht_router("router.utorrent.com", 6881)
ses.add_dht_router("router.bittorrent.com", 6881)
ses.add_dht_router("dht.transmissionbt.com", 6881)
ses.start_dht()

link = "magnet:?xt=urn:btih:"+info_hash
h = lt.add_magnet_uri(ses, link, params)

while (not h.has_metadata()):
    time.sleep(1)

torinfo = h.get_torrent_info()

for x in range(torinfo.num_files()) :
    h.file_priority(x, 0)

but found the files in /tmp/torrent_files/

Kind Regards
Muaaz

[arvidn]

you can set file priorities without the metadata by setting the vector in add_torrent_params.

The reason you end up downloading files now is that there's a race condition. libtorrent will start downloading pieces at the same time it posts the metadata alert to you. By the time you receive the alert, files may have been created already

[fastesol]

Thank you Arvidn for your great support.
I just noticed, there are few torrents that are being downloaded and rest are not. I added this in parameter (Zero priority for each file)

params = {
    'save_path': '/tmp/torrent_files/',
    'file_priorities': [0] * 1000
}

Note: we don't know number of files at the start. I assumed 1000. (I might be wrong)

Kind regards
Muaaz

[sledgehammer999]

That's why in qBittorrent, when we want to load only the metadata from the magnet link we set flag_upload_mode and unset flag_auto_managed. We don't even bother with setting file priorities.

[sledgehammer999]

Also, I am not sure, but setting flag_stop_when_ready might do the same thing as the above.

[arvidn]

at first I was thinking stop_when_ready could be used here too. right now though, downloading metadata counts as downloading for its purposes. It's main use case is to check a torrent without starting it.

[sledgehammer999]

@arvidn maybe its worthwhile to have an explicit flag for this situation?

[fastesol]

well this is what I did

params = {
    'save_path': '/tmp/torrent_files_magneticod/',
    'file_priorities': [0] * 1000,
    'flag_auto_managed': False,
    'flag_stop_when_ready': True,
    'upload_mode': True
}

Due to the shortage of time, I didn't use the combinations of above options to know which option is stopping, the download now. I think, upload_mode is doing the job, (I'll check the cpp code and try with different combinations and post it here) anyway, really appreciated your help.

Thank you!

[arvidn]

leaving stop_when_ready in there will cause a surprise the day you add a torrent whose files you already have, and it goes to check them.

[StanislavNikolov]

After multiple hours reading the docs, I still can't find the answer:
1) auto_managed must be false, otherwise it ignores the other flags (at least upload_mode)
2) file_priorities seems like a crude hack, because there may be more than 1000 files
3) stop_when_ready means not going into "downloading_metadata" state, so it must be false (default)
4) auto_managed false and upload_mode true does not put torrents into "downloading_metadata"

@fastesol's solution does not work (and it makes sense to not work) - but what is the correct way? Probably auto_managed = False and manually moving torrents between queues?

[arvidn]

you can set upload_mode (in add_torrent_params::flags). I've recently improved the documentation to answer this question (but I don't think it's up on the website yet). If you have any suggestions of where in the documentation the answer to this question should be, I can add it. (or better yet, suggest a pull request against RC_1_2)

[arvidn]

auto_managed false and upload_mode true does not put torrents into "downloading_metadata"

what mode does it put the torrent in? paused? (in which case you need to clear the paused flag too)

[StanislavNikolov]

import time, sys, libtorrent as lt

# https://www.libtorrent.org/reference-Alerts.html#alert_category_t
#alert_mask = 0b111111111111111111111111
alert_mask = 0 # lt.alert.category_t.torrent_log_notification
ses = lt.session({
     'upload_rate_limit': 0
    ,'download_rate_limit': 0
    ,'active_downloads': -1
    ,'active_limit': -1
    ,'alert_mask': alert_mask
})

def add_infohash(ih):
    link = 'magnet:?xt=urn:btih:' + ih

    params = {
         'save_path': 'data/'
         ,'auto_managed': False
         ,'upload_mode': True
         #,'stop_when_ready': True
         #,'paused': True
        #, 'file_priorities': [0] * 1000
    }

    handle = lt.add_magnet_uri(ses, link, params)
    handle.resume()

    return handle

torrents = []
for infohash in open('list.txt').readlines():
    infohash = infohash.strip()
    torrents.append((infohash, add_infohash(infohash)))

#print(handle.get_torrent_info())
#print(handle.is_valid(), handle.has_metadata(), handle.trackers())

while True:
    print('-------')

    alerts = ses.pop_alerts()
    for a in alerts: print(a.message())

    for infohash, handle in torrents:
        info = handle.get_torrent_info()
        stat = handle.status()
        size = info.total_size() if info is not None else None
        print(f'{infohash}: size={size}, qp={stat.queue_position}, paused={stat.paused}, state={stat.state}')

    time.sleep(0.5)

The code above produces this output after a couple of minutes:

[redacted]
-------
7976F5BD93F4239C0681BE6850A1B3750BB916C3: size=1479539936, qp=0, paused=False, state=downloading
C7059826B29ADAC1C79F907DB3230179FA15B2C4: size=None, qp=1, paused=False, state=downloading_metadata
4388342AA5AF38BFB7FF48E3A3A353C6D0D8F5A5: size=1490216142, qp=2, paused=False, state=downloading
06C92BC2C1114DD0CA3305E3A45084D34075268F: size=1414016684, qp=3, paused=False, state=downloading
1749DC45DFC52D3F3358DB289894151DC86D6AE1: size=2803883340, qp=4, paused=False, state=downloading

Interesting things to note:
1) Sometimes directories named after the torrents are created in /data. Not always, not reproducible. Maybe there is a race condition somewhere?
2) When state=downloading_metadata, indeed it seems like libtorrent downloads metadata. On the other hand, state=download is misleading, because nothing is actually written to disk. Maybe it caches it to RAM, but didn't see anything like this in the docs.

[StanislavNikolov]

I found the following sentence in the docs https://www.libtorrent.org/reference-Core.html#torrent_flags_t: "In order to stop a torrent once the metadata has been downloaded, instead set all file priorities to dont_download" - looks like this is the direction I should be exploring. Maybe I need a tight loop to look for "downloaded metadata" events, and change priorities. I can't set priorities before I have metadata, because I don't know how many files are there. (Also, the 'file_priorities': [0] * 1000 trick does not work too). But this leaves room for even more race conditions - it would be possible for files to be allocated and pieces downloaded before I have a chance to set their priorities.

[arvidn]

Sometimes directories named after the torrents are created in /data. Not always, not reproducible. Maybe there is a race condition somewhere?

yes, this was discussed here recently.

When state=downloading_metadata, indeed it seems like libtorrent downloads metadata. On the other hand, state=download is misleading, because nothing is actually written to disk. Maybe it caches it to RAM, but didn't see anything like this in the docs.

Well, it is downloading the metadata. The metadata isn't written to disk (because traditionally it existed on disk already). It can be written to disk with save_resume or by asking for the metadata and creating a torrent from it.

[arvidn]

(Also, the 'file_priorities': [0] * 1000 trick does not work too). But this leaves room for even more race conditions - it would be possible for files to be allocated and pieces downloaded before I have a chance to set their priorities.

You would have to set the file priorities as you add the torrent, in add_torrent_params