Libtorrent: [Feature] A way to invalidate pieces?

[sledgehammer999]

Disclaimer:

1. I understand that this might be too late for 1.2.0, so feel free to outright close it.
2. I understand that this might be difficult to implement due to complexity of code invloved, so feel free to close it.

Can you consider adding a way to invalidate certain pieces or range of pieces? And maybe higher function that invalidates pieces belonging to a specific file?

Why?
We get a lot of requests that want a "delete file" option (usually in the list of file priorities) in multifile torrents. We explain to them that currently the only way to implement this is to a)pause torrent, b)delete file, c)force recheck and d) resume. However, we haven't implemented this because a) it adds quite a complexity to instrument this correctly on our side and b)force rechecking is mostly undesirable especially when involving big torrents.

So from qbittorrent's POV the simplest way would be to tell libttorrent "hey consider these pieces not downloaded" and then we go delete those files ourselves. This should avoid the need for forcing a recheck. I don't know how feasible it is to actually implement on your side.

@glassez @Chocobo1 any comments, better ideas? Or should this request be dropped?

[Seeker2]

A problem from the torrent swarm's point-of-view is any remaining files that aren't deleted may be left with missing partial end pieces.

[sledgehammer999]

A problem from the torrent swarm's point-of-view is any remaining files that aren't deleted may be left with missing partial end pieces.

Maybe this could be solved by moving the partial piece of the deleted file into the .parts file.

[Seeker2]

Maybe, although I'm sure a force recheck would break that.

[sledgehammer999]

Maybe, although I'm sure a force recheck would break that.

Why? How does libtorrent currently store edge pieces? It certainly doesn't write out the unselected file.

[Seeker2]

A force recheck probably wouldn't be "aware" of the .parts file.

[glassez]

way to invalidate certain pieces

The way to tell libtorrent "just forget these pieces" is a useful thing in general.

[sledgehammer999]

A force recheck probably wouldn't be "aware" of the .parts file.

This is only true for pre-1.1.0 libtorrent or other bittorrent clients. But we're talking about the same client with at least 1.1.0 libtorrent.

[Seeker2]

Good, it's more up-to-date than my knowledge is. :)

[arvidn]

I agree that this seems like a useful thing to have. It's not all that simple to implement well on the libtorrent side either. There's an extension message to tell a peer a piece was lost, that would potentially end up being a lot of messages. Maybe some clients would treat an extra bitfield message correctly, or maybe all peers should just be disconnected and reconnected.

A somewhat simpler approach to only allow this "offline", i.e. not while connected to a swarm.

However, there is a way to do that currently as well, by manipulating the resume data and remove and re-add the torrent.

Especially in libtorrent-1.2, manipulating resume data is simpler since it's held in the add_torrent_params object, rather than a bencoded buffer.

Specifically, you can clear which pieces the resume data say you have.

[sledgehammer999]

A somewhat simpler approach to only allow this "offline", i.e. not while connected to a swarm.

Maybe disconnect peers that require the invalidated pieces?

However, there is a way to do that currently as well, by manipulating the resume data and remove and re-add the torrent.

This will mess up with the queue position.

[arvidn]

This will mess up with the queue position.

The queue position can be restored though

[elgatito]

I'm also interesting in this feature.

In my case, I'm storing pieces to memory, so it looks like I have a range of pieces, that I want to store, when user gets first piece from that range, I increment the range position, first element can be deleted, new element in the list is downloaded.
Currently, I set all pieces priority to 0, except those that have to be downloaded, they are 7.
If we'd have 10 pieces in the torrent and space for saving only 5 pieces, that would be:
7 7 7 7 7 0 0 0 0 0
And when first item is not needed, it goes to:
0 7 7 7 7 7 0 0 0 0.

When I move the range, I use torrent_handle::prioritize_pieces() to update priorities.
I have tried to call torrent_handle->native_handle()->picker->restore_piece(index) on deleted piece, so when I need this piece once again - I would set priority to > 0 and it gets downloaded.
But trick did not help with that :)

[elgatito]

@arvidn observing the code and it looks like restore_piece() is what I need in my case, it should treat piece as non-downloaded and when I set it's priority to >0 - it should re-download it?

[elgatito]

I have played a little bit more with restore_piece() and it's not helping in my case.
I need to reset piece status, so it become clean, like it was on initialization, if I were not even touching/downloading it, so when priority then goes to >0, it will be started as usual piece.

[arvidn]

restore_piece() is not related to this. It resets the state of a piece if it fails the hash check.
looking at the vicinity of write_dont_have() would probably be more helpful. The piece picker would need to be updated to remove pieces that it previously thought had been downloaded. I don't think that function exists yet.

[elgatito]

@arvidn

restore_piece() is not related to this. It resets the state of a piece if it fails the hash check.

As for me, failing hash would need piece to be re-downloaded (entirely), and until it's done - pieces cannot be available for seeding and should not be considered as downloaded.
That is why I tried this method.

The piece picker would need to be updated to remove pieces that it previously thought had been downloaded. I don't think that function exists yet.

Would be great to have this function. Probably issue reporter would need function with vector input (to reset multiple pieces), and I would need that or function with int piece input.

[elgatito]

@arvidn I have tried pretty dummy method addition to restore pieces in piece_picker:

    void piece_picker::reset_piece(int index)
    {
        auto const download_state = m_piece_map[index].download_queue();
        if (download_state != piece_pos::piece_open) return;

        we_dont_have(index);
        add_download_piece(index);
    }

Of course, not sure which checks do I need, or what else should be done here to reset piece state to what it was on torrent initialization, but in practice, I see piece is being re-downloaded fine after it was deleted, but priority was set again to > 0.

[elgatito]

My custom reset_piece works almost in they way I wanted, when I need to re-download piece - I set priority to >0 and it's downloaded, but sometimes I set priority and piece is not downloading.

I have seen there is erase_download_piece, maybe I need to run it if status is != finished?

[arvidn]

if you want to extend the piece_picker with this functionality, you have to make sure that after removing a piece, it's not considered to be had. There are a few data structures that encode this information (I think somewhat redundantly, for performance reasons). I think the first step would be to look at the unit test and add the cases you would like to be supported

[libTorrentUser]

I just want to add that I am trying to do pretty much what elgatito is also trying to do. An official non-hacky way of saying "look, this piece is gone, I don't have it any more and I am not interested in it" would be great.

Currently I am trying to achieve the same result using another (way uglier) method - simply failing the read call. But in order to do that I also have to change code I shouldn't be touching in order to avoid the torrent being stopped by libTorrent.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.