Libtorrent: improve piece picker support for small pieces

[sserdda-liamE]

(moved from https://github.com/qbittorrent/qBittorrent/issues/10402)

Libtorrent's current automatic piece size calculation is decidedly odd and is running afoul of at least one private tracker's limits. I see there was a request last year for sensible defaults that went unanswered. I'd like to suggest that the piece size be calculated to result in a torrent with either:

between 500 and 1000 pieces

or :

between 1000 and 2000 pieces

uTorrent 2.2.1's default seems to be to create a torrent with between 512 and 1024 pieces and it has created many hundreds of thousands of torrents at those values without any seeming swarm performance issues, so 500 pieces as a lower bound provably works.

Piece sizes of under 4MB can run into performance issues on gigabit connections, but qB's current default won't switch to 4MB until a ~48GiB torrent size. uTorrent's torrent creator would have switched to 4MB pieces @ 2GB.

I'm not aware of any commonly used clients that have difficulty with 8MB or 16MB pieces, so there shouldn't be a compatibility issue with using those sizes.
(There are clients that can't handle 32MB or above, though, so perhaps >16MB should not be chosen automatically.)

[arvidn]

Libtorrent's current automatic piece size calculation is decidedly odd and is running afoul of at least one private tracker's limits.

Would you mind elaborating on this? Are there trackers that disallow piece sizes that are too small? or too large?

uTorrent 2.2.1's default seems to be to create a torrent with between 512 and 1024 pieces and it has created many hundreds of thousands of torrents at those values without any seeming swarm performance issues

Let's avoid making claims like this without any measurements.

Piece sizes of under 4MB can run into performance issues on gigabit connections

Would you mind elaborating a bit here? It sounds like the disk seek time, between 4MiB pieces, becomes the bottleneck at high enough rates. This would (presumably) only apply to seeders. In swarms where most peers are well behaved, in that they "pin" peers to whole pieces at high rates (libtorrent does this for instance).

I'm not aware of any commonly used clients that have difficulty with 8MB or 16MB pieces, so there shouldn't be a compatibility issue with using those sizes.

The piece sizes were not chosen for compatibility. The piece size primarily affect:

1. the size of the .torrent file
2. the speed at which data can propagate through a swarm

and I suppose the disk-seek-latency could be added to this list as well.

Generally, to increase performance (2) the smallest piece size is chosen such that (1) stays within reason.

It sounds like you're suggesting that (2) is a small problem compared to disk-seek-delay between 4MiB pieces on fast connections.

In BitTorrent v2, (1) is no longer a concern, so I would expect piece sizes to go down to 1MiB, or something like that, in the long run.

This is still all under the assumption that the performance issue you raise is caused by the seek time between serving pieces to peers. It seems like this is primarily a property of the downloader. Sure, there is a well established piece picker logic that makes peers request entire pieces at a time at high rates, but in principle, there's nothing preventing a peer from requesting multiple adjacent pieces from a single peer at high rates. (in fact, libtorrent does this from web seeds, since they are especially finicky when it comes to serving contiguous data).

For reference, here are the PR: https://github.com/arvidn/libtorrent/pull/2669

@ssiloti iirc, you came up with the formula. Maybe you have some comments.

[sserdda-liamE]

Would you mind elaborating on this? Are there trackers that disallow piece sizes that are too small? or too large?

Both. Allowed range is between 250 and 4000 pieces. (excepting the bottom and top ends)

Piece sizes of under 4MB can run into performance issues on gigabit connections

Would you mind elaborating a bit here? It sounds like the disk seek time, between 4MiB pieces, becomes the bottleneck at high enough rates. This would (presumably) only apply to seeders. In swarms where most peers are well behaved, in that they "pin" peers to whole pieces at high rates (libtorrent does this for instance).

It applies to peers as well. Large torrents are going to exceed the write cache to a degree where the writes will still largely be random, and being hit by thousands of small writes will hammer a drive more than a few large ones. I had my fastest drive taken to its knees by a 30GB torrent made with 256kB pieces.

The piece sizes were not chosen for compatibility. The piece size primarily affect:

1. the size of the .torrent file
2. the speed at which data can propagate through a swarm

and I suppose the disk-seek-latency could be added to this list as well.

Generally, to increase performance (2) the smallest piece size is chosen such that (1) stays within reason.

It sounds like you're suggesting that (2) is a small problem compared to disk-seek-delay between 4MiB pieces on fast connections.

*smaller than 4MB pieces.
My tests across gigabit LAN showed issues at 2MB and below. My disks could handle 4MB.
So libtorrrent not switching to 4MB pieces until a 48GB torrent means the vast majority of torrents created at the automatic values are going to have a problematically small piece size.

I'd say the overhead and increased I/O load incurred by 80,000 piece torrents seems to outweigh any peering benefits. The "massive swarm with a widely dispersed distributed copy" which would see gains from a high piece count does not often seem to exist in the real world.

[ghost]

@arvidn I agree with the OP's points. Not switching to 4MB pieces until a 48GB torrent means most torrents created by libtorrent are going to hit most people's disks pretty hard. Especially public torrents where people just use auto mode. Private trackers are less likely because some people care about piece size on private trackers, but not much less. The end result of the current auto mode is still likely to result in performance drops for the end downloaders. We need something more reasonable.

[ssiloti]

I agree with @arvidn that it would be better to address poor disk utilization by making the piece picker favor adjacent pieces when the disk buffer pool is under pressure.

[ghost]

@ssiloti Why not both?

[ssiloti]

Because increasing the piece size negatively impacts swarm performance and I don't consider satisfying the arbitrary rules of some private tracker a good reason to change the default.

[sserdda-liamE]

The defaults already fit within those "arbitrary" limits for smaller piece sizes. It leaks out on the top end for no discernible reason.
Why would this 629GB torrent of mine be made with 80,000 8MB pieces? "Swarm performance"? How many snatches do you think it has?
It performed just fine made with 16MB.

I agree with @arvidn that it would be better to address poor disk utilization by making the piece picker favor adjacent pieces when the disk buffer pool is under pressure.

That only helps if there are adjacent pieces and only until they're propagated; it doesn't help the peer being inundated by multiple seeds; and it doesn't address the wear being placed on the disk at any other time.
My upload bandwidth is low enough and files spread across enough drives that I could probably make every torrent with 16kB pieces and not be disk I/O bound. Doesn't mean I'd do such a terrible thing to my drives. I am the lone seeder on around 200 torrents and killing the drive with such a colossal multiplication of seeks would tend to negate any further propagation.

[arvidn]

The defaults already fit within those "arbitrary" limits for smaller piece sizes. It leaks out on the top end for no discernible reason.

As I mentioned in a previous post, it's a trade-off between size of .torrent file and swarm performance (specifically how fast data propagates through the swarm). That's the reason.

Why would this 629GB torrent of mine be made with 80,000 8MB pieces? "Swarm performance"?

Right. because 16 MiB pieces would greatly impact the performance and 4 MiB pieces would make for a very large .torrent file.

It performed just fine made with 16MB.

Compared to what? Do you have measurements?

I agree with @arvidn that it would be better to address poor disk utilization by making the piece picker favor adjacent pieces when the disk buffer pool is under pressure.

That only helps if there are adjacent pieces and only until they're propagated; it doesn't help the peer being inundated by multiple seeds; and it doesn't address the wear being placed on the disk at any other time.

I don't quite understand what you're saying here, but I would like to.

1. As it's the downloader that picks what to download, that's not really a problem. You just pick aligned ranges of pieces. But even if you don't, you'll still have a lot of piece-picks with adjacent open pieces.

2. "only until they're propagated". Regardless of piece size, you have to wait for a piece to be propagated to your peers until you can request it. If they don't propagate fast enough for you to be able to make large requests, the disk I/O is most likely not your bottleneck. Consider the first few pieces you request as "single" pieces. If your disk I/O is slow, your peers will get ahead of you and the next time you pick pieces you'll have multiple adjacent pieces available to pick.

3. "it doesn't help the peer being inundated by multiple seeds;". I may be misunderstanding you here, but it's the downloader that picks pieces. It doesn't cost you anything to be connected to a seed unless you request anything from it. It's up to you to pick pieces optimized for disk I/O or not.

4. "and it doesn't address the wear being placed on the disk at any other time." are you referring to "wear and tear" of physically moving the arm? You could always download sequentially, but you probably won't get very good download performance. It's not obvious that you'll gain that much in terms of drive longevity though, but I'm not an expert on this. The filesystem will fragment files regardless. iirc, the default extent size on ext4 is 4MiB.

My upload bandwidth is low enough and files spread across enough drives that I could probably make every torrent with 16kB pieces and not be disk I/O bound. Doesn't mean I'd do such a terrible thing to my drives. I am the lone seeder on around 200 torrents and killing the drive with such a colossal multiplication of seeks would tend to negate any further propagation.

I would suggest the only reason it would be expensive, disk I/O wise, is because implementation details in the downloaders. Currently implementations have an affinity to pick whole pieces, but it doesn't have to be like that. Although, as the downloader, it can be hard to know whether the uploader is disk I/O bound or not. Maybe there should be a hint.

[sserdda-liamE]

It performed just fine made with 16MB.

Compared to what? Do you have measurements?

A 1.001 ratio on the upload doesn't look too bad to me.

As I mentioned in a previous post, it's a trade-off between size of .torrent file and swarm performance (specifically how fast data propagates through the swarm). That's the reason.

A larger file does not need more pieces, though. Everything takes proportionally longer.

1. As it's the downloader that picks what to download, that's not really a problem. You just pick aligned ranges of pieces. But even if you don't, you'll still have a lot of piece-picks with adjacent open pieces.

You can only request what they have. With random distribution as it's being seeded to the swarm you're not going to have adjacent pieces. Everyone is going to be hammered by tiny seeks.

1. "only until they're propagated". Regardless of piece size, you have to wait for a piece to be propagated to your peers until you can request it. If they don't propagate fast enough for you to be able to make large requests, the disk I/O is most likely not your bottleneck. Consider the first few pieces you request as "single" pieces. If your disk I/O is slow, your peers will get ahead of you and the next time you pick pieces you'll have multiple adjacent pieces available to pick.

Perhaps you're not quite aware of what the Chinese can crank out:

https://i.postimg.cc/DZqmL6J6/Capture.png

Contrast with this one made with qBittorrent @ 2MB piece size. It's pretty sad... except for that one guy who can afford to download Blu-rays to SSDs...

https://i.postimg.cc/NfvSd4yT/Capture.png

1. "and it doesn't address the wear being placed on the disk at any other time." are you referring to "wear and tear" of physically moving the arm? You could always download sequentially, but you probably won't get very good download performance. It's not obvious that you'll gain that much in terms of drive longevity though, but I'm not an expert on this. The filesystem will fragment files regardless. iirc, the default extent size on ext4 is 4MiB.

I think it's 128MB with 4kB block size but neither would stop it from reading/writing sequentially.

I would suggest the only reason it would be expensive, disk I/O wise, is because implementation details in the downloaders. Currently implementations have an affinity to pick whole pieces, but it doesn't have to be like that. Although, as the downloader, it can be hard to know whether the uploader is disk I/O bound or not. Maybe there should be a hint.

What would further segmentation achieve when a piece is contiguous by nature? Breaking it into smaller pieces wouldn't help disk I/O.
Everyone could just request pieces sequentially but that's terrible for swarm health. "Requesting random contiguous segments" is just, "Using a larger piece size," which is why we're here.
This is already a solved problem. You do not need to split a file to the bit level to get it propagated. Splitting into segments of 0.1% (1000 piece torrents) work quite fine at that while keeping .torrent size down and quickly ramping to a piece size that gives you good I/O performance.

[arvidn]

What would further segmentation achieve when a piece is contiguous by nature?

"further segmentation" (i.e. smaller pieces) achieve faster propagation of data through the swarm.
the fact that pieces are contiguous is not really relevant to that achievement.

Breaking it into smaller pieces wouldn't help disk I/O.
Everyone could just request pieces sequentially but that's terrible for swarm health. "Requesting random contiguous segments" is just, "Using a larger piece size," which is why we're here.

I'm suggesting that "piece size" and "contiguous request size" are technically different things. They just happen to be conflated because of implementation details. (and it's quite easy to fix, I intend to do so, probably all the way back to libtorrent 1.1.x).

The piece size affect how long it takes from the time you download a byte to the time you're able to upload that byte to someone else. Larger piece sizes increase this latency.

It's possible to increase the "contiguous request size" without sacrificing the propagation latency caused by larger pieces.

This is already a solved problem. You do not need to split a file to the bit level to get it propagated. Splitting into segments of 0.1% (1000 piece torrents) work quite fine at that while keeping .torrent size down and quickly ramping to a piece size that gives you good I/O performance.

I don't think it is a solved problem (in that a solution has not been implemented). You can have a different bias in your trade-off between propagation latency and contiguous requests and .torrent file sizes.

[sserdda-liamE]

You can have a different bias in your trade-off between propagation latency and contiguous requests and .torrent file sizes.

The current implementation isn't internally consistent, though.
If it was, "Hey, we've decided that between 500,000 and 1 million pieces is optimal because what's a few megabytes in .torrent files between friends," that would be one thing. But the piece number fluctuates from sane values at one piece size to insane ones at another. If you're valuing some hypothetical propagation advantage of 20,000 pieces at 2MB, why does it not use 20,000 at 256kB?

[ssiloti]

As arvid points out here, what really matters for performance is not the number of pieces in the torrent, or even the piece size itself, but the time it takes to download a piece. Thus there is a diminishing return on reducing piece size past a certain point. This is one reason why the default does not reduce piece size further for small torrents.

The other rational behind the current default is that what constitutes a reasonable torrent file size scales with the size of the content. A 400 KB torrent file for 40 GB of content is a drop in the bucket. A 400 KB torrent for 40 MB of content would add about 1% to the total download while providing little benefit.

[sserdda-liamE]

As arvid points out here, what really matters for performance is not the number of pieces in the torrent, or even the piece size itself, but the time it takes to download a piece. Thus there is a diminishing return on reducing piece size past a certain point. This is one reason why the default does not reduce piece size further for small torrents.

It's all proportional, though. A 16GB torrent with 16MB pieces will take 16 times longer to download each piece, 16 times longer to upload each piece, 16 times longer to download the whole thing, and 16 times longer to seed to 1.0 than a 1GB torrent at 1MB piece size. It's the exact same thing except for the scale.
So the only thing that reducing the relative piece size does is stop disk I/O from scaling as well.

The other rational behind the current default is that what constitutes a reasonable torrent file size scales with the size of the content. A 400 KB torrent file for 40 GB of content is a drop in the bucket. A 400 KB torrent for 40 MB of content would add about 1% to the total download while providing little benefit.

It would only make sense to do that if the performance loss from dropping to those sub-1000 piece torrents was less than 1%. So... it doesn't sound like this tracker's limits are very arbitrary if the result is a consumption of 1/10th the resources and maintaining of disk scaling while incurring less than a 1% loss.

[arvidn]

does this look reasonable? https://github.com/arvidn/libtorrent/pull/3753

[arvidn]

It's all proportional, though. A 16GB torrent with 16MB pieces will take 16 times longer to download each piece, 16 times longer to upload each piece, 16 times longer to download the whole thing, and 16 times longer to seed to 1.0 than a 1GB torrent at 1MB piece size. It's the exact same thing except for the scale.
So the only thing that reducing the relative piece size does is stop disk I/O from scaling as well.

No, this is not correct. Sure, it is proportional, but you seem to suggest it's linearly proportional, which it isn't. At small piece sizes other kinds of overhead may become dominant, such as certain book keeping, sending HAVE messages, updating the piece picker etc.

But your assumption that the download time is 16 times longer for a 16 GiB torrent vs. a 1 GiB torrent (regardless of piece size) is clearly not correct. You're ignoring piece propagation delays, as you seem to have done throughout this thread. Is there a good reason to assume that it's negligible?

I would imagine this to be quite easy to show in a simple simulation, but it's also fairly intuitive.

[Seeker2]

Piece propagation issues when streaming: https://github.com/qbittorrent/qBittorrent/issues/182#issuecomment-312121825
Corrections and clarifications in follow-up post by arvidn!

[arvidn]

I'm renaming this ticket to indicate what I think should be done in libtorrent, and I'm assigning it to the 1.2.2 milestone. I hope nobody is offended.

[arvidn]

The piece picker should increase affinity for larger ranges of bytes (say 4 MiB), regardless of piece size. Currently there's an affinity to finish a whole piece, via the "prioritize partials" feature.

Since piece sizes may be very small, in order to improve disk throughput (by allowing larger writes), this affinity should apply to groups of pieces of, say, 4MiB. Ideally these groups would also be aligned to 4MiB boundaries.

4MiB is a good number as it's the size of huge pages on linux (although I don't believe linux supports memory mapped I/O via huge pages yet). It's also the default extent size in ext4, meaning that writing full 4MiB blocks results in significantly less seeking and filling in holes in files later.

It's possible this change end up being too large for a 1.2.x point release, but some part of it might still make it in there.

[arvidn]

Here's an attempt at addressing this. Anyone care to give it a spin? https://github.com/arvidn/libtorrent/pull/3913

[arvidn]

this has landed in RC_1_2 and will be included in libtorrent-1.2.2