qBittorrent 🚀 - Sparse files/preallocation suggestions

@verdy-p since you are well versed in this area, maybe you could create some pull request(s)?

xavier2k6 on 9 Jun 2020

Waht is complicate to add the two IOCTL calls after opening the file, but before making any write to it?
The first IOCTL is to allow the file to become sparse, the second is to perform the preallocation (which does not write to the preallocated area on NTFS, or even on FAT/FAT32 volumes if they are mounted with a "support sparse" option for its filesystem cache; if you've not called the IOCTL to allow the file to be sparse, the preallocation IOCTL will still write zeroes, but much faster and more efficiently than what a cliant program would do because it will be made by an internal thread of the core OS or of the filesystem driver, so applications don't even have to allocate memory buffers; the OS itself will use fast IOCTL to perform the writes of zeroes, if needed, without even allocating and passing any buffer; at the lowest level of drivers, there are even instructions that allow initializing a range of sectors with zeroes without passing any memory buffer, and this is used for example when formatting new volumes).
The same also applies to Linux that also has similar IOCTLs on open file handles. Many filesystem drivers are also smart enough to include the support of sparse files via their own filesystem cache, so that many read/writes of sectors full of zeroes are avoided. And this is used by common applications such as database management systems (SQL/SparQL/Hashstores/hierarchic...); any OS has to manage tons of temporary files that frequently benefit from sparse support at filesystem level (this is just a work to be done by the service that will mount the filesystem and then share it to user applications).
This significantly speeds up the OS in many places, and also helps saving storage space and preserving the lifecycle of SSD/flash storage by avoiding many unnecessary writes!

So just use the IOCTL: if they are not supported, it does not matter! Continue like what you were doing, and the application will see that the current filesize is still not large enough for the fseek() position and so will start sequential writes zeroes to reach that position, until the application can write the data it expected to store. But if it succeeds (instantly), the application can query the current file position after fseek() and will see that it is the one expected, so there's no delay at all to perform the actual write of data (let the filesystem manage the rest of the uninitialized area).
On Win32 this is fully documented (and supported independantly of the underlying type system used, so the applciation does not even have to check which filesystem is used on the target filesystem, which could even be mounted or redirected into the namespace of another filesystem).

In the previous bug report, I indicated the two IOCTL calls you need for Win32.

For Linux, look at: https://unix.stackexchange.com/questions/366373/what-are-functions-to-manipulate-sparse-files-under-linux

verdy-p on 9 Jun 2020

@verdy-p

The bug#12535 recently closed as "'solved" is definitely NOT solved.

It is solved, because the problem OP was experiencing was due to the insufficient auto disk cache (causing severe performance loss) set in Windows systems with >= 16 GiB RAM bug, which was found and fixed in libtorrent a while ago.

You mentioned another potential problems relating to sparse files and preallocation in the other ticket, but remember: one issue per ticket. So it's a good thing that we're now discussing that separate matter in this separate ticket.

Now let me be honest: you write huge walls of text that take me and everyone else a long time to read, and no doubt a long time and effort for you to write. Why not invest some of that time into producing a verifiable test case + benchmark to illustrate your point (like this user did, for example: https://github.com/qbittorrent/qBittorrent/issues/10999#issuecomment-573456493)? If it shows good results, an accompanying PR implementing the corresponding positive changes would also be nice.

Also, the things you write about preallocation/sparse files probably concern libtorrent more so than qBittorrent. The best person to ask about all of the details of how preallocation/sparse files are handled in libtorrent is @arvidn, should you have any questions after reading the relevant documentation: https://libtorrent.org/manual-ref.html#storage-allocation.

Plus, both in the latest tagged release as well as latest master (and as far back as I can remember) qBittorrent has used the sparse mode by default (which is also the default in libtorrent):

https://github.com/qbittorrent/qBittorrent/blob/2c65b796401f924d6889aadde958de3d2846bb4b/src/base/bittorrent/session.cpp#L2234

FranciscoPombal on 10 Jun 2020

@verdy-p It's difficult to interpret and follow your posts. I would encourage you to try to boil them down and make it clear what the problem is, starting from the high level symptom and then describe in more detail what your investigation has revealed is causing the symptom at inreasingly lower level, followed by a proposed solution, and maybe even an explanation how the proposed change solves the problem.

If you then want to add more back-story or references, do that later in the post. Right now things are so mixed up that it's hard to know what you're proposing as a solution and what you're suggesting is the problem.

We still have the very slow start because of the very long delay to initialize the large file, when downloading large files in random order.

It sounds like your files are not sparse, and hence are allocated and zeroed up-front. Is that a correct understanding?

Then there's a list of your suggestions:

generate "sparse files" if possible (e.g. on NTFS), which is very fast

This is the default in libtorrent. Are you sure you've configured the storage to use sparse files?
Did you look at the libtorrent source code, specifically those IOCTLs to understand why the existing code is insufficient or doesn't work?

you don't even need to know if the filesystem supports it, just try it with an IOCTL on the open file handle.

This doesn't sound like a suggestion, which makes it harder to interpret this bullet list of suggestions.

the assumption that sparse files creates fragmentation on disk is WRONG, if you use it properly: sparse files can be preallocated on disk (the allocated area is not read/written as physically as long as it remains virtually at 0).

This isn't a suggestion either, and it seems it's drifting away from the point (I think) you're making.

Once fully downloaded, the sparse file an be converted back to a non-sparse file (it won't need any reallocation or fragment moves) by an IOCTL. But it is not really needed, you can close the file (the areas reamin allocated even if they are not physically read/written because they contain only zeroes).

I'll interpret this as suggesting doing this. This is already the default in libtorrent. Did you look at the code and understand why it's insufficient or why it doesn't work?
What's the symptom you see that suggests that it doesn't work?

If you convert back the sparse file to a normal file, all remaining areas filled with zeroes will have to be written (but they are already allocated on disk).

Also not a suggestion (and probably shouldn't be part of this bullet list)

If the downbload is complete, it's best to keep the file in sparse state, then ask Windows to deallocate the uninitialized parts that were allocated but are still virtually at zero: this will save storage space, but it is the only case where the file will appear to be fragmented (only at positions of holes in the sparse file), so the file will still be fast to access (you don't really need to coalesce the sparse fragments together; the initially preallocated areas that were freeds can be left to store other files)

This is quite unclear. If the download is complete, there are no holes in the file, by definition, right? Maybe I'm misunderstanding what you mean by "uninitialized parts that were allocated but are still virtually zero". This also seems to be drifting off-toping from the ticket.

Preallocation of sparse files is extremely fast (very few I/O, only a small area in the allocation bitmaps of the filesystem, and a small area of the MFT describing the file record). This is also true even for FAT/FAT32, except that FAT/FAT32 does not keep a registry of the unitialized areas as it does not have sparse files. These unitilized areas can then contain random data (from prior contents of former deleted files or from the last disk cleaning/formatting), but qBitTorrent also knows which part were downloaded or not, so it never needs these areas to be preinitialized to zeroes.

This is also not a suggestion.

Preallocation ensures that there will be enough storage space long before starting to download the source: qBitTorrent would inform immediartely the user early before he attempts to download some large file: as the storage space will be allocated once, no other process or download will compete to use a part of the initaillizy estimated free space.

Also not a suggestion.

The bullets I could interpret as suggestions are already being done by libtorrent. You're sort of implying that it doesn't work, but you haven't really been explicit about what symptoms you see that suggests that it doesn't work.

arvidn on 12 Jun 2020

In the previous bug report, I indicated the two IOCTL calls you need for Win32.

I have no idea which ticket is the "previous" one. On windows, the way you set the sparse flag is via DeviceIoControl() with the FSCTL_SET_SPARSE control code. I assume that's what you're referring to bye "IOCTL". It's a bit confusing though, since it's not actually using the ioctl() call on windows.

For Linux, look at: https://unix.stackexchange.com/questions/366373/what-are-functions-to-manipulate-sparse-files-under-linux

Please explain how this link is relevant. What operations are you suggesting be done that requires querying holes in sparse files?

Still though, after two very long posts, it's still unclear (at least to me) whether this is a feature request of some kind or a bug report against files not having their sparse flag set. You've mentioned a lot of other things, but I get the impression sparse file flag is at the core of the ticket.

arvidn on 12 Jun 2020

@arvidn -> https://github.com/qbittorrent/qBittorrent/issues/12535#issuecomment-620405492

xavier2k6 on 12 Jun 2020

thanks. @verdy-p I believe libtorrent implements all the suggestions you bring up in that comment. If you have any reason to believe any of that is done incorrectly or doesn't work, please share that reason!

arvidn on 12 Jun 2020

When a download is "complete" it may have downloaded areas of a file that
are full of zeroes: if the file is sparse, these were actually NOT written
to disk, the area is still marked as sparse (the filesystem does not bother
writing to disk these areas). So when you keep the file as is, its
preallocated areas are kept, but you can use an IOCTL to free the
preallocated area, this won't change the logical state of the file as the
areas are still logically read as zeroes without performing the actual I/O,
just from the sparse allocation map of the file. This means that the
preallocated area that are mapped to zeroes can be safely freed.

Several IOCTL do that:

one for declaring the file to become sparse: any write of zeroes will be
discarded replaced by an entry in the sparse map. This is a requirement
before the 3 other IOCTL have an effect. This does not change the free
space on disk, or the logical file size. It is instantaneous (note that
existing areas already filled with zeroes are kept as is, the filesystem
will not read the file to detect these areas and convert them to sparse
areas. After this call, any writes of zeroes in the file will convert these
areas to sparse (but the allocation is still kept).
one for preallocating the file size (this is instantaneous: all extended
filesize will be added to the sparse map only. This does not change the
free space on disk, only the logical filesize. It only writes in the sparse
map of the file. It is instantaneous.
one for freeing the allocated areas that are still parse (it does not
matter if the application wrote zeroes in them). This is instantaneous and
like the first IOCTL, no attempt is made to read the file.
the last one if for converting a file back to non-sparse: this is were
the sparse areas will be physically allocated on disk if they were still
not, and will be filled with zeroes, to create a contiguous file: after
this calls all writes of zeroes will be physically written.

In summary a file fragment in a file can be in 3 state (actually two):

unallocated and after the filesize: no entry in the sparse map, there's
no logical data in these areas. Reading from it, or seeking to these
positions will return an error (read past end of file)
allocated (before the filesize) but logically 0: the physical data on
disk is kept in random, but the fragment is marked in the sparse file and
will logically read as zeroes.
allocated and with actual data; the physical data is read from disk.
Wring is performed as well but writing zeroes will mark the area into the
sparse map (leaving the allocated space with its past content, but hidden)

For filesystems that don't have native support of sparse file, the OS will
add a 4th state, for areas that are allocated but whose data is still in
random state: this state is transient and used by the OS internally inside
a worker thread to initilize these areas as 0: the sparse map remains
virtually in memory (in the filesystem cache) but temporarily (the OS or
filesystem driver could keep a registry stored somewhere about the state of
its space intializing worker thread, so it can survive the OS
shutdown/restart, but this requires the filesystem to be mounted with the
same option for emulating the sparse support (this is that windows does for
FAT32 volumes but I don't know if it resists the OS reboot or wher it
kjeeps the registry, possibly only in side the Windows registry for one of
its services implementing volume manager).

For now in qBitTorrent, it is visible that you can download very fast about
128MB randomly and write it randomly: but then it stalls because the write
queue is full, waiting for Gigabytes of zeroes to be written on disk before
the pending 128MB in the write queue is finalized. This is a problem when
your internet speed is faster than about 40-50% of the disk write speed. If
your disk can only write 30 MebiByte/s (about 300 MegaBits/s) your initial
download speed is limited to about 170 Megabit/s (even if the internet
access allows for 1Gigabit/s): you instantly experiment the long delays
(waiting for Gigabytes of zeroes to be written) if you've started to
download the file in random order (with soms fragments to be written
several Gigabytes aways from the start of file. You'll get then the maximum
download speed (300 MegaBit/s only when the areas on disk have been
initialized to zeroes).

And so with the current version you can see that qB download about 128
megabytes (in random order) very fast in a fraction of second (in fact a
bit less, around 127MB, because the write queue is partilly used by the OS
for its intialization of zeroes where it uses an internal buffering for I/O
requests writing 1MB segments of zeroes), then stalls for long minutes
until the whole file is "cleared" with zeroes. This is this step which is
very lengthy and causes the heavy workload on the target disk.

Le ven. 12 juin 2020 à 01:55, Arvid Norberg notifications@github.com a
écrit :

@verdy-p https://github.com/verdy-p It's difficult to interpret and
follow your posts. I would encourage you to try to boil them down and make
it clear what the problem is, starting from the high level symptom and
then describe in more detail what your investigation has revealed is
causing the symptom at inreasingly lower level, followed by a proposed
solution, and maybe even an explanation how the proposed change solves the
problem.

If you then want to add more back-story or references, do that later in
the post. Right now things are so mixed up that it's hard to know what
you're proposing as a solution and what you're suggesting is the problem.

We still have the very slow start because of the very long delay to
initialize the large file, when downloading large files in random order.

It sounds like your files are not sparse, and hence are allocated and
zeroed up-front. Is that a correct understanding?

Then there's a list of your suggestions:

generate "sparse files" if possible (e.g. on NTFS), which is very
fast

This is the default in libtorrent. Are you sure you've configured the
storage to use sparse files?
Did you look at the libtorrent source code, specifically those IOCTLs to
understand why the existing code is insufficient or doesn't work?

you don't even need to know if the filesystem supports it, just try
it with an IOCTL on the open file handle.

This doesn't sound like a suggestion, which makes it harder to interpret
this bullet list of suggestions.

the assumption that sparse files creates fragmentation on disk is
WRONG, if you use it properly: sparse files can be preallocated on disk
(the allocated area is not read/written as physically as long as it remains
virtually at 0).

This isn't a suggestion either, and it seems it's drifting away from the
point (I think) you're making.

Once fully downloaded, the sparse file an be converted back to a
non-sparse file (it won't need any reallocation or fragment moves) by an
IOCTL. But it is not really needed, you can close the file (the areas
reamin allocated even if they are not physically read/written because they
contain only zeroes).

I'll interpret this as suggesting doing this. This is already the default
in libtorrent. Did you look at the code and understand why it's
insufficient or why it doesn't work?
What's the symptom you see that suggests that it doesn't work?

If you convert back the sparse file to a normal file, all remaining
areas filled with zeroes will have to be written (but they are already
allocated on disk).

Also not a suggestion (and probably shouldn't be part of this bullet list)

If the downbload is complete, it's best to keep the file in sparse
state, then ask Windows to deallocate the uninitialized parts that were
allocated but are still virtually at zero: this will save storage space,
but it is the only case where the file will appear to be fragmented (only
at positions of holes in the sparse file), so the file will still be fast
to access (you don't really need to coalesce the sparse fragments together;
the initially preallocated areas that were freeds can be left to store
other files)

This is quite unclear. If the download is complete, there are no holes in
the file, by definition, right? Maybe I'm misunderstanding what you mean by
"uninitialized parts that were allocated but are still virtually zero".
This also seems to be drifting off-toping from the ticket.

Preallocation of sparse files is extremely fast (very few I/O, only
a small area in the allocation bitmaps of the filesystem, and a small area
of the MFT describing the file record). This is also true even for
FAT/FAT32, except that FAT/FAT32 does not keep a registry of the
unitialized areas as it does not have sparse files. These unitilized areas
can then contain random data (from prior contents of former deleted files
or from the last disk cleaning/formatting), but qBitTorrent also knows
which part were downloaded or not, so it never needs these areas to be
preinitialized to zeroes.

This is also not a suggestion.

Preallocation ensures that there will be enough storage space long
before starting to download the source: qBitTorrent would inform
immediartely the user early before he attempts to download some large file:
as the storage space will be allocated once, no other process or download
will compete to use a part of the initaillizy estimated free space.

Also not a suggestion.

The bullets I could interpret as suggestions are already being done by
libtorrent. You're sort of implying that it doesn't work, but you haven't
really been explicit about what symptoms you see that suggests that it
doesn't work.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/qbittorrent/qBittorrent/issues/12991#issuecomment-642987405,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKSUG3MNQUQRQ7U33NT7YTRWFVIRANCNFSM4NZXWENA
.

verdy-p on 12 Jun 2020

When a download is "complete" it may have downloaded areas of a file that
are full of zeroes:

That really seems like a fringe use case. Do you have any example of where this would be common?

arvidn on 12 Jun 2020

It sets an internal field of the torrent session object (this is just a
wrapper):

 p.storage_mode = isPreallocationEnabled() ? lt::storage_mode_allocate

: lt::storage_mode_sparse;

But this is not used at all by qBitTottent that ignores this setting in the
wrapper object.

There's still not IOCTL performed when the file is open(), seek() and then
written, because this is not part of the generic class and must be ported
for each OS.

And I don't see any code using it in the Win32 port (it' not to libtorrent,
which is portable, to manage this adaptation as it does not perform the
file I/O itself, it just handles the protocol and the internal map of
fragments to download or to hash; it also does not perform itself the
download/upload)
You need a file I/O adapter inside qBittorrent itself (what you adapted to
Win32 was just the "filelock" primitive, the local enumeration of directory
contents, a "file system watcher" to detect new or modified files).
Libtorrent is light and portable (mostly independant of the OS).

Le mer. 10 juin 2020 à 19:42, Francisco Pombal notifications@github.com a
écrit :

@verdy-p https://github.com/verdy-p

The bug#12535 recently closed as "'solved" is definitely NOT solved.

It is solved, because the problem OP was experiencing was due to the insufficient
auto disk cache (causing severe performance loss) set in Windows systems
with >= 16 GiB RAM bug, which was found and fixed in libtorrent a while
ago.

You mentioned another potential problems relating to sparse files and
preallocation in the other ticket, but remember: one issue per ticket. So
it's a good thing that we're now discussing that separate matter in this
separate ticket.

Now let me be honest: you write huge walls of text that take me and
everyone else a long time to read, and no doubt a long time and effort for
you to write. Why not invest some of that time into producing a verifiable
test case + benchmark to illustrate your point (like this user did, for
example: #10999 (comment)
https://github.com/qbittorrent/qBittorrent/issues/10999#issuecomment-573456493)?
If it shows good results, an accompanying PR implementing the corresponding
positive changes would also be nice.

Also, the things you write about preallocation/sparse files probably
concern libtorrent more so than libtorrent. The best person to ask about
all of the details of how preallocation/sparse files are handled in
libtorrent is @arvidn https://github.com/arvidn, should you have any
questions after reading the relevant documentation:
https://libtorrent.org/manual-ref.html#storage-allocation.

Plus, both in the latest tagged release as well as latest master (and as
far back as I can remember) qBittorrent has used the sparse mode by default
(which is also the default in libtorrent):

https://github.com/qbittorrent/qBittorrent/blob/2c65b796401f924d6889aadde958de3d2846bb4b/src/base/bittorrent/session.cpp#L2234

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/qbittorrent/qBittorrent/issues/12991#issuecomment-642159020,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKSUG6OK7FBZRJK62LGIGDRV7AXVANCNFSM4NZXWENA
.

verdy-p on 12 Jun 2020

It sets an internal field of the torrent session object (this is just a
wrapper):
 p.storage_mode = isPreallocationEnabled() ? lt::storage_mode_allocate
: lt::storage_mode_sparse;

But this is not used at all by qBitTottent that ignores this setting in the
wrapper object.

I'm not sure what that was in response to. If p is an lt::add_torrent_params object, I would expected qBittorrent not to "use" that field (i.e. read it). It's used to communicate with libtorrent.

And I don't see any code using it in the Win32 port

Right here, you can use github's search function:

https://github.com/arvidn/libtorrent/search?q=DeviceIoControl&unscoped_q=DeviceIoControl

(it' not to libtorrent,
which is portable, to manage this adaptation as it does not perform the
file I/O itself, it just handles the protocol and the internal map of
fragments to download or to hash; it also does not perform itself the
download/upload)
You need a file I/O adapter inside qBittorrent itself (what you adapted to
Win32 was just the "filelock" primitive, the local enumeration of directory
contents, a "file system watcher" to detect new or modified files).
Libtorrent is light and portable (mostly independant of the OS).

Sorry, I have no idea what those paragraphs are about.

arvidn on 12 Jun 2020

"You have no idea"... This thread opens directly with a reference to bug#12535...
(yes it is "DeviceIoControl" in Win32, this is obvious and explicited in the last bug, and I also explicitly saifd that the name of this API call depends on the OS!)

As well, I can see that your search suggestion reveals that the IOCTL is still not used. There's still no correlation at all between "file" object open mode and the "storage_mode' field set by the torrent session object. The link between the two is still not done.

verdy-p on 12 Jun 2020

@verdy-p

It sets an internal field of the torrent session object (this is just a
wrapper):

p.storage_mode = isPreallocationEnabled() ? lt::storage_mode_allocate
: lt::storage_mode_sparse;

If you look at https://github.com/qbittorrent/qBittorrent/blob/release-4.2.5/src/base/bittorrent/session.cpp, every p is an lt::add_torrent_params object (see the docs for it here: https://libtorrent.org/reference-Core.html#add_torrent_params). So, this code that you mention works as expected to tell libtorrent which storage mode to use. Besides, even if this did not actually do anything, like you say, the storage mode would still be set to sparse, because that is the default in libtorrent.

I think you may be misunderstanding what the "preallocation" settings means/does in qBittorrent. If it is active, then it is _supposed to_ write all zeros to disk, as much as the size of the files in question. That's it. No sparse files or anything like that. It is expected that this can significantly delay the start of big torrents. On the other hand, if it is inactive, then it uses the sparse allocation, and the start delay should be unnoticeable (assuming of course your storage is configured to use, and supports, sparse files). Whatever the case, it leverages libtorrent's capabilities for this. There is no code in qBittorrent that directly handles preallocation/sparse files/etc, other than just telling libtorrent to use one or the other allocation modes (which is 1 line of code).

Feel free to point out any bug in libtorrent's implementation of the feature if you find any. Alternatively, if it is possible that in session.cpp there is any code path that does not set the preallocation mode correctly, point us directly to it as well.

FranciscoPombal on 12 Jun 2020

@verdy-p you keep mentioning https://github.com/qbittorrent/qBittorrent/issues/12535, but I already told you that the cause of the report was due to another bug which has already been fixed. Otherwise, how would you explain that the OP said the problem got fixed simply by upgrading to 4.2.4 (which included the fix to the cache bug), with no other changes?

FranciscoPombal on 12 Jun 2020

"what the preallocation settings means/does in qBittorrent. If it is active, then it is supposed to write all zeros to disk"

There's no such assumption to do and in fact it's the worst thing it can do. Using sparse files to preallocate the area (without writing any zeroes) is what must be done, and it is exactly the puprose of this bug here.

Writing zeroes on very large files is extremely bad, notably when downloading from torrents in random order. It makes no sense at all to write these zeroes that will be overwritten anyway when the download completes, so it is just a massive waste of I/O.

Allocating space for a file does NOT mean that the space MUST be filled with zeroes: the space can be allocated FAST and still be marked as an unwritten area: it is a reserve and any attempt ot read from it will return a buffer filled with zeroes in memory while the associated allocated area is left for being used alter when there' will be non-zero contents to write there.

And this bug is still not solved. Really present in version 4.2.5, easy to observe: just download any large torrent (several gigabytes) from a Gigabit internet access and try storing it to harddisk (whose write speed is slower by several factors (writing to hard disks are now about 15% of the internet download speed). Today, the Internet is faster than local storage!

And this is very easy to reproduce: jsut try downloading to a device with a slower writing speed (e.g. download to a slow USB key or an external SATA drive). You'll immediately see the qB download very fast 127Megabytes, then immediately stalls for several minutes (the time to write gigabytes of zeroes).

verdy-p on 12 Jun 2020

As well, I can see that your search suggestion reveals that the IOCTL is still not used. There's still no correlation at all between "file" object open mode and the "storage_mode' field set by the torrent session object. The link between the two is still not done.

What makes you say that? Can you support that claim?

arvidn on 12 Jun 2020

There's no such assumption to do and in fact it's the worst thing it can do. Using sparse files to preallocate the area (without writing any zeroes) is what must be done, and it is exactly the puprose of this bug here.

I thought this ticket was about you suggesting using sparse files (which is already the case, but you seem to suggest that there's some bug that causes it not to work, but you don't seem to want to say that outright, so I still don't know).

But that sentence there suggests that this ticket is about when enabling full allocation mode, you also say files should be sparse. If you want sparse files, can't you just leave the allocation mode setting at the default?

I'm still confused.

arvidn on 12 Jun 2020

Even for the "full allocation" mode the file should be sparse as possible (fully allocated instantly, but still not written with dummy zeroes: the OS will allocate the space, will ensure that this area is reserved and won't be used for other files; it is mapped to the file, but not read/written physically, that space is marked in the sparse-map as being allocated but not initialized).

As I said there are three kinds of "holes" in sparse files:

(1) holes that are unallocated on disk (will still be read virtually as zeroes) - they cause a file to be fragmented and store to become insufficient later (very bad for torrents! don't ever use them for any download in random order!)
(2) holes that are preallocated on disk but not initialized (will be read virtually as zeroes) - they are used to limit the fragmentation and ensure the storage will be fully available (BEST choice for torrrents)
(3) holes that are allocated on disk and initialized with zeroes (still no I/O is needed to read them as zeroes) - This is a "housekeeping state" that can be used only for the conversion between sparse and non-sparse storage modes: it allows unallocating them fast if keeping the file in sparse mode (reducing the storage needed by suppressing these areas and creating holes of the 1st kind above), or allows converting the file to plain storage without writing the same zeroes again on the same area).
(There's a fourth state for normal, non-zero data)

Mode (2) is definitly the solution and can be applyed in all cases by default. It allows faster starts of downlaods, uses much less I/O, won't cause their system to become instantly very unresponsive for long minutes after they try downloading a very large file: even if you have a fast storage, it takes considerable time to complete the write of gigabytes of zeroes (consider the time required for example to format a large volume or RAID array: this is the same order of time, even with the fastest disks and even if you don"t have a very fast internet access).

It NEVER makes sense to massively write all these zeroes, that are completely useless (and will need to be rewritten later for the actual data once it is effectively downloaded). We just need to preallocate the space (and with sparse files this can be done instantly with a single IOCTL API call).

Mode (1) would be used only if users want fragmented files (I doubt they will want that) and partial downloads being interrrupted in the middle because of insufficient space on the target volume (once again, users will never want that!)

ALL Torrents shouls be usable and downloadable in random order independantly of their size: there are aplpications that will want torrents for files up to 1 terabyte or more (sharing a full fiesystem dump, or a large database, or a large volume ISO file for applications, or for backup/mirroring purpose)

verdy-p on 12 Jun 2020

@verdy-p

Sounds like what you want for your use case is to just leave that setting off, so as to use the sparse allocation. In qBittorrent preallocation is off by default, so to me it just looks like you are creating your own problems by turning it on, even though it is not what you want. From what I have seen, qBittorrent users that turn this on usually do so for the following reason (listed in the libtorrent documentation):

No risk of a download failing because of a full disk during download, once all files have been created.

For those users, the initial delay in download start and extra I/O is a worthwhile trade-off for this guarantee. But that is _their_ use case. Yours is different, apparently.

However, if you are saying this:

But that sentence there suggests that this ticket is about when enabling full allocation mode, you also say files should be sparse.

That's another discussion entirely, that concerns libtorrent, not qBittorrent. To be honest I did not know it was possible to have sparse files that also reserve logical space. I read up on it a little bit - are you talking about fallocate (on Linux), for example?

FranciscoPombal on 12 Jun 2020

Preallocating space without writing it exists in BOTH Windows and Linux (probably as well in Android, MacOS, iOS, FreeBSD, but I did not check), and it exists since long as well in various Unix'es (I was present several decennials agos, notably for installing RDBMS engines and creating databases, managing their storage space...). It has been used since long by "logical volume managers" (to create virtual filesystems stored in a file of another underlying filesystem).

Windows and Linux even supports sparse files on filesystems that don't have native support for it (it is a volume mounting option: the sparse files registry will be kept by creating a special file on the volume, and that file will be managed by the filesystem's own cache). It works for example with FAT filesystems (that has no native per-file sparse support, but can use fast preallocation of a few sectors in the FAT map without writing anywhere else on the volume).

On NTFS, per-file sparse support is builtin (there's a sparse file-mapping in the MFT records or one of its "extended attributes" if the 1KB MFT record for 1 file is not large enough to hold this map: NTFS supports it esily because all files are a collection of typed-"streams", small strems can be packed together in the 1KB MFT record if they are "resident", or in the extended records allocated elsewhere: the MFT record has a resident stream for the common file names, or basic security and file types, or for the main allocation map and the "sparse" stream is located there). This is a feature that initially came from Digital's "VMS" (and IBM's "MVS" where they first appeared on mainframes), it was then ported to OS/2 and NTFS. It was added since long in Apple's modern filesystems as well. It was integrated as well on IBM AiX, HP-UX, SunOS and Solaris... (And there's an extension to support it as well on NFS). All OSes that support "logical volume managers" have it (notably all OSes supporting virtual filesystems for hypervisors, or that support common RDBMS systems like Oracle, Sybase, SQLServer, Informix, and so on as they can implement their own space allocators and efficiently manage their storage space)! On NTFS, the "MFT" is itself a sparse file (only the first few records are at a static position and one of the first 1KB entries is for describing the file containind the allocation maps and their own streams for additional attributes, including a filename "$MFT", basic attributes like the MFT logical size, and security attributes), as opposed to FAT where the allocation map is a static area allocated at start of the volume and not allocating itself.

So definitely you assume that sparse files only supported type (1) holes (see my previous description). You're wrong: type (2) also exist since long and are much better!

But in qBittorrent, if we currently set the storage mode to "partial file" only, nothing is preallocated (you use only type (1) and this causes fragmentation on disk and failure in the middle of a large download due to lack of space to finish the download). This is a very bad behavior and this should be off by default (as it only causes problems). All files should be preallocated by default (without necessarily initializing them with zeroes, and it is possible almost always, the filesystem will take care of initializing this space only if it cannot avoid it due to lack of support on the mounted volume!).

verdy-p on 12 Jun 2020

@verdy-p I see, thanks for the explanation. @arvidn can't libtorrent use this variety of sparse files? On Windows, if I understood this correctly, looks like the only remaining thing is to call FSCTL_SET_ZERO_DATA (https://docs.microsoft.com/en-us/windows/win32/api/winioctl/ni-winioctl-fsctl_set_zero_data) after having created/set the file as sparse (which libtorrent already does: https://github.com/arvidn/libtorrent/blob/ce0a85f783bec37484776a37fe3662279091ecc5/src/file.cpp#L560)

FranciscoPombal on 12 Jun 2020

@verdy-p Can you please confirm that everything that you have written so far in this ticket (and I guess in the previous ticket) boils down to:

libtorrent, instead of calling SetFileValidData() to allocate files, should call DeviceIoControl() with FSCTL_SET_ZERO_DATA, to fill the file with virtual zeroes.

Followed by some rationale. If that's the case, I would really encourage you to work on improving the precision in your communication. Whatever your message is, it's really diluted and drowned in noise.

Now, assuming that is what you are suggesting, I would be really curious to know why that would be better. The documentation for FSCTL_SET_ZERO_DATA IOCTL specifically says (in the second sentence no less):

If the file is sparse or compressed, the NTFS file system may deallocate disk space in the file.

Which is definitely not what you would want to happen if you set storage mode to "allocate". But, perhaps I'm missing some subtle detail.

Furthermore, if it would really be more efficient to write virtual zeroes, than to truncate a file to some size where it appears to be full of zeroes; why wouldn't the OS implement the latter in terms of the former?

arvidn on 12 Jun 2020

this can be common in some very large (uncompressed) bitmap data, or in
databases and ISO images of a full filesystem.
files that have large chunks of zeroes are not exceptional, in fact they
are quite common and using sparse files to store them is very efficient.
(I wonder if the network torrent protocol itself has a way to save network
bandwidth for not transmitting most zeroes, and "compress" such ranges,
without depending on compression on the transport layer.)

Le ven. 12 juin 2020 à 13:35, Arvid Norberg notifications@github.com a
écrit :

When a download is "complete" it may have downloaded areas of a file that
are full of zeroes:

That really seems like a fringe use case. Do you have any example of where
this would be common?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/qbittorrent/qBittorrent/issues/12991#issuecomment-643224212,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKSUG6TALETAIPABZ324ULRWIHJ5ANCNFSM4NZXWENA
.

verdy-p on 13 Jun 2020

Ths OS does not use sparse files by default, for only compatibility reasons
with some programs that expect a precise layout on disk (but this can be
hidden to them by using a mounting option for the volume).
Declaring a file to allow sparse storage does not mean it will become
sparse immediately: you have to instruct the OS about areas that it does
not need to write but just has to allocate.

Also When I wrote: "If the file is sparse or compressed, the NTFS file
system may deallocate disk space in the file."
I was speaking about the holes of type (3) that are created when you write
zeroes or allocated areas that were never written with other data. The
unallocation will occur with another IOCTL that will remove these allocated
areas.

This is never occuring automatically, but can be performed to remove these
remaining areas (eg. when the download is complete: you may have downloaded
ranges of zeroes and attempted to write them, but the OS discarded these
writes, keeping the area allocated but not even written; using the IOCTL
can discard these ranges from the allocation; the file will remain sparse
but this time the holes will no longer be allocated and will be usable for
other files: this "apparently" fragment your file but only where it has
large holes, this is still more efficient for reading these files because
these areas "reside" in memory, not on disk). The OS may eventually partly
defragment the successive data segments between holes, to keep them grouped
at least within some multiple of clusters (but as far as I know, NTFS does
not do that, only some defragmenters will collect and pack the sparse data
segments together; but some virtual disk drivers are doing that, notably
the virtual volume manager for Hyper-V, so that it can repack a large
virtual volume into a smaller file containing only non-zero sectors).

Le sam. 13 juin 2020 à 00:00, Arvid Norberg notifications@github.com a
écrit :

@verdy-p https://github.com/verdy-p Can you please confirm that
everything that you have written so far in this ticket (and I guess in the
previous ticket) boils down to:

libtorrent should, instead of calling SetFileValidData() to allocate
files, should call DeviceIoControl() with FSCTL_SET_ZERO_DATA IOCTL, to
fill the file with virtual zeroes.

Followed by some rationale. If that's the case, I would really encourage
you to work on improving the precision in your communication. Whatever your
message is, it's really diluted and drowned in noise.

Now, assuming that is what you are suggesting, I would be really curious
to know why that would be better. The documentation for FSCTL_SET_ZERO_DATA
IOCTL specifically says (in the second sentence no less):

If the file is sparse or compressed, the NTFS file system may deallocate
disk space in the file.

Which is definitely not what you would want to happen if you set
storage mode to "allocate". But, perhaps I'm missing some subtle detail.

Furthermore, if it would really be more efficient to write virtual zeroes,
than to truncate a file to some size where it appears to be full of zeroes;
why wouldn't the OS implement the latter in terms of the former?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/qbittorrent/qBittorrent/issues/12991#issuecomment-643499761,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKSUGY7NXQJOTWYUAOJ4XDRWKQOXANCNFSM4NZXWENA
.

verdy-p on 13 Jun 2020

@verdy-p can you please confirm that my description of what you are trying to say is correct?

arvidn on 13 Jun 2020

I can jsut confirm the initial request (which was clear from the start of this bug and of the initial bug that was closed):

first declare the open file to the OS so that it will support sparse storage with a single fast IOCTL call
preallocate the storage space (this won't write tons of zero) for the whole file size with a single fast IOCTL call
continue normally : you can write areas in random order until the download completes using normal file.write() calls (including if these are buffers full of zeroes: the OS will discard these writes automatically and will return immediately, making that range part of the hole)
when the downlaod is complete, you may either:
- remove the sparse file support from it with an extra IOCTL (this may be slow as the OS will write zeroes in the these areas which were already allocated but never written, but typical files will not have many such holes, so low volumes of writes will occur; you may then close the file; the OS will fill the areas in the background, and will remove the sparse map when it finishes, but the file is still immediately usable "as is" even if the sparse areas are still not fully filled with zeroes and the sparse support and the sparse map is still not fully removed; let the filesystem do that for you).
- or keep the file sparse but instruct the OS to deallocate the areas still reserved for holes (this will give back these areas usable for storing other files); this will be very fast as well (only updating the allocation map of the filesystem; eventually the OS will manage to group some used areas together, but it generally won't do that: the only interest is to defragment the free space usable for other files); a single IOCTL can do that.
  
  The OS (or the filesystem driver with its internal cache) is smart enough to choose the fastest option that minimize the necessary volume of I/O and can better schedule the operations than a single application running in user mode.

verdy-p on 13 Jun 2020

I can jsut confirm the initial request (which was clear from the start of this bug and of the initial bug that was closed):

I don't think it's clear at all. You have a long description of operations on a file. So what? what's the context? which file should have this treatment? under what conditions/settings? is this a feature request? is it a bug report? why should files be treated this way?

Everything you write (at a quick glance) is already done in libtorrent. why are you writing this?

arvidn on 13 Jun 2020

Because All demonstrates EASILY and REPRODUCTIBLY that downloads of big
files start by downloading very fast (a few seconds) 127 megabytes, then
the torrent will stall, stop downloading anything for long minutes while
the disk is 10% busy for writing arbitrarily large amount of zeroes
(gigabytes or more): as long as ALL these zertoes are not fully written,
nothing is downloaded (and the first 127 megabytes downloaded from random
position are still not written on disk, they are pending in memory waiting
for the end of writes of all these zeroes).
And this happens ALWAYS when downloading in random order from
arbitrary positions of the large file.

The only solution for now is to download sequentially (or not from any
position that is too far away (if you don't download any fragment which is
located more than about 64 megabytes from the start, you set a higher limit
on the amount of zeroes to write. writing 128 megabytes of zeroes is not
very long, so you won't stall for long. But it's then impossible to "swarm"
file by a torrent across a set of hosts on the network, if that file is not
first splitted in smaller chunks that will be distributed in separate
torrents.

This means that for now qBittorent is almost unusable for large files (all
files larger than 128 megabytes) and very slow start for all files (it
makes excessive writes of zeroes, always unneeded because these zeroes will
be rewritten a second time with the actual data). This is not a defect of
the torrent protocol, but of the implementation for writing to local
storage.

qBittorrent is then unsuitable as a distribution method for large files
like ISOs (DVD images), or VM snapshots, or large database dumps, or host
backup/replication.

Le sam. 13 juin 2020 à 02:13, Arvid Norberg notifications@github.com a
écrit :

I don't think it's clear at all. You have a long description of operations
on a file. So what? what's the context? which file should have this
treatment? under what conditions/settings? is this a feature request? is it
a bug report? why should files be treated this way?

Everything you write (at a quick glance) is already done in libtorrent.

Visibly NO, it is NOT already in libtorrent (at least not the version integrated in qBittorrent v 4.2.5)

You apparently can't see the problem because your tests are apparently limited to:

downloading only small files (less than 128 megabytes each)
from a slow internet access (100 megabit/s or lower, not a gigabit fiber internet or better)
to a fast SSD storage (which is much faster for writes than your internet download speed but that has limited storage capacity, so that you would not use them for downloading/sharing very large files: if you want to download share large files or large collections of files, you would use a local RAID array, or LAN storage such as NFS or CIFS mounted volumes: their write speed is generally around 30-50 megabytes/s at best, but frequently around 10 megabyte/s, much slower than the internet download)

verdy-p on 13 Jun 2020

Because All demonstrates EASILY and REPRODUCTIBLY that downloads of big
files start by downloading very fast (a few seconds) 127 megabytes, then
the torrent will stall, stop downloading anything for long minutes while
the disk is 10% busy for writing arbitrarily large amount of zeroes
(gigabytes or more): as long as ALL these zertoes are not fully written,
nothing is downloaded (and the first 127 megabytes downloaded from random
position are still not written on disk, they are pending in memory waiting
for the end of writes of all these zeroes).
And this happens ALWAYS when downloading in random order from
arbitrary positions of the large file.

I have never seen this happen, but that doesn't really matter. It sounds like it's reproducible on your computer. You seem to have an idea of what the problem is. Could you gather some information of what's actually happening, say with process monitor?

qBittorrent is then unsuitable as a distribution method for large files
like ISOs (DVD images), or VM snapshots, or large database dumps, or host
backup/replication.

I really think that you should file a separate ticket about this. If I understand you, this is about an optimization where zeroes are potentially not transferred and not written to disk (but "virtually" written to disk). It's not obvious that having a special case for this would carry its own weight, but that discussion belongs in a separate ticket as far as I can tell.

Everything you write (at a quick glance) is already done in libtorrent.

Visibly NO, it is NOT already in libtorrent (at least not the version integrated in qBittorrent v 4.2.5)

Please share how you established this. How is it visible? For example:

did you step through the program in a debugger and saw that the calls in question were not made or with some invalid argument?
did you run ProcessMonitor or strace to see that the calls were not made or made with some invalid arguments?
did you inspect the state of files from a partial download and see it be in some state you wouldn't expect?

The way you describe it, it sounds like you're seeing a symptom and speculating what the cause is, rather than collecting evidence. For example, this:

You apparently can't see the problem because your tests are apparently limited to:

downloading only small files (less than 128 megabytes each)

from a slow internet access (100 megabit/s or lower, not a gigabit fiber internet or better)

to a fast SSD storage

That's speculation, and you're wrong. But I am most likely not using the same operating system or file system as you, I don't doubt that you experience these symptoms. It would be really helpful if you could provide some evidence pointing at what might be happening and what could be done about it.

arvidn on 13 Jun 2020

That's not speculation and the source code shows clearly that there's NO
preallocation at all performed with the necessary IOCTL API call.
The source code also shows clearly that the settings partly written for it
is NOT used at all.

Le sam. 13 juin 2020 à 09:12, Arvid Norberg notifications@github.com a
écrit :

Because All demonstrates EASILY and REPRODUCTIBLY that downloads of big
files start by downloading very fast (a few seconds) 127 megabytes, then
the torrent will stall, stop downloading anything for long minutes while
the disk is 10% busy for writing arbitrarily large amount of zeroes
(gigabytes or more): as long as ALL these zertoes are not fully written,
nothing is downloaded (and the first 127 megabytes downloaded from random
position are still not written on disk, they are pending in memory waiting
for the end of writes of all these zeroes).
And this happens ALWAYS when downloading in random order from
arbitrary positions of the large file.

I have never seen this happen, but that doesn't really matter. It sounds
like it's reproducible on your computer. You seem to have an idea of what
the problem is. Could you gather some information of what's actually
happening, say with process monitor?

qBittorrent is then unsuitable as a distribution method for large files
like ISOs (DVD images), or VM snapshots, or large database dumps, or host
backup/replication.

I really think that you should file a separate ticket about this. If I
understand you, this is about an optimization where zeroes are potentially
not transferred and not written to disk (but "virtually" written to disk).
It's not obvious that having a special case for this would carry its own
weight, but that discussion belongs in a separate ticket as far as I can
tell.

Everything you write (at a quick glance) is already done in libtorrent.

Visibly NO, it is NOT already in libtorrent (at least not the version
integrated in qBittorrent v 4.2.5)

Please share how you established this. How is it visible? For example:

did you step through the program in a debugger and saw that the
calls in question were not made or with some invalid argument?

did you run ProcessMonitor or strace to see that the calls were not
made or made with some invalid arguments?

did you inspect the state of files from a partial download and see
it be in some state you wouldn't expect?

The way you describe it, it sounds like you're seeing a symptom and
speculating what the cause is, rather than collecting evidence. For
example, this:

You apparently can't see the problem because your tests are apparently
limited to:

downloading only small files (less than 128 megabytes each)

from a slow internet access (100 megabit/s or lower, not a gigabit
fiber internet or better)

to a fast SSD storage

That's speculation, and you're wrong. But I am most likely not using the
same operating system or file system as you, I don't doubt that you
experience these symptoms. It would be really helpful if you could provide
some evidence pointing at what might be happening and what could be done
about it.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/qbittorrent/qBittorrent/issues/12991#issuecomment-643582806,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAKSUG4G4TLWJTGIKTHIRODRWMRHTANCNFSM4NZXWENA
.

verdy-p on 13 Jun 2020

That's not speculation

Well, you were saying things about my tests that were incorrect.

and the source code shows clearly that there's NO
preallocation at all performed with the necessary IOCTL API call.
The source code also shows clearly that the settings partly written for it
is NOT used at all.

I don't think it's clear to anyone else participating in this thread. Could you please links to the source? URLs to file and line in github would be great, along with an explanation how it's wrong.

arvidn on 13 Jun 2020

How can I link to a source code that does not even exist ?

E.g. searching for the necessary Windows API call:
https://github.com/qbittorrent/qBittorrent/search?q=DeviceIoControl

I can look at source code for file objects or session objects, there's NOTHING.

(And Im not speaking about your own fork in GitHub where there's some different code for your tests, which is also not conclusive and is also missing the necessary API)

All I can say is that the official source code does not contain ANY call to the necessary API and it is easy to see (using the search function of GitHub, but NOT within your forked project like you did above, with then non-relevant replies)

So I'm not speculating: I'm really speaking about things that don't exist in official qB (but you reply using a different version that you've forked: YOU are speculating by stating this is my local configurtion that is specific, when in fact it is YOUR configuration which is specific)

And given the fact that you admitted to ignore the fact that we can allocate space for a file WITHOUT writing data to it (provided the file storage more is converted to allow sparse allocation with a prior IOCTL), I think your fork has no chance of being integrated as is.

You should read again MSDN and the link I provided (for Windows), as well look for equivalent API for other OS (fallocate() is a possiblity, but first check the library used, in doubt use the native API).

https://docs.microsoft.com/en-us/windows/win32/api/winioctl/ni-winioctl-fsctl_set_zero_data

verdy-p on 13 Jun 2020

How can I link to a source code that does not even exist ?

What? How can there be a bug in source code that doesn't exist? :)

E.g. searching for the necessary Windows API call:
https://github.com/qbittorrent/qBittorrent/search?q=DeviceIoControl

qBittorrent is the wrong repository to search. It's not responsible for creating these files. If you were genuinely interested I'm sure you would have learned this by now.

(And Im not speaking about your own fork in GitHub where there's some different code for your tests, which is also not conclusive and is also missing the necessary API)

The link I posted was not to qBittorrent, it was to libtorrent. The repository that's responsible for creating the files you speak of.

All I can say is that the official source code does not contain ANY call to the necessary API and it is easy to see (using the search function of GitHub, but NOT within your forked project like you did above, with then non-relevant replies)

I think you have to define "official". In my mind I provided proof of the contrary a few posts ago; the github search link you refer to.

So I'm not speculating: I'm really speaking about things that don't exist in official qB (but you reply using a different version that you've forked: YOU are speculating by stating this is my local configurtion that is specific, when in fact it is YOUR configuration which is specific)

Which version of libtorrent are you using when you build qBittorrent?

And given the fact that you admitted to ignore the fact that we can allocate space for a file WITHOUT writing data to it (provided the file storage more is converted to allow sparse allocation with a prior IOCTL), I think your fork has no chance of being integrated as is.

I don't know what you're talking about. my branch of what?

You should read again MSDN and the link I provided (for Windows), as well look for equivalent API for other OS (fallocate() is a possiblity, but first check the library used, in doubt use the native API).

I don't think any of this matters. We're clearly talking past each other at a different level.

arvidn on 13 Jun 2020

And given the fact that you admitted to ignore the fact that we can allocate space for a file WITHOUT writing data to it (provided the file storage more is converted to allow sparse allocation with a prior IOCTL), I think your fork has no chance of being integrated as is.

I don't know what you're talking about. my branch of what?

https://github.com/arvidn/libtorrent/search?q=DeviceIoControl&unscoped_q=DeviceIoControl

This is the branch that you used to reply above (in your own fork). This code is NOT used in qBittorrent. and anyway it still does not use the required IOCTL for preallocation.

So my bug here is fully relevant to qBittorrent. And your suggestion in your version of libtorrent does not apply and in fact it does not work at all:

https://github.com/arvidn/libtorrent/search?q=FSCTL_SET_ZERO_DATA

(as long as you don't use this Windows API, you'll physically write tons of zeroes for all preallocations)

verdy-p on 13 Jun 2020

https://github.com/arvidn/libtorrent/search?q=DeviceIoControl&unscoped_q=DeviceIoControl

This is the branch that you used to reply above (in your own fork). It is NOT used in qBitTorrent.

Can you please support that statement?

Which version of libtorrent do you build qBittorrent with? or which version of libtorrent does your qBittorrent binary link against?

@FranciscoPombal do you know what he means?

arvidn on 13 Jun 2020

https://github.com/arvidn/libtorrent/search?q=DeviceIoControl&unscoped_q=DeviceIoControl
This is the branch that you used to reply above (in your own fork). It is NOT used in qBitTorrent.

Can you please support that statement?

Which version of libtorrent do you build qBittorrent with? or which version of libtorrent does your qBittorrent binary link against?

Since the beginning I'm using the version built in the official release of qBittorrent (v2.4.5 today, but the bug started with previous versions, and is still not fixed/implemented).

verdy-p on 13 Jun 2020

Since the beginning I'm using the version built in the official release of qBittorrent (v2.4.5 today, but the bug started with previous versions, and is still not fixed/implemented).

I believe the current version of libtorrent is 1.2.6 (which roughly corresponds to the RC_1_2 branch of libtorrent in the repository I linked you to).

It sounds like there's an older version of qBT that doesn't have this issue, is that right?
Do you know which version of libtorrent was the last one that didn't have the issue?

arvidn on 13 Jun 2020

Since the beginning I'm using the version built in the official release of qBittorrent (v2.4.5 today, but the bug started with previous versions, and is still not fixed/implemented).

I believe the current version of libtorrent is 1.2.6 (which roughly corresponds to the RC_1_2 branch of libtorrent in the repository I linked you to).

It sounds like there's an older version of qBT that doesn't have this issue, is that right?
Do you know which version of libtorrent was the last one that didn't have the issue?

ALL past and present versions of qBittorrent have this problem. Preallocations without writing zeroes has never been implemented for now. (and yes it is a problem for downloading large files in random order, which is the default behavior highly desirable in a healthy torrent network: forcing the sequential download order to solve the initial performance problem is really bad: only the initial part of the files are swarmed, the end of the file remains dependant on the initial seeder, that must be a single server with high capacity: this server is rapidly overwelmed by many concurrent attempts to get the end of file, and those that got the end of file are rapidly disconnected from the torrent network and no longer participate: this is no longer a "torrent" if only one or very few seeders can share the whole file, most downloaders will then stall for long, waiting for capacity from some of the few seeders that have the whole file)

verdy-p on 13 Jun 2020

@FranciscoPombal already alluded to this to a degree here https://github.com/qbittorrent/qBittorrent/issues/12991#issuecomment-643369573

My understanding from reading/following this is that FSCTL_SET_SPARSE "IS" available but not fully implemented or is missing FSCTL_SET_ZERO_DATA in order to fully implement what is being asked here.

So this to me is a "feature request"

@verdy-p Is this a correct understanding?

xavier2k6 on 13 Jun 2020

@FranciscoPombal already alluded to this to a degree here #12991 (comment)

My understanding from reading/following this is that FSCTL_SET_SPARSE "IS" available but not fully implemented or is missing FSCTL_SET_ZERO_DATA in order to fully implement what is being asked here.

So this to me is a "feature request"

@verdy-p Is this a correct understanding?

Yes, this is what is needed. FSCTL_SET_SPARSE is not enough, you also need FSCTL_SET_ZERO_DATA for fast preallocations (if you use it, don't use FileWrite with buffers full of zeroes as it is still physically writing zeroes, even inside sparse preallocated holes: these holes will be cleared, removing or reducing these holes)

verdy-p on 13 Jun 2020

don't use FileWrite with buffers full of zeroes as it is still physically writing zeroes, even inside sparse holes

That has never been done in any version of libtorrent. I would expect you to know that (but you also seem to think the source code for libtorrent can be found in some other place, yet to be shared with the thread)

arvidn on 13 Jun 2020

@verdy-p If I make a patch that employs FSCTL_SET_ZERO_DATA, when full allocation is set. Could you benchmark it before and after and share the results?

arvidn on 13 Jun 2020

@arvidn @FranciscoPombal Would anything need to be done on qBittorrent's end to fully support/implement that patch?

I could provide a "windows test build" when all necessary requirements are met unless @FranciscoPombal would like to do it through vcpkg nightlies?

xavier2k6 on 13 Jun 2020

@verdy-p If I make a patch that employs FSCTL_SET_ZERO_DATA, when full allocation is set. Could you benchmark it before and after and share the results?

Thanks for trying it. It will definitely improve the situation:

no longer any need to download in sequential order
full preallocation at start of downloads: no longer any "file system full" in the middle of a download, and minimal fragmentation of the allocated space as it is done in one call (the filesystem will just allocate the largest fragments can can fit in the available free spaces), so there's also improved I/O speed (downloaded files will be mostly read later in sequential physical order, this is great for storing to hard disks or RAID arrays, and not SSD/flash/ramdisk).
if full preallocation fails (due to lack of free space on the target volume, a dialog may inform the user immediately: this dialog may propose to either cancel the download, or start it with partial allocation and sequential order only (the download may still fail in the middle later due to lack of space for additional data)
fast start of all downloads (including in random order)
constant sharing of downloaded fragment at any position while the download is in progress: more seeders available on the healthier torrent).
the "partial allocation" flag may be kept only for users that want to download to a FAT partition (because it does not support sparse allocation, and it will will write tons of zeroes for downloads in random order, however it can cause excessive fragmentation or file system full errors in the middle: those users will want to use sequential downloads only, in which case "partial allocation" has no use at all), but it should be off by default (on Windows, most people use NTFS volumes)
only the "sequential download" flag may be useful (only for users of FAT volumes or some network mounted volumes without sparse file support) but it should be off also by default.
now this removes the huge dependency on a single initial seeder for large files (where torrents are ineffective to distribute the workload with the help of all downloaders that have partial fragments).
the code you wrote to check the list of allocated ranges (or holes) is not needed at all (it should be disabled compltely if full allocation is set).
if you change the setting from "partial" to "full allocation", the whole filesize may need to be reallocated for the missing fragments (this is where you need transitorily a loop to query the allocated ranges and then use FSCTL_SET_ZERO_DATA to allocate them, and only them without "clearing" the existing downloaded data)
you may optionally use FSCTL_SET_ZERO_DATA instead of FileWrite() with buffers full of zeroes (if the downloaded data range is full of zeroes): this saves unnecessary writes.
for initial tests/benchmarks, an advanced setting can be used to enable the FSCTL_SET_ZERO_DATA preallocation, without it, writes of zeroes will still occur. But once fully tested, this setting should become on by default.
used properly, this cannot fail (it is already widely used in volume managers, RDBMS managers, Hyper-V and other hypervisors...): preallocation without writes works on all version of Windows since Windows XP; it also exists in Linux, FreeBSD and many proprietary Unixes or other OSes since decennials (SunOS, Solaris, AiX, HP-UX, VMS, MVS... All those that can run an hypervisor or supported by major database systems like Oracle, MySQL and others)
on FAT volumes, it is also possible to preallocate a file without writing it (it requires locking the volume to modify the FAT table, there are some libaries that implement it, or some command line tools that can do that: the content of the allocated file will be random data or old data from older files); but I'm not suggesting implementing it (by default FSCTL_SET_ZERO_DATA will still write zeroes to initiliaze the preallocated space, but it performs better than what you would do in a user application as this is done securely by the OS in a separate internal worker thread, inside the volume manager: in such situation, FSCTL_SET_ZERO_DATA may not return immediately and will be blocking for long: you may instead use a loop of FSCTL_SET_ZERO_DATA with a maximum size of 64MB so that the blocking will not be too long, it may generate some fragmentation but it is minor if preallocated fragments are 64MB at least).

It can only be better.

verdy-p on 13 Jun 2020

@verdy-p From what I read on the internet (which isn't all that much yet) it sounds like the DeviceIoControl with FSCTL_SET_ZERO_DATA is essentially the same as fallocate() with FALLOC_FL_PUNCH_HOLE on linux. (man page)

i.e. it's a way to deallocate ranges of a file, to make them sparse, which is the default mode for files on linux and files marked as sparse on windows. Can you please explain how I'm misunderstanding these articles, or how they are wrong?

https://docs.microsoft.com/en-us/windows/win32/api/winioctl/ni-winioctl-fsctl_set_zero_data

says:

Fills a specified range of a file with zeros (0). If the file is sparse or compressed, the NTFS file system may deallocate disk space in the file. This sets the range of bytes to zeros (0) without extending the file size.

http://www.flexhex.com/docs/articles/sparse-files.phtml

says:

You may ask: "What if we set a new end-of-file marker without calling DeviceIoControl?", for example, by executing the following function calls:

::SetFilePointer(hFile, 0x1000000, NULL, FILE_END);
::SetEndOfFile(hFile);

What will we find in those 16 megabytes between the old and the new end-of-file markers? Sparse zeros, real zeros, just some junk? The right answer is: sparse zeros. You don't need to call DeviceIoControl/FSCTL_SET_ZERO_DATA to create a sparse zero block in the end of the file - simply moving the end-of-file marker will do the trick.

arvidn on 13 Jun 2020

Perhaps to clarify.

it also exists in Linux, FreeBSD and many proprietary Unixes or other OSes since decennials

Could you say what the linux equivalent is? is it fallocate() with FALLOC_FL_PUNCH_HOLE?

arvidn on 13 Jun 2020

So in summary, for the "full allocation" mode:

first open the file for creation
call immediately FSCTL_SET_SPARSE, then:
- if FSCTL_SET_SPARSE succeeded call FSCTL_SET_ZERO_DATA in one operation to allocate the full size
- if FSCTL_SET_SPARSE failed (e.g. on FAT volume), call FSCTL_SET_ZERO_DATA by blocks of maximum size 64MB (to avoid these calls to block for too long)

64MB is half the size of the maixmum write queue size for all threads in the current process. All threads are serializing their I/O into the same output queue. All these I/O are normally executed in order unless they are flagged as asynchronous I/O (in which case the OS will select the I/O that can be executed the fastest).

My benchmarks in qBittorrent shows that it stalls each time after downloading 128MB (the effective output queue is about 127.1MB but this varies a bit, depending on concurrent threads, or the amount of other writes needed to update other parts of the disk: the allocation map, the directory entries for updating the filesizes, the other threads writing to log files, the other threads writing to the Windows registry hives possibly indirectly via some other API calls).

128MB for the write quue is the limit set by the OS. A process with enough privileges may request to extend its output queue (but this is undesirable in general as this affects the performance of all other concurrent processes on the same OS).

This limit may sometimes be shorter than 128MB (notably for worker processes used in application servers, so that a single or few clients will not exceed the total I/O supported).

This 128MB limit is OS-dependant and may be different in Linux/Unix and changing it may require other privileges.

Using writes (including FSCTL_SET_ZERO_DATA on non-sparse files) with size 64MB if reasonnable, it does not produce excessive fragmentation. For sparse files (with FSCTL_SET_SPARSE enabled), FSCTL_SET_ZERO_DATA can be used with any size, it is not limited.

verdy-p on 13 Jun 2020

here's a first pass. please take a look: https://github.com/arvidn/libtorrent/pull/4764

arvidn on 13 Jun 2020

if FSCTL_SET_SPARSE succeeded call FSCTL_SET_ZERO_DATA in one operation to allocate the full size

The msdn article is quite explicit about this call not extending the file size.

This sets the range of bytes to zeros (0) without extending the file size.

Surely one would have to call SetFilePointerEx() and SetEndOfFile() first, to set file size.

arvidn on 13 Jun 2020

if FSCTL_SET_SPARSE failed (e.g. on FAT volume), call FSCTL_SET_ZERO_DATA by blocks of maximum size 64MB (to avoid these calls to block for too long)

I'm not doing that, and I'm not particularly interested in spending a lot of time to support filesystems that can't do sparse files.

arvidn on 13 Jun 2020

"The FALLOC_FL_PUNCH_HOLE flag must be ORed with FALLOC_FL_KEEP_SIZE in mode; in other words, even when punching off the end of the file, the file size (as reported by stat(2)) does not change."

So FSCTL_SET_ZERO_DATA is the equivalent of (FALLOC_FL_PUNCH_HOLE|FALLOC_FL_KEEP_SIZE): there's an allocation, no deallocation.

Note that the POSIX fallocate() always initializes the allocated areas with zeroes. This is not the case with the Linux API, or other APIs specific to other OSes.

Yes you'd need to use SetEndOfFile() to extend the file size on Windows

verdy-p on 13 Jun 2020

if FSCTL_SET_SPARSE failed (e.g. on FAT volume), call FSCTL_SET_ZERO_DATA by blocks of maximum size 64MB (to avoid these calls to block for too long)

I'm not doing that, and I'm not particularly interested in spending a lot of time to support filesystems that can't do sparse files.

You may be wrong: the OS performs this faster (and securely, including for managing priorities between processes) than you would do, with your own file writes using large memory buffers.

My own benchmarks for other applciations showed that it was faster on FAT volumes to use FSCTL_SET_ZERO_DATA with blocks of 64MB and no buffers at all, than using file-writes with so large buffers in user's virtual memory (this is not a question of I/O, but of memory use: this affects how virtual memory is paged to/from the paging file, when memory is low). The OS uses its own internal zeroed memory buffers, that are preallocated,, read-only, and shared across all processes, properly aligned with driver constraints or bus constraints, for example the buffer may need to reside in a specific limited region of physical memory, and just the size to fit the needs supported by the underlying filesystem drivers or physical disk drivers. The OS will then perform the necessary loop of physical writes itself.

The application should not care about the optimal size of these buffers (or preallocation sizes in non-sparse files), except they should not use more than half the maximum size of the per-process write-queue (128MB by default in Windows) for any blocking I/O (async I/O don't have this limit)

If you use SetEnfOfFile() which is a blocking I/O, the extension of filesizes for non-sparse file should also be done by maximum steps of 64MB (otherwise it will block for very long as it will also internally use writes of zeroes for non-sparse files and it will fill up the total queue size of the current process, so all other threads that want to do their own writes will be also blocked: with 64MB, at least the concurrent threads will be able to get their writes not blocked for long and they'll continue working; for example other concurrent torrent downloads/uploads, or the applicaton logging in the main GUI thread).

verdy-p on 13 Jun 2020

I'm not doing that, and I'm not particularly interested in spending a lot of time to support filesystems that can't do sparse files.

You may be wrong:

I'm saying I'm not very interested in spending my time working on supporting filesystems that don't support sparse files. What is there to be wrong about?

the OS performs this faster (and securely, including for managing priorities between processes) than you would do, with your own file writes using large memory buffers.

Well, I'm not doing that either (which we've already established). So I think we're good.

arvidn on 13 Jun 2020

@verdy-p

Ok honestly this is starting to get on my nerves. Yet more walls of text with virtually no actionable information, no relevant tests, benchmarks, or results, confusing statements, and strange claims that I'm tired of wasting my time reading.

Let's see:

These comments (https://github.com/qbittorrent/qBittorrent/issues/12991#issuecomment-643610374, https://github.com/qbittorrent/qBittorrent/issues/12991#issuecomment-643614041, https://github.com/qbittorrent/qBittorrent/issues/12991#issuecomment-643615068) make it seem like you think libtorrent is fork of qBittorrent or maybe that the qBittorrent repository is using a separate fork of libtorrent for its builds. By this time you should have realized this is completely wrong, and the fact that you would think this is the case after spamming so much text defies all reason.
In these comments you also make it seem that qBittorrent does not use any appropriate DeviceIoControl()-related calls, perhaps due to not properly using the libtorrent API, which I have already demonstrated to false in https://github.com/qbittorrent/qBittorrent/issues/12991#issuecomment-643297161
I finally managed to dissect what you _most likely_ meant in your walls of text in https://github.com/qbittorrent/qBittorrent/issues/12991#issuecomment-643369573. The very creator of libtorrent then had the patience to continue the discussion with you, arguing the merits of your claims and even going as far as submitting a PR implementing what you apparently want: https://github.com/arvidn/libtorrent/pull/4764. This is the sort of thing that you should have already done yourself, tested, and provided results for, as I've suggested in https://github.com/qbittorrent/qBittorrent/issues/12991#issuecomment-642159020, instead of spewing text like this. You say you're interested in fixing the problem, and it's obvious you have the time, so why not just do it?? Or, if you don't have the ability to, asking for concrete help with the implementation?
In https://github.com/qbittorrent/qBittorrent/issues/12991#issuecomment-643218999 there is this strange claim about downloads slowing down massively after 127 MiB or something strange like that or whatever, something that no one has ever seen. Additionally, there is the demonstrably false claim that "the problem manifests when the network speed is faster than 40-50% of the disk write speed because blah blah blah". I had never had a problem downloading at gigabit speeds (~ 100 MiB/s) to HDDs with nominal write speeds of 120 MiB/s and others. This is a weird issue on your end (or you still haven't understood what the preallocation checkbox currently does in qBittorrent), but that doesn't even matter, it's another issue for another ticket.

Let's focus on the 3rd bullet point (the new libtorrent PR), which is the most important for the discussion at hand:

Since you've wasted the time of many people with these walls of text for what apparently could have otherwise been clearly described in 10 lines or less, your next posts (here, or in the libtorrent PR) _better be_ constructive and actually useful (e.g. reporting actual results of using the PR, suggesting concrete changes in the code, etc). Go test it and provide feedback, or implement stuff yourself. Any other action on your part, such as more useless text spam is unacceptable at this point.
If you keep up this attitude of refusing to be helpful and to properly collaborate with others, even after being told exactly what to do and how to do it, and after others have taken their time to do stuff for you, I'll close this. I'm done wasting my time with this and sick of these useless walls of text. You really need to learn how to communicate and collaborate more effectively.

FranciscoPombal on 14 Jun 2020

for the record, I find it extremely unlikely that @verdy-p is correct in his assertions about the impact of FSCTL_SET_ZERO_DATA. For these reasons:

It would not make any sense for microsoft to implement poor performance by default
Microsoft's documentation suggests that nothing is pre-allocated by making this call, but in fact the opposite. It's turning allocated disk space into a sparse region.

But, I'm eagerly awaiting results from the benchmark.

arvidn on 15 Jun 2020

@arvidn @FranciscoPombal Does anything need to be done on qBittorrent's side?, If not - I will go ahead & make a test build with this patch so that it can be then benchmarked.

xavier2k6 on 15 Jun 2020

Ideally @verdy-p would review the patch to ensure it's what he has in mind

arvidn on 15 Jun 2020

👍1

Ideally @verdy-p would review the patch to ensure it's what he has in mind

Ok, build "ON HOLD" until further feedback/direction.

xavier2k6 on 15 Jun 2020

👍1

Sparse file created by qbittorrent on my linux system with btrfs file system (copy-on-write turned off for the download directory) created a huge amount of fragmentation, and it also creates a slight/moderate amount of fragmentation on Ext4 filesystem.

I have been trying to switch to full pre-allocation by _storage_mode_allocate_ default when adding torrents:
1) replace default value _storage_mode_sparse_ with _storage_mode_allocate_ in libtorrent code (session.cpp, read_resume_data.cpp, torrent_status.hpp, session_handle.hpp, add_torrent_params.hpp),
2) toggle on _BITTORRENT_SESSION_KEY("Preallocation")_ (changing value from _false_ to _true_) in qbittorrent code (session.cpp).
I built from the altered code and made sure pre-allocation is enabled in qbittorrent options, and found that sparse file was still being used for new downloads and fragmentation (number of extents of a file) was increasing as the download progressed.

Could anyone confirm that by toggling on “Pre-allocate disk space for all files” (in qBittorrent options - downloads tab) _storage_mode_allocate_ will be sent as parameter to libtorrent, and that the failure to invoke fallocate() in Linux is a bug on libtorrent side?

Reference:
1) Excerpt from https://github.com/arvidn/libtorrent/blob/RC_1_2/include/libtorrent/storage_defs.hpp
(My thought: _storage_mode_allocate_ should invoke fallocate() in Linux)

    // types of storage allocation used for add_torrent_params::storage_mode.
    enum storage_mode_t
    {
        // All pieces will be written to their final position, all files will be
        // allocated in full when the torrent is first started. This is done with
        // ``fallocate()`` and similar calls. This mode minimizes fragmentation.
        storage_mode_allocate,

        // All pieces will be written to the place where they belong and sparse files
        // will be used. This is the recommended, and default mode.
        storage_mode_sparse
    };

2) Excerpt from https://github.com/qbittorrent/qBittorrent/blob/master/src/base/bittorrent/session.cpp
(My thought: if _isPreallocationEnabled()_ is true, _storage_mode_allocate_ will be used for _p.storage_mode_ and transmit to libtorrent as parameter when adding and downloading torrrents)

    // Flags
    // Preallocation mode
    if (isPreallocationEnabled())
        p.storage_mode = lt::storage_mode_allocate;
    else
        p.storage_mode = lt::storage_mode_sparse;

pureair on 22 Jun 2020

On ext4, i believe all files are sparse (in that they can be sparse, as opposed to windows where you need to set a flag first). Preallocating a file doesn’t guarantee that it will have a single extent, or that the extents will be contiguous. It’s just assumed to make it more likely.

Especially large files on drives with relatively little space left, I would think are always likely to have multiple extents.

Furthermore, I would expect the file system to optimize for performance, not minimising the number of extents.

So, I don’t think it’s obvious, from your observation, that there is a problem.

Convincing evidence would be a benchmark of a file downloaded with qbt, compared with one created by dd or some other tool that creates a “better” file.

arvidn on 22 Jun 2020

Could anyone confirm that by toggling on “Pre-allocate disk space for all files” (in qBittorrent options - downloads tab) _storage_mode_allocate_ will be sent as parameter to libtorrent, and that the failure to invoke fallocate() in Linux is a bug on libtorrent side?

On the qBittorrent side, that checkbox only toggles the parameter, it does nothing else. If it is in toggled state, storage_mode_allocate is used (passed to libtorrent), as expected. Anything beyond that happens in libtorrent - so qBittorrent cannot guarantee anything about whether or not fallocate is used.

FranciscoPombal on 22 Jun 2020

Convincing evidence would be a benchmark of a file downloaded with qbt, compared with one created by dd or some other tool that creates a “better” file.

Good suggestion. I downloaded Ubuntu 20.04 live-CD image using qbittorrent and copied the file using cp and fallocate+dd. Here's the result:

File downloaded using qbittorrent with “Pre-allocate disk space for all files” option checked.

# ls -l ubuntu-20.04-desktop-amd64.iso
-rw-r--r--    1 777      111      2715254784 Jul  7 15:17 ubuntu-20.04-desktop-amd64.iso
# filefrag ubuntu-20.04-desktop-amd64.iso
ubuntu-20.04-desktop-amd64.iso: **1043 extents** found

File copied using cp

# cp ubuntu-20.04-desktop-amd64.iso  ubuntu-20.04-desktop-amd64.iso~cp
# filefrag ubuntu-20.04-desktop-amd64.iso~cp
ubuntu-20.04-desktop-amd64.iso~cp: **42 extents** found

File copied using fallocate+dd

# fallocate -l 2715254784 ubuntu-20.04-desktop-amd64.iso~fallocate
# filefrag ubuntu-20.04-desktop-amd64.iso~fallocate
ubuntu-20.04-desktop-amd64.iso~fallocate: 2 extents found
# dd if=ubuntu-20.04-desktop-amd64.iso of=ubuntu-20.04-desktop-amd64.iso~fallocate
5303232+0 records in
5303232+0 records out
# filefrag ubuntu-20.04-desktop-amd64.iso~fallocate
ubuntu-20.04-desktop-amd64.iso~fallocate: **154 extents** found
# lsattr ubuntu-20.04-desktop-amd64.iso~fallocate
---------------C-- ubuntu-20.04-desktop-amd64.iso~fallocate

File copied using fallocate+dd (with conv=notrunc)

# fallocate -l 2715254784 ubuntu-20.04-desktop-amd64.iso~fallocate_dd_notrunc
# filefrag ubuntu-20.04-desktop-amd64.iso~fallocate_dd_notrunc
ubuntu-20.04-desktop-amd64.iso~fallocate_dd_notrunc: 2 extents found
# dd if=ubuntu-20.04-desktop-amd64.iso of=ubuntu-20.04-desktop-amd64.iso~fallocate_dd_notrunc conv=notrunc
5303232+0 records in
5303232+0 records out
# filefrag ubuntu-20.04-desktop-amd64.iso~fallocate_dd_notrunc
ubuntu-20.04-desktop-amd64.iso~fallocate_dd_notrunc: **2 extents** found

It appears qbittorrent/libtorrent does created a more fragmented file on btrfs file system (note that the torrent only has 2590 pieces of 1M piece, and on ext4 the fragmentation is much less severe) and if per-allocation is properly implemented, the fragmentation could be greatly reduced.

It's very likely the fragmentation results from that the file being a sparse file instead of a per-allocated file and that btrfs (or kernel implementation on btrfs) is not very friendly on sparse files.

Regarding performance, it took a long time for rm to complete on a fragmented file. (More precisely, when I tried to delete a directory containing ~100 x 2G file, each of ~1000 extents. It took more than 1 hour for rm to complete.) I am not sure if it's the fragmentation that actually causes the performance degradation, but that's my best guess. The drive is an external drive mounted through USB, I think the slow USB connection make the degradation more noticeable.

On the qBittorrent side, that checkbox only toggles the parameter, it does nothing else. If it is in toggled state, storage_mode_allocate is used (passed to libtorrent), as expected. Anything beyond that happens in libtorrent - so qBittorrent cannot guarantee anything about whether or not fallocate is used.

Good to know. Thanks.

pureair on 7 Jul 2020

👍1

@pureair thanks for the investigation. Do you have any concrete ideas of what to do, instead of posix_fallocate() to reduce the number of extents?

or is the current use of or detection of posix_fallocate() not working for your platform? (there's a separate ticket on libtorrent about how builds with musl failed to detect posix_fallocate() being available).

if strace qbtm you should be able to see whether posix_fallocate() is called, and whether it looks like it's called incorrectly.

arvidn on 7 Jul 2020

@pureair thanks for the investigation. Do you have any concrete ideas of what to do, instead of posix_fallocate() to reduce the number of extents?

or is the current use of or detection of posix_fallocate() not working for your platform? (there's a separate ticket on libtorrent about how builds with musl failed to detect posix_fallocate() being available).

if strace qbtm you should be able to see whether posix_fallocate() is called, and whether it looks like it's called incorrectly.

@arvidn After apply your recent patch on libtorrent (probably instead of detecting whether posix_fallocate() is available, detect w… ), pre-allocation new behaves as expected, fragmentation is greatly reduced on both ext4 and btrfs, both show around 20 extents for a 7GB file. (I just built from the latest code, not committing each patch separately, so not sure which commit actually solves the problem)
I guess the problem was truly about the detection of posix_fallocate().

pureair on 11 Jul 2020

👍1

Well, the original issue report concerns Windows systems, so I don't think we should close this just yet.

@verdy-p ping. https://github.com/qbittorrent/qBittorrent/issues/12991#issuecomment-644086394 https://github.com/qbittorrent/qBittorrent/issues/12991#issuecomment-644096478

FranciscoPombal on 11 Jul 2020

If anyone's interested, I played with this idea in rust, here's the code:

In my tests, the code above works great on both Windows and Linux. It's in rust, but the core idea should be easy to grasp, and it should be even more compact in C++.

MOZGIII on 9 Sep 2020

@MOZGIII it's not clear to me what "this idea" refers to. The calls you make in your rust code is already in libtorrent (and has been for a very long time).

I think @verdy-p has had sufficient time to demonstrate that the behaviour of FSCTL_SET_ZERO_DATA on windows is any different than what's documented (and in fact provides a performance improvement), and that it would make sense to close this ticket, along with the path I offered to be benchmarked. https://github.com/arvidn/libtorrent/pull/4764

arvidn on 9 Sep 2020

I think @verdy-p has had sufficient time to demonstrate that the behaviour of FSCTL_SET_ZERO_DATA on windows is any different than what's documented (and in fact provides a performance improvement), and that it would make sense to close this ticket, along with the path I offered to be benchmarked. arvidn/libtorrent#4764

Indeed. @MOZGIII if you have any new ideas about this, please open a new issue in the libtorrent issue tracker to discuss that.

FranciscoPombal on 9 Sep 2020

Qbittorrent: Sparse files/preallocation suggestions

All 68 comments

Related issues