Qbittorrent: Recheck and move multiple torrents one-by-one

[a-raccoon]

Current Behavior: Performing a Force Recheck on multiple selected torrents results in concurrent scans of multiple files across multiple torrents all at the same time. On platter drives, this means lots of random seeking (slow). Same holds true when multiple torrents are being moved from temp to completion folder, or via category or destination folder change.

Desired Behavior: Sequential (one at a time) file rechecking or moving by implementing a queuing system.

[Syst3mSh0ck]

Not exactly ideal when you have a lot of torrents that require a recheck. I would also prefer the option to choose between serial/parallel processing for force rechecks.

[FranciscoPombal]

Related and relevant discussion: https://github.com/qbittorrent/qBittorrent/issues/9061
@arvidn @airium

[arvidn]

this is the behaviour for torrents that are "force-started" (i.e. not auto-managed). Is it possible that qBitTorrent clears the auto-managed flag whenever the user force-rechecks a torrent?
Otherwise, perhaps all your torrents were force-started before you force-rechecked them.

[a-raccoon]

@Syst3mSh0ck : Your statement is confusing. At first it sounds like you think my proposal is rubbish, but then it sounds like you mean the current behavior is rubbish. Please clarify.

@arvidn : I don't know about any of that. I don't use "Force Start", but I'll use "Force Recheck" on a selection of torrents that have completed, where I change Set Location to another harddrive, and then I use Force Recheck to confirm that the files still have integrity (as a file-copy verification).

In any event, it seems always-stupid, under any circumstance, for qBittorrent to recheck more than one torrent at a time, or more than one torrent from the same storage device. This just destroys any hope of performance.

[slrslr]

@sledgehammer999 In my case (qbt 4.1.1, WIn10) when i highlight 2 torrents and do Force recheck,
then one torrent is stalled at 0%, other is 99%. Then when i recheck "0%" one, it works and ends at 99%.

[dakusan]

Same here. Just imported from uTorrent and am having to recheck torrents 1 at a time due to it trying to recheck everything in parallel, causing thrashing from non-sequential reads.

[Edit] I realized that if I first force recheck them all, then pause them, and then unpause them, it does the rechecking serially. So while still a [bug/feature], not debilitating.

As a feature, it would also be nice if the rechecking could be done in parallel per mounted partition.

[a-raccoon]

Thanks for that find @dakusan ! I will have to try that trick next time... Select multiple torrents, choose Force Recheck, then Pause, then Resume.

[iheartcsharp]

Based on the libtorrent docs there is a separate queue for torrents that are in checking status. The default value for it is supposed to be 1, so it should check 1 torrent at a time. So not sure how qBittorent is checking them all. I checked the latest source code and it doesn't seem to be setting the active_checking value anywhere.

[arvidn]

torrents where the auto_managed flag is not set, are not subject to any of the queuing logic

[slrslr]

torrents where the auto_managed flag is not set, are not subject to any of the queuing logic

earlier in this thread i mentioned i am having this issue on Windows 10, 64b, qbt 4.1.1
now i tried on Windows 7, 64b, qbt 4.1.4 and no problem, i tried to rechceck multiple manual, multiple automatic management torrents and mixture. These torrents was all 100% completed and it always checked 1 torrent at a time, but when i did same for incomplete (like lets say 10% downloaded), it checked multiple torrents at same time, but apparently again without problem of setting torrent to 0% downloaded as described in my previous reply..

[a-raccoon]

Someone mentioned earlier in this thread that the problem is with FORCE Recheck. You have to turn off FORCE and then it queues the rechecks one-by-one. You cannot begin a recheck without first doing a FORCE recheck. I have verified this behavior myself.

[slrslr]

You cannot begin a recheck without first doing a FORCE recheck

I do not understand what you describe and how to "turn OFF FORCE", but in this thread when i spoken about "recheck" i always meant "Force recheck" option triggerred manually by the user from the torrent right click menu.

[a-raccoon]

You turn off FORCE by clicking it again. Perform the select, right-click menu routine twice. Then it will queue rechecks one-by-one.

Force -> Un-force -> Force -> Un-force -> Force -> Un-force ...

[thalieht]

From what i've seen, rechecking paused torrents clears the auto_managed flag for them. Here's a patch that fixes that although i have a feeling it's a workaround. ping @qbittorrent/demigods

--- a/src/base/bittorrent/torrenthandle.cpp
+++ b/src/base/bittorrent/torrenthandle.cpp
@@ -1270,13 +1270,21 @@ void TorrentHandle::forceRecheck()
 {
     if (!hasMetadata()) return;

+    if (hasError())
+        m_nativeHandle.clear_error();
+    if (hasMissingFiles())
+        m_hasMissingFiles = false;
+    if (isForced()) {
+        m_startupState = NotStarted;
+        m_needsToStartForced = true;
+    }
+
+    m_nativeHandle.auto_managed(true);
     m_nativeHandle.force_recheck();
     m_unchecked = false;

-    if (isPaused()) {
+    if (isPaused())
         m_nativeHandle.stop_when_ready(true);
-        resume_impl(true, true);
-    }
 }

 void TorrentHandle::setSequentialDownload(bool b)

[glassez]

@thalieht, I do not have time to see it in detail yet, I will do it later. At least touching m_startupState is totally incorrect and it breaks related logic. "startup state" is for handling initial startup of torrent. When initialization is completed the torrent is always in "started" state.

[thalieht]

That's why i said workaround. I just wanted to get to
https://github.com/qbittorrent/qBittorrent/blob/b87987ec89bee54b2462f6a529db472b2511f65e/src/base/bittorrent/torrenthandle.cpp#L1558

[glassez]

From what i've seen, rechecking paused torrents clears the auto_managed flag for them.

It's because of resume_impl(true, true);. Can you try resume_impl(false, true); instead?

[thalieht]

Doh! I knew that but didn't think to try it. However i found this now: #8711

To get around this the torrent is placed in upload_mode and out of
automanagement.

[glassez]

Doh! I knew that but didn't think to try it. However i found this now: #8711

But since my last changes we no more rely on "checked" alert. Now it is driven by "stop_when_ready" flag and libtorrent pauses it synchronously. So we can try to revert it and use resume_impl(false, false); here.

[thalieht]

Here https://github.com/arvidn/libtorrent/issues/2759#issue-293012044 @sledgehammer999 says he tried various stuff including "stop_when_ready". Maybe wait for some feedback?

[iheartcsharp]

Ideally libtorrent should limit heavy disk operations such as moving/checking to 1 per disk. I don't understand any reason to check 2+ torrents on the same drive all at the same time. Even with SSDs performance suffers if you allow that. Also auto torrent management shouldn't really apply to these heavy disk operations. I think a better solution would be for qBittorrent to manage its own queue so that libtorrent only checks 1 torrent at a time.

For example, let's say you select 5 torrents and click Force Recheck. qBittorrent should have a separate queue where it puts those torrents in a list. It then takes the 1st torrent from that list and update the state in libtorrent. It waits until the torrent check is done, then it takes then next torrent and switches the state.

I wish I could help out but I haven't touched C/C++ in a very long time. Been a C# developer the last 15 years.

[arvidn]

@iheartcsharp I may have missed something earlier. Is there a reason the libtorrent queue is not sufficient? you can set active_checking to 1 (which is the default). Torrents that are not auto managed (i.e. have the torrent_flags::auto_managed flag set) aren't subject to any limits. Is it important that the checking limit be a special case to allow non-auto-managed torrents to be subject to it? if so, why?

The main benefit I can see with qbt managing checking torrents itself is that it can try to assess which drive a torrent belongs to, and have separate queues per drive. This is a surprisingly complicated thing to do though. Perhaps it can be made to work in the most common cases. But there's nothing that says that all files in a torrent are stored on the same drive. In fact, if you have a logical volume where the block device span multiple drives, even a single file can live on multiple drives.

[a-raccoon]

Given the present-day extreme of forced rechecking of every torrent at the same time, the ideal extreme would be to only check 1 torrent at a time, regardless of where the files live.

As I mentioned earlier, this latter behavior is possible, but you have to issue the Re-check action twice, because the first action is "Force Recheck" and then the second action is "Recheck"(without force).

So the solution, for now, is to remove that Force toggle from the first action. That's the fastest fix we can accomplish and release by next version.

[thalieht]

@sledgehammer999 in https://github.com/arvidn/libtorrent/issues/2759#issue-293012044 you said you tried various stuff including stop_when_ready() which is now used by the way. What didn't work with it? Was it the thing that patch fixed? If yes, and since we'll only support some some version after 1.1.7 as the minimum, can auto_managed be enabled again on rechecking paused torrents?
In #8711 you mention about the race condition that stop_when_ready() prevents.
From the docs:

The stop-when-ready flag fixes the inherent race condition of waiting
for the state_changed_alert and then call pause().

[glassez]

So we can try to revert it and use resume_impl(false, false); here.

My quick test didn't show any problems. It checks torrent one by one and then pauses them again.
Now I want to embed this patch in my personal build and post the PR here to get more people involved in testing.

[glassez]

10242

[Ryrynz]

Not sure if planned but would be nice to have this fix added to the 4.1.x branch.. It's kinda a big deal.

[TRaSH-]

it's closed but still not solved in 4.1.5 ?

[glassez]

How can it be fixed in version 4.1.5? This version is published before the problem has been fixed.

[TRaSH-]

/me bashes head against wall,
didn't look at the date of the last reply, though it would be fixed sooner being that it's a issue for such a long time.
thnx for the fix and i hope to see it in a release soon but before the 4.2 branch

[mzso]

@glassez commented on 2019. febr. 1. 06:53 CET:

#10242

I think this behavior should be considered for moving files via Qbittorrent. (eg: setting the location, or QBs automatic movements between download folder and target)

It looks quite obvious that it's done in parallel. I just moved a couple of torrents, not even big ones. The HDD was worked heavily, and it took a long time before even one has finished moving. After that it others finished rather soon.

[a-raccoon]

I agree mzso, but I was of the impression that file moving uses the queue system. At least it appears that way to me since I asked for the "Moving" status to be added. It may be possible that torrents with the "Force" flag set that they move concurrently instead of in queue. (Force is a flag that gets switched on and off per each torrent, which is used to override the torrent queuing system.)

[mzso]

@a-raccoon commented on 2019. máj. 6. 22:03 CEST:

I agree mzso, but I was of the impression that file moving uses the queue system. At least it appears that way to me since I asked for the "Moving" status to be added. It may be possible that torrents with the "Force" flag set that they move concurrently instead of in queue. (Force is a flag that gets switched on and off per each torrent, which is used to override the torrent queuing system.)

I didn't use use forcing for a long while now. But as per the symptoms I described It seems quite obvious that they're not queued.

[Zentriert]

You claimed this issue is closed for new version 4.1.6?

https://i.imgur.com/rExR3QJ.png

Definitely not fixed for me, I have 69 torrents with "missing files" because my external disconnected at some point and qBitorrent decided they all have to be checked again ... well it's checking them all at once and absolutely murdering the I/O speeds.

I updated to new version, it automatically started doing what's in the screenshot and I visibly watched the "checked percentage" go up for multiple torrents at once. I had to manually pause them to make it stop checking multiple at once.

[amloessb]

I can confirm the behavior that @Zentriert is seeing on version 4.1.6. I had my NAS disconnect momentarily, and I quit qBittorrent to minimize the damage. However, once I started it back up, every torrent that was disrupted is now re-checking simultaneously.

https://i.imgur.com/mnHgmBk.png

[Ryrynz]

@glassez could you take a look at this one please?

[megapro17]

@glassez could you take a look at this one please?

This program is dead, nobody gives a shit. A lot of bugs still exists in years

[Ryrynz]

Far from it, Libtorrent has significant development and this client has at least five people working on it in their own time. It's going pretty strong.

[glassez]

I'm working on it currently. But it's hard for me to fix a problem that I am not exposed personally to in everyday life. I have to recreate it on purpose. But I have very little free time.

[amloessb]

Thank you, Vladimir. I really appreciate it.

Please let me know if there is anything I can do to make it easier for you
to reproduce.

On Fri, Jun 28, 2019, 22:09 Vladimir Golovnev [e-mail]
wrote:

I'm working on it currently. But it's hard for me to fix a problem that I
am not exposed personally to in everyday life. I have to recreate it on
purpose. But I have very little free time.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/qbittorrent/qBittorrent/issues/9120?email_source=notifications&email_token=AB2XLJ3JFKZDZCH4III54Q3P43VALA5CNFSM4FGHH472YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODY3RTAQ#issuecomment-506927490,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AB2XLJ4YBADSWJIMHYRKB7TP43VALANCNFSM4FGHH47Q
.

[glassez]

Please let me know if there is anything I can do to make it easier for you to reproduce.

It would be nice if someone from affected's helped test this on its system when my job is completed. You should compile it yourself or I can provide binary (for Windows x64) with my patch.

[megapro17]

Far from it, Libtorrent has significant development and this client has at least five people working on it in their own time. It's going pretty strong.

2,526 issues isn't saying anything?

[mzso]

Far from it, Libtorrent has significant development and this client has at least five people working on it in their own time. It's going pretty strong.

2,526 issues isn't saying anything?

Well, you can't expect much more from free and open source stuff. Even Firefox, supported by an organization is plagued with bugs big and small dragged on for years and decades even.

[arvidn]

Far from it, Libtorrent has significant development and this client has at least five people working on it in their own time. It's going pretty strong.

2,526 issues isn't saying anything?

It's saying that there are a lot more users than developers. And probably some users that create a ticket for an issue that already has a ticket, from time to time.

[arvidn]

@glassez I think the solution is to not touch the auto_managed flag at all. Except for "force-start" probably, but as soon as the torrent leaves the force-started state, the auto_managed flag should be put back.

[Ryrynz]

2,526 issues isn't saying anything?

Yeah, some of these may already been long since resolved and others may not be issues with the actual client at all. Things are actually getting a lot better both on Libtorrent's side as well as qBt's, so I expect this number to dwindle down, don't let that dissuade you from qBt. 4.1.7 is coming and 4.2.0 looks like to be a big improvement, won't be long now.

[glassez]

@glassez I think the solution is to not touch the auto_managed flag at all. Except for "force-start" probably, but as soon as the torrent leaves the force-started state, the auto_managed flag should be put back.

I don't know your exact reasons for suggesting it. But we have to "play" with auto_managed/paused flags when we start a torrent to meet our needs. Of course, we should return it to its original form after all the preparations are done. Unfortunately, libtorrent does not allow us to do these things any other way.

[glassez]

Can someone test #10871?

[a-raccoon]

It would be nice if someone from affected's helped test this on its system when my job is completed. You should compile it yourself or I can provide binary (for Windows x64) with my patch.

@glassez : Any win64 binary you drop in this thread will get eager download and use. Also, would you care to reopen this issue?

[glassez]

https://yadi.sk/d/BA5Gd1y_kGS2VA

[mzso]

@a-raccoon commented on 2019. jún. 30. 14:54 CEST:

It would be nice if someone from affected's helped test this on its system when my job is completed. You should compile it yourself or I can provide binary (for Windows x64) with my patch.

@glassez : Any win64 binary you drop in this thread will get eager download and use. Also, would you care to reopen this issue?

Can you check moving torrents (setting a location on a different drive) as well?

[glassez]

Binary above is updated.

[glassez]

Can you check moving torrents (setting a location on a different drive) as well?

Maybe next time I can take a few hours off my personal life...

[mzso]

@glassez commented on 2019. jún. 30. 19:36 CEST:

Can you check moving torrents (setting a location on a different drive) as well?

Maybe next time I can take a few hours off my personal life...

Well, I was replying to @a-raccoon who seemed eager to test.

But since that's bugged as well (not enqueued at all) , if it's to be fixed a dev needs to look at it at some point.

[amloessb]

@glassez: I tried the binary provided, but it crashed immediately after asking if I wanted to make it the default application for torrents and magnet links. I have a minidump, so let me know if there's a way I can send it to you.

[Ryrynz]

Any chance of someone building a 4.1.7 release after this fix and whatever is left to backport? Do you need Sledgehammer for that? Seems he's gone off grid.

[thalieht]

This is finally fixed in 4.1.7 right?

[Ryrynz]

You did the testing didn't you? I think it's safe to close.

[a-raccoon]

Was Moving... fixed too, or do moves still happen in parallel instead of sequentially?

[Ryrynz]

They still happen in parallel.. checked with 4.20 Alpha as well. :(

[thalieht]

@Ryrynz one or more of those torrents were Forced right? I can reproduce in that case.

@a-raccoon still parallel.

[Ryrynz]

Nope, didn't force any. Simply added two torrents, downloaded and moved.

[thalieht]

Moved? This is about parallel rechecking.

[Ryrynz]

Add that to the list too then.

[arvidn]

@Ryrynz typically it's better to have tickets that are specific, and that can be closed when that specific issue has been resolved. You should probably create (or comment in an existing ticket, I'm sure there is one) about parallel moving of torrents. The topic of this ticket is checking of torrents in parallel.

[mzso]

@thalieht commented on 2019. szept. 8. 12:03 CEST:

Moved? This is about parallel rechecking.

Moving was mentioned as well, because it's a similar issue with just as painful consequences.

[mzso]

@arvidn commented on 2019. szept. 8. 13:22 CEST:

@Ryrynz typically it's better to have tickets that are specific, and that can be closed when that specific issue has been resolved. You should probably create (or comment in an existing ticket, I'm sure there is one) about parallel moving of torrents. The topic of this ticket is _checking_ of torrents in parallel.

Because of the obscene amount of issues it's not too likely for a new one to get attention.

[mzso]

@arvidn commented on 2019. szept. 8. 13:22 CEST:

@Ryrynz typically it's better to have tickets that are specific, and that can be closed when that specific issue has been resolved. You should probably create (or comment in an existing ticket, I'm sure there is one) about parallel moving of torrents. The topic of this ticket is _checking_ of torrents in parallel.

Anyway there already appears to be an issue (#9158) opened by him, then went unnoticed.

And avoiding needless moves would also help: #7916

[a-raccoon]

I think moving was also mentioned here because @glassez (et al) pull #10871 was perhaps intended to fix both issues of sequential rechecking and sequential moving. It only makes sense that these two file operations should even share the same code, especially given the qbt option to recheck a torrent after a torrent has completed (AND presumably moved from \temp to \complete).

@iheartcsharp commented on Jan 8

Ideally libtorrent should limit heavy disk operations such as moving/checking to 1 per disk. I don't understand any reason to check 2+ torrents on the same drive all at the same time. Even with SSDs performance suffers if you allow that. Also auto torrent management shouldn't really apply to these heavy disk operations. I think a better solution would be for qBittorrent to manage its own queue so that libtorrent only checks 1 torrent at a time.

File moving has been a part of this thread since the beginning.

[CyrusNajmabadi]

Is there any way for file moving to be addressed at all? I find it somewhat baffling that a library and tool written around the idea of extremely heavy IO would also be written in such a way as to hurt performance so heavily.

Does qbitorrent not have any sort of internal queues where async-io can be done? it seems like it would jsut need to enqueue these moves to that queue to be done sequentially.

It seems like this must be possible given that it was possible to "check" files one at a time. If we can "check" one a time... why can't hte same be done for moves?

[sgmihai]

Please enable some option to enqueue moves, I have 3GB/s ssd temporary download location and 1GB/s capable disk array, yet when I get 7+ moves happening all at once, everything goes to shit, takes forever to move, and even torrents transfers stop, probably because of the disk overload detection.

[Ryrynz]

I just changed a location for a label I have and rather than offer to move the torrents or just not do it (like I hoped it would) it just started moving all of them at once.

Looking at this closer though it appears as if it's actually moving them sequentially, all the torrents are marked as "Moving" but my disk performance is anywhere from 60MB/s to 90MB/s which it sure wouldn't be if it was moving them all at once. The disk contents also appear to be confirming this to be the case.

So it looks like ideally we'd just have the torrents marked "Moving" when they're being moved and when not, marked as Queued for Move.

[a-raccoon]

Ideally the progress indicator might do something too.

[ghost]

@FranciscoPombal this should be closed now right? There's a queueing system in place now for moving torrents.

[FranciscoPombal]

@an0n666 yes. The requested features are now available thanks to:
https://github.com/qbittorrent/qBittorrent/pull/10242
https://github.com/qbittorrent/qBittorrent/pull/10266
https://github.com/qbittorrent/qBittorrent/pull/12035
https://github.com/qbittorrent/qBittorrent/pull/12188