Jackett: [enhancement] asynchronous search - reduce the impact of slow or down indexers

[ekaunt]

Environment

OS: Arch Linux. kernel 5.5

Jackett Version: 0.14.418.0

Are you using a proxy or VPN? no

Description

Searches take about 2 minutes to complete. this makes this service nearly unusable because if i mispell something i need to wait 2 full minutes before i get the wrong results back, then another 2 minutes till the correct results show. if i need to switch search terms then thats another 2 minutes, and if i need to switch it up again because im not sure how torrents have the title so i need to check various different wordings for the same thing, i need to wait 2 minutes between each search…

I have 130 indexers, of which, 5 fail the Test (125/130 actually work).

When i only select one indexer to search it goes really quickly. the more i enable the slower the search gets.

my proposed solution is to make searches go asynchronously. instead of waiting for all trackers to complete, and only then show the results, whichever tracker finishes the search first gets to show its results, and as all the trackers finish searching, they populate the search results list. there can maybe be an indicator that shows when the last tracker finished loading.

[garfield69]

Are you talking about the aggregated indexer ALL ?
I do not believe that asynchronous calls to each separate indexer for the ALL aggregate indexer is suitable, because Jackett is primarily a Torznab provider to apps, with the manual dashboard search just one of the features, but not the main reason for Jackett existence.
So Apps like Sonarr, Radarr, DuckieTV etc, expect that a Torznab query they make to Jackett via the Torznab API will return a single complete result response.
Having partial results returned as each asynchronous tasks completes, would probably cause these Apps to either accept the first response or the last response, and ignore all the others.

I invite the c# devs to confirm or correct my statement, @cadatoiva @ngosang your input welcome.
Thanks.

[ekaunt]

Are you talking about the aggregated indexer ALL

Yes

Maybe there should be a difference between the gui and the API so ppl like me can use it as an indexer of indexers

[cadatoiva]

Current implementation does not support the ability to implement this enhancement without a large amount of difficulty involved, because of what @garfield69 mentioned. A feature similar to this is something in the works though, however it involves some major changes to the entire code base before you'll be able to see the difference on the front end.

What we can do once we get the underlying support is to allow all of the trackers to run asynchronously until the requested limit is reached, and then pause processing unless/until more is requested. This would mean that since Sonarr/Radarr only request ~100 results come back, running /all will stop and return what it's got so far once 100 non-filtered results are grabbed. Similarly, if you aren't sorting your results, the web interface defaults to only displaying 20-100 results at a time, so we can pause the search after the first page is populated until you browse to the next page of results, where it will then process the next 20-100 results.

[ekaunt]

Similarly, if you aren't sorting your results, the web interface defaults to only displaying 20-100 results at a time, so we can pause the search after the first page is populated until you browse to the next page of results, where it will then process the next 20-100 results.

That sounds like a great idea! Is this already available, or is this the thing thats still in the works? if its still being worked on, what an eta? (no rush. you guys are doing it for free on your own free time after all)

[cadatoiva]

This is a thing that's in the works, and a long way off. Possibly months or a year or more.

[ekaunt]

@cadatoiva do you know if theres a program that can search through many trackers (including private trackers) simultaneously? like jackett, but one that doesnt take 2m each search?

[RoloSoze]

@ekaunt I have over 150 indexers configured in Jackett, and it typically takes me less than 1 minute per query to search them all. There were times when it used to take more than that, even though I had fewer configured indexers, and I could always trace it to one or two public indexers that were slowing the whole search down, in which case I would just remove them permanently because ironically, I found that most of the public trackers that were slowing me down were useless for my searches anyways. This is precisely why I choose not to blindly add all supported public trackers.

If most of your configured indexers are public trackers, you may want to remove the ones that are consistently slowing you down, especially if they don't usually return your desired results. It can be tedious to identify the slow ones, especially that just because a tracker is slow right now does not necessarily mean it will always be slow, but doing so will cut down your average search duration. Follow the steps that @ilike2burnthing suggested in the comment right below this one; that would be a good way to start systematically. Most of my configured indexers are private, and while private trackers are obviously not immune from downtimes or the occasional degrading performance, I have found them to be more reliable and better performing than public ones, for the most part.

Something else to consider - Even if you decide to keep a large number of configured indexers, you don't always need to search them all. I have learned over time that the 80/20 rule applies accurately to trackers - you're likely to get your desired search results 80% of the time, or more, from 20% of your configured trackers. You just need to develop a better understanding of which of your 125 functioning indexers return your desired results more often. For this reason, I mostly search a little over half of my configured indexers (~80) using a workaround I came up with (https://github.com/Jackett/Jackett/issues/661#issuecomment-574404283). I honestly don't even need to search half since my top 20 private ones return what I want 90%+ of the time, but I search half anyway (all but 3 are private) because it's fast enough - takes 20 to 30 seconds per search. This faster search duration compared to searching all my 150 indexers isn't only because I'm searching fewer indexers, but also because all but 3 are private. I understand this workaround is not ideal, but until a developer adds changes to the UI to allow users to do #661 | #2185 | #4787 (3rd point), this will do for now.

I still think your suggestion is valid, and the improvements @cadatoiva talked about are crucial to speeding up Jackett's overall performance, but until that happens, I suggest considering my 2 suggestions above.

[ilike2burnthing]

If you're only using it for manual searches and downloads, rather than passing on to another application (e.g. Sonarr), and you're happy with a predefined set of (public) indexers, then try DuckieTV and enable Torrent Dialog multiple SE in Settings > Torrent Search Engines.

You can link Jackett to DuckieTV as well to expand the number of indexers. As searches (using the built-in indexers) are asynchronous, if you get your results early then great, if not then you can wait for Jackett to finish to find them.

Honestly, I think a better solution for you would be to trim down the number of indexers you're using in Jackett. Start with the 5 that don't work, then test the rest individually and see which ones take the longest. Remove them (either from Jackett or just from a custom search) and see if that improves things for you.

EDIT: Rolo beat me to it by a few seconds 😋

[cadatoiva]

@ekaunt, to add to what @RoloSoze and @ilike2burnthing, I don't know of any other torrent aggregator that does what Jackett does, which is a big reason why I jumped in as a developer. Everything I've seen that comes close tends to be either aggregator sites (like Torrentz2) which are conveniently included in Jackett, or some kind of bundled addon for a torrent client and I like my current client too much to move away from it. The only contenders are programs like Cardigann which has been abandoned and pulled into Jackett.

[ekaunt]

What torrent client has which addon thats an aggregator?

[cadatoiva]

@ngosang develops for qBittorrent and that has one. Vuze has a search built in for sure and I believe μTorrent as well.