Medusa pushing the same episode to downloader multiple times

[OmgImAlexis]

Apparently this is a known issue that we've had with sab. Currently Medusa seems to push the same file to sab 1 - 4 times.

@duramato @labrys can confirm

[labrys]

I know this happens with failed downloads. I think it also happens with backlog. I do not believe daily causes it though.

[fernandog]

@OmgImAlexis @Labrys
Logs are pretty descriptive now (including backlog). Also tons of tests for the methods

So without logs there is nothing to do and I will remove the bug label.

Increase number of logs and size. There is no excuse about logs roll over.

[fernandog]

@labrys if you know it happens why no report with logs?

[OmgImAlexis]

Gist of search "Gotham.S03E13": https://gist.github.com/OmgImAlexis/8771fbc15f79867490169763e087e741

[OmgImAlexis]

Nothing at all in my failed history and you can see that in the screenshot Medusa is snatching the same file twice.

[fernandog]

Come on. You are not a user.
Need logs around the time both snatches.
Cant be PP loga as PP doesnt snatch it.

[OmgImAlexis]

@fernandog and I think we found the issue and it's that somewhere show updater or show refresh is removing the snatched state on episodes when it gets run.

The episodes for example would be on DOWNLOADED 1080p HDTV and then snatch a file so it'd then be SNATCHED 1080p WEB-DL but then the show update/refresh would run and set it back to DOWNLOADED 1080p HDTV meaning it gets snatched again.

Edit: That's wrong we found the issue.

[fernandog]

@labrys is there a reason to re-open this?

[labrys]

Yes, because the issue still exists. If you have a fix for it, then PR the fix and reference the issue. If someone finds this issue then they see something other than just a closed issue, but that a fix was submitted.

[fernandog]

Its already in develop. Is already referenced

See this above:
fernandog referenced this issue from commit 6966046 about 10 hours ago
fernandog referenced this issue from commit 51ff22a about 9 hours ago

Closing again

If you have logs after that commit please send it

[labrys]

Now its referenced. See above my comment? That's a proper reference since it can be found from the issue.

[labrys]

Odd. Your references to the commit don't show for me unless I log out.
Actually now I remember why I have you on my ignore list.

[labrys]

Anyways, reopening as this is still happening on current develop.

If you want logs I'll e-mail them to you but im not posting 100 MB worth of logs.

[labrys]

Here's a grep of the logs showing lines with The, 100, and s04e01.
https://gist.github.com/labrys/df4524252c36f392244f947a77ae6324

[OmgImAlexis]

Looks like it's gotta be another issue somewhere else since it went down to only snatching a single duplicate for me so at least the snatched best seems to be fixed.

[fernandog]

@labrys @OmgImAlexis
its not an issue:

each NZB has different size so its a NEW nzb. Just check logs:
ping @duramato

2017-02-02 04:28:53 DB       SEARCHQUEUE-RETRY-268592 :: [DOGnzb] :: [34f47b6] failed.db: SELECT release FROM failed WHERE release=? AND size=? AND provider LIKE ? LIMIT 1 with args [u'The_100_S04E01_720p_HDTV_x264_AVS', 1754851025, u'DOGnzb']
2017-02-02 04:28:53 DB       SEARCHQUEUE-RETRY-268592 :: [NZB.Cat] :: [34f47b6] failed.db: SELECT release FROM failed WHERE release=? AND size=? AND provider LIKE ? LIMIT 1 with args [u'The_100_S04E01_720p_HDTV_x264_AVS', 1040404801, u'NZB.Cat']
2017-02-02 04:28:53 DB       SEARCHQUEUE-RETRY-268592 :: [SickBeard Index] :: [34f47b6] failed.db: SELECT release FROM failed WHERE release=? AND size=? AND provider LIKE ? LIMIT 1 with args [u'The_100_S04E01_720p_HDTV_x264_AVS', 1755032493, u'SickBeard Index']

[labrys]

@fernandog It is an issue if its queuing multiple copies of an episode. It doesn't matter if they are different releases, its queuing more than one release at a time. Sure if a user used manual selection to send multiple then it would not be an issue. Automation grabbing multiples of the same episode though most certainly is an issue.

[fernandog]

Doesnt matter if is same episode. It another release. All others failed.

@duramato can you explain please?

[p0psicles]

What's the issue here?
Does it:
snatch -> download -> pp (failed) -> snatch -> download .
or
snatch -> download -> snatch -> download -> pp(failed) -> pp (failed)

If the second, yes that's an issue. If the first, I suspect it's working as designed.

[fernandog]

As all releases have same quality is scenario 1
medusa can't snatch a new one unless is a preferred quality OR manually snatched

[labrys]

@p0psicles more like the second since the same episode is queued multiple times concurrently.
The bottom image is my history. The first download failed, the 2nd, 3rd, and 4th passed, the 5th failed and I still have the episode in my queue 3 more times (the top image).

[duramato]

The issue is it snatches other releases for the same episode regardless if it's already snatched. This is an issue and has happen to me in the past.

[fernandog]

It only snatched second and third because of the failed!
Check logs! RETRY thread that snatched it

[fernandog]

Wasn't daily or backlog that snatched but SEARCHQUEUE-RETRY because it was a failed release

So unless someone show me logs for DAILY or BACKLOG snatching a second release for the same quality when there current status is SNATCHED, i'm closing this.

Logs:

2017-02-02 04:21:16 INFO     SEARCHQUEUE-RETRY-268592 :: [34f47b6] Downloading The.100.S04E01.PROPER.720p.HDTV.x264-FLEET from NZB.Cat
2017-02-02 04:24:59 INFO     SEARCHQUEUE-RETRY-268592 :: [34f47b6] Downloading The.100.S04E01.PROPER.720p.HDTV.x264-FLEET from NZB.Cat
2017-02-02 04:28:53 INFO     SEARCHQUEUE-RETRY-268592 :: [34f47b6] Downloading The.100.S04E01.PROPER.720p.HDTV.x264-FLEET from DOGnzb
2017-02-02 05:15:53 INFO     SEARCHQUEUE-RETRY-268592 :: [34f47b6] Downloading The.100.S04E01.720p.HDTV.x264-AVS from DOGnzb

[labrys]

@fernandog Regardless of if it was snatched because of a retry, daily or backlog, there should still only ever be a single active snatch unless a user forcibly added more than one to their queue (e.g. through selecting multiple results in manual snatch). The automation should never queue up more than one snatch at a time.

[labrys]

Also if you notice the logs don't cover all the snatches due to roll-over, and no I'm not extending my logs further because I'm not logging more than 100 MB of logs (50x 2MB).

[fernandog]

@labrys

this is the flow:
snatch -> download -> pp (failed) -> snatch -> download >> pp OK

PP that marked as failed then the RETRY thread that snatchs another (only one active snatch)

This is how is it today.

If you don't agree please write down a new flow so we all can discuss it

[labrys]

I am telling you thats not the way it is actually working or there would not be 3 active copies in my queue at once. If it worked correctly as you described it would not be a bug.

[labrys]

Your flow that you describe is how its supposed to work. But that is not what is actually happening.

[fernandog]

@labrys prove then. you are opening the issue: logs, screenshot of history with two snatches and etc

don't waste more my time with only threory and guessing

[labrys]

How much better proof than a screenshot of multiple copies in queue at a single time? Stop closing an issue just because you don't understand it.

[fernandog]

I asked for medusa SS if you cant provide, so sorry

[p0psicles]

In stead of playing this yes/no game. Let's try to use these Brains, and come up with a plan to isolate the issue, and come with a solution okay? So for the people who are sure the recently experienced this. Let's analyze these logs?

[labrys]

@p0psicles I've cleared my logs and purged my sab queue and have reset all my shows in a snatched state. All I can do currently is wait, but if I don't catch it soon after it occurs then I won't have any logs available.

[p0psicles]

The increase your log file size. Set it to 25mb per file. You'll have up to 5 or 7 log files. Trust me, you'll have plenty of time.

[OmgImAlexis]

@p0psicles I made mine 10 logs with 50mb each and there wasn't anything. It looks like logging may be missing from whatever is broken here.

[fernandog]

@OmgImAlexis @labrys fixed right?

[labrys]

I havent had the time to check yet, so I can neither confirm nor deny.

[fernandog]

will close, if happens again, open.

[OmgImAlexis]

From what I can see nothing has double snatched since this. 👍

[muheras]

I hate to be the bearer of bad news, but this seems to be happening again, lately downloading 4 or 5 versions of the same file.

Link to a full 3-day log that shows duplicated downloads for Bob's Burgers and Suites Episodes as well - https://gist.github.com/muheras/8a81b290dc0c4757c1a4f2c868a596d0

2019-02-14 04:41:22 INFO SNATCHQUEUE-SNATCH-311714 :: [918cfe7] Downloading This.Is.Us.S03E12.1080p.WEB.H264-METCON with size: 3.35 GB from NZBs.org, through a SearchType.DAILY_SEARCH search 2019-02-14 04:41:22 INFO SNATCHQUEUE-SNATCH-311714 :: [918cfe7] Beginning to snatch release: This.Is.Us.S03E12.1080p.WEB.H264-METCON 2019-02-14 04:41:20 INFO SEARCHQUEUE-DAILY-SEARCH :: [918cfe7] Downloading This.Is.Us.S03E12.1080p.WEB.H264-METCON with size: 3.35 GB from NZBs.org 2019-02-14 04:41:19 INFO SEARCHQUEUE-DAILY-SEARCH :: [NZBs.org] :: [918cfe7] Picked This.Is.Us.S03E12.1080p.WEB.H264-METCON as the best result. 2019-02-14 04:41:19 INFO SEARCHQUEUE-DAILY-SEARCH :: [NZBs.org] :: [918cfe7] Quality of This.Is.Us.S03E12.1080p.WEB.H264-METCON is 1080p WEB-DL 2019-02-13 12:41:26 INFO SNATCHQUEUE-SNATCH-311714 :: [918cfe7] Downloading This.Is.Us.S03E12.1080p.WEB.H264-METCON with size: 3.35 GB from NZBs.org, through a SearchType.DAILY_SEARCH search 2019-02-13 12:41:26 INFO SNATCHQUEUE-SNATCH-311714 :: [918cfe7] Beginning to snatch release: This.Is.Us.S03E12.1080p.WEB.H264-METCON 2019-02-13 12:41:25 INFO SEARCHQUEUE-DAILY-SEARCH :: [918cfe7] Downloading This.Is.Us.S03E12.1080p.WEB.H264-METCON with size: 3.35 GB from NZBs.org 2019-02-13 12:41:25 INFO SEARCHQUEUE-DAILY-SEARCH :: [NZBs.org] :: [918cfe7] Picked This.Is.Us.S03E12.1080p.WEB.H264-METCON as the best result. 2019-02-13 12:41:25 INFO SEARCHQUEUE-DAILY-SEARCH :: [NZBs.org] :: [918cfe7] Quality of This.Is.Us.S03E12.1080p.WEB.H264-METCON is 1080p WEB-DL 2019-02-13 09:19:48 INFO SEARCHQUEUE-BACKLOG-311714 :: [NZB.Cat] :: [918cfe7] Quality of This.Is.Us.S03E12.HDTV.x264-SVA is SDTV 2019-02-13 09:19:48 INFO SEARCHQUEUE-BACKLOG-311714 :: [NZB.Cat] :: [918cfe7] Quality of This.Is.Us.S03E12.720p.HDTV.x264-AVS is 720p HDTV 2019-02-13 09:19:48 INFO SEARCHQUEUE-BACKLOG-311714 :: [NZB.Cat] :: [918cfe7] Quality of This.Is.Us.S03E12.720p.HDTV.x264-AVS is 720p HDTV 2019-02-13 09:19:48 INFO SEARCHQUEUE-BACKLOG-311714 :: [NZB.Cat] :: [918cfe7] Quality of This.Is.Us.S03E12.720p.HDTV.x264-AVS is 720p HDTV 2019-02-13 09:19:48 INFO SEARCHQUEUE-BACKLOG-311714 :: [NZB.Cat] :: [918cfe7] Quality of This.Is.Us.S03E12.720p.HDTV.x264-AVS is 720p HDTV 2019-02-13 09:19:48 INFO SEARCHQUEUE-BACKLOG-311714 :: [NZB.Cat] :: [918cfe7] Quality of This.Is.Us.S03E12.iNTERNAL.720p.WEB.H264-AMRAP.English is 720p WEB-DL 2019-02-13 09:19:48 INFO SEARCHQUEUE-BACKLOG-311714 :: [NZB.Cat] :: [918cfe7] Quality of This.Is.Us.S03E12.iNTERNAL.720p.WEB.H264-AMRAP is 720p WEB-DL 2019-02-13 09:19:48 INFO SEARCHQUEUE-BACKLOG-311714 :: [NZB.Cat] :: [918cfe7] Quality of This.Is.Us.S03E12.720p.HDTV.x264-AVS is 720p HDTV 2019-02-13 09:19:47 INFO SEARCHQUEUE-BACKLOG-311714 :: [NZB.Cat] :: [918cfe7] Quality of This.Is.Us.S03E12.iNTERNAL.720p.WEB.H264-AMRAP is 720p WEB-DL 2019-02-13 09:19:47 INFO SEARCHQUEUE-BACKLOG-311714 :: [NZB.Cat] :: [918cfe7] Quality of This.Is.Us.S03E12.720p.HDTV.x264-AVS is 720p HDTV 2019-02-13 03:20:30 INFO DAILYSEARCHER :: [918cfe7] Setting status (Wanted) for show airing today: This Is Us - S03E12 - Songbird Road (2)

[labrys]

@muheras Even though the issue was closed in Feb of 2017 the above fix never fixed the issue, I just got tired of arguing about it.

[p0psicles]

No this issue was fixed. But a bug got reintroduced in master branch. But has already been fixed in develop.

[Forceflow]

I'm seeing this behavior again as well. (latest main version, no dev)

When I manually set an (already downloaded) episode to wanted, thus pushing it into the backlog, I see Medusa furiously queing it for download multiple times.

@p0psicles : Could you point me to the specific commit in which this was fixed?

[p0psicles]

Here you go https://github.com/pymedusa/Medusa/commit/63d31d45e07e69e65355ccf395368092ecd03827

[Forceflow]

@p0psicles: Thanks man, appreciate it. Any way of getting this fix in the current release already? It's bothering me, but switching to develop branch completely probably comes with its own problems...

Here's a log of the problem: https://pastebin.com/Bp6HJ2kk

[p0psicles]

Actually a release is not that far away

[Forceflow]

@p0psicles Good to hear. Keep on doing excellent work!

I see you'd rather have your donations go to the Red Cross. I'll donate now, but you sure you also don't want some beer money?

[p0psicles]

Nah, I actually much rather have others getting involved like your self, helping making our software better. Tnx!

[Forceflow]

@p0psicles : I know python, but the Medusa codebase at the moment is a bit too difficult for me to grasp. I'd gladly help, but this issue seemed pretty complicated. I'll see if I can start working on some smaller issues instead.