Medusa: Getting subtitles for Archived episodes --> Too many requests

[Olivier6767]

Branch: master
Commit: b16e0e5eb4ded727eebd2ab2cd9683ab597662f5
Version: v0.1.17

I get constant errors from several subtitle providers: Too many requests.
After turning on debug, I found out that Medusa is regularly trying to download subtitles for shows and episodes marked as Archived. Since I have many of them, it generates too many requests to the subtitle providers and eventually it exceeds their request quota.

More information:
I have many shows where I had downloaded the first seasons in local language (french). I did this by adding in the required words field in the Advanced season settings.
Once I had watched one season, I marked all the episodes for this season as Archived.
For the latest seasons, since the French version is usually not available upon release, I remove the required keyword and I enable Search subtitles in the Main season settings.

What I found out by reviewing the debug logs, is that Medusa still wants to download all missing subtitles for old seasons marked as Archived. This represents a huge list of subtitles.
This exceeds the subtitle provider's quota and generates daily errors for 'Too Many Requests'.
Therefore no subtitles are downloaded for newly fetched episodes.

Is this a normal behavior?
Is there a setting that would avoid Medusa trying to download old subtitles (for episodes marked as Archived)?

[p0psicles]

It's definitely not expected behavior. Will put up a fix for it shortly.

[p0psicles]

@fernandog in our sql query for episodes that should search for subtitles, i found this: "AND (e.status LIKE '%4' OR e.status LIKE '%6') ".
It looks like where looking explicitly for episodes that have a status of archived. You have any idea why?

[fernandog]

ARCHIVED means It wont snatch any other quality but If show has subtitle enabled and EP is missing subs It will download it. So ARCHIVED is not related with subs

[p0psicles]

okay, but is the right? I mean, when you archive a show, your mostly done with it right?

[fernandog]

Also It dont pick ALL missing Episodes in a each run. It limits.

Addic7ed active blocks subliminal. It can be after 1 request or 10. Please try opensubtiles

[fernandog]

You are done with snatches. Not subs. Changing this wont make Addic7ed works.

[p0psicles]

No I understand. But users assume that when archiving certain episodes, medusa is done with them. I'm just challenging if there is a valid use case, where you archive certain episodes, but still want to keep searching subtitles for them?

Alternative could be that we inform users somehow, that changing episodes to archived, will still have medusa search for them.

There is currently no option for users to configure, not to have medusa search for a subset of episodes of a show. You can disable subtitles only on a show level. Same goes for disabling a show.

[Olivier6767]

Addic7ed actually works fine, it's NOT blocking subliminal.
I get subtitles downloaded from addic7ed from time to time IF it has been a few days after it was flooded by a full search or achived eipodes missing subtitles. I also tried manually using subliminal to confirm this.
For information I have benn using sickrage for years before swithcing to Medusa and never had issues getting subtitles frrom addic7ed or other providers. It uses the same subliminal engine. I guess it did not try to get older subtitles for achived shows.

I'm also using open subtitles but I suspect they also have limits because I get very often this message
2017-11-09 03:26:29 WARNING POSTPROCESSOR :: [b16e0e5] Unexpected error in provider u'opensubtitles'
Traceback (most recent call last):
File "/home/Services/cinema/medusa/Medusa/ext/subliminal/core.py", line 119, in list_subtitles_provider
return self[provider].list_subtitles(video, provider_languages)
File "/home/Services/cinema/medusa/Medusa/ext/subliminal/core.py", line 69, in __getitem__
provider.initialize()
File "/home/Services/cinema/medusa/Medusa/ext/subliminal/providers/opensubtitles.py", line 139, in initialize
'subliminal v%s' % __short_version__))
File "/usr/lib/python2.7/xmlrpclib.py", line 1233, in __call__
return self.__send(self.__name, args)
File "/usr/lib/python2.7/xmlrpclib.py", line 1591, in __request
verbose=self.__verbose
File "/usr/lib/python2.7/xmlrpclib.py", line 1273, in request
return self.single_request(host, handler, request_body, verbose)
File "/usr/lib/python2.7/xmlrpclib.py", line 1321, in single_request
response.msg,
ProtocolError:

The question is really if it makes sense to still search for subtitles for archived episodes, because typically you're down with this show.

If not, is there another status that can be used for this?

[p0psicles]

@pymedusa/developers anyone wants to pitch in here?

[Olivier6767]

Could we implement a checkbox in the subtitles settings asking if we should still search for subtitles for Archived episodes?
This way everyone can choose the best option for their use case.

[p0psicles]

Sickrage only downloads subtitles for episodes with a status of downloaded.

[fernandog]

It blocks subliminal based on how many subliminal users are requesting subs not only single users. Trust me, we did tons of tests

[Olivier6767]

I trust you. I'm just saying that the current settings of trying to gather subtitles for older/archived episodes generates more requests, thus hitting the subliminal limits quicker.
If there is a use case to still search for subtitles for Archived episodes, maybe the checkbox option is the most flexible.

[p0psicles]

First i'd like to challenge the requirement of downloading subs for archived eps. If nobody can come up with a use case, we just change it. Then as a backup a checkbox can always be an option. But if possible i'd like to not include another option.

[Olivier6767]

ok, thank you.

[fernandog]

The use case is when user decide to download subs after get all episodes downloaded.

In this issue, user installed Medusa after he had all episodes already downloaded.
So by default, Medusa will try to download subtitles because subtitles are enabled.
Personally I don't see an issue with that, its by design.

[Olivier6767]

If the user user decides to download subs after all episodes are downloaded, this will happen automatically as long as the episodes are in Downloaded status. The question is if the subs should get downloaded even after episodes are marked as Archived. I think it should not.

Then, episodes can still be changed back from Archived to Downloaded if the user wants to fetch subs for them after the fact.

This would provide a more flexible option allowing for both uses cases.

[NicoLeOca]

Hi,
I use subtitles on a regular basis and I agree that Medusa shouldn't search for subtiles of archived episodes.

[medariox]

The change has been merged to develop and will be available in master in the next release.
https://github.com/pymedusa/Medusa/pull/3266

Thanks for the feedback!

[fernandog]

Latest subliminal which is already in master wont raise Too many requests anymore. It will debug log that provider didn't replied with any data

It will only raise that when a REAL Too Many Requests happens