Medusa: TorrentLeech provider not working correctly after their redesign

[rssfed23]

TL recently did a redesign. As a result I think some of their category numbers/APIs have changed and this is causing issues finding shows. Medusa either can no longer find anything that I know exists in TL or can only find 1080p (even when I know a 720p version exists).

Branch/Commit: 77b20916ab577b82112ebc624f816054788c63f2
OS: Fedora 27
What you did: Initiated a manual search for an episode I know exists on TorrentLeech
What happened: It will either return no results due to the below error or it will return only 1080p results when I know that 720p exists

Logs:

2018-04-23 21:13:12 INFO     SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] Performing episode search for Family Guy
2018-04-23 21:13:12 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] Search mode: Episode
2018-04-23 21:13:12 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] Search string: Family Guy S16E16
2018-04-23 21:13:12 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] GET URL: https://classic.torrentleech.org/torrents/browse?query=Family+Guy+S16E16&categories=2%2C7%2C35%2C26%2C32%2C44 [Status: 503]
2018-04-23 21:13:12 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] User-Agent: Medusa/0.2.2 (Linux; 4.15.17-300.fc27.x86_64; 1fa73858-4731-11e8-9b9c-9e0a979a5796)
2018-04-23 21:13:12 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] CloudFlare protection detected, trying to bypass it
2018-04-23 21:13:20 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] GET URL: https://classic.torrentleech.org/torrents/browse?query=Family+Guy+S16E16&categories=2%2C7%2C35%2C26%2C32%2C44 [Status: 503]
2018-04-23 21:13:20 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:60.0) Gecko/20100101 Firefox/60.0
2018-04-23 21:13:20 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] The response returned a non-200 response while requesting url https://classic.torrentleech.org/torrents/browse. Error: HTTPError(u'503 Server Error: Service Temporarily Unavailable for url: https://classic.torrentleech.org/torrents/browse?query=Family+Guy+S16E16&categories=2%2C7%2C35%2C26%2C32%2C44',)
2018-04-23 21:13:20 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] No data returned from provider
2018-04-23 21:13:20 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] Found 0 unique items
2018-04-23 21:13:20 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] Found qualities: []
2018-04-23 21:13:20 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] No unknown qualities in results
2018-04-23 21:13:20 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] Fallback season pack search initiate
2018-04-23 21:13:21 INFO     SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] Performing season pack search for Family Guy
2018-04-23 21:13:21 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] Found 0 unique items
2018-04-23 21:13:21 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] Found qualities: []
2018-04-23 21:13:21 DEBUG    SEARCHQUEUE-MANUAL-75978 :: [TorrentLeech] :: [77b2091] No unknown qualities in results

[ward0]

@rssfed23 why don't you use Jackett?

[sharkykh]

I can probably fix the provider + test it, I just need some time to work on it.

[ghost]

Same issue here

[djfoley01]

@sharkykh Thank you for working on this issue, I'm still experiencing an issue I'm hoping you can point me in the right direction. Here is the debug log output from medusa:

2018-04-30 02:09:13 DEBUG    SEARCHQUEUE-MANUAL-269586 :: [TorrentLeech] :: [77b2091] GET URL: https://www.torrentleech.org/torrents/browse/list/categories/2,7,35,26,32,44/query/Brooklyn%20Nine-Nine%20S05E19/orderby/seeders/order/desc/ [Status: 503]
2018-04-30 02:09:13 DEBUG    SEARCHQUEUE-MANUAL-269586 :: [TorrentLeech] :: [77b2091] User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0
2018-04-30 02:09:13 DEBUG    SEARCHQUEUE-MANUAL-269586 :: [TorrentLeech] :: [77b2091] The response returned a non-200 response while requesting url https://www.torrentleech.org/torrents/browse/list/categories/2,7,35,26,32,44/query/Brooklyn Nine-Nine S05E19/orderby/seeders/order/desc/. Error: HTTPError(u'503 Server Error: Service Temporarily Unavailable for url: https://www.torrentleech.org/torrents/browse/list/categories/2,7,35,26,32,44/query/Brooklyn%20Nine-Nine%20S05E19/orderby/seeders/order/desc/',)
2018-04-30 02:09:13 DEBUG    SEARCHQUEUE-MANUAL-269586 :: [TorrentLeech] :: [77b2091] No data returned from provider
2018-04-30 02:09:13 DEBUG    SEARCHQUEUE-MANUAL-269586 :: [TorrentLeech] :: [77b2091] Found 0 unique items
2018-04-30 02:09:13 DEBUG    SEARCHQUEUE-MANUAL-269586 :: [TorrentLeech] :: [77b2091] Found qualities: []
2018-04-30 02:09:13 DEBUG    SEARCHQUEUE-MANUAL-269586 :: [TorrentLeech] :: [77b2091] No unknown qualities in results
2018-04-30 02:09:13 DEBUG    SEARCHQUEUE-MANUAL-269586 :: [77b2091] 269586: Formatting pattern: %SN - S%0SE19 - %EN -> Brooklyn Nine-Nine - S05E19 - Bachelor-ette Party
2018-04-30 02:09:13 INFO     SEARCHQUEUE-MANUAL-269586 :: [77b2091] Unable to find manual results for: Brooklyn Nine-Nine - S05E19 - Bachelor-ette Party

Hitting the url in the debug output manually I get results.

{"numFound":2,"torrentList":[{"fid":"1072978","filename":"Brooklyn.Nine-Nine.S05E19.720p.WEB.x264-TBS.torrent","name":"Brooklyn Nine-Nine S05E19 720p WEB x264-TBS","addedTimestamp":"2018-04-30 00:51:52","categoryID":32,"size":570217452,"completed":379,"seeders":350,"leechers":63,"numComments":0,"tags":["Action","comedy","Crime"],"new":true,"imdbID":"","rating":0,"genres":"","tvmazeID":"e1430058","igdbID":""},{"fid":"1072977","filename":"Brooklyn.Nine-Nine.S05E19.WEB.x264-TBS.torrent","name":"Brooklyn Nine-Nine S05E19 WEB x264-TBS","addedTimestamp":"2018-04-30 00:51:29","categoryID":26,"size":263742517,"completed":99,"seeders":88,"leechers":15,"numComments":0,"tags":["Action","comedy","Crime"],"new":true,"imdbID":"","rating":0,"genres":"","tvmazeID":"e1430058","igdbID":""}],"facets":{"tags":{"name":"tags","title":"Tags","type":"field","items":{"Action":2,"Crime":2,"comedy":2}},"size":{"name":"size","title":"Size","type":"range","items":{"[0 TO 786432000]":{"count":2,"label":"0MB - 750MB"},"[786432000 TO 1610612736]":{"count":0,"label":"750MB - 1.5GB"},"[1610612736 TO 4831838208]":{"count":0,"label":"1.5GB - 4.5GB"},"[4831838208 TO 16106127360]":{"count":0,"label":"4.5GB - 15GB"},"[16106127360 TO *]":{"count":0,"label":"More than 15GB"}}},"added":{"name":"added","title":"Added","type":"range","items":{"[NOW\/MINUTE-24HOURS TO NOW\/MINUTE+1MINUTE]":{"count":0,"label":"Last 24hrs"},"[NOW\/MINUTE-48HOURS TO NOW\/MINUTE+1MINUTE]":{"count":0,"label":"Last 48hrs"},"[NOW\/MINUTE-72HOURS TO NOW\/MINUTE+1MINUTE]":{"count":0,"label":"Last 72hrs"},"[NOW\/HOUR-7DAYS TO NOW\/HOUR+1HOUR]":{"count":0,"label":"Last week"},"[NOW\/HOUR-14DAYS TO NOW\/HOUR+1HOUR]":{"count":0,"label":"Last 2 weeks"},"[NOW\/HOUR-1MONTH TO NOW\/HOUR+1HOUR]":{"count":0,"label":"Last month"}}},"seeders":{"name":"seeders","title":"Seeders","type":"range","items":{"[0 TO 50]":{"count":0,"label":"0 - 50"},"[51 TO 200]":{"count":2,"label":"51 - 200"},"[201 TO *]":{"count":0,"label":"More than 200"}}},"name":{"name":"name","title":"HD Resolution","type":"range","items":{"*720p*":{"count":1,"label":"720p"},"*1080p*":{"count":0,"label":"1080p"},"*2160p*":{"count":0,"label":"2160p"}}}},"facetswoc":{"categoryID":{"name":"categoryID","title":"CategoryID","type":"field","items":{"26":1,"32":1}}},"order":"desc","orderBy":"seeders","page":1,"perPage":100,"lastBrowseTime":"1525048733","userTimeZone":"UTC"}

[sharkykh]

As I wrote on the Pull Request which is not yet merged, the provider currently doesn't work ~because of an issue in the CloudFlare-protection bypass package we use, it needs to get updated before this provider can be used again.~

[sharkykh]

Actually @djfoley01, how did you install Medusa? What OS are you using?

[djfoley01]

@sharkykh I ran the docker build, so the base image is the alpine.python:3.7 but the host OS is CentOS 7 (3.10.0-693.21.1.el7.x86_64)

[ghost]

@sharkykh
Manually adding the fix you mentioned (Using the developer build) gives me an error when starting Medusa:

2018-05-01 00:26:43 INFO     EVENT-QUEUE :: [349a87f] Restarting Medusa with ['/usr/bin/python', u'/storage/.kodi/addons/service.medusa/medusa/start.py', '--datadir=/storage/.kodi/userdata/addon_data/service.medusa', '--port=8085', '--nolaunch']
['/usr/bin/python', u'/storage/.kodi/addons/service.medusa/medusa/start.py', '--datadir=/storage/.kodi/userdata/addon_data/service.medusa', '--port=8085', '--nolaunch']
Traceback (most recent call last):
  File "/storage/.kodi/addons/service.medusa/medusa/start.py", line 6, in <module>
    from medusa.__main__ import main
  File "/storage/.kodi/addons/service.medusa/medusa/medusa/__main__.py", line 67, in <module>
    from medusa import (
  File "/storage/.kodi/addons/service.medusa/medusa/medusa/providers/__init__.py", line 16, in <module>
    from medusa.providers.torrent import (
  File "/storage/.kodi/addons/service.medusa/medusa/medusa/providers/torrent/__init__.py", line 42, in <module>
    from medusa.providers.torrent.json import (
ImportError: cannot import name torrentleech
.kodi/addons/service.medusa/medusa/

[sharkykh]

@djfoley01 Sorry, nevermind. I thought the issue was something else. Your install is probably using git so you should be fine once we fix the issue with cloudflare.

@brendanssound I don't know what you did exactly. You seem to be missing the main file which is medusa/providers/torrents/json/torrentleech.py. Note that the folder is json, and not html - the file was moved.

[rssfed23]

I feel I should add there’s two issues here: the CF workaround as detailed above but the catagories themselves are still incorrect so when not experiencing the CF issue you may still hit that one (it’ll return 1080p results only when doing a manual search even though 720p ones are present when the show itself is set to 720p). Just don’t want the Cf fix to merge and have this issue closed as a result :)

[djfoley01]

@sharkykh Thank you I will keep an eye out, I appreciate your work on this

[sharkykh]

@rssfed23
I promise to keep this issue open until the issues are resolved.
But IMO there's a chance the CloudFlare fix will fix both issues.

[ghost]

@sharkykh
Yep you pretty much nailed it. I assumed the html file you added was re-done and just made a new .py file in that folder with the content you added.. Fixing that by getting rid of the file in html folder and putting it in json folder got Medusa starting again but back to

2018-05-01 02:15:01 INFO     SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] Performing episode search for Modern Family
2018-05-01 02:15:01 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] Creating UNIQUE URL index for torrentleech
2018-05-01 02:15:01 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] POST URL: https://www.torrentleech.org/user/account/login [Status: 503]
2018-05-01 02:15:01 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] User-Agent: Medusa/0.2.2 (Linux; 4.11.12; 77473c19-4c91-11e8-9fd0-74d435a92093)
2018-05-01 02:15:01 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] With post data: username=**********&remember_me=on&login=submit&password=**********
2018-05-01 02:15:01 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] CloudFlare protection detected, trying to bypass it
2018-05-01 02:15:11 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] POST URL: https://www.torrentleech.org/user/account/login [Status: 302]
2018-05-01 02:15:11 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0
2018-05-01 02:15:11 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] With post data: username=**********&remember_me=on&login=submit&password=**********
2018-05-01 02:15:12 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] GET URL: https://www.torrentleech.org/ [Status: 200]
2018-05-01 02:15:12 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0
2018-05-01 02:15:12 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] CloudFlare successfully bypassed.
2018-05-01 02:15:12 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] Search mode: Episode
2018-05-01 02:15:12 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] Search string: Modern Family S09E18
2018-05-01 02:15:12 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] GET URL: https://www.torrentleech.org/torrents/browse/list/categories/2,7,35,26,32,44/query/Modern%20Family%20S09E18/orderby/seeders/order/desc/ [Status: 503]
2018-05-01 02:15:12 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] User-Agent: Medusa/0.2.2 (Linux; 4.11.12; 77473c19-4c91-11e8-9fd0-74d435a92093)
2018-05-01 02:15:12 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] CloudFlare protection detected, trying to bypass it
2018-05-01 02:15:30 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] GET URL: https://www.torrentleech.org/torrents/browse/list/categories/2,7,35,26,32,44/query/Modern%20Family%20S09E18/orderby/seeders/order/desc/ [Status: 503]
2018-05-01 02:15:30 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:59.0) Gecko/20100101 Firefox/59.0
2018-05-01 02:15:30 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] The response returned a non-200 response while requesting url https://www.torrentleech.org/torrents/browse/list/categories/2,7,35,26,32,44/query/Modern Family S09E18/orderby/seeders/order/desc/. Error: HTTPError(u'503 Server Error: Service Temporarily Unavailable for url: https://www.torrentleech.org/torrents/browse/list/categories/2,7,35,26,32,44/query/Modern%20Family%20S09E18/orderby/seeders/order/desc/',)
2018-05-01 02:15:30 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] No data returned from provider
2018-05-01 02:15:30 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] Found 0 unique items
2018-05-01 02:15:30 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] Found qualities: []
2018-05-01 02:15:30 DEBUG    SEARCHQUEUE-FORCED-95011 :: [TorrentLeech] :: [349a87f] No unknown qualities in results
2018-05-01 02:15:30 DEBUG    SEARCHQUEUE-FORCED-95011 :: [349a87f] 95011: Formatting pattern: %SN - S%0SE18 - %EN -> Modern Family - S09E18 - Daddy Issues
2018-05-01 02:15:30 INFO     SEARCHQUEUE-FORCED-95011 :: [349a87f] Unable to find forced results for: Modern Family - S09E18 - Daddy Issues

[sharkykh]

@brendanssound
If you wish, you can try it now. I've updated #4142 to include the latest develop branch with the CloudFlare fix. I personally haven't got around to testing the TorrentLeech provider yet, but it should work.

[ghost]

@sharkykh
With the latest develop build this is now working for me. Thanks!

[connexx2]

Same here! Many thanks.

[sharkykh]

@rssfed23 Just making sure, have you tried the develop branch? Did it resolve your issue?
While trying out a manual search I was able to get results in all the qualities.

[rssfed23]

@sharkykh thanks for checking! - Yep; all looks good now thanks :)