It looks like it could be just an issue on theTVDB's end after migrating to the new site, today. Although, there is a new v3 API version they just cut over it; v1 and v2 is supposed to still work.

https://forums.thetvdb.com/viewtopic.php?f=3&t=60048

cgatesman on 15 Nov 2019

the fix from @kmartin36 works

5687

Wally73 on 15 Nov 2019

Geting this same error/problem as well I was about to add 10 or 11 shows and it does not work.

Snowaks on 15 Nov 2019

Hope they get this fixed asap! as this breaks sickchill. Got to love it that they did noting wrong.
Are we ever going to get a alternative to TVDB ?

Snowaks on 15 Nov 2019

@wally73 Thanks for checking, I will merge it into develop when I'm behind a desktop

WebSpider on 15 Nov 2019

This has been merged in develop. Please test the develop branch on your install and give some feedback, thanks!

WebSpider on 15 Nov 2019

After change, works but not importing correctly artwork

2019-11-15 13:35:07 INFO SEARCHQUEUE-BACKLOG-205281 :: No needed episodes found during backlog search for: [Falling Skies]
2019-11-15 13:35:05 INFO SEARCHQUEUE-BACKLOG-205281 :: [Rarbg] :: Performing episode search for Falling Skies
2019-11-15 13:34:56 INFO SHOWQUEUE-ADD :: Done cache check
2019-11-15 13:34:52 WARNING SHOWQUEUE-ADD :: There was an error trying to retrieve the image, aborting
2019-11-15 13:34:52 INFO SHOWQUEUE-ADD :: Request failed: 403 Client Error: Forbidden for url: https://artworks.thetvdb.com/banners/_cache/graphical/205281-g18.jpg ()
2019-11-15 13:34:48 WARNING SHOWQUEUE-ADD :: There was an error trying to retrieve the image, aborting
2019-11-15 13:34:48 INFO SHOWQUEUE-ADD :: Request failed: 403 Client Error: Forbidden for url: https://artworks.thetvdb.com/banners/_cache/posters/5c56073e550e3.jpg ()
2019-11-15 13:34:43 INFO SEARCHQUEUE-BACKLOG-205281 :: Beginning backlog search for: [Falling Skies]
2019-11-15 13:34:42 INFO SHOWQUEUE-ADD :: 205281: Updating NFOs for show with new indexer info
2019-11-15 13:34:42 INFO SHOWQUEUE-ADD :: Launching backlog for this show since its episodes are WANTED
2019-11-15 13:33:41 INFO SHOWQUEUE-ADD :: Setting all episodes to the specified default status: 5
2019-11-15 13:33:40 INFO SHOWQUEUE-ADD :: 205281: Unable to find the show in the database
2019-11-15 13:33:37 INFO SHOWQUEUE-ADD :: theTVDB: {u'useZip': True, u'apikey': u'F9C450E78D99172E', u'cache': u'C:\SickChill\Data\cache\indexers\theTVDB', u'language': 'en'}
2019-11-15 13:33:37 INFO SHOWQUEUE-ADD :: Starting to add show by ShowDir: \xxxFalling Skies (2011)

ca-z on 15 Nov 2019

after switching to develop I do get a nfo and jpg file again!

skaynl on 15 Nov 2019

Hi skaynl, thanks

still getting same error for old series, the only issue is the artwork cover

2019-11-15 14:22:16 INFO SHOWQUEUE-REFRESH :: Done cache check
2019-11-15 14:22:16 WARNING SHOWQUEUE-REFRESH :: There was an error trying to retrieve the image, aborting
2019-11-15 14:22:16 INFO SHOWQUEUE-REFRESH :: Request failed: 403 Client Error: Forbidden for url: https://artworks.thetvdb.com/banners/_cache/graphical/250136-g.jpg ()
2019-11-15 14:22:15 WARNING SHOWQUEUE-REFRESH :: There was an error trying to retrieve the image, aborting
2019-11-15 14:22:15 INFO SHOWQUEUE-REFRESH :: Request failed: 403 Client Error: Forbidden for url: https://artworks.thetvdb.com/banners/_cache/posters/250136-2.jpg ()
2019-11-15 14:22:14 INFO SHOWQUEUE-REFRESH :: 250136: Updating NFOs for show with new indexer info
2019-11-15 14:21:59 INFO SHOWQUEUE-REFRESH :: Performing refresh on Dynamo: Magician Impossible
2019-11-15 14:21:14 INFO CHECKVERSION :: Checking for updates using GIT
2019-11-15 14:21:14 INFO TORNADO :: Starting SickChill on http://0.0.0.0:8081/
2019-11-15 14:21:14 INFO MAIN :: Forcing web server to port 8081
2019-11-15 14:21:14 INFO MAIN :: Starting SickChill [develop] using 'C:SickChillDataconfig.ini'
2019-11-15 14:21:06 INFO EVENT-QUEUE :: Restarting SickChill with ['C:\SickChill\Python\python.exe', u'C:\SickChill\SickChill\SickBeard.py', '--nolaunch', '--port=8081', '--datadir=C:\SickChill\Data']
2019-11-15 14:21:06 INFO EVENT-QUEUE :: Shutting down Tornado

ca-z on 15 Nov 2019

Hi,
Same thing for me, I apply manually the kmartin36-patch-5686, I can get the show now, but the covers doesn't work.

2019-11-15 09:41:36 INFO SHOWQUEUE-ADD :: Request failed: 403 Client Error: Forbidden for url: https://artworks.thetvdb.com/banners/_cache/graphical/5daddd383f070.jpg ()
2019-11-15 09:41:36 WARNING SHOWQUEUE-ADD :: There was an error trying to retrieve the image, aborting

potrios on 15 Nov 2019

I will test it now

Snowaks on 15 Nov 2019

I did not receive a error and show did not add. Log below

`2019-11-15 09:36:03 INFO SHOWQUEUE-ADD :: Show neXt (2019) is on theTVDB but contains no season/episode data.

2019-11-15 09:35:59 INFO SHOWQUEUE-ADD :: theTVDB: {u'useZip': True, u'apikey': u'F9C450E78D99172E', u'cache': u'/usr/local/sickchill/cache/indexers/theTVDB', u'language': 'en'}

2019-11-15 09:35:59 INFO SHOWQUEUE-ADD :: Starting to add show by ShowDir: /media/TV/neXt`

Snowaks on 15 Nov 2019

It is fixed I can add show's in dev mode. neXt is error out not sure why. Home, Posters for new show's are MIA.

Snowaks on 15 Nov 2019

@Snowaks FYI I confirm that I have the same condition than you here!

potrios on 15 Nov 2019

Same here. How do I get into develop mode? And when I enter develop mode, this will automatically be fixed? Or will I have to do something else? (kind of a noob here)

LordXLordX on 15 Nov 2019

It is fixed I can add show's in dev mode. neXt is error out not sure why. Home, Posters for new show's are MIA.

Your show has no information on TheTVDB, so this is expected behavior. See here: https://www.thetvdb.com/series/nextfox

chesh1r3k on 15 Nov 2019

Project development noob question :-) :
Do I have to wait for this fix until the develop branch is merged into the master branch, or will this fix be applied to the master branch as a patch?

Zw3rv3r on 15 Nov 2019

On my side, all shows can be add now with the patch of the "tvdb_api.py". However it seem something can no be associated with the info received from TVDB and the atworks... See logs below

2019-11-15 09:41:36 WARNING SHOWQUEUE-ADD :: There was an error trying to retrieve the image, aborting
2019-11-15 09:41:36 INFO SHOWQUEUE-ADD :: Request failed: 403 Client Error: Forbidden for url: https://artworks.thetvdb.com/banners/_cache/graphical/5daddd383f070.jpg ()
2019-11-15 09:41:36 WARNING SHOWQUEUE-ADD :: There was an error trying to retrieve the image, aborting
2019-11-15 09:41:36 INFO SHOWQUEUE-ADD :: Request failed: 403 Client Error: Forbidden for url: https://artworks.thetvdb.com/banners/_cache/posters/5d85d693a958a.jpg ()
2019-11-15 09:41:34 INFO SHOWQUEUE-ADD :: 360733: Updating NFOs for show with new indexer info
2019-11-15 09:41:24 INFO SHOWQUEUE-ADD :: Setting all episodes to the specified default status: 5
2019-11-15 09:41:24 INFO SHOWQUEUE-ADD :: 360733: Unable to find the show in the database
2019-11-15 09:41:24 INFO SHOWQUEUE-ADD :: theTVDB: {u'useZip': True, u'apikey': u'F9C450E78D99172E', u'cache': u'/config/cache/indexers/theTVDB', u'language': u'en'}
2019-11-15 09:41:23 INFO SHOWQUEUE-ADD :: theTVDB: {u'useZip': True, u'apikey': u'F9C450E78D99172E', u'cache': u'/config/cache/indexers/theTVDB', u'language': u'en'}
2019-11-15 09:41:23 INFO SHOWQUEUE-ADD :: Starting to add show by Indexer Id: 360733
2019-11-15 09:41:22 INFO ThreadPoolExecutor-0_3 :: theTVDB: {u'useZip': True, u'apikey': u'F9C450E78D99172E', u'cache': u'/config/cache/indexers/theTVDB', u'language': u'en'}
2019-11-15 09:41:08 INFO SHOWQUEUE-REMOVE-SHOW :: Some notifications could not be sent. Continuing with postProcessing...
2019-11-15 09:41:08 INFO SHOWQUEUE-REMOVE-SHOW :: Deleted show folder /mnt/DNS-320L/Volume_1/Tv_Shows/Watchmen
2019-11-15 09:41:08 INFO SHOWQUEUE-REMOVE-SHOW :: Attempt to delete show folder /mnt/DNS-320L/Volume_1/Tv_Shows/Watchmen
2019-11-15 09:41:08 INFO SHOWQUEUE-REMOVE-SHOW :: Attempt to delete cache file /config/cache/images/360733.fanart.jpg
2019-11-15 09:41:08 INFO SHOWQUEUE-REMOVE-SHOW :: Attempt to delete cache file /config/cache/images/360733.banner.jpg
2019-11-15 09:41:08 INFO SHOWQUEUE-REMOVE-SHOW :: Attempt to delete cache file /config/cache/images/360733.poster.jpg
2019-11-15 09:41:08 INFO SHOWQUEUE-REMOVE-SHOW :: Removing Watchmen
2019-11-15 09:40:54 INFO SHOWQUEUE-REFRESH :: Done cache check
2019-11-15 09:40:54 WARNING SHOWQUEUE-REFRESH :: There was an error trying to retrieve the image, aborting
2019-11-15 09:40:54 INFO SHOWQUEUE-REFRESH :: Request failed: 403 Client Error: Forbidden for url: https://artworks.thetvdb.com/banners/_cache/graphical/5daddd383f070.jpg ()
2019-11-15 09:40:54 WARNING SHOWQUEUE-REFRESH :: There was an error trying to retrieve the image, aborting
2019-11-15 09:40:54 INFO SHOWQUEUE-REFRESH :: Request failed: 403 Client Error: Forbidden for url: https://artworks.thetvdb.com/banners/_cache/posters/5d85d693a958a.jpg ()
2019-11-15 09:40:53 INFO SHOWQUEUE-REFRESH :: 360733: Updating NFOs for

potrios on 15 Nov 2019

Problem is back, the

REFRESH the show

2019-11-15 16:56:26 WARNING SHOWQUEUE-FORCE-UPDATE :: Unable to contact theTVDB, aborting: Bad zip file received from thetvdb.com, could not read it

ADD Show

AA tvdb_error: Bad zip file received from thetvdb.com, could not read it
AA raise tvdb_error(e)
AA File "C:SickChillSickChilllibtvdb_apitvdb_api.py", line 629, in _getetsrc
AA epsEt = self._getetsrc(url, language=language)
AA File "C:SickChillSickChilllibtvdb_apitvdb_api.py", line 874, in _getShowData
AA self._getShowData(key, self.config['language'], True)
AA File "C:SickChillSickChilllibtvdb_apitvdb_api.py", line 946, in __getitem__
AA s = t[self.indexer_id]
AA File "C:SickChillSickChillsickbeardshow_queue.py", line 324, in run
AA Traceback (most recent call last):
2019-11-15 16:59:47 ERROR SHOWQUEUE-ADD :: [e094759] Unable to look up the show in \QNAP563-02tvOFFFriends (1994) on theTVDB using ID 79168, not using the NFO. Delete .nfo and try adding manually again.: Bad zip file received from thetvdb.com, could not read it
2019-11-15 16:59:46 INFO SHOWQUEUE-ADD :: theTVDB: {u'useZip': True, u'apikey': u'F9C450E78D99172E', u'cache': u'C:\SickChill\Data\cache\indexers\theTVDB', u'language': 'en'}
2019-11-15 16:59:46 INFO SHOWQUEUE-ADD :: Starting to add show by ShowDir: Friends (1994)

ca-z on 15 Nov 2019

Same here. How do I get into develop mode? And when I enter develop mode, this will automatically be fixed? Or will I have to do something else? (kind of a noob here)

*1st add your github account and save.

General Configuration> Advanced Settings> Branch Version> Select develop> Checkout Branch (will restart)

ca-z on 15 Nov 2019

👍1

Same here. How do I get into develop mode? And when I enter develop mode, this will automatically be fixed? Or will I have to do something else? (kind of a noob here)

*1st add your github account and save.

General Configuration> Advanced Settings> Branch Version> Select develop> Checkout Branch (will restart)

Thank you so much - I am such a noob, how do I add my account?

LordXLordX on 15 Nov 2019

Same here. How do I get into develop mode? And when I enter develop mode, this will automatically be fixed? Or will I have to do something else? (kind of a noob here)

*1st add your github account and save.
General Configuration> Advanced Settings> Branch Version> Select develop> Checkout Branch (will restart)

Thank you so much - I am such a noob, how do I add my account?

No prob, your gifhub account, the same you using to login here, or create new one

https://github.com/join

ca-z on 15 Nov 2019

Ok, got it - and your steps worked - thank you so much.

So what is the next step here? Wait for a new release and then switch back over to master?

LordXLordX on 15 Nov 2019

Ok, got it - and your steps worked - thank you so much.

So what is the next step here? Wait for a new release and then switch back over to master?

Yep, pretty much that

chesh1r3k on 15 Nov 2019

hey everyone, i keep getting this error when i try to turn to develop mode git checkout -f develop returned : error: pathspec 'develop' did not match any file(s) known to git. .
What am i doing wrong?

Mixanologos on 15 Nov 2019

I'm on develop branch and still getting the API Zip errors:

22:16:31 DEBUG::ThreadPoolExecutor-0_3 :: Searching for Show with searchterm(s): ['The Mandalorian', 'Mandalorian'] on Indexer: theTVDB
22:16:38 INFO::SHOWQUEUE-ADD :: Starting to add show by ShowDir: /tv/The Mandalorian
22:16:38 INFO::SHOWQUEUE-ADD :: theTVDB: {u'useZip': True, u'apikey': u'F9C450E78D99172E', u'cache': u'/data/cache/indexers/theTVDB', u'language': 'en'}
22:16:38 ERROR::SHOWQUEUE-ADD :: [5aad224] Unable to look up the show in /tv/The Mandalorian on theTVDB using ID 361753, not using the NFO. Delete .nfo and try adding manually again.: Bad zip file received from thetvdb.com, could not read it
Traceback (most recent call last):
File "/app/sickchill/sickbeard/show_queue.py", line 324, in run
s = t[self.indexer_id]
File "/app/sickchill/lib/tvdb_api/tvdb_api.py", line 947, in __getitem__
self._getShowData(key, self.config['language'], True)
File "/app/sickchill/lib/tvdb_api/tvdb_api.py", line 875, in _getShowData
epsEt = self._getetsrc(url, language=language)
File "/app/sickchill/lib/tvdb_api/tvdb_api.py", line 630, in _getetsrc
raise tvdb_error(e)
tvdb_error: Bad zip file received from thetvdb.com, could not read it

fatalglitch on 15 Nov 2019

I can confirm that this problem persists on the develop branch

AA tvdb_error: Bad zip file received from thetvdb.com, could not read it
AA raise tvdb_error(e)
AA File "*****/sickchill/lib/tvdb_api/tvdb_api.py", line 630, in _getetsrc
AA epsEt = self._getetsrc(url, language=language)
AA File "*****/sickchill/lib/tvdb_api/tvdb_api.py", line 875, in _getShowData
AA self._getShowData(key, self.config['language'], True)
AA File "*****/sickchill/lib/tvdb_api/tvdb_api.py", line 947, in __getitem__
AA s = t[self.indexer_id]
AA File "*****/sickchill/sickbeard/show_queue.py", line 324, in run
AA Traceback (most recent call last):
2019-11-16 00:53:02 SHOWQUEUE-ADD :: [e094759] Unable to look up the show in ***** on theTVDB using ID *****, not using the NFO. Delete .nfo and try adding manually again.: "There is no item named 'en.xml' in the archive"

sickchilluser on 16 Nov 2019

Please note that there are multiple issues on thetvdb's end regarding backward API compatibility.

The official list of known issues is (as of _Nov 16 2019 12:29 AM GMT_):

Wrong artwork format returned
Wrong artwork language returned
API delivering zip file discrepancies
Clients aren't handling 301s
Default language handling

(cf: https://forums.thetvdb.com/viewtopic.php?f=17&t=60106)

"API V2 broken today"
https://forums.thetvdb.com/viewtopic.php?f=122&t=60034

LeeThompson on 16 Nov 2019

What where they thinking when they switch to this new version? if it was this broken seem's like it should have been tested more be for tvdb made any change's. Look like there API system is all jacked up now from what I am reading. (None of this is about sickchill)

Snowaks on 16 Nov 2019

Would be really nice if we could get a back up system to tvdb so if they decide to break there stuff again, API_V2 we do not get hit hard.

Snowaks on 16 Nov 2019

👍1

Is it fixed?
I see fixed label but its not working for me.
Neither master nor dev branch are working properly, both give error on force update

2019-11-16 11:30:55 WARNING SHOWQUEUE-FORCE-UPDATE :: Unable to contact theTVDB, aborting: Bad zip file received from thetvdb.com, could not read it
2019-11-16 11:30:52 DEBUG SHOWQUEUE-FORCE-UPDATE :: 361753: Loading show info from theTVDB
2019-11-16 11:30:52 DEBUG SHOWQUEUE-FORCE-UPDATE :: Retrieving show info from theTVDB
2019-11-16 11:30:52 DEBUG SHOWQUEUE-FORCE-UPDATE :: Beginning update of The Mandalorian
2019-11-16 11:30:40 WARNING SHOWQUEUE-FORCE-UPDATE :: Unable to contact theTVDB, aborting: Bad zip file received from thetvdb.com, could not read it
2019-11-16 11:30:37 DEBUG SHOWQUEUE-FORCE-UPDATE :: 367553: Loading show info from theTVDB
2019-11-16 11:30:37 DEBUG SHOWQUEUE-FORCE-UPDATE :: Retrieving show info from theTVDB
2019-11-16 11:30:37 DEBUG SHOWQUEUE-FORCE-UPDATE :: Beginning update of Dollface
2019-11-16 11:29:46 WARNING SHOWQUEUE-FORCE-UPDATE :: Unable to contact theTVDB, aborting: Bad zip file received from thetvdb.com, could not read it
2019-11-16 11:29:42 DEBUG SHOWQUEUE-FORCE-UPDATE :: 295829: Loading show info from theTVDB
2019-11-16 11:29:42 DEBUG SHOWQUEUE-FORCE-UPDATE :: Retrieving show info from theTVDB
2019-11-16 11:29:42 DEBUG SHOWQUEUE-FORCE-UPDATE :: Beginning update of The Man in the High Castle

papampi on 16 Nov 2019

After switching to the develop branch, update and clean install I still receive the following error:

2019-11-16 10:34:19 SHOWQUEUE-ADD :: [5aad224] Unable to look up the show in /share/Series/... (2018) on theTVDB using ID 343253, not using the NFO. Delete .nfo and try adding manually again.: Bad zip file received from thetvdb.com, could not read it

NiteAngyl on 16 Nov 2019

Yes it is broken again! returned this error in Dev mode.

2019-11-16 03:44:41 ThreadPoolExecutor-0_15 :: Show id 361868 not found on theTVDB, not adding the show: error Bad zip file received from thetvdb.com, could not read it 2019-11-16 03:44:09 ThreadPoolExecutor-0_12 :: Show id 361868 not found on theTVDB, not adding the show: error Bad zip file received from thetvdb.com, could not read it

Snowaks on 16 Nov 2019

👀1

I can no longer add tv shows again try 2 different shows.

Snowaks on 16 Nov 2019

Same here

2019-11-16 10:22:07 SHOWQUEUE-ADD :: [5aad224] Unable to look up the show in The Blacklist Redemption (2017) on theTVDB using ID 311903, not using the NFO. Delete .nfo and try adding manually again.: Bad zip file received from thetvdb.com, could not read it

ca-z on 16 Nov 2019

In the known issues list at thetvdb I suspect "API delivering zip file discrepancies" is what's causing "Bad zip file received from thetvdb.com, could not read it".

UPDATE: No it's much worse than that, series information isn't working.. at all.

LeeThompson on 16 Nov 2019

What might be worth a try is to see if the API v3 calls are working properly, if so quickest way forward might be to add API v3 support to SickChill. (I have not looked at the API documentation so I don't know how trivial this would be.)

I do agree with other comments and feature requests that adding support for other data sources for TV shows (perhaps even allowing some manual, local data editing) (so that thetvdb isn't the only source of data) would be a very good direction to go in so cases like this would be a brief nuisance instead of breaking things completely.

LeeThompson on 16 Nov 2019

👍1

There's not (only) something wrong with the zip compression - neither forcing self.config['useZip'] = False in lib/tvdb_api/tvdb_api.py, nor forcing 'useZip': False, in sickbeard/indexers/indexer_config.py helps - still getting (EDIT: to be clear: on the develop branch):

2019-11-16 13:44:22 WARNING  SHOWQUEUE-ADD :: Show in **** has no name on theTVDB, probably searched with the wrong language. Delete .nfo and add manually in the correct language.
2019-11-16 13:44:22 INFO     SHOWQUEUE-ADD :: theTVDB: {u'useZip': False, u'apikey': u'F9C450E78D99172E', u'cache': u'/***/sickchill/cache/indexers/theTVDB', u'language': 'en'}
2019-11-16 13:44:22 INFO     SHOWQUEUE-ADD :: Starting to add show by ShowDir: ******

sickchilluser on 16 Nov 2019

hey everyone, i keep getting this error when i try to turn to develop mode git checkout -f develop returned : error: pathspec 'develop' did not match any file(s) known to git. .
What am i doing wrong?

Same issue here. I'm quite new to this so am thinking I'm doing something wrong.

edspeds on 16 Nov 2019

I do agree with other comments and feature requests that adding support for other data sources for TV shows (perhaps even allowing some manual, local data editing) (so that thetvdb isn't the only source of data) would be a very good direction to go in so cases like this would be a brief nuisance instead of breaking things completely.

Even better a musicbrainz-like public and distributable database...

ncl2fth on 16 Nov 2019

(This was initially mark fixed due to the changes in the 'develop' branch but still has issues due to problems on thetvdb.com side.)

Even if you separately apply portions of the patch (from kmartin36 above—Thanks!) by...

altering the code to seek 'series.xml' from the received zip archive (instead of '%s.eml' % language, which resolves to 'en.xml')...

or

choosing 64bit zip encoding (or not, leaving the prior default)...

...it fails to find the requisite information. (I'm testing this by GUI-selecting "Force Full Update" on an established show page in my local DB.)

It appears to me that thetvdb.com has now reverted to shipping the previous zip format (not 64bit encoded) so you have to take OUT the 64bit mods from the patch to receive valid archive data. But it still cannot find what it seeks in the received decoded data.

If I knew what thetvdb.com was actually shipping inside the new archive (not 'en.xml', not 'series.xml') I'd try hacking the patch to look for the real contents. I cannot find this documented for API v2 at thetvdb.com, as their API inspector/executer page is only showing v3 right now.

If I were more of a pythonista I would simply write the received zipdata to a local file and inspect it manually to see what the internal structure is. This could yield a replacement string for 'en.xml' that could possibly be extracted. Alas, I am not a pythonista.

(Similar failures occur trying to add a new show (or deleting an existing one and adding it back). No supripse there as it all goes thru lib/tvdb_api/tvdb_api.py.)

But the received zip info neither contains "en.xml" nor "series.xml" so the same failure occurs in the end.

mmhere on 16 Nov 2019

Their api documentation says this:
Versioning
You may request a different version of the API by including an Accept header in your request with the following format: Accept:application/vnd.thetvdb.v$VERSION. This documentation automatically uses the version seen at the top and bottom of the page.

I've never looked at the sickchill source before today, would it be terribly difficult to slap in the extra header pointing to the known good version of the api?

Apologies if this has already been considered.

jwshive on 16 Nov 2019

👍1

forging ahead.

I found by adding some logging code in lib/tvdb_api/tvdb_apy.py that the necessary XML returned is named 'en.zip.xml' where before it was 'en.xml'. Changing the lookup to this (as return xmltodict.parse(myzipfile.read('%s.zip.xml' % language), postprocessor=process)) lets it get a little further, where it actually gets back some XML, but now it cannot find the XML key seriesname -- it looks like the XML has completely changed format.

The log reports

2019-11-16 15:51:43 ERROR SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 78998, but attribute 'seriesname' was empty.

So going after the header to specify version (which klainn just mentioned) seems like a good idea.

In particular, sickchill submits this URL, which you can stick directly into a browser to test: http://thetvdb.com/api/F9C450E78D99172E/series/78998/all/en.zip and gets back a package called en.zip.zip containing the following files, all of which are uncompressed HTML (after you unpack the zip archive) in spite of being named `en.zip.xml' within the zip archive-->

-rw-r--r-- 1 username 803 19791231.0000.00 actors.xml
-rw-r--r-- 1 username 5.6K 19791231.0000.00 banners.xml
-rw-r--r-- 1 username 346K 19791231.0000.00 en.zip.xml

Sigh.

mmhere on 17 Nov 2019

I tried the following to pass in a header pair (python dictionary) to the session.get() request, but the same info comes back whether such header info is passed, apparently newer and unparseable by current version of sickchill

was:
resp = session.get(url.strip(), params=params)

tried:

# various attempts commented:
#headerV2={'Accept': 'application/vnd.thetvdb.v2'}
#headerV2={'Accept': 'application/vnd.thetvdb.v2.2.2'}
headerV2={'Accept': 'application/vnd.thetvdb.v2.0.0'}
resp = session.get(url.strip(), params=params, headers=headerV2)

Note that this is within the _loadUrl() method of class Tvdb in tvdb_api.py (passing headers according to Python documentation for import requests)

Further, if you do manually plug a URL such as http://thetvdb.com/api/F9C450E78D99172E/series/78998/all/en.zip into a browser, and inspect the en.zip.xml file inside the archive that is returned, you in fact find no occurrences of the XML tag 'seriesname' (or even 'SeriesName') that sickchill is looking for.

At this point, I strongly suspect that thetvdb.com is not actually heeding API version requests, and so the V2 API is pretty much unavailable (or at least portions of it are missing and are mapped to returning newer V3 info).

Clients like ours expecting V2 may be stuck until thetvdb.com provides better backward compatibility support for the real/expected V2 data.

mmhere on 17 Nov 2019

I tried the following to pass in a header pair (python dictionary) to the session.get() request, but the same info comes back whether such header info is passed, apparently newer and unparseable by current version of sickchill

was:
resp = session.get(url.strip(), params=params)

tried:
# various attempts commented:
#headerV2={'Accept': 'application/vnd.thetvdb.v2'}
#headerV2={'Accept': 'application/vnd.thetvdb.v2.2.2'}
headerV2={'Accept': 'application/vnd.thetvdb.v2.0.0'}
resp = session.get(url.strip(), params=params, headers=headerV2)
Note that this is within the _loadUrl() method of class Tvdb in tvdb_api.py (passing headers according to Python documentation for import requests)

Further, if you do manually plug a URL such as http://thetvdb.com/api/F9C450E78D99172E/series/78998/all/en.zip into a browser, and inspect the en.zip.xml file inside the archive that is returned, you in fact find no occurrences of the XML tag 'seriesname' (or even 'SeriesName') that sickchill is looking for.

At this point, I strongly suspect that thetvdb.com is not actually heeding API version requests, and so the V2 API is pretty much unavailable (or at least portions of it are missing and are mapped to returning newer V3 info).

Clients like ours expecting V2 may be stuck until thetvdb.com provides better backward compatibility support for the real/expected V2 data.

I made the exact same modifications to tvdb_api.py hoping to request the v2 api.

Same results.

zcpnate on 17 Nov 2019

Apologies for so many posts. Just noticed that the XML tag 'seriesname' ('SeriesName', but sickchill downcases it before doing lookup) is actually present in the en.zip.xml data that is returned, but the data for the tag is empty. In fact the XML is malformed.

So I think having sickchill look for 'en.zip.xml' gets us pretty close and we just have to wait for thetvdb.com to fix the data being returned by their API (whether client tries to specify V2 or not).

This link at forums.thetvdb.com correctly mentions that "All series information [is] blank except people".

If I look at the en.zip.xml file contents returned in the zip archive by hitting http://thetvdb.com/api/F9C450E78D99172E/series/78998/all/en.zip in a browser, I indeed see stuff like this:

  <Series>
    <id>0</id>
    <Actors>|Tiff Needell|Vicki Butler-Henderson|Tom Ford|Jason Plato|Jonny Smith|Tim Shaw|Jon Bentley|</Actors>
    <Airs_DayOfWeek/>
    <Airs_Time/>
    <ContentRating/>
    <FirstAired/>
    <Genre/>
    <IMDB_ID/>
    <Language/>
    <Network/>
    <NetworkID/>
    <Overview/>
    <Rating>0</Rating>
    <RatingCount>0</RatingCount>
    <Runtime/>
    <SeriesID/>
    <SeriesName/>
    <Status/>
    <added/>
    <addedBy/>
    <banner/>
    <fanart/>
    <lastupdated>0</lastupdated>
    <poster/>
    <zap2it_id/>
  </Series>

Note that the <id> and <Actors> tags are syntactically correct, and the <Actors> tag has real data (oddly the <id> value is 0) but everything after that is broken and is malformed XML. In particular, note that there is a <SeriesName/> (XML closing tag) but there is no corresponding leading <SeriesName> (XML opening tag) and... THERE IS NO DATA!

The error in the sickchill log looks like the following, so it corroborates the fact that sickchill properly gets the XML, parses it, but finds blank entries, so verily and truly punts.

17:26:28 ERROR::SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 78998, but attribute 'seriesname' was empty.
Traceback (most recent call last):
  File "/home/username/src/SickChill/sickbeard/show_queue.py", line 648, in run
    self.show.loadFromIndexer(cache=not self.force)
  File "/home/username/src/SickChill/sickbeard/tv.py", line 873, in loadFromIndexer
    "Found {0}, but attribute 'seriesname' was empty.".format(self.indexerid))
tvdb_attributenotfound: Found 78998, but attribute 'seriesname' was empty.

QED. We wait for the Boffins™ at thetvdb.com.

(™ I use the term Boffins very loosely.)

mmhere on 17 Nov 2019

tried to switch to the develop branch but i continue to get errors:
2019-11-16 23:16:34 SHOWQUEUE-FORCE-UPDATE :: Unable to contact theTVDB, aborting: Bad zip file received from thetvdb.com, could not read it

no fix for that?

Matthewserta on 17 Nov 2019

@Matthewserta Not yet, unfortunately the problem is at thetvdb who launched their v3 API and appear to have broken it all. (If you scroll up a bit there are links to their forum threads.)

LeeThompson on 17 Nov 2019

(@mmhere it's worse than that, v3 isn't working either.)

I just did my own testing on TheTVDB's API tester (swagger) and I can get episode data but not series data.

Using BoJack Horseman as an example, a request for series data (https://api.thetvdb.com/series/282254) returns:

{
  "Error": "ID: 282254 not found"
}

With headers:

{
  "cache-control": "private, max-age=300",
  "content-length": "32",
  "content-type": "application/json; charset=utf-8",
  "date": "Sun, 17 Nov 2019 05:10:13 GMT",
  "vary": "Accept-Language",
  "via": "1.1 1b74ccf4cb51eacf97a0e6d60ae46a3f.cloudfront.net (CloudFront)",
  "x-amz-cf-id": "(Redacted)",
  "x-amz-cf-pop": "SEA19-C2",
  "x-cache": "Error from cloudfront",
  "x-powered-by": "Thundar!",
  "x-thetvdb-api-version": "3.0.0"
}

Whereas "https://api.thetvdb.com/series/282254/actors" works properly.

Those wishing to do their own testing:
https://api.thetvdb.com/swagger

(You'll need a thetvdb account name, account key- both are on your account dashboard/info and an api key. After logging in with your credentials, add the JWT token to the top of the page. In my personal case it didn't let me add the token in Chrome so I used Edge. Yeah, blech.)

UPDATE: Posts on tvdb's forum indicate API v1 is broken as well.

LeeThompson on 17 Nov 2019

tried to switch to the develop branch but i continue to get errors:
2019-11-16 23:16:34 SHOWQUEUE-FORCE-UPDATE :: Unable to contact theTVDB, aborting: Bad zip file received from thetvdb.com, could not read it

If you pull the 64bit encoding change, and instead request file 'en.zip.xml' it gets the XML but the XML itself is _syntactically malformed_ ←←strike that, it is valid but empty -- see further posts below.

Still waiting on thetvdb.com to get their act together or crumble under pressure and go back to API v2 like they did earlier this year when they let v3 out of the barn for the first time.

mmhere on 17 Nov 2019

Thanks for all the hard work guys, I can (obviously) confirm that it's broken here too. I'll just sit and wait for the time being.

jaxjexjox on 17 Nov 2019

@mmhere Just wanted to say thank you for your testing on this. That said, your comment about the XML being malformed is false. The XML you posted is perfectly valid XML.

Examples:

 <id>0</id>
 <id 0 />

The above are equivalent when parsed by an XML validator.
This format: <id/> would simply mean the id tag exists, but there is no data.

Unless SickChill uses a specific version of XML where this is not the case?

CovertLeopard on 17 Nov 2019

👍2

I just switched to developer branch to try and bypass the error and now i get this message:

2019-11-17 09:11:39 SHOWQUEUE-ADD :: [5aad224] Unable to look up the show in D:New CompletedShowName on theTVDB using ID XXXXXX, not using the NFO. Delete .nfo and try adding manually again.: Bad zip file received from thetvdb.com, could not read it

cadmuslabslex on 17 Nov 2019

_SickChill Broken till TVDB gets there act together!!!_ To be honest This type of behavior is appalling to drop some thing in it's current forum this broken. it is almost like they are taking after the gaming industry!

There is really noting sick chill can do to fix TVDB at this moment other then doping a hot fix to use some other database then tvdb. I do not know of any other databases that compare with tvdb. Also I do not hold the skills to do this type of a fix, Any Idea's ?

Snowaks on 17 Nov 2019

imdb has some information for tv shows too, but not sure if they have a nice API.

sjorge on 17 Nov 2019

Maybe we need to ask for a Feature Request/High Priority that gives us a back up Database/API system for our Series/Shows data.

Snowaks on 17 Nov 2019

This worked for me:

edit /lib/tvdb_api/tvdb_api.py

find all %s.xml
replace all with %s.zip.xml

restart sickchill

now able to add shows.

Just tried this - didn't work for me.

LordXLordX on 17 Nov 2019

Seeing this in my docker install from Linuxserver.io
AA tvdb_error: "There is no item named 'en.xml' in the archive" AA raise tvdb_error(e) AA File "/app/sickchill/lib/tvdb_api/tvdb_api.py", line 629, in _getetsrc AA epsEt = self._getetsrc(url, language=language) AA File "/app/sickchill/lib/tvdb_api/tvdb_api.py", line 874, in _getShowData AA self._getShowData(key, self.config['language'], True) AA File "/app/sickchill/lib/tvdb_api/tvdb_api.py", line 946, in __getitem__ AA s = t[self.indexer_id] AA File "/app/sickchill/sickbeard/show_queue.py", line 324, in run AA Traceback (most recent call last): 2019-11-17 13:41:34 ERROR SHOWQUEUE-ADD :: [e094759] Unable to look up the show in /tv/See on theTVDB using ID 361565, not using the NFO. Delete .nfo and try adding manually again.: "There is no item named 'en.xml' in the archive" 2019-11-17 13:41:34 INFO SHOWQUEUE-ADD :: theTVDB: {u'useZip': True, u'apikey': u'F9C450E78D99172E', u'cache': u'/config/cache/indexers/theTVDB', u'language': 'en'} 2019-11-17 13:41:34 INFO SHOWQUEUE-ADD :: Starting to add show by ShowDir: /tv/See

gadgetusaf on 17 Nov 2019

The zip isn't base64 anymore. I removed that code, then change the read file to en.zip.xml. I still get an error about it missing the show name, but seems to be farther along.

zipdata.write(resp.content)
myzipfile = zipfile.ZipFile(zipdata)
return xmltodict.parse(myzipfile.read('en.zip.xml'), postprocessor=process)

nevafuse on 17 Nov 2019

The above are equivalent when parsed by an XML validator.
This format: <id/> would simply mean the id tag exists, but there is no data.

Ah! Thanks! I was not aware of that. As we see, the tags are therefore empty of data, which we know to exist but simply isn't appearing in this access mode. Geez, if they'd just give me a SQL login to the back end ;-)

thetvdb.com forums mention this [empty data, showing similar XML to what I quoted above] in the post over there. So at least someone in thetvdb commentary community has reported it. Dunno if those capable of changing it to produce real data have made any progress (or have accepted the specific details of the issue as a _TODO_ item).

Current Status Summary

I hit a sample link just now (http://thetvdb.com/api/F9C450E78D99172E/series/78998/all/en.zip, in a browser.)

It does still return an archive called en.zip.zip (although if they revert to V2 this might change back to en.zip).

The archive is not base64 encoded, which it had been for a brief time when they first unleashed V3.

The archive en.zip.zip contains the target XML file currently named en.zip.xml, which itself is uncompressed XML.

That XML contains many empty XML tags of the form <tag/>

SickChill punts when it finds that the <SeriesName/> tag is devoid of data (it reports this as "seriesname" in the error log).

∴ not yet fixed :-(

mmhere on 17 Nov 2019

@mpretz
This worked for me:
edit /lib/tvdb_api/tvdb_api.py
find all %s.xml
replace all with %s.zip.xml
restart sickchill
now able to add shows.

@LordXLordX
Just tried this - didn't work for me.

IIRC, some posts in thetvdb forums indicate that certain shows return real data while others return empty data. Perhaps @mpretz got lucky but @LordXLordX did not. With the show I'm using as an example (ID 78998 in URL in previous post) it still returns empty fields inside en.zip.xml.

@mpretz -- can you identify which show (ID number) succeeded?

I'm starting to wonder if larger amounts of data cause thetvdb API to fail. E.g. the show I keep mentioning has 28 seasons and many episodes. Maybe it fails to return data because some threshold is exceeded as the API interface layer grunges around in the database back-end.

That said, it isn't a _huge_ amount of data by any stretch. The example I cite returns an archive/container en.zip.zip that is under 50KiB while the extracted en.zip.xml is under 350KiB.

mmhere on 17 Nov 2019

I'm starting to wonder if larger amounts of data cause thetvdb API to fail. E.g. the show I keep mentioning has 28 seasons and many episodes. Maybe it fails to return data because some threshold is exceeded as the API interface layer grunges around in the database back-end.

That said, it isn't a huge amount of data by any stretch. The example I cite returns an archive/container en.zip.zip that is under 50KiB while the extracted en.zip.xml is under 350KiB.

I'm also getting this error for shows with only one season, so I don't think the amount of data is the problem.

sjorge on 17 Nov 2019

I to am adding a show with only one season and getting an error as well.

gadgetusaf on 17 Nov 2019

thetvdb update

The broken series endpoint has been brought up in numerous threads on the forums and has yet to be acknowledged by any of their staff or dev team. The site itself is presently down (503 Service Temporarily Unavailable).

The latest official statement is (Sun Nov 17, 2019 17:53 GMT):

Hello users,

We do apologize for some of our latency issues on Friday evening. We've optimized several queries and they seem to have stabilized the site.

Our developers are continuing to work on API-related issues and we are continuing to ticket your bugs on the frontend.

We've deployed the mirrors xml path

Most favorites have been deployed. However, if you are missing yours, please PM me and we will try to restore them.

We will keep you posted as more fixes go out. Thank you for your patience.

LeeThompson on 17 Nov 2019

By making the %s.xml to %s.zip.xml for the language variable I was able to add show id 351959 but it threw a warning trying to get the poster. Just tossing this id out there as one that worked for comparisons.

jwshive on 17 Nov 2019

By making the %s.xml to %s.zip.xml for the language variable I was able to add show id 351959 but it threw a warning trying to get the poster. Just tossing this id out there as one that worked for comparisons.

Where do you change that at?

Matthewserta on 17 Nov 2019

From @mpretz earlier.

This worked for me:
edit /lib/tvdb_api/tvdb_api.py
find all %s.xml
replace all with %s.zip.xml
restart sickchill
now able to add shows.

jwshive on 17 Nov 2019

Changing to %s.zip.xml worked for me aswell.
Also looks like they have fixed something over at thetvdb. I downloaded the en.zip for the ID 78998 and it looks like the XML is fine now compared to a 3 hour old copy i have.
Have been able to add 3 shows so far, only thing that doesn't work is retrieving images
Request failed: 403 Client Error: Forbidden for url: https://artworks.thetvdb.com/banners/_cache/posters/78998-2.jpg

reggie88 on 17 Nov 2019

👍2

I download http://thetvdb.com/api/F9C450E78D99172E/series/78998/all/en.zip right now and still see empty <SeriesName/>. A forced update inside the SickChill GUI thusly still fails.

I wonder if the cloudfront caching is getting in the way here. I'll try again in a little bit.

On my end the response headers when refreshing inside SickChill GUI show:

{
    'x-amzn-RequestId': '0ed140d7-dd19-4f4b-9eca-bb9b2f857cf6',
    'Content-Rating': '',
    'Content-Length': '47202',
    'Via': '1.1 f126db770f21f288439f44d357978a27.cloudfront.net (CloudFront), 1.1 400e19a7f70282e0817451f6606ca8f9.cloudfront.net (CloudFront)',
    'X-Cache': 'Hit from cloudfront',
    'Content-Disposition': 'attachment; filename="en.zip.zip"',
    'Content-Transfer-Encoding': 'binary',
    'Age': '1420',
    'x-amz-apigw-id': 'DRZ2hEJ0PHcFwsA=',
    'X-Amzn-Trace-Id': 'Root=1-5dd07029-626e3a00be61cea0e51f6080;Sampled=0',
    'Connection': 'keep-alive',
    'X-Amz-Cf-Id': '0G0lrb8Yjutzcr-5rqgKD1JwaE1ad78oJD4NNsIDPV7WkpqaUICC-g==',
    'Content-Description': 'File Transfer',
    'Date': 'Sat, 16 Nov 2019 21:54:50 GMT',
    'Content-Type': 'application/zip'
}

mmhere on 18 Nov 2019

This worked for me:
edit /lib/tvdb_api/tvdb_api.py
find all %s.xml
replace all with %s.zip.xml
restart sickchill
now able to add shows.

This worked for me as well (show updating works), of course it's in a docker container so whenever that updates it will revert.

LeeThompson on 18 Nov 2019

The API is live even though the GUI based interactive website seems up and down today.

API v3 hacks -- how to access valid data the new way

I'm not really suggesting anyone Pythonize the following logic inside tvdb_api.py but an enterprising Pythonista (not me!) might be able to make some headway along these lines.

Doing stuff like the following shell fü to access API v3 does yield correct results. The data can only be obtained in JSON though. I tried replacing the header Accept: application/json with Accept: application/xml as well as Accept: application/XML but it always returns JSON.

If we wanted to hot fix/patch and use v3 right now we'd need to change tvdb_api.py to handle JSON data in Python. I'm not really suggesting anyone do this, it's just that API v3 does seem to be returning well formed data for that show ID (78998) that I've been using as a piñata. ;-)

Also, API v3's so-called _route_ /series/NNNNN (simply an URL such as https://api.thetvdb.com/series/NNNNN) only returns series info and does not include all of the episode details in the single chunk of data that is returned. In API v2 that SickChill uses, a single call returns series header data as well as all episode details as XML; this is probly parsed in one go by SickChill.)

Therefore tvdb_api.py would also need to make subsequent requests to an URL such as https://api.thetvdb.com/series/NNNNN/episodes in order to get all of the episode data. This is also JSON so needs to be parsed via whatever Python JSON library, instead of using the XML Python library as it does today.

Further! /series/NNNNN/episodes only returns a page at a time so there's a bit of pagination loop logic required to slurp up all episode info for long running series that have many episodes.

OK, don't shoot me, but here is an ugly bash script proof of concept that actually gets back all valid data (but in JSON). Definitely not ready for prime time, but it shows that reasonable data is coming back using the newer API.

Don't shoot me! This is fugly shell and piping logic in Linux, relying on bash, curl, sed, grep and python -m json.tool, and writes to local files. Yuck! (Dons kevlar vest)

#!/bin/bash
# interactive API interrogation page;  use URL in browser to manually play (trailing '#' matters):
#   https://api.thetvdb.com/swagger#

#username='YourUserName'    # TODO you must uncomment and fill this in
#userkey='XXXXXXXXXXXXXXXX' # TODO you must uncomment and fill this in
#apikey='YYYYYYYYYYYYYYYY'  # TODO you must uncomment and fill this in

credentials="{\"apikey\":\"$apikey\",\"userkey\":\"$userkey\",\"username\":\"$username\"}"

o=$(curl -s -X POST --header 'Content-Type: application/json'   \
         --header 'Accept: application/json'                    \
         -d "$credentials"                                      \
         'https://api.thetvdb.com/login')

# returns nearly 500 chars in place of ZZZ...Z
# {"token":"ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ"}

# parse out of the above return:  #Fugly
token=$(printf "$o"|sed -e 's/^{"token":"//' -e 's/"}$//')

SeriesID='78998' # TODO you must fill this in, or it could be an arg to this script
language='en'

# get JSON series data:
rm -f series.json # cleanup anything prior
curl  -s -X GET --header 'Accept: application/json'         \
      --header "Accept-Language: ${language}"               \
      --header "Authorization: Bearer ${token}"             \
      "https://api.thetvdb.com/series/${SeriesID}"          |
      python -m json.tool                                   > series.json

# get JSON episode data:
episodes_page=episodes_page.json
episodes_file=episodes.json
rm -f $episodes_page $episodes_file # cleanup anything prior
page=1
get_next_page=1
while ((get_next_page)); do
    curl    -s -X GET --header 'Accept: application/json'   \
            --header "Authorization: Bearer ${token}"       \
            "https://api.thetvdb.com/series/${SeriesID}/episodes?page=${page}" > $episodes_page
    # the end of the page range returns JSON:  {"Error":"No results for your query: map[]"}
    # #EvilShellFoo:
    grep -c '"Error".*No results' $episodes_page > /dev/null; grep_rc=$?
    if ((grep_rc)) ; then # non-zero value implies grep did not see last page condition
        python -m json.tool < $episodes_page >> $episodes_file
    else # grep saw last page condition so stop the loop
        get_next_page=0
    fi
    ((page++))
done
rm -f $episodes_page

mmhere on 18 Nov 2019

FYI on the develop branch for me forcing self.config['useZip'] = False in lib/tvdb_api/tvdb_api.py now works for me except for images.

sickchilluser on 18 Nov 2019

Hah! Excellent catch. I'm debugging further, but requesting non-zip data does indeed refresh the show in the GUI. Update in a moment in this post.

mmhere on 18 Nov 2019

I only find two instances of %s.xml in the .py file... should there be more? Or am I replacing s.xml without the %?

LordXLordX on 18 Nov 2019

There were

I only find two instances of %s.xml in the .py file... should there be more? Or am I replacing s.xml without the %?

There were three instances in mine

zamz84 on 18 Nov 2019

I believe the TBDB guys are re-working this, so it'll be fixed soon anyhow? or will we have a patch written for sickchill?

jaxjexjox on 18 Nov 2019

I only find two instances of %s.xml in the .py file... should there be more? Or am I replacing s.xml without the %?

Include the %

mpretz on 18 Nov 2019

Just to be clear: The "replace %s.xml with %s.zip.xml"-fix is meant against the master-branch, not against develop, right? I guess https://github.com/SickChill/SickChill/pull/5687 changed the relevant occurrence as well... Haven't checked myself, since I'm fine with the self.config['useZip'] = False-fix for the time being.

sickchilluser on 18 Nov 2019

Right. It seems that @sickchilluser's change in tvdb_api.py to request non-zipped data
(self.config['useZip'] = False)
is a great temporary way forward. See notes on _Configuration Info_ below.

When SickChill requests non-zipped data (or you do the same thing in a browser or cmdline tool like wget or curl) correct info comes back (minus pictures) in API v2 format.

In tvdb_api.py, if you temporarily change this as @sickchilluser noted:

        self.config['useZip'] = useZip

to this:

        self.config['useZip'] = False

...then alternate logic fires further on in tvdb_apy.py, where it FIRST requests this URL:
http://thetvdb.com/api/F9C450E78D99172E/series/78998/en.xml

Then SECOND requests this URL:
http://thetvdb.com/api/F9C450E78D99172E/series/78998/all/en.xml

Instead of making a single request to get back a zip package with all data in one place (and which is presently broken by thetvdb.com, containing empty data in many cases), SickChill makes a couple of URL requests to get back uncompressed XML that it (SickChill) goes on to happily parse and use.

A "Force Full Update" in the SickChill web GUI now works for me!

Note that setting self.config['useZip'] = False does not require changing %s.xml to %s.zip.xml because that section of code involving zip data never even runs.

_Configuration Info_

This change should probably be set into the configuration info (haven't found that yet) rather than hard-coding the change in tvdb_api.py——but this hard-coded change does work for now.

Full Details

If you simulate this in a shell you can easily see that all the correct data comes back. Note that there are two URLs below, and both request straight up XML [non-zipped] data:

#!/bin/bash
SeriesID=78998 # can make this a script arg
SickChillAPIkey='F9C450E78D99172E'
wget -O s.xml "http://thetvdb.com/api/${SickChillAPIkey}/series/${SeriesID}/en.xml"
wget -O e.xml "http://thetvdb.com/api/${SickChillAPIkey}/series/${SeriesID}/all/en.xml"
xmllint --format s.xml > s_formatted.xml
xmllint --format e.xml > e_formatted.xml

(Now have a look at the files s_formatted.xml and e_formatted.xml, or if you don't have xmllint, just look at s.xml and e.xml).

Here's what happens inside SickChill:

When the FIRST URL http://thetvdb.com/api/F9C450E78D99172E/series/78998/en.xml is requested inside SickChill, it receives the following headers and the subsequent fully populated XML series data.

Additionally the SECOND URL http://thetvdb.com/api/F9C450E78D99172E/series/78998/all/en.xml requested inside SickChill comes back with the 2nd set of headers/data shown further below. The episode data continues on in its entirety but is curtailed in the snippet shown below.

series request headers

{
  'x-amzn-RequestId': 'd657aeb1-7624-422f-bd6c-61005056c138',
  'Content-Rating': 'TV-G',
  'x-amz-apigw-id': 'DU_xoH56vHcFdZw=',
  'Via': '1.1 fda3b2797d2719576f6b916583a28e52.cloudfront.net (CloudFront), 1.1 5afc8eca980390e71a86518c6f90001a.cloudfront.net (CloudFront)',
  'X-Cache': 'Hit from cloudfront',
  'Transfer-Encoding': 'chunked',
  'Age': '587',
  'Content-Encoding': 'gzip',
  'Vary': 'Accept-Encoding',
  'X-Amzn-Trace-Id': 'Root=1-5dd1e00a-d3f156d05dd48d9f35f01578;Sampled=0',
  'Connection': 'keep-alive',
  'X-Amz-Cf-Id': 'k1fX8HG2BIOzHj9GnyGGT_R89EBhIIZWUE4mCa1JMUDy8zlOZxR9jQ==',
  'Date': 'Mon,
  18 Nov 2019 00:04:26 GMT', 'Content-Type': 'application/xml'
}

series request XML data

<Data>
  <Series>
    <id>78998</id>
    <Actors>|Tiff Needell|Vicki Butler-Henderson|Tom Ford|Jason Plato|Jonny Smith</Actors>
    <Airs_DayOfWeek>Thursday</Airs_DayOfWeek>
    <Airs_Time>9:00 PM</Airs_Time>
    <ContentRating>TV-G</ContentRating>
    <FirstAired>2002-04-08</FirstAired>
    <Genre>|Documentary|Sport|</Genre>
    <IMDB_ID>tt0324679</IMDB_ID>
    <Language>en</Language>
    <Network>Quest</Network>
    <NetworkID>961</NetworkID>
    <Overview>Fifth Gear is a British show and as implied by the title, it's about cars.</Overview>
    <Rating>0</Rating>
    <RatingCount>0</RatingCount>
    <Runtime>60</Runtime>
    <SeriesID>71360</SeriesID>
    <SeriesName>Fifth Gear</SeriesName>
    <Status>Continuing</Status>
    <added/>
    <addedBy/>
    <banner>graphical/78998-g4.jpg</banner>
    <fanart>fanart/original/78998-7.jpg</fanart>
    <lastupdated>1573181733</lastupdated>
    <poster>posters/78998-2.jpg</poster>
    <zap2it_id/>
  </Series>
</Data>

episodes request headers

{
  'x-amzn-RequestId': 'ccfe1cb1-efa5-4a4c-a9ab-840c77301ab8',
  'Content-Rating': 'TV-G',
  'x-amz-apigw-id': 'DU_xrG7pvHcFl0g=',
  'Via': '1.1 0cf6c59c77f0fff670ae085179adc459.cloudfront.net (CloudFront), 1.1 5afc8eca980390e71a86518c6f90001a.cloudfront.net (CloudFront)',
  'X-Cache': 'Hit from cloudfront',
  'Transfer-Encoding': 'chunked',
  'Age': '587',
  'Content-Encoding': 'gzip',
  'Vary': 'Accept-Encoding',
  'X-Amzn-Trace-Id': 'Root=1-5dd1e00a-65d25226059ecc5814a0c344;Sampled=0',
  'Connection': 'keep-alive',
  'X-Amz-Cf-Id': 'OCyBk3ASSPZbX8u1j2nfvD4AQRG6HrDSpN3h2re71WhbycuJn2uu_A==',
  'Date': 'Mon, 18 Nov 2019 00:04:26 GMT',
  'Content-Type': 'application/xml'
}

episodes request XML data (curtailed for brevity)

<Data>
  <Series>
    <id>78998</id>
    <Actors>|Tiff Needell|Vicki Butler-Henderson|Tom Ford|Jason Plato|Jonny Smith</Actors>
    <Airs_DayOfWeek>Thursday</Airs_DayOfWeek>
    <Airs_Time>9:00 PM</Airs_Time>
    <ContentRating>TV-G</ContentRating>
    <FirstAired>2002-04-08</FirstAired>
    <Genre>|Documentary|Sport|</Genre>
    <IMDB_ID>tt0324679</IMDB_ID>
    <Language>en</Language>
    <Network>Quest</Network>
    <NetworkID>961</NetworkID>
    <Overview>Fifth Gear is a British show and as implied by the title, it's about cars.</Overview>
    <Rating>0</Rating>
    <RatingCount>0</RatingCount>
    <Runtime>60</Runtime>
    <SeriesID>71360</SeriesID>
    <SeriesName>Fifth Gear</SeriesName>
    <Status>Continuing</Status>
    <added/>
    <addedBy/>
    <banner>graphical/78998-g4.jpg</banner>
    <fanart>fanart/original/78998-7.jpg</fanart>
    <lastupdated>1573181733</lastupdated>
    <poster>posters/78998-2.jpg</poster>
    <zap2it_id/>
  </Series>
  <Episode>
    <id>306314</id>
    <Combined_episodenumber>10</Combined_episodenumber>
    <Combined_season>7</Combined_season>
    <DVD_chapter>0</DVD_chapter>
    <DVD_discid/>
    <DVD_episodenumber>0</DVD_episodenumber>
    <DVD_season>0</DVD_season>
    <Director/>
    <EpImgFlag/>
    <EpisodeName>Episode 10</EpisodeName>
    <EpisodeNumber>10</EpisodeNumber>
    <FirstAired>2005-05-23</FirstAired>
    <GuestStars/>
    <IMDB_ID/>
    <Language>en</Language>
    <Overview>Tiff drives the incredible new BMW M5 to the famous Pau circuit in the South of France.
...

mmhere on 18 Nov 2019

To simplify. Using the suggestion from @sickchilluser I applied this single line diff in tvdb_apy.py and it all works for me:

$ cd src/SickChill/lib/tvdb_api
$ diff tvdb_api-OLD.py tvdb_api.py
465c465,466
<         self.config['useZip'] = useZip
---
>         #self.config['useZip'] = useZip
>         self.config['useZip'] = False

No other changes were required.

I'm not saying this is a permanent fix (and it will ship more data when doing updates), but at least new episodes can now be populated. Other users have reported that new shows can be added.

_Configuration info_

It would probably be better to change this by altering the data in the indexerConfig section in src/SickChill/sickbeard/indexers/indexer_config.py

That would mean changing this line:

            'useZip': True,

to this:

            'useZip': False,

...but I haven't tried that and then followed it down into the actual code in tvdb_api.py to ensure it arrives there as expected. An exercise left for others smarter than I. Still probly not a permanent solution.

mmhere on 18 Nov 2019

👍2

Worked for me - thanks guys!

LordXLordX on 18 Nov 2019

I know! I talk too much!!

Just as a confirmation, the one-liner self.config['useZip'] = False change hard-coded in tvdb_api.py just now allowed me to refresh (_Force Full Update_ button in the SickChill web GUI) another show: _Madam Secretary_

My local DB for that show did not include the latest episodes and therefore was not prepared to automatically seek the one that is going out tonight. My local DB was only up to date thru last week's episode. Those latest episodes were added into thetvdb.com since all of these problems began two days ago.

Now, however, clicking _Force Full Update_ allowed that show entry to update and be aware of tonight's outing as well a few future episodes. SickChill should now seek them out in its usual automated fashion.

Q.E.D. # for now

mmhere on 18 Nov 2019

I would like to suggest that SickChill adds self.config['useZip'] to the ini file so it can be changed at runtime without code changes (as an advanced user setting obviously).

LeeThompson on 18 Nov 2019

Thanks for the
tvdb_api.py
self.config['useZip'] = False
hack, this worked and I was able to add a show I've been trying to add for days. Also confirm, the pictures didn't load into sickchill ,but that's fine.

vjbonan on 18 Nov 2019

@LeeThompson—Yeah, that's a great idea. SickChill/sickbeard/indexers/indexer_config.py is clearly configuration data. I'm not expert enough to figure out how to let that wend its way out thru all the GUI layers (that mako stuff is complicated).

One certainly shouldn't have to edit a .py file to effect this change. Some of the other config data could be brought out as well. language and apikey come to mind. Possibly even the api URL base_url although that's potentially a bit more risky unless a user is truly an expert.

mmhere on 18 Nov 2019

I wouldn't even put useZIP in the GUI, the .ini would be fine.

LeeThompson on 18 Nov 2019

I'd prefer not to be hand editing files to fix this kind of stuff, then again - it's a pretty infrequent issue, isn't it.

jaxjexjox on 18 Nov 2019

Here's a sketch for an _untested_ horrible hack using global-level config.ini settings with debug as a model. It'd be better to fully support a [THETVDB] specific section in config.ini but I haven't dug far enough to see how to do that.

In sickbeard/__init__.py there are a bunch of globals.

They're declared using the global statement just inside this method declaration: def initialize(consoleLogging=True):, so notice there's a , DEBUG in there.

Then they are initialized to default values further up in that file, note DEBUG = False.

Then they are overridden in code like this further on in the initialize() method. We care about the DEBUG = here but the other stuff is present for context:

        check_section(CFG, 'SABnzbd')
        check_section(CFG, 'KODI')
...
        DEBUG = check_setting_bool(CFG, 'General', 'debug')
        GUI_LANG = check_setting_str(CFG, 'GUI', 'language')
        INDEXER_DEFAULT_LANGUAGE = check_setting_str(CFG, 'General', 'indexerDefaultLang', 'en')
...

(The following is not great because the portion of the name "thetvdb" is bound into the key instead of having a [THETVDB] section with a key use_zip inside of it) but if you wanted to name the config.ini setting thetvdb_use_zip so that you could set it to 1 or 0 like other booleans in config.ini then:

All in sickbeard/__init__.py:

add THETVDB_USE_ZIP = True above def initialize(consoleLogging=True):
add , THETVDB_USE_ZIP in the global statement inside initialize()
add THETVDB_USE_ZIP = check_setting_bool(CFG, 'General', 'thetvdb_use_zip') in the body of initialize() near other similar settings retrievals.

The hack we presently have in tvdb_api.py:

        #self.config['useZip'] = useZip
        self.config['useZip'] = False

could then presumably be replaced with:

        #self.config['useZip'] = useZip
        #self.config['useZip'] = False
        # Look up the global from config.ini:
        self.config['useZip'] = sickbeard.THETVDB_USE_ZIP

Now in config.ini add:

thetvdb_use_zip = 0

Yuck! Aberrant use of evil globals like I was taught to avoid ages ago!! But the precedents are already set.

mmhere on 18 Nov 2019

%s.zip.xml

This worked. I am now able to add new shows.

Thank you.

troypressley on 18 Nov 2019

(about 7 hours ago I said)
I download http://thetvdb.com/api/F9C450E78D99172E/series/78998/all/en.zip right now and still see empty <SeriesName/>. A forced update inside the SickChill GUI thusly still fails.

I wonder if the cloudfront caching is getting in the way here. I'll try again in a little bit.

I just re-tried this now. This time the zip compressed results come back with actual data, where before most of the fields were empty. Maybe thetvdb is making progress?

This means it would work with the %s.zip.xml change proposed a few days earlier.

<Data>
  <Series>
    <id>78998</id>
    <Actors>|Tiff Needell|Vicki Butler-Henderson|Tom Ford|Jason Plato|Jonny Smith|Tim Shaw|Jon Bentley|</
    <Airs_DayOfWeek>Thursday</Airs_DayOfWeek>
    <Airs_Time>9:00 PM</Airs_Time>
    <ContentRating>TV-G</ContentRating>
    <FirstAired>2002-04-08</FirstAired>
    <Genre>|Documentary|Sport|</Genre>
    <IMDB_ID>tt0324679</IMDB_ID>
    <Language>en</Language>
    <Network>Quest</Network>
    <NetworkID>961</NetworkID>
    <Overview>Fifth Gear is a British show and as implied by the title, it's about cars. The hosts have m
    <Rating>0</Rating>
    <RatingCount>0</RatingCount>
    <Runtime>60</Runtime>
    <SeriesID>71360</SeriesID>
    <SeriesName>Fifth Gear</SeriesName>
    <Status>Continuing</Status>
    <added/>
    <addedBy/>
    <banner>graphical/78998-g4.jpg</banner>
    <fanart>fanart/original/78998-7.jpg</fanart>
    <lastupdated>1573181733</lastupdated>
    <poster>posters/78998-2.jpg</poster>
    <zap2it_id/>
  </Series>
...

Note that <SeriesName>Fifth Gear</SeriesName> is now filled in where it was missing earlier.

So maybe the zip mode is OK going forward?

I'm sticking with the cleartext XML mod I have in place for now.

mmhere on 18 Nov 2019

Update from thetvdb staff

Emby appears to be expecting en.xml, and is throwing an error because the file is actually named en.zip.xml.

We're going to fix it. It's already been ticketed for our API team. - ChristyEzzell

LeeThompson on 18 Nov 2019

Isn't it like Day 3 for them? ...

jaxjexjox on 18 Nov 2019

Isn't it like Day 3 for them? ...

Yup, and they haven't been communicating well either. Pretty embarrassing fiasco.

LeeThompson on 18 Nov 2019

By this time the last time around when V3 escaped out the barn doors, they had corralled it, put it back, and reverted to V2.

mmhere on 18 Nov 2019

This worked for me:
edit /lib/tvdb_api/tvdb_api.py
find all %s.xml
replace all with %s.zip.xml
restart sickchill
now able to add shows.

This worked for me as well (show updating works), of course it's in a docker container so whenever that updates it will revert.

I'm a bit gobbled up in work to do this myself, but could you see if you can add a new show with this hack>

WebSpider on 18 Nov 2019

The massupdate itself is working for me now. Thanks for all your hints.
The Traktsync isn't working anymore, any ideas for this to?
WARNING TRAKTCHECKER :: Could not check trakt progress for xxxxx because the imdb id is missing from tvdb data, skipping

WebBuddha on 18 Nov 2019

Just a note, as we await thetvdb fixing their stuff, that images are still not loading.

I mention it not because it is anything new, but because some stuff has begun working as changes occur over at thetvdb. For example populating new episodes into a show page or adding a new show weren't working earlier but began to work some time last night (as long as you use the %s.zip.xml change touched upon several times above).

I believe image loading issues are unrelated to SickChill's particular use of the API as other clients (such as Kodi) are noted elsewhere to have the same image loading issues.

mmhere on 18 Nov 2019

For reference, thetvdb staff finally started a thread consolidating all known issues and their status:

https://forums.thetvdb.com/viewtopic.php?f=122&p=163719#p163719

LeeThompson on 19 Nov 2019

👍2

Wow, that's kind of late, day 4?

jaxjexjox on 19 Nov 2019

I just tried again - I still ahve my 'docker' setup - which is a day or two old. "stock" Sickchill is still unable to do this, so I'm guessing the API backend issues are still in play, regardless of the fixes here.

jaxjexjox on 19 Nov 2019

I just tried again - I still ahve my 'docker' setup - which is a day or two old. "stock" Sickchill is still unable to do this, so I'm guessing the API backend issues are still in play, regardless of the fixes here.

Do we think "stock" should ever work again? I mean, if the change to return en.zip.xml is always going to be present going forward, then seeking en.xml on our side will always fail.

Perhaps we should double clutch. Check one way first, then the second upon failure. If thetvdb reverts to shipping en.xml like they used to, it'd work either way.

We could also make non-zip a fallback maybe. I.e., zip fails in any way (encoding doesn't work as we expect, or the file sought isn't present), go after non-zip and seek en.xml. That [non-zip, seeking en.xml] is actually the setup I've had running in my instance successfully for a couple of days now.

mmhere on 19 Nov 2019

@LeeThompson
For reference, thetvdb staff finally started a thread consolidating all known issues and their status:
https://forums.thetvdb.com/viewtopic.php?f=122&p=163719#p163719

Thanks for finding that. Very useful.

If thetvdb has gone so far to collect all of that info and claim to be working on various parts (and they haven't punted back to V2 entirely for the site and API like they did when they first attempted this earlier this year) I'd say it's safe to say they really are going forward with V3 this time. No putting that horse back in the barn then.

mmhere on 19 Nov 2019

"Do we think "stock" should ever work again? I mean, if the change to return en.zip.xml is always going to be present going forward, then seeking en.xml on our side will always fail."

Stock will 100% work again, sickchill devs are a little slower than sonar but (to my knowledge) haven't retired, this one will be fixed at this end I suspect but also, I expect TVDB will infact fix this mess over the coming days. Dragging their feet but it's a free service isn't it.

jaxjexjox on 19 Nov 2019

I think I wrote something confusing. I meant will the existing version unmodified work again. That's what I meant when I quoted "stock" from your earlier post.

I know that this project as a whole will work again once the fixes are applied, and then that will newly become "stock". Apologies for any misunderstanding.

I believe several of us already have variants of fixes running, so it's just a matter of getting a combined/vetted version of those into mainline code after appropriate review and testing.

Cheers!

mmhere on 19 Nov 2019

TL;DR version, please. What do we need to do to be able to add shows again? Or do we just sit tight and wait?

qbitza on 19 Nov 2019

TL;DR version, please. What do we need to do to be able to add shows again? Or do we just sit tight and wait?

Lazy method, sit and wait

edit /lib/tvdb_api/tvdb_api.py
find all %s.xml
replace all with %s.zip.xml
restart sickchill

This method will fix it, but you wont get any show pictures until its officially fixed.

Matthewserta on 19 Nov 2019

👍4

Perhaps we should double clutch. Check one way first, then the second upon failure. If thetvdb reverts to shipping en.xml like they used to, it'd work either way.

We could also make non-zip a fallback maybe. I.e., zip fails in any way (encoding doesn't work as we expect, or the file sought isn't present), go after non-zip and seek en.xml. That [non-zip, seeking en.xml] is actually the setup I've had running in my instance successfully for a couple of days now.

I think this makes a lot of sense (all of it), SickChill should do fallbacks (perhaps even API level fallbacks) as needed to handle web service data hiccups/issues.

SickChill also needs to have support for alternative data sources (and perhaps an advanced manual add/edit although that would require a lot of UI work). While not perfect: themoviedb (it's TV data kinda sucks but might be better than nothing if there's an outage at thetvdb), imdb, tribune media services (perhaps via Schedules Direct [http://www.schedulesdirect.org/], which is a pay service but they do give free dev keys so it could at least be coded and available).

LeeThompson on 19 Nov 2019

TL;DR version, please. What do we need to do to be able to add shows again? Or do we just sit tight and wait?

Lazy method, sit and wait

edit /lib/tvdb_api/tvdb_api.py
find all %s.xml
replace all with %s.zip.xml
restart sickchill

This method will fix it, but you wont get any show pictures until its officially fixed.

This worked for me.

cadmuslabslex on 19 Nov 2019

This worked for me but now when i do a mass update I am getting an error. The error is: Unable to write file to D:TVShow Name Hereseries.xml - are you sure the folder is writable? error 13 : Permission denied.

I have given every possible permission I can think of to that directory. If anyone knows what Is wrong please let me know.

cadmuslabslex on 19 Nov 2019

This worked for me but now when i do a mass update I am getting an error. The error is: Unable to write file to D:TVShow Name Hereseries.xml - are you sure the folder is writable? error 13 : Permission denied.

I have given every possible permission I can think of to that directory. If anyone knows what Is wrong please let me know.

HI, try to run the service Sickchill specifying the account (account should have r/w access to the file repository)

ca-z on 19 Nov 2019

TL;DR version, please. What do we need to do to be able to add shows again? Or do we just sit tight and wait?

Lazy method, sit and wait

edit /lib/tvdb_api/tvdb_api.py
find all %s.xml
replace all with %s.zip.xml
restart sickchill

This method will fix it, but you wont get any show pictures until its officially fixed.

Thanks for providing this!

Currently I'm pressed for time, but if someone could make a merge request, that would help enormously.

Edit: Just seen #5701 reviewing that now

WebSpider on 19 Nov 2019

Guys, please test the develop branch, it has #5701

WebSpider on 19 Nov 2019

develop branch fixes the issue for me.

mschout on 19 Nov 2019

Hi,

Can you tell me please if docker image (https://hub.docker.com/r/sickchill/sickchill/) can be used again?
Can I use develop branch there also?

Best regards

kretzu77 on 19 Nov 2019

Develop branch is now working for me. Reverted any changes I made before pulling fresh develop version. However, one issue I'm having is with metadata. Seems to be retrieving foreign metadata for images on certain shows. Show posters seem to be worst affected. Examples include Travel Man & The IT crowd. The IT Crowd for instance is pulling a hebrew poster.

jodsclass on 19 Nov 2019

@qbitza
TL;DR version, please.

TL;DR version A

As mentioned above, in lib/tvdb_api/tvdb_api.py, find this particular line (iine#614 in main branch):

                return xmltodict.parse(myzipfile.read('%s.xml' % language), postprocessor=process)

and replace %s.xml with %s.zip.xml.

This final result is what's on the develop branch that others are using successfully:

                return xmltodict.parse(myzipfile.read('%s.zip.xml' % language), postprocessor=process)

_Restart SickChill after changes._

TL;DR version B

In sickbeard/indexers/indexer_config.py find 'useZip': True, and replace with 'useZip': False,.

_Restart SickChill after changes._

mmhere on 19 Nov 2019

The develop branch is working for me. However I did notice some warnings and errors when adding a test show.

2019-11-19 12:13:35 ERROR ThreadPoolExecutor-0_7 :: [e9a82d9] Missing time zone for network: Disney+. Check valid network is set in indexer (theTVDB) before filing issue.
2019-11-19 12:12:40 INFO SEARCHQUEUE-BACKLOG-361753 :: Beginning backlog search for: [The Mandalorian]
2019-11-19 12:12:39 INFO SHOWQUEUE-ADD :: Done cache check
2019-11-19 12:12:39 WARNING SHOWQUEUE-ADD :: There was an error trying to retrieve the image, aborting
2019-11-19 12:12:39 INFO SHOWQUEUE-ADD :: Request failed: 403 Client Error: Forbidden for url: https://thetvdb.com/banners//series/361753/backgrounds/61998170.jpg ()
2019-11-19 12:12:38 WARNING SHOWQUEUE-ADD :: There was an error trying to retrieve the image, aborting
2019-11-19 12:12:38 INFO SHOWQUEUE-ADD :: Request failed: 403 Client Error: Forbidden for url: https://thetvdb.com/banners//series/36175362000530.jpg ()
2019-11-19 12:12:37 WARNING SHOWQUEUE-ADD :: There was an error trying to retrieve the image, aborting
2019-11-19 12:12:37 INFO SHOWQUEUE-ADD :: Request failed: 403 Client Error: Forbidden for url: https://thetvdb.com/banners//series/36175362000530.jpg ()
2019-11-19 12:12:36 INFO SHOWQUEUE-ADD :: Copying from /media/TV/The Mandalorian/cover.jpg to /var/db/sickrage/cache/images/361753.poster.jpg
2019-11-19 12:12:36 INFO SHOWQUEUE-ADD :: 361753: Updating NFOs for show with new indexer info
2019-11-19 12:12:35 INFO SHOWQUEUE-ADD :: Launching backlog for this show since its episodes are WANTED
2019-11-19 12:12:26 INFO SHOWQUEUE-ADD :: Setting all episodes to the specified default status: 5
2019-11-19 12:12:25 INFO SHOWQUEUE-ADD :: 361753: Unable to find the show in the database
2019-11-19 12:12:25 INFO SHOWQUEUE-ADD :: theTVDB: {u'useZip': True, u'apikey': u'REMOVED KEY', u'cache': u'/var/db/sickrage/cache/indexers/theTVDB', u'language': 'en'}
2019-11-19 12:12:25 INFO SHOWQUEUE-ADD :: Starting to add show by ShowDir: /media/TV/The Mandalorian

The show I added did show up in the Poster view correctly with a correct picture. Now I have blank spot between two TV shows. I inspect the source and the simple layout. There should be no show where the blank post is. See the picture below.

JeremiahGillis on 19 Nov 2019

@JeremiahGillis Some stuff is still broken at thetvdb's end. If you scroll up a bit you'll see a link to thetvdb's forum where they are now tracking all known issues and give their status. (Most of it is still "In progress").

LeeThompson on 19 Nov 2019

@LeeThompson @JeremiahGillis —
Image stuff is generally still broken at thetvdb. People report Kodi also having issues getting images.

mmhere on 19 Nov 2019

@LeeThompson @mmhere Thanks. I see this now. Also Plex is having issues grabbing the images. Fun stuff!

JeremiahGillis on 19 Nov 2019

@qbitza
TL;DR version, please.

TL;DR version A

As mentioned above, in lib/tvdb_api/tvdb_api.py, find this particular line (iine#614 in main branch):
                return xmltodict.parse(myzipfile.read('%s.xml' % language), postprocessor=process)
and replace %s.xml with %s.zip.xml.

This final result is what's on the develop branch that others are using successfully:
                return xmltodict.parse(myzipfile.read('%s.zip.xml' % language), postprocessor=process)
_Restart SickChill after changes._

TL;DR version B

In sickbeard/indexers/indexer_config.py find 'useZip': True, and replace with 'useZip': False,.

_Restart SickChill after changes._

My problem is that I'm too noob to do that inside docker (on synology) :(

kretzu77 on 19 Nov 2019

@qbitza
TL;DR version, please.

TL;DR version A

As mentioned above, in lib/tvdb_api/tvdb_api.py, find this particular line (iine#614 in main branch):
                return xmltodict.parse(myzipfile.read('%s.xml' % language), postprocessor=process)
and replace %s.xml with %s.zip.xml.
This final result is what's on the develop branch that others are using successfully:
                return xmltodict.parse(myzipfile.read('%s.zip.xml' % language), postprocessor=process)
_Restart SickChill after changes._

TL;DR version B

In sickbeard/indexers/indexer_config.py find 'useZip': True, and replace with 'useZip': False,.
_Restart SickChill after changes._
My problem is that I'm too noob to do that inside docker (on synology) :(

Within SickChill goto Settings -> General -> Advanced Settings -> GitHub subsection
Fill in your GitHub credentials and click Save Changes
Once changes are saved, in the same GitHub section, drop down Branch Version and select Develop and then click Checkout Branch.
SickChill will download the develop branch and restart. You'll now be on the temporary fix.

chesh1r3k on 19 Nov 2019

Synology Docker edit (note, of course if the container gets updated, these changes will be lost)
(This is a general guide, not for SickChill specifically)

Log into DSM
Open the Docker app
Click "Containers" in the left pane
Highlight the container you wish to edit and click on "Details" on the top button bar.
Click the "Terminal" tab
Click the "Create" button
In the list of terminals, you should now see one called "bash", click on it.
You are now in the shell of the container.
When you're done, closing the detail screen will close any terminals.

NOTE: The commands available to you in the container will depend on what has been provided. The linuxserver/sickchill container does come with vi etc so you can edit files.

LeeThompson on 20 Nov 2019

the new image location at https://artworks.thetvdb.com.

so not more thetvdb.com/banners/etc

On Wed, Nov 20, 2019 at 12:12 AM Lee Thompson notifications@github.com
wrote:

Synology Docker edit (note, of course if the container gets updated, these
changes will be lost)
(This is a general guide, not for SickChill specifically)

Log into DSM

Open the Docker app

Click "Containers" in the left pane

Highlight the container you wish to edit and click on "Details" on
the top button bar.

Click the "Terminal" tab

Click the "Create" button

In the list of terminals, you should now see one called "bash"
click on it.

You are now in the shell of the container.

NOTE: The commands available to you in the container will depend on what
has been provided. The linuxserver/sickchill container does come with vi
etc so you can edit files.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/SickChill/SickChill/issues/5686?email_source=notifications&email_token=AATWOEPLRQ5DBFUG7I32KW3QURXF7A5CNFSM4JNUGBY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEEQDMNA#issuecomment-555759156,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AATWOEOV6Q4SJFICDT6HGA3QURXF7ANCNFSM4JNUGBYQ
.

Wally73 on 20 Nov 2019

I had done the mod where you change %s.xml to %s.zip.xml and all was working until today.. I tried to go to the develop branch to see if there was a better fix in place, nothing worked.. So i went back to master, and after redoing the %s.zip.xml It doesnt work at all now..

AA tvdb_attributenotfound: Found 332747, but attribute 'seriesname' was empty.
AA "Found {0}, but attribute 'seriesname' was empty.".format(self.indexerid))
AA File "C:SickChillSickChillsickbeardtv.py", line 853, in loadFromIndexer
AA self.show.loadFromIndexer(cache=not self.force)
AA File "C:SickChillSickChillsickbeardshow_queue.py", line 651, in run
AA Traceback (most recent call last):

Im guessing something has changed on the TVdb's end of things, But who knows at this point..

Matthewserta on 20 Nov 2019

@Matthewserta Try it again, it should work (I just did it manually with the API test tool)... if it doesn't it's possible that the series name may be throwing something (thetvdb or sickchill) for a loop due to an accented character:

{
  "data": {
    "id": 332747,
    "seriesId": "",
    "seriesName": "90 Day Fiancé: Before The 90 Days",
    "aliases": [],

As for thetvdb status I would bookmark this URL: https://forums.thetvdb.com/viewtopic.php?f=122&t=60239

LeeThompson on 20 Nov 2019

Ones it is fixed will the meta data poster fix them self, Or will we need to readd the show ?

Snowaks on 20 Nov 2019

Ones it is fixed will the meta data poster fix them self, Or will we need to readd the show ?

Nah, you can force an update and it’ll download all the missing.

Matthewserta on 20 Nov 2019

The TL;DR version B change above (use no zip, so uncompressed) has been working for me for a few days. This was also working when the zip side of the API was returning empty data even _with_ the %s.zip.xml change.

mmhere on 20 Nov 2019

Switching to dev branch failed for me (Synology). Couldn't download, didn't restart. I'll try method A when I'm not so bleary-eyed, if the main branch isn't updated first.

tinyhands01 on 20 Nov 2019

I think TL;DR version A worked for me. I tried to add an existing show and got an error -- but not related to getting any data from TVDB. I went and added a random show from popular imdb lists and it added with no errors. Thank you.

didyouexpectthat on 20 Nov 2019

Switching to dev branch failed for me (Synology). Couldn't download, didn't restart. I'll try method A when I'm not so bleary-eyed, if the main branch isn't updated first.

It's the same for my on my Synology NAS. No possibility to change to "devolop" via GUI.

WebBuddha on 20 Nov 2019

Have switched to develop branch and the searching is working for me now. Getting content errors though that are hopefully unrelated to sickchill. EG:

2019-11-20 18:25:23 WARNING  SHOWQUEUE-ADD :: Show in /media/Video/TV/The Witcher has no name on theTVDB, probably searched with the wrong language. Delete .nfo and add manually in the correct language.
2019-11-20 18:25:22 INFO     SHOWQUEUE-ADD :: theTVDB: {u'useZip': True, u'apikey': u'XXXX', u'cache': u'/home/*****/SickRage/cache/indexers/theTVDB', u'language': 'en'}
2019-11-20 18:25:22e INFO     SHOWQUEUE-ADD :: Starting to add show by ShowDir: /media/Video/TV/The Witcher
2019-11-20 18:22:32 INFO     SHOWQUEUE-ADD :: Show The Mandalorian is on theTVDB but contains no season/episode data.
2019-11-20 18:22:32 INFO     SHOWQUEUE-ADD :: theTVDB: {u'useZip': True, u'apikey': u'XXXX', u'cache': u'/home/*****/SickRage/cache/indexers/theTVDB', u'language': 'en'}
2019-11-20 18:22:32 INFO     SHOWQUEUE-ADD :: Starting to add show by ShowDir: /media/Video/TV/The Mandalorian

Adding WWE NXT worked fine though

brendanmig on 20 Nov 2019

@brendanmig that's odd, as this URL http://thetvdb.com/api/F9C450E78D99172E/series/361753/all/en.zip, which is what SickChill uses to pull up series data en masse (the API key is tied to SickChill itself, the six digit ID is the one for _The Mandalorian_) presently brings back the file en.zip.zip if you plug the URL into a browser. In turn, en.zip.zip contains en.zip.xml which appears to show fully populated data. I retrieved this live (just now) from thetvdb.com

Is it possible your cache (/home/*/SickRage/cache/indexers/theTVDB) contains old data from a few days ago when in fact thetvdb.com was returning an XML tree but with empty data?

I mean, maybe it's hitting local cache, getting old/blank info, and having issues then.

mmhere on 20 Nov 2019

@mmhere My cache directory is empty. I also haven't tried downloading The Witcher before, that one was a brand new test. But that's showing a different error in any case

brendanmig on 20 Nov 2019

@brendanmig Huh. Got me curious so I checked http://thetvdb.com/api/F9C450E78D99172E/series/362696/all/en.zip for The Witcher, which also returns valid en.zip.zip with en.zip.xml inside, populated with real data. Presumably you were after the 2019 series.

So I tried adding that show to my queue and it worked! Mostly; note the warning below.

For completeness my setup is using TL;DR version B I posted above. Haven't tried version A to see if that matters.

It does produce the following in the warning log, but we know there are presently image problems at thetvdb.com. Oddly it did get the little banner image, it just couldn't get the portrait-oriented poster.

2019-11-20 00:28:11 DEBUG    SHOWQUEUE-ADD :: Fetching image from http://thetvdb.com/banners//series/362696/backgrounds/1398887.jpg
2019-11-20 00:28:12 INFO     SHOWQUEUE-ADD :: Request failed: 403 Client Error: Forbidden for url: https://thetvdb.com/banners//series/362696/backgrounds/1398887.jpg
2019-11-20 00:28:13 WARNING  SHOWQUEUE-ADD :: There was an error trying to retrieve the image, aborting

For someone working on image stuff in the near future: note that the 2nd line (INFO) shows a client auth error. This smells of a need to use API V3 on this side (or better: for thetvdb to finish their support for V2 so it doesn't need the full double handshake with login tokens the way V3 does it -- that is, there should be a single /banners/... URL that includes the API key shown above that would get the image).

mmhere on 20 Nov 2019

Gotta be honest, after this many days, getting unhappy with tvdb at this point.

Realise that this isn't sick chill problem at all, obv

jaxjexjox on 20 Nov 2019

@mmhere Even stranger, both The Witcher (and yes, 2019 series) and The Mandalorian have the same set if elements in their <Series> element, although a bunch of The Witcher ones are empty I guess cause it's hasn't actually aired yet. Both have SeriesName elements which I assume is what it's looking for. Double checked with Watchmen which does have episodes (360733) and that also gives the 'no name' error

brendanmig on 20 Nov 2019

Double checked I was on develop branch and had a clean up to date clone.

x@y:~/SickRage$ git status
# On branch develop
nothing to commit (working directory clean)
x@y: SickRage$ git pull
Already up-to-date.

Looked at the WWE NXT xml (354505 which does get added correctly) and can't see any fundamental difference between it and the other shows I tried (well, NXT has an empty Actors element). Also had a quick parse through the code, and again, can't see anything obviously wrong there. Not that I've done more than grep for the error messages I'm seeing and poking at what might cause the problem.

brendanmig on 20 Nov 2019

@brendanmig Makes me wonder if some of the network reliability services (cloudflare, et al) are getting in the way. When I was investigating this stuff during the early stages of this outage, I dumped the http response headers to the log. On my end it came thru some Amazon (AWS) network layers with cloudflare showing cache hits. Apologies for leaning on the cache idea again, but I wonder if some of that sort of thing could be getting in the way. Always hard to debug, that.

Maybe after a wait of some hours for propagation it would start working for you? When the data was partially empty for me back then, I waited and it started getting good data the next day.

As an aside I wish there were a way to tell those network cache services to flush. I mean, it's nice to have the redundancy, locality, and DOS-attack prevention, but sometimes you want the real client and the real server to talk to each other.

(If you look far above in this issue page right here you can see the headers I saw at that time.)

mmhere on 20 Nov 2019

@mmhere, yeah, I know what you mean, caching is hard. And cache-busting is even harder. Would generally expect a browser request and a programmatic request from the same location to get the same thing, but who the hell knows what cloudfront etc is doing under the hood. And caching is definitely a legit suspect here, even from my basic understanding of the XML parsing I can't see how I could be getting the errors I am if it's getting the XML my browser is returning. Will see how things go tomorrow, and if I get a chance will try some headers and response dumps.

brendanmig on 20 Nov 2019

@brendanmig You know another cache suspect is the SQLite cache.db file that SickChill keeps around on disk in its local dir. The python code actively seeks to avoid network hits off the box by looking there first and only goes around that after a pretty long timeout. They don't want to blow whatever request-count budget they may have at the various indexers.

I know that when I've edited meta data at www.thetvdb.com and wanted it to show up as soon as possible in SickChill, I would have to shut down SickChill, remove sickchill/cache.db and restart SickChill. If I didn't do this there would be a noticeable delay before any forced updates in the SickChill GUI showed new results locally.

$ cd $HOME/SickChill # or wherever
$ ls -oh cache.db
-rw-r--r-- 1 username 304K Nov 20 03:17 cache.db
# shutdown SickChill
$ rm -f cache.db
# start SickChill

The cache.db does live across restarts so has to be manually removed to truly clear it. The removal of cache.db is sanctioned by notes I read on the wiki long ago.

I know you mentioned your local cache(s) were clean, but I wonder if you tried this trick?

If I'm fast enough I select restart in the GUI and blow away cache.db while the automatic restart is trundling along. The cache file is created right away again (empty).

mmhere on 20 Nov 2019

@mmhere, nup, tried blowing cache.db out and it didn't help. get the same results as before.

brendanmig on 20 Nov 2019

I switched to develop branch, and deleted cache/ and cache.db and that seems to work fine.

m1dnight on 20 Nov 2019

Switched to develop branch and and it is slowly updating all shows. I'm also able to add new shows.

nielsaers on 20 Nov 2019

Anyone here using dockSTARTer? wondering if we have a work around for that yet?

mildfuzz on 20 Nov 2019

2019-11-20 16:11:29 ThreadPoolExecutor-0_2 :: [e9a82d9] Failed doing webui callback: Traceback (most recent call last):
File "/app/sickchill/sickchill/views/index.py", line 180, in async_call
result = function(**kwargs)
TypeError: set_site_message() takes exactly 4 arguments (1 given)

kretzu77 on 20 Nov 2019

Verified I was able to add a show now, but still some 403's here and there. It also downloaded some Russian poster. Appreciate everyones work here.

2019-11-20 06:39:46 WARNING SHOWQUEUE-ADD :: There was an error trying to retrieve the image, aborting
2019-11-20 06:39:46 INFO SHOWQUEUE-ADD :: Request failed: 403 Client Error: Forbidden for url: https://thetvdb.com/banners//series/36175362000530.jpg ()
2019-11-20 06:39:41 WARNING SHOWQUEUE-ADD :: There was an error trying to retrieve the image, aborting
2019-11-20 06:39:41 INFO SHOWQUEUE-ADD :: Request failed: 403 Client Error: Forbidden for url: https://thetvdb.com/banners//series/36175362000530.jpg ()
2019-11-20 06:39:38 INFO SHOWQUEUE-ADD :: Copying from /Volumes/RAIDSHIT/TV Shows/The Mandalorian/folder.jpg to /Applications/SickChill/cache/images/361753.poster.jpg
2019-11-20 06:39:38 INFO SHOWQUEUE-ADD :: 361753: Updating NFOs for show with new indexer info
2019-11-20 06:39:32 INFO SHOWQUEUE-ADD :: Launching backlog for this show since its episodes are WANTED
2019-11-20 06:39:25 INFO SHOWQUEUE-ADD :: Setting all episodes to the specified default status: 5
2019-11-20 06:39:24 INFO SHOWQUEUE-ADD :: 361753: Unable to find the show in the database
2019-11-20 06:39:22 INFO SHOWQUEUE-ADD :: theTVDB: {u'useZip': True, u'apikey': u'F9C450E78D99172E', u'cache': u'/Applications/SickChill/cache/indexers/theTVDB', u'language': 'en'}

the-davidtran on 20 Nov 2019

I can't activate the develop branch: 2019-11-20 17:37:27 ThreadPoolExecutor-0_1 :: [e094759] git checkout -f develop returned : error: pathspec 'develop' did not match any file(s) known to git.

Any thoughts?

zkye on 20 Nov 2019

I can't activate the develop branch: 2019-11-20 17:37:27 ThreadPoolExecutor-0_1 :: [e094759] git checkout -f develop returned : error: pathspec 'develop' did not match any file(s) known to git.

Any thoughts?

You can try navigating to the root directory of SickChill and running git fetch -pa to see if it updates any refs. It may help, but it could also be your git installation causing problems, so it would be a good idea to check if git is in your PATH env and properly installed.

Also, I am still having issues adding The Mandalorian, trying to add it via the "IMDB's Popular Shows" and clicking the "Add Show" button results in the following in the logs only:
2019-11-20 17:14:33 INFO ThreadPoolExecutor-0_3 :: theTVDB: {u'useZip': True, u'apikey': u'F9C450E78D99172E', u'cache': u'/media/sdk1/home/*****/SickChill/cache/indexers/theTVDB', u'language': u'en'}

And then nothing is actually added to my show list.

I'm on the develop branch with neither of the above fixes manually applied. I was able to add several other shows with the above IMDB method, just not The Mandalorian.

When I try to add it manually via "Add New Show" I see the following in the logs:
2019-11-20 17:19:37 INFO SHOWQUEUE-ADD :: Starting to add show by ShowDir: /media/sdk1/home/*****/Shows/The Mandalorian 2019-11-20 17:19:37 INFO SHOWQUEUE-ADD :: theTVDB: {u'useZip': True, u'apikey': u'F9C450E78D99172E', u'cache': u'/media/sdk1/home/*****/SickChill/cache/indexers/theTVDB', u'language': 'en'} 2019-11-20 17:19:37 WARNING SHOWQUEUE-ADD :: Show in /media/sdk1/home/*****/Shows/The Mandalorian has no name on theTVDB, probably searched with the wrong language. Delete .nfo and add manually in the correct language.

I've also manually deleted the local directory for The Mandalorian after several failed attempts to add it properly, and I've also attempted to add it after deleting the cache.db and contents of the cache directory. Seems to be an issue with this show specifically.
I wonder if The Mouse is involved? 😆

Spartanaco on 20 Nov 2019

I can't activate the develop branch: 2019-11-20 17:37:27 ThreadPoolExecutor-0_1 :: [e094759] git checkout -f develop returned : error: pathspec 'develop' did not match any file(s) known to git.
Any thoughts?

You can try navigating to the root directory of SickChill and running git fetch -pa to see if it updates any refs. It may help, but it could also be your git installation causing problems, so it would be a good idea to check if git is in your PATH env and properly installed.

Nope doesnt work still. I hope they fix it soon

zkye on 20 Nov 2019

I’m able to get develop branch through sickchill settings. Have you tried
that?

On Wed, Nov 20, 2019 at 9:35 AM zkye notifications@github.com wrote:

I can't activate the develop branch: 2019-11-20 17:37:27
ThreadPoolExecutor-0_1 :: [e094759
https://github.com/SickChill/SickChill/commit/e094759640ca61c5c033bf1ff9d33bc7fc1ba857]
git checkout -f develop returned : error: pathspec 'develop' did not match
any file(s) known to git.
Any thoughts?

You can try navigating to the root directory of SickChill and running git
fetch -pa to see if it updates any refs. It may help, but it could also
be your git installation causing problems, so it would be a good idea to
check if git is in your PATH env and properly installed.

Nope doesnt work still. I hope they fix it soon

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/SickChill/SickChill/issues/5686?email_source=notifications&email_token=AAPUHZAFMP64JQLTPXPLQCDQUVYPRA5CNFSM4JNUGBY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEETC26Q#issuecomment-556150138,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAPUHZA3QMD4TGS2OSI37CLQUVYPRANCNFSM4JNUGBYQ
.

>

David Tran
QA Engineer
[email protected] :: www.truex.com

the-davidtran on 20 Nov 2019

Anyone here using dockSTARTer? wondering if we have a work around for that yet?

I think dockSTARTer will have to update their docker images, so please be patient while the updates percolate through to all these third party services.

m1dnight on 20 Nov 2019

@mmhere, develop branch working this morning, must have just been cached up somewhere. still got poster issues but that's expected.

brendanmig on 20 Nov 2019

I use sickchill image (https://hub.docker.com/r/sickchill/sickchill) and develop branch works (changed in sickchill web interface with github user and password).

stop container
delete chache.db and failed.db
start container

input user and password in config-general-Advanced Settings page
save changes
refresh page
select develop in Branch version field
checkout branch button is a must (sickchill is restarted with new branch)

kretzu77 on 20 Nov 2019

How do you pull a dev version of the docker image? tag:dev?

jaxjexjox on 20 Nov 2019

How do you pull a dev version of the docker image? tag:dev?

You have to first go to their docker repository and see if they even have a dev version and see what it is tagged as. If there is one, you would change the tag in your docker container configuration and then download the new container.

chesh1r3k on 20 Nov 2019

👍1

I have SickChill Master installed from git in a FreeNAS jail (not iocage) but I don't have an option other than master in the drop down under branch version. Any suggestions? Otherwise just wait and eventually master will be updated? Thanks.

natetheskate on 20 Nov 2019

@natetheskate You have to put in your github credentials in the appropriate fields before you can checkout other branches

samapico on 21 Nov 2019

👍2

@mmhere, develop branch working this morning, must have just been cached up somewhere.
still got poster issues but that's expected.

Huzzah! That's similar to what I observed in early debugging with _Fifth Gear_. Good to know that cloudflare (et al) can be at fault and sometimes do seem to get in the way when transferring .zip files that are automatically assembled from a database query on the other side. weird that. it's not really file transfer, it is DATABASE LOOKUP! Don't cache that sh*t!

mmhere on 21 Nov 2019

@mmhere, develop branch working this morning, must have just been cached up somewhere.
still got poster issues but that's expected.

Huzzah! That's similar to what I observed in early debugging with _Fifth Gear_. Good to know that cloudflare (et al) can be at fault and sometimes do seem to get in the way when transferring .zip files that are automatically assembled from a database query on the other side. weird that. it's not really file transfer, it is DATABASE LOOKUP! Don't cache that sh*t!

It is a bit strange. I think I should still have the zips I downloaded in browser in my temp so might check if they have different data today to what they did yesterday. Does seem bizarre that a URL that contains an API key would have any sort of caching header set on it. Though I guess if it's a relatively short age it's not too much of an issue. Wonder how that's supposed to work in the V3 api though where you have to have a JWT token instead.

brendanmig on 21 Nov 2019

@mmhere, OK, so cloud front does all sorts of weird things, including behaving differently depending on user agent it seems. First I requested mandalorian info via firefox and got these headers:

TTP/2.0 200 OK
content-type: application/zip
content-length: 3326
date: Thu, 21 Nov 2019 04:02:03 GMT
x-amzn-requestid: 1eb6bd18-cdce-40bb-b1d9-1eb93740be51
content-transfer-encoding: binary
content-disposition: attachment; filename="en.zip.zip"
content-description: File Transfer
content-rating: TV-PG
x-amz-apigw-id: DfbZWGlcPHcFd6g=
x-amzn-trace-id: Root=1-5dd60c3b-ce663e28e7c22a90446aade8;Sampled=0
via: 1.1 7a7cbcc9a496cf341e54c90ad14e02d4.cloudfront.net (CloudFront), 1.1 e3cb2b95dc77970fa884677fa82b833f.cloudfront.net (CloudFront)
x-amz-cf-pop: SYD1-C1
x-cache: Hit from cloudfront
x-amz-cf-pop: SYD1-C2
x-amz-cf-id: sacZCaLiZv_yJW6TNeNMWgZyRt4U-UmA0zIxmeKQZ_S1r_w2PNyK5Q==
X-Firefox-Spdy: h2

Then made a curl request as follows:

$ curl -D -  https://thetvdb.com/api/F9C450E78D99172E/series/361753/all/en.zip -o /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0HTTP/2 200
content-type: application/zip
content-length: 3341
date: Thu, 21 Nov 2019 04:06:09 GMT
x-amzn-requestid: cf485fd5-5631-40e7-9fea-3b8d9ea41f4e
content-transfer-encoding: binary
content-disposition: attachment; filename="en.zip.zip"
content-description: File Transfer
content-rating: TV-PG
x-amz-apigw-id: Dfb_xH20PHcFW7g=
x-amzn-trace-id: Root=1-5dd60d31-41ac1850e77dd1f6cb346d94;Sampled=0
via: 1.1 360bc380530e42ff8d4114ee99dd6212.cloudfront.net (CloudFront), 1.1 359a113ca166631b42f31a0f2e6a1aab.cloudfront.net (CloudFront)
x-amz-cf-pop: SYD4-C1
x-cache: Miss from cloudfront
x-amz-cf-pop: SYD1-C1
x-amz-cf-id: QnRRJQhhvFWk9L0WBbw6YJlzOALz8JIA4fz9YlGxkJ1KLuVeVy3cRA==

So firefox hit the cloudfront cache whereas curl missed. And curl returned a bigger file (had a single changed banner url).

Repeated downloads in firefox kept giving the same smaller file, curl kept giving the larger but started cache hitting.

At least this explains why I could have been getting different data in browser downloads to what sickchill was seeing.

brendanmig on 21 Nov 2019

@brendanmig Interesting. It doesn't make for much of a reliable RPC system. Call once, get something. Call again with different user agent (or maybe hit a different intermediate cache storage node) get a different answer. I'm all for distributed load and I/O leveling but give me a consistent response. Maybe they prefer cache busting for certain user agents or something. I.e., advertise that you are a robot (not an end user's browser) and it ...what... _tries harder_?

mmhere on 21 Nov 2019

@brendanmig It would be interesting to see the results if you had curl use Firefox's user agent etc.

LeeThompson on 21 Nov 2019

@mmhere And I'm not sure how revoking an API key is supposed to work if you're caching responses in a CDN.

brendanmig on 21 Nov 2019

@brendanmig And surely the user agent is configurable in the import requests library used in SickChill for methods like request.get(). I dunno what that defaults to.

mmhere on 21 Nov 2019

@mmhere I don't know what cloudfront is doing actually. I can get different results from the exact same request:

$ curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0" -D -  https://thetvdb.com/api/F9C450E78D99172E/series/361753/all/en.zip -o a.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0HTTP/2 200
content-type: application/zip
content-length: 3341
date: Thu, 21 Nov 2019 04:06:09 GMT
x-amzn-requestid: cf485fd5-5631-40e7-9fea-3b8d9ea41f4e
content-transfer-encoding: binary
content-disposition: attachment; filename="en.zip.zip"
content-description: File Transfer
content-rating: TV-PG
x-amz-apigw-id: Dfb_xH20PHcFW7g=
x-amzn-trace-id: Root=1-5dd60d31-41ac1850e77dd1f6cb346d94;Sampled=0
via: 1.1 360bc380530e42ff8d4114ee99dd6212.cloudfront.net (CloudFront), 1.1 098fddbcdf00e65b8479d1d17b41d28a.cloudfront.net (CloudFront)
x-amz-cf-pop: SYD4-C1
x-cache: Hit from cloudfront
x-amz-cf-pop: SYD1-C1
x-amz-cf-id: dbRhB1KPsHimErqgwFHLM2mRJJUWR_fsV-uC9cy3Q2gn6tZ20H-opg==
age: 3706

100  3341  100  3341    0     0   3341      0  0:00:01 --:--:--  0:00:01 30651

$ curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0" -D -  https://thetvdb.com/api/F9C450E78D99172E/series/361753/all/en.zip -o a.zip
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0HTTP/2 200
content-type: application/zip
content-length: 3326
date: Thu, 21 Nov 2019 05:06:06 GMT
x-amzn-requestid: 4e361147-1ece-488d-9e55-e86b67c7e1e7
content-transfer-encoding: binary
content-disposition: attachment; filename="en.zip.zip"
content-description: File Transfer
content-rating: TV-PG
x-amz-apigw-id: DfkxxF4vPHcF3bQ=
x-amzn-trace-id: Root=1-5dd61b3e-205aea86c03814a0e7a3a0c8;Sampled=0
via: 1.1 a4e03b25c402f8e111eba098232bf16f.cloudfront.net (CloudFront), 1.1 3a6d09c229b46334ae8150e9562036de.cloudfront.net (CloudFront)
x-amz-cf-pop: SIN2-C1
x-cache: Hit from cloudfront
x-amz-cf-pop: SIN2-C1
x-amz-cf-id: LIgQ-hO3ATBbsFR3JWANNVgPiU3mt1AJsQfIXpy37_TMAjrCU3Jn8g==
age: 223

100  3326  100  3326    0     0   3326      0  0:00:01 --:--:--  0:00:01  7716

brendanmig on 21 Nov 2019

Ahhh, it depends which cloud front edge it hits!

brendanmig on 21 Nov 2019

@mmhere @brendanmig

import requests

url = 'SOME URL'

headers = {
    'User-Agent': 'My User Agent 1.0',
    'From': '[email protected]'  # This is another valid field
}

response = requests.get(url, headers=headers)

I believe the default is the python versions/platform etc...User-Agent': 'python-requests/2.2.1 CPython/2.7.5 Windows/7'

clinton-hall on 21 Nov 2019

It's not user agent differences I'm encountering but routing. Depending on which way my ISP decides to send my requests changes which Cloudfront edge I'm hitting, and hence what response I get!

brendanmig on 21 Nov 2019

example user-agent on an old version of linux:
python-requests/1.1.0 CPython/2.6.6 Linux/2.6.32-431.3.1.el6.x86_64

(https://evanhahn.com/python-requests-library-useragent/)

mmhere on 21 Nov 2019

For me the current develop branch works again, even including images now.

sickchilluser on 21 Nov 2019

I have sickchill and sabnzb both in docker. I updated to develop in sickchill finaly. But I cant get my sickchill to connect to sabnzb for some reason. tried a few SABnzbd server URL inputs like localhost etc but it will not connect. The api from sabnzb that you put in sick is correct. I am losing my mind, i just want to update and add new series :(

zkye on 21 Nov 2019

For me the current develop branch works again, even including images now.

Regarding images, thetvdb has been updating their API behind the scenes(*) to deliver images. At this point some come through and some do not. The key point is that without any client API changes on SickChill's side, some images have begun coming thru. Over time and subject to caching issues I'd expect more images to be available via [a] force-updating an existing show, or [b] adding a new show.

(*) That is, with API V2 client accesses unchanged, the underlying database has begun to ship images. It still fails sometimes but it looks like thetvdb are gradually providing coverage for most if not all series. After things settle it may be necessary to poke them in thetvdb forums regarding specific series.

mmhere on 21 Nov 2019

New fun this morning, SickChill's daily update of series is failing. I opened a separate thread for this issue ( #5717 ) I don't know if this is an intended change by thetvdb or if this is just another bug.

SickChill's Log:

2019-11-21 07:42:55 INFO SHOWUPDATER :: Could not get the recently updated show data from theTVDB. Retrying later. Url was: http://thetvdb.com/api/Updates.php?type=series&time=978336000
2019-11-21 07:42:55 INFO SHOWUPDATER :: Request failed: 403 Client Error: Forbidden for url: https://thetvdb.com/api/Updates.php?type=series&time=978336000

If you manually use the URL, it is failing authentication (presumably the new API v3 JWT Token):
{"message":"Missing Authentication Token"}

LeeThompson on 21 Nov 2019

I can confirm the same on my end:

2019-11-21 01:49:26 DEBUG    SHOWUPDATER :: GET URL: https://thetvdb.com/api/Updates.php?type=series&time=978336000 [Status: 403]
2019-11-21 01:49:26 INFO     SHOWUPDATER :: Request failed: 403 Client Error: Forbidden for url: https://thetvdb.com/api/Updates.php?type=series&time=978336000
2019-11-21 01:49:26 INFO     SHOWUPDATER :: Could not get the recently updated show data from theTVDB. Retrying later. Url was: http://thetvdb.com/api/Updates.php?type=series&time=978336000

Just adding my two cents worth to prove it happens to more than one of us.

This happened under automation overnight. Note that with debug enabled there is an additional line that confirms the HTTP response code to be 403 (which maps to "Forbidden" in the HTTP spec).

mmhere on 21 Nov 2019

Earlier I said:

Regarding images, thetvdb has been updating their API behind the scenes(*) to deliver images. At this point some come through and some do not. The key point is that without any client API changes on SickChill's side, some images have begun coming thru. Over time and subject to caching issues I'd expect more images to be available via [a] force-updating an existing show, or [b] adding a new show.

(*) That is, with API V2 client accesses unchanged, the underlying database has begun to ship images. It still fails sometimes but it looks like thetvdb are gradually providing coverage for most if not all series. After things settle it may be necessary to poke them in thetvdb forums regarding specific series.

As of today the poster images that were failing to come across the V2 interface now do appear after a force-update in the GUI. I had added _The Witcher_ as a test, which found background images and horizontal banner images, but reported a WARNING and failed to display the poster.

There is still a WARNING in the log (haven't traced that) but the poster now appears.

mmhere on 22 Nov 2019

Can we add new shows yet?

jaxjexjox on 22 Nov 2019

@jaxjexjox new show adding is working for me with the develop change to lib/tvdbapi/tvdbapi.py.

I haven't tried all that many shows.

Some shows still get empty data from thetvdb, failing to add or force-update. Additionally some shows do not receive all of the images. So it's hard to declare it "done" while thetvdb continues working on their plumbing.

Some shows that may not update one day started working the next day for other users. With the data changing rapidly at thetvdb it seems that network caching between client and server can get in the way making you wait a while until all of the data settles (can take hours).

My biggest concern at this point is that the nightly auto updates don't seem to be working. See https://github.com/SickChill/SickChill/issues/5717. This implies force-updating to get new episodes populated until the API V2 data supply is fixed "on the other side". Blech,

mmhere on 22 Nov 2019

Force update for The Mandalorian wasn't getting all the artwork, I had the background image but no poster in series page or show list. Had to remove and re-add the series to get poster.

brendanmig on 22 Nov 2019

Force update is even more broken I think:
2019-11-22 08:42:50 SHOWQUEUE-FORCE-UPDATE :: Unable to contact theTVDB, aborting: "There is no item named u'en.xml' in the archive"

dirkronsmans on 22 Nov 2019

👍1

Force update is fixed by the same mod to tvdb_api.py presently on the develop branch. Specifically this error is addressed: "There is no item named u'en.xml' in the archive"

mmhere on 22 Nov 2019

So who will update the docker instance?

jaxjexjox on 22 Nov 2019

👍1

The docker instance will get updated once this patch is no longer in develop, but sent to master.

I'm not seeing an all out 'yes this works in all cases' right now, and I'm not convinced TVDB wont do some breaking stuff within a few days, so a bit more patience, or run the develop branch for a bit.

WebSpider on 22 Nov 2019

hey everyone, i keep getting this error when i try to turn to develop mode git checkout -f develop returned : error: pathspec 'develop' did not match any file(s) known to git. .
What am i doing wrong?

Same issue here. I'm quite new to this so am thinking I'm doing something wrong.

same here. Is there a solution?

nikram77 on 23 Nov 2019

Sorry to say but I put in the time and yeah I'm out. Many years behind the project and I know this is TVDB not sickchills fault, but Medusa fixed it faster and supports 3 sources.

sudo docker run -d
--name=medusa
-v /dockerconfigs/medusa:/config
-v /mnt/TVdownOnSSD:/downloads
-v /mnt/TVonARRAY/:/tv
-e PUID=0
-e PGID=0
-e TZ=Australia/Melbourne
-p 8111:8081
pymedusa/medusa

Done :/ up and running and working well after an hour or two.

jaxjexjox on 24 Nov 2019

👎3

I get the error [e094759] Unable to connect to theTVDB while creating meta files - skipping - "There is no item named u'en.xml' in the archive" and was directed here as a duplicate bug.

I can confirm that switching to the development branch resolves the issue.

dracupg on 24 Nov 2019

I can not update to the develop branch. :-( Getting the following error:

"git checkout -f develop returned : error: pathspec 'develop' did not match any file(s) known to git."

Smiggel on 24 Nov 2019

In order to successfully update to the develop branch, I first had to run
sudo git fetch --all
then
git checkout -f develop
There are still issues at TheTVdb, but at least I can add and update shows/series until the issues are resolved.

consenti on 24 Nov 2019

In order to successfully update to the develop branch, I first had to run
sudo git fetch --all
then
git checkout -f develop
There are still issues at TheTVdb, but at least I can add and update shows/series until the issues are resolved.

Ok, thanks. Will see if Incan update it on my NAS this way.

Smiggel on 24 Nov 2019

Hello
I´m new to this and learning everyday byt i cant find were and how to do branch changes in the tvdbapi.py file?
Someone that can guide a newbe?
Thanks

MrYoda77 on 24 Nov 2019

@MrYoda77 The post two above yours https://github.com/SickChill/SickChill/issues/5686#issuecomment-557911025 (thanks @consenti) shows how get branch changes. Switch from main to develop branch (this worked for me):
cd SickChill
git fetch --all
git checkout -f develop

Alternately you can edit your local file SickChill/lib/tvdb_api/tvdb_api.py. Find this line (assuming you're on main it is line# 614):

                return xmltodict.parse(myzipfile.read('%s.xml' % language), postprocessor=process)

change to (as it is Python don't change the indented leading spaces):

                return xmltodict.parse(myzipfile.read('%s.zip.xml' % language), postprocessor=process)

In particular change '%s.xml' to '%s.zip.xml'

Restart SickChill (obviously).

mmhere on 24 Nov 2019

👍4

Hmm didnt work for me... i have change the file. SickChill/lib/tvdb_api/tvdb_api.py

Error:
"There is no item named 'de.zip.xml' in the archive"

long form:
2019-11-24 23:55:22 SHOWQUEUE-ADD :: [e094759] Unable to look up the show in /volume1/share/Serien/nameofserie on theTVDB using ID 75299, not using the NFO. Delete .nfo and try adding manually again.: "There is no item named 'de.zip.xml' in the archive"

thezepter on 25 Nov 2019

👍1

Maybe that show has no Deutsche content?

mmhere on 25 Nov 2019

shure ;-)

thezepter on 25 Nov 2019

thetvdb has been working on some problems where 'en' gets sent back or even data gets dropped when other languages are requested. Very amurriken

mmhere on 25 Nov 2019

ok.. so just wait a few days and hope the problem solves itself ? I just can't add any new series to Sickchill anymore. :-( I wouldn't care about the series information here. Is it possible to add series manually to my database without thetvdb, so that sickchill searches bluntly with the name at the indexer ? (sorry for my bad english)

thezepter on 25 Nov 2019

^^ Can report that the above changes (either development branch, or by editing the .py by hand) seems to only result in changing the error from "no item named en.xml.." _to_ "no item named en.zip.xml".

daleus on 25 Nov 2019

👍1

^^ Same experience as daleus. Edited the .py and getting same message

ronrosenfeld on 25 Nov 2019

Just switch over to development branch.

Edit: take that back.. I tried it couple of days ago and it worked but not anymore.

dachoeks3 on 25 Nov 2019

TVDB has changed it again according to their forum, archive is now en.xml.zip containing files en.xml, actor.xml, and banners.xml

pjplaypus on 25 Nov 2019

Yep, confirmed what @pjplaypus is seeing. Tested with the following and am seeing en.xml [https://thetvdb.com/api/F9C450E78D99172E/series/361753/all/en.zip]

brendanmig on 25 Nov 2019

not what I see

$ wget -O en.zip 'https://thetvdb.com/api/F9C450E78D99172E/series/361753/all/en.zip'
--2019-11-24 21:13:15--  https://thetvdb.com/api/F9C450E78D99172E/series/361753/all/en.zip
Resolving thetvdb.com (thetvdb.com)... 99.86.135.160
Connecting to thetvdb.com (thetvdb.com)|99.86.135.160|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3566 (3.5K) [application/zip]
Saving to: ‘en.zip’
2019-11-24 21:13:16 (878 MB/s) - ‘en.zip’ saved [3566/3566]

$ ls -l
total 4
-rw-r--r-- 1 username 3566 Nov 24 21:13 en.zip

$ unzip en.zip
Archive:  en.zip
  inflating: en.zip.xml              
  inflating: banners.xml             
  inflating: actors.xml              

$ ls -l
total 36
-rw-r--r-- 1 username  1105 Dec 31  1979 actors.xml
-rw-r--r-- 1 username  5767 Dec 31  1979 banners.xml
-rw-r--r-- 1 username  3566 Nov 24 21:13 en.zip
-rw-r--r-- 1 username 16499 Dec 31  1979 en.zip.xml
#----------------------------------------^^^^^^^^^^

The en.zip.xml that is extracted is what the latest change on develop branch here supports. From the above URL it contains non-empty series data for The Mandalorian.

beware cloudfront cache hits though. headers for the above retrieval show this:

  Content-Type: application/zip
  Content-Length: 3566
  Connection: keep-alive
  Date: Sun, 24 Nov 2019 07:05:12 GMT
  x-amzn-RequestId: 3a6d898e-8508-4c8f-ad8b-a09f2c0eac91
  Content-Transfer-Encoding: binary
  Content-Disposition: attachment; filename="en.zip.zip"
  Content-Description: File Transfer
  Content-Rating: TV-PG
  x-amz-apigw-id: DpvCSGFPvHcFUwQ=
  X-Amzn-Trace-Id: Root=1-5dda2ba8-fc12c3f09a09cfc03638b1c0;Sampled=0
  Via: 1.1 64f86ae1c24221f3a2e4d653d6dbc416.cloudfront.net (CloudFront), 1.1 92e604c539993adca02dc86bcca48800.cloudfront.net (CloudFront)
  X-Amz-Cf-Pop: SEA19-C2
  X-Cache: Hit from cloudfront
  X-Amz-Cf-Pop: HIO51-C1
  X-Amz-Cf-Id: 3NMiB-pMKVivKHTnF2_6fBTIf_TQRKfnN-2KIXa_giyxkfylRsk4vA==
  Age: 413

mmhere on 25 Nov 2019

Seem like you're the one encountering caching is hard now mmhere :)

$ wget -O en.zip 'https://thetvdb.com/api/F9C450E78D99172E/series/361753/all/en.zip'
--2019-11-25 16:20:48--  https://thetvdb.com/api/F9C450E78D99172E/series/361753/all/en.zip
Resolving thetvdb.com (thetvdb.com)... 13.227.242.230, 2600:9000:20ec:7c00:8:ce2f:ba8a:8a61, 2600:9000:20ec:7400:8:ce2f:ba8a:8a61, ...
Connecting to thetvdb.com (thetvdb.com)|13.227.242.230|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 3054 (3.0K) [application/zip]
Saving to: 'en.zip'

100%[====================================>] 3,054       --.-K/s   in 0s

2019-11-25 16:20:48 (121 MB/s) - 'en.zip' saved [3054/3054]

$ unzip en.zip
Archive:  en.zip
  inflating: en.xml
  inflating: banners.xml
  inflating: actors.xml
$ ls -la
total 36
drwxrwxr-x  2 username username  4096 Nov 25 16:20 .
drwxr-xr-x 35 username username  4096 Nov 25 16:20 ..
-rw-rw-r--  1 username username   819 Dec 31  1979 actors.xml
-rw-rw-r--  1 username username  4071 Dec 31  1979 banners.xml
-rw-rw-r--  1 username username 14154 Dec 31  1979 en.xml
#--------------------------------------------------^^^^^^
-rw-rw-r--  1 username username  3054 Nov 25 16:20 en.zip

brendanmig on 25 Nov 2019

@brendanmig heh.

surely there must be some header one can pass in with a request to say Do Not Cache.

mmhere on 25 Nov 2019

I'll note again that if you disable use of zip (which I did a week ago) it just gives you straight up en.xml with all of the data SickChill needs. I've been running this way for a week and have not been affected by en.xml → en.zip.xml → en.xml

Looks like thetvdb has decided to go back to original output to support legacy apps (I assume) that are hard to change.

mmhere on 25 Nov 2019

Which is what should have happened in the first place! Why the hell did anything in the legacy API change at all!

brendanmig on 25 Nov 2019

cuz they can't test in all scenarios for all clients, and someone on their end injected a bug where it provided 'en.zip.xml' then didn't find it. that's always been an incorrect filename anyway, because after unpacking the delivered zip archive the 'en.zip.xml' is just plain uncompressed xml. that naming is a bug.

I'm making assumptions here.

mmhere on 25 Nov 2019

That's what I mean, the name changing to en.zip.xml to start with could have been easily caught with basic regression testing. Which given they were releasing a new API version you would have thought that validation that the old APIs hadn't been broken would have been part of the procedure.

brendanmig on 25 Nov 2019

You'd have thought. They clearly were steaming ahead with and caring most about V3

mmhere on 25 Nov 2019

But despite my complaining, this should mean that master works again now right?

brendanmig on 25 Nov 2019

I guess (master working) (subject to cache). Hopefully they'll fix the Updates.php one soon so nightly updates will magically work again.

mmhere on 25 Nov 2019

They put out a call earlier this year (maybe even last year) to test V3 before they even got to the Throwing Of The Big Switch. I'm guessing many clients just didn't bother until things started to break.

mmhere on 25 Nov 2019

Yeah, well, you don't typically go and update to use a new API when things aren't broken in the old one, and if there's no new features you want in the new.

brendanmig on 25 Nov 2019

Is there UI to disable the zip download or is this an ini file thing?

brendanmig on 25 Nov 2019

disabling zip is neither .ini nor UI sadly. the easiest way is in the config.py file. let's see...

mmhere on 25 Nov 2019

sickbeard/indexers/indexer_config.py hand edit 'useZip': True, to False.

Far above I proposed a way to get this into config.ini, although it was slightly inelegant.

mmhere on 25 Nov 2019

👍1

Yeah, that did the trick. Thanks!

brendanmig on 25 Nov 2019

https://github.com/SickChill/SickChill/issues/5686#issuecomment-554844139 is my config.ini proposal. I'm not happy with it because it puts a setting at the global level in config.ini (and I never nor did anyone else follow through on implementing it). It'd be better to have a [THETVDB] section with settings specific to that service so that they are separated and grouped away from others. That is more involved.

It would let you say

thetvdb_use_zip = 0

but this (more involved) would be better

[THETVDB]
use_zip = 0

mmhere on 25 Nov 2019

What a clusterf**k.

This is _exactly_ how you do not roll out an update/new API version.

LeeThompson on 25 Nov 2019

👍2

Ah, yeah, I see what you mean, not nice :)

brendanmig on 25 Nov 2019

Heh. I've gotten in better in Python in the last ten days.

mmhere on 25 Nov 2019

😄1 👍1

And we can soon close this issue once the original commits are reverted and attempt to purge all memory of this horrendous experience!

brendanmig on 25 Nov 2019

The last thing I know about that needs addressed (and thetvdb has committed to putting it back) is the stuff over in #5717 to get nightly updates working properly again(*). If thetvdb follows through on this and does it correctly, the current main should be working again as is. Like, what, one month later eventually?!? :wink:

(*) I got a V3 version of that stuff working in Python as a safety valve should we need to use it. It also functions as a model for how we might eventually move to V3 entirely (but several places need changed).

mmhere on 25 Nov 2019

Or just do what @jaxjexjox did and decamp to Medusa. Can I say that here? I don't want to decamp.

mmhere on 25 Nov 2019

I've already decamped at least twice since the old sickbeard days, would prefer not to do so again :)

brendanmig on 25 Nov 2019

I really didn't wanna leave to be fair, sickchill has been good to me, but 3 sources for medusa and more frequent commits.

The blame lays squarely on tvdb - not sickchill, none the less medusa has improved a lot since my last check a year ago.
Sorry all :(

jaxjexjox on 25 Nov 2019

@jaxjexjox Did it import your sickbeard.db acceptably?

mmhere on 25 Nov 2019

@mmhere it imports without problems but you might as well re-add most shows to switch indexer and they also have a few more/different settings.

Hate to say it but I updated my medusa docker as well and running that now.
Wasn't a fan in the past but the codebase seems to be fixed faster and also they tend to add more features. (not everything is smiles and rainbows but at least my shows are being downloaded now :))

This is not me telling people to move but I just don't have the time to fix this one and I don't see it changing/moving forward fast any time soon.

dirkronsmans on 25 Nov 2019

I think if thetvdb finishes the work on Updates.php (for nightly updates) they committed to fix then it will be all done. Now that they made this most recent change back to en.xml our main branch is OK again.

So I'll beg to politely disagree with your "any time soon" assessment.

mmhere on 25 Nov 2019

Didn't mean to make it sound like that.
My "any time soon" was more pointed towards further development such as V3 and other providers :)

dirkronsmans on 25 Nov 2019

Ah, on that we can agree. (V3 and providers beyond thetvdb)

mmhere on 25 Nov 2019

TVDB has changed it again according to their forum, archive is now en.xml.zip containing files en.xml, actor.xml, and banners.xml

If i hit https://thetvdb.com/api/F9C450E78D99172E/series/361753/all/en.zip in the browser now, it gives me a en.xml.zip with the en.xml, actor.xml, and banners.xml.
The link https://thetvdb.com/api/F9C450E78D99172E/series/361753/all/en.xml.zip works aswell.

After reverting the changes in tvdb_api.py (%s.zip.xml back to %s.xml) and changing %s.zip to %s.xml.zip in the tvdb_api.py at line 533, I am now able to add shows again. Useing main branch.

reggie88 on 25 Nov 2019

Regarding decamping and other sick* forks...

There are several alternatives. I have usually tried to keep out of pushing favourites, and try to support all current branches.

The reality is I am supporting some NAS builds for Medusa, SickChill, SickGear etc, as well as supporting Sick* forms and Sonarr in nzbToMedia.

Any project is going to be hurt by an external change like what TVDB did, and there was significant disruption to many (most) projects
While some other projects may have had more official commits, I saw this community come up with several solutions to keep things moving.

These are open-source community projects, so feel free to discuss, send pull requests etc. Check out various projects and use what you like. But don't feel the need to abandon something that has worked for a long time, simply because a 3rd party caused major disruption and the "official" team didn't provide rapid fixes to something that was changing without clear guidance from the 3rd party.

clinton-hall on 25 Nov 2019

👍6

@reggie88 Regarding https://github.com/SickChill/SickChill/issues/5686#issuecomment-558030689 -- yup, thetvdb seems to have changed it back to en.xml. Note that subject to cloudfront caching (proven above) some people are still receiving the previous form (circa yesterday or the day before). I'd expect that by tomorrow morning it should always give back consistent results and main will work as before all of this started. You can dump headers to see whether you got a cache hit and thus old data or not. wget -S or curl -D will show headers.

See #5717, however, as the Updates.php link is still not working due to thetvdb's issues. They've committed to restoring this.

mmhere on 25 Nov 2019

sickbeard/indexers/indexer_config.py hand edit 'useZip': True, to False.

Far above I proposed a way to get this into config.ini, although it was slightly inelegant.

solved my Problems ! Thanks

thezepter on 25 Nov 2019

@mmhere Just looks like they also changed the name of the zip to en.xml.zip. Guess it's time for me to change useZip to false till they decide what to name it.

https://thetvdb.com/api/F9C450E78D99172E/series/361753/all/en.zip
```
Content-Type: application/zip
Content-Length: 3054
Connection: keep-alive
Date: Mon, 25 Nov 2019 08:38:14 GMT
x-amzn-RequestId: 0619d785-d6f5-4a36-9a1a-7befb671256e
Content-Transfer-Encoding: binary
Content-Disposition: attachment; filename="en.xml.zip"
Content-Description: File Transfer
Content-Rating: TV-PG
x-amz-apigw-id: DtPmeF0nvHcF4Ow=
X-Amzn-Trace-Id: Root=1-5ddb92f6-e198c7d0f8aae6dc54b0b6e0;Sampled=0
Via: 1.1 9c7c26f5beeb09381cea450ea3581b37.cloudfront.net (CloudFront), 1.1 23776effa8a63b2e2dccd702e73b0c86.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: FRA50-C1
X-Cache: Miss from cloudfront
X-Amz-Cf-Pop: AMS54-C1
X-Amz-Cf-Id: hyDe61TyBu-et119jo6qUePFHGlFjeKD2ZoPTQ1TzWkFQEo0A9RM1g==

reggie88 on 25 Nov 2019

Yah, the zip stuff is screwed up on their end (has been since the beginning and seems still in flux). Requesting non-zip has been working this whole time (knocks wood).

That said, the SickChill Python code doesn't care what the headers say about the filename (en.xml.zip) above, because the code simply gets the zip data and unpacks it to reveal the files inside. It never really mentions the name of the zip file itself, it just knows it has obtained zip data.

I.e., in pseudo-code terms it does: zipped_thing=get_zip(); unpack zipped_thing; look in unpacked items for en.xml.

What matters is the contents of [files inside] delivered zip, which thetvdb seems now to have reverted to en.xml like it was before they started doing all this... um... stuff (refains from swearing)

mmhere on 25 Nov 2019

disabling zip is neither .ini nor UI sadly. the easiest way is in the config.py file. let's see...

I'll go ahead and make this an INI thing. Not going to expose this in UI for the moment tho. This should fix current master, and make it a bit more flexible for future indexer breakages.

WebSpider on 25 Nov 2019

👍2

disabling zip is neither .ini nor UI sadly. the easiest way is in the config.py file. let's see...

I'll go ahead and make this an INI thing. Not going to expose this in UI for the moment tho. This should fix current master, and make it a bit more flexible for future indexer breakages.

Super cool. :heart:

You saw my suggested hack well above modeled on config.ini's debug=1 setting right? You can probly do it better than what I suggested

mmhere on 25 Nov 2019

Clinton I do not begruge sickchill in any capacity, this problem is clearly due to TVDB, 100% I didn't want to go, but TVDB took so astoundingly long to fix this. I tried waiting, but at this point it's fully ridiculous.

As to the person asking, I didn't import my database as much as I just pointed to the files on the TV drive and imported the shows- couple of hours work at most.

jaxjexjox on 25 Nov 2019

I get this is not a sickchills problem but this is almost 11 days with out be able to add new show's. There is no reason sickchill can not fix this by jumping ship TVDB? Why haven't we switch or added a different database ? Like https://simkl.com/ medusa is using Imbd not sure how.

Snowaks on 25 Nov 2019

just added a show and for me the master commit e094759 works

lofteddwarf on 25 Nov 2019

New show are fine but forced updated on existing ones still : SHOWQUEUE-FORCE-UPDATE :: Unable to contact theTVDB, aborting: "There is no item named u'en.zip.xml' in the archive"

olafkfreund on 25 Nov 2019

Branch: develop
Commit: e9a82d932c30f9614bc4658fba03a745afc705ef
Database Version: 44.0

Snowaks on 25 Nov 2019

There is no reason sickchill can not fix this by jumping ship TVDB? Why haven't we switch or added a different database ? Like https://simkl.com/ medusa is using Imbd not sure how.

There is non-trivial code to be written to adapt to any other database format and the API standing in front of it. Surely others such as simkl.com have a different API and different database format(s) underneath. I mean in the end the data describing TV series and their episodes is all pretty understandable but there are choices designers make differently when encoding it and creating a database.

Given the open source nature of other sick* forks, however, we would be free to borrow existing implementation ideas from other projects that have already done something similar. That's also a non-trivial task.

So there's an enhancement opportunity here for a motivated developer.

mmhere on 25 Nov 2019

👍2

New show are fine but forced updated on existing ones still : SHOWQUEUE-FORCE-UPDATE :: Unable to contact theTVDB, aborting: "There is no item named u'en.zip.xml' in the archive"

thetvdb switched back to the original way from before TheDebacle™. So main is good as is for adding a new show and Force-Update on an existing show. Nightly automatic updates still await thetvdb fixes; for that see link_at_thetvdb and #5717.

mmhere on 25 Nov 2019

Now its working again with master branch,
Thanks for help ppl!

MrYoda77 on 25 Nov 2019

This worked for me:
edit /lib/tvdb_api/tvdb_api.py
find all %s.xml
replace all with %s.zip.xml
restart sickchill
now able to add shows.

This worked for me as well (show updating works), of course it's in a docker container so whenever that updates it will revert.

For those of us that changed this ^^ in the master, do we revert it back?

zamz84 on 26 Nov 2019

Disabling zip can fail in corner cases!

More thetvdb shenanigans. This time it means setting useZip=False can fail and only for some series. Went back to main now that thetvdb reverted to shipping proper data like they did two weeks ago.

Just a fair warning to others that disabling zip may bite you but only on some series. I've traced this to special character encoding.

(details)

I had been running with zip turned off which causes SickChill to request the 2nd URL shown below. Up thru Saturday night this was all fine.

Now I discovered that at least one show (Madam Secretary) started failing to update and could not be Force-Updated. Switched zip back on and continued to let SickChill request en.zip which once again has inside of it the end-desired file en.xml. As others have done, this is effectively a switch back to our current main, which is functional.

These two return slightly different contents in the two en.xml files that are eventually produced.

wget -S -O en.zip 'http://thetvdb.com/api/F9C450E78D99172E/series/281623/all/en.zip'
wget -S -O en.zip 'http://thetvdb.com/api/F9C450E78D99172E/series/281623/all/en.xml'

The first URL returns en.zip which when unpacked contains the en.xml that we've been using for years. This works fine and reflects what our main branch is doing.

The second URL does directly return an en.xml file, which is uncompressed XML, but special characters are encoded differently in it! This causes various parsing to fail. The show never updates.

So fair warning that in a small number of cases (and subject to the ever in-flux river of unannounced changes that is thetvdb) switching off zip may cause problems. (That said, it still makes sense to have this switchability as a config.ini setting for future recovery from breakage -- it did work fine for me for two weeks.)

mmhere on 26 Nov 2019

@zamz84 yup, if you locally modified main, change it back to %s.xml.

mmhere on 26 Nov 2019

Regarding https://github.com/SickChill/SickChill/issues/5686#issuecomment-558462104 different encodings and failure to parse meaning series don't update or add.

The one that parses OK (requesting en.zip and unpacking it then running it thru xmllint --format) produces output like this:

<?xml version="1.0"?>
<Data>
  <Series>
    <id>281623</id>
    <Actors>|Wallis Currie-Wood|T&#xE9;a Leoni|Zeljko Ivanek|Keith Carradine|Bebe Neuwirth|Geoffrey Arend|Sebastian Arcelus|Erich Bergen|Kathrine Herzer|Patina Miller|Evan Roe|Tim Daly|</Actors>
...continues...

The one that fails to parse (requesting en.xml, which is delivered directly), then running it thru xmllint --format produces output like this:

<?xml version="1.0" encoding="UTF-8"?>
<Data>
  <Series>
    <id>281623</id>
    <Actors>|Wallis Currie-Wood|Téa Leoni|Zeljko Ivanek|Keith Carradine|Bebe Neuwirth|Geoffrey Arend|Sebastian Arcelus|Erich Bergen|Kathrine Herzer|Patina Miller|Evan Roe|Tim Daly|</Actors>
...continues...

The code in tvdb_api.py does seem to be prepared to parse UTF-8 (it makes calls to this effect) but it apparently does not handle it (the 2nd one). The first one where special characters are instead encoded as &#x__; and &#x____; (both one byte and multibyte versions) does parse OK

mmhere on 26 Nov 2019

Now I discovered that at least one show (Madam Secretary) started failing to update and could not be Force-Updated. Switched zip back on and continued to let SickChill request en.zip which once again has inside of it the end-desired file en.xml. As others have done, this is effectively a switch back to our current main, which is functional.

Using the developer branch should fix this issue (i had the same with the same show last week). However, there are other shows now that fail with the "Unable to connect to theTVDB while creating meta files - skipping - error "There is no item named u'en.zip.xml' in the archive"" error. (in this case an anime show called "Black Clover".

dporem on 26 Nov 2019

Any ETA on an update to master? I'm not sure how to switch to the development branch.

Tonkapug on 28 Nov 2019

Now I discovered that at least one show (Madam Secretary) started failing to update and could not be Force-Updated. Switched zip back on and continued to let SickChill request en.zip which once again has inside of it the end-desired file en.xml. As others have done, this is effectively a switch back to our current main, which is functional.

Using the developer branch should fix this issue (i had the same with the same show last week). However, there are other shows now that fail with the "Unable to connect to theTVDB while creating meta files - skipping - error "There is no item named u'en.zip.xml' in the archive"" error. (in this case an anime show called "Black Clover".

For one of the shows I am finding that there are additional episodes listed on the TVDB but not showing in SickChill - (SickChill shows up to ep 7 but the TVDB lists up to ep 9)

This is after a force full update on both Master & Dev -

Other shows are working fine.

zamz84 on 28 Nov 2019

I should have some time to spend on this coming Friday

WebSpider on 28 Nov 2019

Hiya. I'm still experiencing this same issue. The Forced Update solution worked for a number of shows, but I've still got at least 4 shows that are still missing episodes which have been aired.

I also tried switching to the dev branch. That did not help either.

Any ideas please?

benjaminu on 30 Nov 2019

Weird stuff happening,
It just renamed one of my son series from "Teen Titans go" to "The Dawn is Your Enemy: Teen Titans Go! is Dead!!!!!!!!"
I removed the show and it added it again, same weird name, not updating episodes from thetvdb and shows 63 episodes for season 5 and no episode for season 6 and also a season 666.
It has the correct tvdb id and link though http://thetvdb.com/?tab=series&id=266775

It seems its TVDB API cache problem not sickchill

papampi on 30 Nov 2019

Hiya. I'm still experiencing this same issue. The Forced Update
solution worked for a number of shows, but I've still got at least 4
shows that are still missing episodes which have been aired.

I also tried switching to the dev branch. That did not help either.

I have seven shows that fail to update when the nightly update runs and updates ALL shows(*). Pretty sure there is some data still bad at thetvdb, or at least coming thru the API as empty/bad.

You should probly mention the specific shows in thetvdb forums. They have previously fixed individual database records (and delivery thru the API) behind the scenes. They've done this for users' "favorites"; maybe they can do this per show as well. Their database seems to have had a small number of records compromised during their transition. Also:

I've also observed situations where data displayed in their website GUI looks fine but comes through the API empty. This is the case with some of my lucky seven.

The main branch is otherwise working OK over here given that the large majority of shows update acceptably. (I've just been waiting for my seven shows to be fixed by Magix™. Perhaps this is not the best approach. ;-) )

[(*)My nightly update code has a hack that forces all shows updated til selective updating of changed shows using thetvdb's Updates.php URL is fixed.]

mmhere on 30 Nov 2019

I don't know anything about coding. I have three shows that will not show posters but the posters are there under SickChillDatacacheimages and they show if you click on the poster under the show in SickChill. But I still get the 2019-12-15 20:57:02 SHOWQUEUE-REFRESH :: There was an error trying to retrieve the image, aborting error. The images are being retrieved.

krash1220 on 16 Dec 2019

I don't know anything about coding. I have three shows that will not show posters but the posters are there under SickChillDatacacheimages and they show if you click on the poster under the show in SickChill. But I still get the 2019-12-15 20:57:02 SHOWQUEUE-REFRESH :: There was an error trying to retrieve the image, aborting error. The images are being retrieved.

I am seeing a similar issue with new shows that have been added since the new API went into effect. Most recently, The Witcher. Episodes and all seem to work ok, but downloading SOME art does not work. Here's the excerpt from the log. Looks to me like it is using _cache in the path name for some reason...not sure if that is normal or what, but wanted to mention it here. I posted on the TheTVDB Forum about it as well in regard to another show.

Here's the log:

2019-12-25 21:53:27 INFO SHOWQUEUE-REFRESH :: Done cache check
2019-12-25 21:53:27 WARNING SHOWQUEUE-REFRESH :: There was an error trying to retrieve the image, aborting
2019-12-25 21:53:27 INFO SHOWQUEUE-REFRESH :: Request failed: 403 Client Error: Forbidden for url: https://artworks.thetvdb.com/banners/_cache/graphical/5d1a2cb6138b3.jpg ()
2019-12-25 21:53:27 WARNING SHOWQUEUE-REFRESH :: There was an error trying to retrieve the image, aborting
2019-12-25 21:53:27 INFO SHOWQUEUE-REFRESH :: Request failed: 403 Client Error: Forbidden for url: https://thetvdb.com/banners/series/362696/_cache/posters/62003331.jpg ()
2019-12-25 21:53:26 INFO SHOWQUEUE-REFRESH :: 362696: Updating NFOs for show with new indexer info
2019-12-25 21:53:26 INFO SHOWQUEUE-REFRESH :: Performing refresh on The Witcher
2019-12-25 21:52:48 INFO SHOWQUEUE-REFRESH :: Done cache check
2019-12-25 21:52:48 WARNING SHOWQUEUE-REFRESH :: There was an error trying to retrieve the image, aborting
2019-12-25 21:52:48 INFO SHOWQUEUE-REFRESH :: Request failed: 403 Client Error: Forbidden for url: https://artworks.thetvdb.com/banners/_cache/graphical/5d1a2cb6138b3.jpg ()
2019-12-25 21:52:48 WARNING SHOWQUEUE-REFRESH :: There was an error trying to retrieve the image, aborting
2019-12-25 21:52:48 INFO SHOWQUEUE-REFRESH :: Request failed: 403 Client Error: Forbidden for url: https://thetvdb.com/banners/series/362696/_cache/posters/62003331.jpg ()
2019-12-25 21:52:46 INFO SHOWQUEUE-REFRESH :: Performing refresh on The Witcher

Here's a link to my post on thetvdb forum:

https://forums.thetvdb.com/viewtopic.php?p=166281#p166281

Thanks.

ulrick65 on 26 Dec 2019

Completely fixed in develop branch. ;)

miigotu on 8 Jan 2020

Awesome! I don't use develop...so I will wait for it to hit the big time in master!

Thanks...great work you do here...I really appreciate it.

ulrick65 on 8 Jan 2020

This is an example of the message i get while doing a force update. I get this with every show in my sickrage while doing a mass update. This happens in develop and in master since this morning.
Branch: master
Commit: 17e379df7dadc7711db1c76695411267f245a961
Version: v2020.01.03-1
Database Version: 44.1

2020-01-10 08:51:26 SHOWQUEUE-FORCE-UPDATE :: [17e379d] Data retrieved from theTVDB was incomplete, aborting: Found 332331, but attribute 'seriesname' was empty.
2020-01-10 08:51:15 SHOWQUEUE-FORCE-UPDATE :: [17e379d] Data retrieved from theTVDB was incomplete, aborting: Found 277928, but attribute 'seriesname' was empty.
2020-01-10 08:51:10 SHOWQUEUE-FORCE-UPDATE :: [17e379d] Data retrieved from theTVDB was incomplete, aborting: Found 354265, but attribute 'seriesname' was empty.
2020-01-10 08:45:30 SHOWQUEUE-FORCE-UPDATE :: [17e379d] Data retrieved from theTVDB was incomplete, aborting: Found 361315, but attribute 'seriesname' was empty.

McEnnes on 10 Jan 2020

The (thetvdb) are at it again. server side back end stuff is currently borked. Affects not only SickChill but other consumers of their data, such as various media centers. Additional symptoms in SickChill include inability to add new shows right now.

https://forums.thetvdb.com/viewtopic.php?f=17&t=61698&p=168243

mmhere on 10 Jan 2020

Yeah I was just trying to do some testing and couldnt smh. It effects APIv3 as well

miigotu on 10 Jan 2020

Also, their regularly updated status page (tracking the debacle that is V3 "upgrade") mentions these:

Here are the current known API-related issues with fixes in progress:
• Users Reporting Shows Removed on API — _Researching_
...
Here are the current known site-related issues with fixes in progress:
• Ongoing 504 errors causing site delays and "Whoops" error messages — _Researching_

That status page is here, updated daily (and twice today so far)
https://forums.thetvdb.com/viewtopic.php?f=122&t=60239

mmhere on 10 Jan 2020

https://twitter.com/thetvdb/status/1196636721581969408
Develop is completely broken right now.

API authentication w/o username & key — Released 11/15: You'll need to authenticate with your API key, not your username.
And:
API key page unaccessible (caching issue) — Released 11/27
Are broken again

miigotu on 10 Jan 2020

From the top of their twitter feed https://twitter.com/thetvdb they claim this:

TheTVDB.com @thetvdb -- 4 hours ago

Anyone experiencing outages on the site or API last night –
we've located the root & have made some adjustments to alleviate it.
Should also take care of any 502 and 504 caching errors.

We're also looking into some issues in the v1 API now where it's returning empty records.

It's still completely broken though. API doesn't respond or responds with empty data.

mmhere on 10 Jan 2020

Same here, can't add any show (on master), I am assuming it is because of what @miigotu mentioned.

2020-01-10 19:01:47 DEBUG ThreadPoolExecutor-0_1 :: Searching for Show with searchterm(s): ['game of thrones'] on Indexer: theTVDB
2020-01-10 19:01:31 DEBUG ThreadPoolExecutor-0_0 :: Searching for Show with searchterm(s): ['for all mankind'] on Indexer: theTVDB
2020-01-10 19:01:20 DEBUG ThreadPoolExecutor-0_4 :: Searching for Show with searchterm(s): ['dragon ball'] on Indexer: theTVDB
2020-01-10 19:01:07 DEBUG ThreadPoolExecutor-0_6 :: Searching for Show with searchterm(s): ['the rookie', 'rookie'] on Indexer: theTVDB

gomera on 10 Jan 2020

AuthenticationError: Unknown error while authenticating. Check your api key or your user/userkey

miigotu on 10 Jan 2020

Similar things are happening on media centers so this is not limited to SC use of the API.

Something is broken deep in their DB system because the website presentation layer also suffers various issues ("Whoops - something went wrong") when trying to modify shows and so on.

They at least appear to know about it (or did and think they finished) because their tweet claims they changed something. Best to keep the noise level up in their forums so they know they still have work to do.

I'm thinkin, good luck with commercializing this platform, something that is one of their eventual goals...

mmhere on 10 Jan 2020

I tried authenticating right on their api site https://api.thetvdb.com/swagger#!/Authentication/post_login and it does exactly what SC does

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>504 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
CloudFront attempted to establish a connection with the origin, but either the attempt failed or the origin closed the connection.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: rPX0vbmZrVyVviOfFnyjFRnRN-VpVd-60SgOT_lKkqG8VJoh7EJbPA==
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>

miigotu on 10 Jan 2020

Heh, you beat me to it. I suspected their own swagger# page would fail. It does!

That's the most direct feedback to send to them.

mmhere on 10 Jan 2020

@mmhere feel free to post that to them, I never get a response lol

miigotu on 11 Jan 2020

I'm seeing success again with SC (main branch); I did a force-update from the GUI on one show. Others are reporting other API usage working again. Bet it doesn't last.

https://forums.thetvdb.com/viewtopic.php?f=17&t=61720

mmhere on 11 Jan 2020

thetvdb admins claim this:

Hi there, we have recently pushed a fix and we're seeing the API back to normal now.
Please continue to report if you see any other errors. Thanks so much.

https://forums.thetvdb.com/viewtopic.php?f=17&t=61720&p=168336#p168336

mmhere on 11 Jan 2020

Further, the overall summary page that collects the full extent of their issues (https://forums.thetvdb.com/viewtopic.php?f=122&t=60239) claims via a bold blue note at the top:

* The API Issues from 1/10 should be fully resolved as of this most recent update. *

mmhere on 11 Jan 2020

Api login is working now

miigotu on 11 Jan 2020

Api login is working now

Hi @miigotu and @mmhere I'm not to familiar with python and I'm using sickbeard with python in Windows. I'm having issues editing the tvdb_api.py file. Can either of you assist me?

usenet135 on 11 Jan 2020

@usenet135 there is no tvdb_api in develop anymore, Ive rewritten it all to use the new api

miigotu on 11 Jan 2020

@usenet135 what is it that you are trying to accomplish?

miigotu on 11 Jan 2020

Well this stinks. API at thetvdb is better but still not healthy.

The overnight automatic show update gets "seriesname empty" errors for many shows. This only occurs for some shows (about 10% of mine). Those shows fall out of the local database, losing all airdates and therefore are lost from the schedule page. UPCOMING SHOWS WILL NOT GET DOWNLOADED.

What's detailed below was an update situation, but the "seriesname empty" error also causes adding of new shows to fail for certain shows. I tried "Deadwater Fell" (failed) while "Wisting" worked.

The symptom in SC is evidence of the API finding a show but returning at least partially empty data. In particular the seriesname field is empty. Note that this is similar to problems during the early phases of this V3 API _adventure_.

There is a post at thetvdb https://forums.thetvdb.com/viewtopic.php?f=17&t=61736 that talks about similar problems. So theoretically thetvdb has been informed, assuming they read that thread (and other similar ones).

The real problem then becomes that all of these shows get punted out of the local SC database. In particular their scheduled next air times are lost. The shows that still exist on disk locally are reloaded so they appear in the GUI. The schedule, however, and THEREFORE ALL FUTURE DOWNLOAD ATTEMPTS are lost until such time that the show can be restored with valid airdates (to the local db).

"Doctor Who (2005)" for example, has its upcoming episodes completely lost from the schedule, ones on disk locally are still retained, but all airdate info is lost.

I haven't dug any deeper into how much of this could be down to the crappy cloudfront caching behavior, but this looks pretty bad.

Details below:

(It only does FORCE as below for shows that are indicated updated at thetvdb according to a preceding query against the "updated shows since datestamp" API method)

2020-01-11 00:26:32 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 362696, but attribute 'seriesname' was empty. 2020-01-11 00:26:31 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 363955, but attribute 'seriesname' was empty. 2020-01-11 00:26:07 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 360295, but attribute 'seriesname' was empty. 2020-01-11 00:25:47 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 370139, but attribute 'seriesname' was empty. 2020-01-11 00:25:42 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 364093, but attribute 'seriesname' was empty. 2020-01-11 00:24:36 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 354963, but attribute 'seriesname' was empty. 2020-01-11 00:23:15 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 360261, but attribute 'seriesname' was empty. 2020-01-11 00:21:59 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 73730, but attribute 'seriesname' was empty. 2020-01-11 00:20:03 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 270633, but attribute 'seriesname' was empty. 2020-01-11 00:19:53 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 356317, but attribute 'seriesname' was empty. 2020-01-11 00:18:17 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 347115, but attribute 'seriesname' was empty. 2020-01-11 00:18:06 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 338947, but attribute 'seriesname' was empty. 2020-01-11 00:17:16 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 350068, but attribute 'seriesname' was empty. 2020-01-11 00:16:00 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 326866, but attribute 'seriesname' was empty. 2020-01-11 00:11:28 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 328569, but attribute 'seriesname' was empty. 2020-01-11 00:10:43 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 332027, but attribute 'seriesname' was empty. 2020-01-11 00:10:38 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 326791, but attribute 'seriesname' was empty. 2020-01-11 00:10:33 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 73388, but attribute 'seriesname' was empty. 2020-01-11 00:10:28 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 73762, but attribute 'seriesname' was empty. 2020-01-11 00:09:46 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 71814, but attribute 'seriesname' was empty. 2020-01-11 00:05:13 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 295759, but attribute 'seriesname' was empty. 2020-01-11 00:04:49 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 80660, but attribute 'seriesname' was empty. 2020-01-11 00:03:48 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 313270, but attribute 'seriesname' was empty. 2020-01-11 00:02:21 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 311711, but attribute 'seriesname' was empty. 2020-01-11 00:02:16 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 311714, but attribute 'seriesname' was empty. 2020-01-11 00:01:06 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 306719, but attribute 'seriesname' was empty. 2020-01-10 23:58:19 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 305220, but attribute 'seriesname' was empty. 2020-01-10 23:57:39 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 262980, but attribute 'seriesname' was empty. 2020-01-10 23:53:57 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 273181, but attribute 'seriesname' was empty. 2020-01-10 23:53:47 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 299139, but attribute 'seriesname' was empty. 2020-01-10 23:53:37 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 295760, but attribute 'seriesname' was empty. 2020-01-10 23:51:56 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 255192, but attribute 'seriesname' was empty. 2020-01-10 23:50:56 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 295640, but attribute 'seriesname' was empty. 2020-01-10 23:46:32 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 257655, but attribute 'seriesname' was empty. 2020-01-10 23:44:26 SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 269613, but attribute 'seriesname' was empty.

Sigh.

mmhere on 11 Jan 2020

@mmhere errors on master are expected. I rewrote everything in SC to use api v2/3 in develop and cannot release it until it is more tested and they stop fucking with it on their end. Try develop, its working good.

Master is using api v1 with only the show updater checking which shows to update by asking api v2 (sucky)
Develop is entirely api v2/3

miigotu on 11 Jan 2020

@miigotu cool. maybe I will switch to develop then. You are right that I'm using master.

They (you know who) claim they're supporting V1 but we all know how well that support is actually working.

mmhere on 11 Jan 2020

@miigotu Does develop bump the DB version (the sqlite stuff in sickbeard.db)? I'm wondering if it is safe to switch back and forth.

mmhere on 11 Jan 2020

Nope, its safe to switch. Worst that could happen is buggered nfo or xml, but those have been tested quite a bit and have been fine for me so far

miigotu on 11 Jan 2020

cool. I don't use local nfo/xml so that's not a concern.

mmhere on 11 Jan 2020

I'm pretty sure I got all the issues with nfo/xml/images fixed, I've been testing and working on it for weeks and have had a few other people riding develop also.

miigotu on 11 Jan 2020

P.S.: I really do appreciate all of your hard work on this.

mmhere on 11 Jan 2020

One thing that's really nasty is if the API fails to return seriesname info, it summarily deletes all data from the local DB (sqlite stuff in sickbeard.db) then repopulates from files only. All the series meta data thus falls away. The future scheduled episodes fall away too.

I ended up going back to last night's backup of sickbeard.db after tonight's autoupdate failed 36 separate shows.

As a separate task it might be nice to protect against this so that what's already in the local DB is kept instead of being dropped.

As it stands, errors at thetvdb (and I know this is V1 related so should get better when develop comes online for everyone) cause the local instance to just fall over. When future V3 failures crop up it'd be nice not to have lots of shows' info just go poof.

mmhere on 11 Jan 2020

@usenet135 what is it that you are trying to accomplish?

@miigotu I'm just trying to add shows to sickbeard but get the following errors:

"Show in WINDOWS LOCATION has no name on TVDB, probably the wrong language used to search with."

It's able to check for episodes for existing... but not able to add new shows. This just began recently as I was able to add a show about a week ago

usenet135 on 11 Jan 2020

@usenet135 because tvdb site itself broke it, not SickChill. Just update and try again, I rewrote the entire indexer code to use their new apu (which is still unstable even)

miigotu on 11 Jan 2020

"Show in WINDOWS LOCATION has no name on TVDB, probably the wrong language used to search with."

That's the same symptom of the V1 API being currently broken by thetvdb. This happens for some shows but not all.

When V1 API returns empty data like this, it also causes Force-Update to fail, whether invoked manually from the GUI for a single show, or when nightly automatic updates run for many shows.

A bad side effect of the latter is that SC deletes all meta data for such an updated show from the local sqlite sickbeard.db. Following this, future episodes of shows are lost so no updates occur when the time comes for the show to be aired.

mmhere on 11 Jan 2020

Here is an example of the nightly update failing a well known show followed by deleting its local data in sickbeard.db:

2020-01-11 00:09:46 DEBUG SHOWQUEUE-FORCE-UPDATE :: Beginning update of Will & Grace 2020-01-11 00:09:46 DEBUG SHOWQUEUE-FORCE-UPDATE :: Retrieving show info from theTVDB 2020-01-11 00:09:46 DEBUG SHOWQUEUE-FORCE-UPDATE :: 71814: Loading show info from theTVDB{'useZip': True, 'cache': u'/home/bgeske/src/SickRage/cache/indexers/theTVDB', 'apikey': 'F9C450E78D99172E', 'language': 'en'} 2020-01-11 00:09:46 ERROR SHOWQUEUE-FORCE-UPDATE :: [a71302b] Data retrieved from theTVDB was incomplete, aborting: Found 71814, but attribute 'seriesname' was empty. Traceback (most recent call last): File "/home/bgeske/src/SickRage/sickbeard/show_queue.py", line 648, in run self.show.loadFromIndexer(cache=not self.force) File "/home/bgeske/src/SickRage/sickbeard/tv.py", line 873, in loadFromIndexer "Found {0}, but attribute 'seriesname' was empty.".format(self.indexerid)) tvdb_attributenotfound: Found 71814, but attribute 'seriesname' was empty.
I had to revert my local sickbeard.db to the previous night's backup.

mmhere on 11 Jan 2020

V1 isn't in use AT ALL anymore. Just update, develop was merged to master.

miigotu on 11 Jan 2020

@usenet135 because tvdb site itself broke it, not SickChill. Just update and try again, I rewrote the entire indexer code to use their new apu (which is still unstable even)

do i need to switch from sickbeard to sickchill?

usenet135 on 11 Jan 2020

@usenet135
If you are going to post on the SickChill site, then yes, switch to SickChill

Otherwise post at SickBeard (https://github.com/midgetspy/Sick-Beard) for support with that version... I do note that SickBeard has not been updated since 2016 and "issues" are disabled.

clinton-hall on 11 Jan 2020

@miigotu said:
V1 isn't in use AT ALL anymore. Just update, develop was merged to master.

@miigotu Unless I missed an announcement, I wasn't aware until the post above just now that develop —with all the tré coolio enhancements to use thetvdb V3 API [kinda working? we hope??]— has been merged to main.

Would you mind terribly confirming that main is blessed with V3 enhancements before I undertake the switch?

I have a few minor enhancements in my own code (notification handling, subtitle retrieval post processing) that I re-apply. I've been waiting to update and apply those until I heard word that main is enhanced.

mmhere on 11 Jan 2020

@usenet135
If you are going to post on the SickChill site, then yes, switch to SickChill

Otherwise post at SickBeard (https://github.com/midgetspy/Sick-Beard) for support with that version... I do note that SickBeard has not been updated since 2016 and "issues" are disabled.

sickbeard was referenced which is why i asked. are there installation instructions anywhere?

usenet135 on 11 Jan 2020

@usenet135

SickChill was forked from SickBeard many years ago.... but it really is quite different now.

Installation instructions:
https://github.com/SickChill/SickChill/wiki/Installation-&-Configuration-Guides

clinton-hall on 11 Jan 2020

thank you! i tried this last night but still had the error but will try again

usenet135 on 11 Jan 2020

@mmhere yes, I merged about 80k lines of changed code to master last night.

miigotu on 11 Jan 2020

@miigotu Most cool, sir. R-E-S-P-E-C-T

mmhere on 11 Jan 2020

@mmhere yes, I merged about 80k lines of changed code to master last night.

@miigotu Thank you sooo much! I was driving myself crazy last night trying to get that to work... SickChill is working and migrating shows over. You just make this girl very happy!

usenet135 on 11 Jan 2020

As an aside, and having not updated to latest master yet, the V1 show updates seem to be working again given the data coming thru from thetvdb. I believe that cloudfront had bad data in their caches for many hours after the latest round of unf**king that thetvdb did yesterday.

mmhere on 12 Jan 2020

Is there still a problem with TVDB? I don't seem to get any new episodes from any of the shows I'm tracking in SickChill. When I do a 'force full update' an a show i get lots of errors in de log file:

2020-03-01 10:44:45 INFO SHOWQUEUE-REFRESH :: Metadata writer is unable to find episode 1x16 of Doctor Who (2005) on theTVDB...has it been removed? Should I delete from db?

Also any new series like Star Trek Picard and Fauda are not refreshed and being downloaded.

Manfred73 on 1 Mar 2020

i do not have any issues anymore, i do a regular force full update, that is all.
Maybe remove and add show to db, or try a re install of sickchill. i had a lot of updates last weeks, so if you did not try the re install.

McEnnes on 1 Mar 2020

Hi,

I did a full update on some of the new shows. It seems to refresh the list with new episodes from the TVDB now. However, the new shows are still not being downloaded, even when I search them manually by clicking the glass/finder icon for an episode.

I suspect this is because I've set preferred quality of some newly added shows to UHD-4K and there's hardely any show available in that quality I guess.

If I set preferred quality to UHD-4K, will it then only search for that quality and not find 1080p for example?

Update

I set preferred quality to HD1080p and did manual search for the episodes. They are downloading now.

Manfred73 on 9 Mar 2020

Sickchill: TVDB updated their API and now SickChill can't find anything

Before submitting your issue:

Most helpful comment

All 309 comments

5687

Current Status Summary

API v3 hacks -- how to access valid data the new way

Full Details

TL;DR version A

TL;DR version B

TL;DR version A

TL;DR version B

TL;DR version A

TL;DR version B

>

Update

Related issues