Bazarr: Bad subtitle encoding

Created on 12 Jan 2018  路  54Comments  路  Source: morpheus65535/bazarr

Hi

when bazarr download czech or slovak subtitles where is diacritics it show like this,

na jej膫颅 p脛鈥簍ilet膫漏 misi.
Zkoumat nezn膫藝m膫漏 nov膫漏 sv脛鈥簍y,

bug library related
Source
picture halali

All 54 comments

It's strange because I explicitly ask to Subliminal module to save content of subtitles in utf-8: https://github.com/morpheus65535/bazarr/blob/97652cf0728bb8ab858c4adafb1963344a6565e0/get_subtitle.py#L38

It's been tested a lot (like a thousand time) without any problem. Do you have this problem with only one subtitles or more?

morpheus65535 picture morpheus65535 on 12 Jan 2018

more, all that i download...

had same problem with sub-zero and they solve it somehow...
but i dont use sub-zero anymore, it download lot of shit subtitles

halali picture halali on 12 Jan 2018

@zeusy123 You had problems with those languages?

morpheus65535 picture morpheus65535 on 12 Jan 2018

@halali Still have this issue? You can point me on an Sub-Zero issue where I can find more information on what they've done to fix it? Thanks!

morpheus65535 picture morpheus65535 on 17 Jan 2018

Yes still have it

try this
https://github.com/pannal/Sub-Zero.bundle/commit/e274a542c116e64d9804139f0b03725705911a49
https://github.com/pannal/Sub-Zero.bundle/commit/419bee76e203ff8689bb605008f3a1e87eb93105

halali picture halali on 17 Jan 2018

Ok seems they are using their own patched version of subliminal. I would prefer not to go down this path...

I've hear that @Diaoul is working on a new major version of Subliminal (v3). I hope he'll fix that kind of stuff in it.

morpheus65535 picture morpheus65535 on 17 Jan 2018

@halali Could you switch to dev branch and update. You're going to be using the last development version of Subliminal. Please report back.

morpheus65535 picture morpheus65535 on 23 Apr 2018

yes it work now

halali picture halali on 23 Apr 2018
👍1

This is a problem for me too that started recently. Attaching the original files from the subtitle sites downloaded and the subtitle file that has been processed by Bazarr. Basically when the subtitle is re-coded from ANSI to UTF-8, it displays incorrect local characters on both of my Samsung TVs as well as on Windows 10 using BSPlayer.
Is there a chance that re-coding can be made optional please with a switch? Currently this makes Bazarr unusable for me because I have to re-download all subtitles again.
Attaching a few examples. They have been downloaded from Supersubtitles (hungarian provider).
conversations.with.a.killer.bazarr.zip
Conversations.with.a.Killer.The.Ted.Bundy.Tapes.S01.720p.NF.WEB-DL.DD5.1.x264-QOQ.hun.zip
chernobyl.zip

gszigethy picture gszigethy on 11 May 2019

What can be done with this? All my subtitles downloaded that are re-coded from ANSI to UTF-8 are incorrect, basically Bazarr became unusable. I'm using the docker image. No local characters are coded well.

Thanks.

gszigethy picture gszigethy on 23 May 2019

@pannal could we make this optionnal?
https://github.com/morpheus65535/bazarr/blob/59ac0a8b910189c6a958270e91080aab16560e81/libs/subliminal_patch/core.py#L312

morpheus65535 picture morpheus65535 on 23 May 2019

Converting to UTF-8 is not an issue at all. If it's done correctly.

This might be a special case for Hungarian, I'll check.

pannal picture pannal on 23 May 2019
👍1

Checking chernobyl.zip:
Both look similar to me, for example line 12: 艕 volt a legjobb v谩laszt谩s. vs. 脮 volt a legjobb v谩laszt谩s.

Are your clients perhaps not capable of displaying UTF-8?

Edit: Oh, they're not. 脮 vs. 艕, is that what you're talking about?

pannal picture pannal on 23 May 2019

Thank you guys for looking into it. It is not just those characters, let me show you. It is visible with all characters.

1) Bazarr-normalized subtitle
This has been normalized by Bazarr and converted to UTF-8.

  • BSPlayer.
    image
  • Samsung TV
    image
  • This is the subtitle:
    Chernobyl.S01E01.720p.WEBRip.x264-TBS.bazarr.zip
    As you see, two clients (and my other TV as well) interprets them the same way, incorrectly.
  • Notepad++ on Windows says this subtitle is UTF-8:
    image
  • If I check the file in Debian:
root@llamaduck:/srv/tomb/jbod/data/sorozat/Chernobyl/Season 01# uchardet Chernobyl.S01E01.720p.WEBRip.x264-TBS.bazarr.srt 
UTF-8

2) Original subtitle downloaded from the provider:
This is Windows-1252 encoded.

  • BSPLayer
    image
  • Samsung TV
    image
  • This is the subtitle:
    Chernobyl.S01E01.720p.WEBRip.x264-TBS.original.zip
    As you see, two clients (and my other TV as well) interprets them the same way, and this time, correctly.
  • Notepad++ on Windows says this subtitle is encoded in ANSI:
    image
  • If I check the file in Debian:
root@llamaduck:/srv/tomb/jbod/data/sorozat/Chernobyl/Season 01# uchardet Chernobyl.S01E01.720p.WEBRip.x264-TBS.original.srt 
windows-1252

It looks to me that very likely the input character encoding may not be properly detected and then the translation to UTF-8 results in garbage. This can be even with incorrect input file encoding as well, but again, that's something we don't really have influence on. So I believe the option not to normalize but keep the original file would be great for some exotic languages :)

I have also noticed that for me this started beginning February. Big Bang Theory is one of the series I watch and Bazarr started encode the file incorrectly at that point. I can't track it back unfortunately which version of container have I been using before, still trying to figure it out, but it is very likely out of sync with your release cycle.

gszigethy picture gszigethy on 24 May 2019

Ok, so I managed to narrow it down. This is related to UTF-8 BOM. As soon as the BOM is present, everything works as expected. In fact I had UTF-8 subtitles from the past, they just worked flawlessly as they had UTF-8-SIG encoding so (UTF-8 with BOM).

To add UTF-8 BOM in existing files:
sed -i '1s/^/\xef\xbb\xbf/' $filename
Where filename is obviously is the filename.

Not sure if this makes sense to include in the code by default or as an option? I still think though that re-coding should be optional as plenty of devices (older TVs) don't support Unicode and there is nothing wrong with people using their local code tables/encoding.

Another possible workaround is to use Bazarr's post processing feature invoking this little script, calling with the {{subtitles}} parameter and of course using the user's local encoding.

#!/bin/bash
encoding=`uchardet "$1"`;
if [ "$encoding" == "UTF-8" ]; then
        recode UTF-8..windows-1250 "$1";
fi;

This has a drawback of uchardet missing from the pre-compiled docker container. But with some effort this can be worked around.

gszigethy picture gszigethy on 24 May 2019

@gszigethy I'll look into it. We might throw away the BOM by accident. I agree about the normalization to UTF-8 being optional - this might be a use case for Bazarr, but it wasn't for Sub-Zero, which is the basis for Bazarr (through subliminal_patch), before.

pannal picture pannal on 24 May 2019

You are awesome and deserve a cookie and a beer!

gszigethy picture gszigethy on 24 May 2019

Reference for this: https://docs.python.org/2/library/codecs.html#encodings-and-unicode
On decoding utf-8-sig will skip those three bytes if they appear as the first three bytes in the file. In UTF-8, the use of the BOM is discouraged and should generally be avoided.

So in general it should work perfectly when I use UTF-8-SIG as a replacement decode encoding for UTF-8, then use UTF-8 for the target encoding as the BOM isn't needed anymore, after decoding correctly.

Will test tonight.

pannal picture pannal on 24 May 2019

Yep, that's exactly what I was expecting: https://stackoverflow.com/questions/8898294/convert-utf-8-with-bom-to-utf-8-with-no-bom-in-python

pannal picture pannal on 24 May 2019

I'm not sure the problem is clear. The identification of the source file is ok, no issue there. The problem is when you normalize and write the UTF-8 subtitle, you don't write BOM in the output and many devices, including Samsung TVs are unable to identify the UTF-8 without the BOM. BOM is not mandatory part of UTF-8 but is widely used and accepted.

https://en.wikipedia.org/wiki/Byte_order_mark

The Unicode Standard permits the BOM in UTF-8,[3] but does not require or recommend its use.[4] Byte order has no meaning in UTF-8,[5] so its only use in UTF-8 is to signal at the start that the text stream is encoded in UTF-8, or that it was converted to UTF-8 from a stream that contained an optional BOM

As the BOM is accepted but not recommended, maybe introducing BOM for the output in general may work but not necessarily the right approach. Probably your original idea, ie. making normalization and re-coding optional is the better way. In this case switching off normalization would result only in find-match-rename-download action only, not touching the file content.

gszigethy picture gszigethy on 24 May 2019

@gszigethy If you enable debug logging and download a subtitle, what is the guessed encoding?

logger.debug("Changing encoding: to %s, from %s", encoding, ge)

morpheus65535 picture morpheus65535 on 24 May 2019 
24/05/2019 14:56:26|INFO    |subliminal_patch.subtitle       |Guessing encoding for language hu|
24/05/2019 14:56:26|DEBUG   |subliminal_patch.subtitle       |Trying encodings ['utf-8', 'windows-1250', 'iso-8859-2']|
24/05/2019 14:56:26|INFO    |subliminal_patch.subtitle       |Guessed encoding windows-1250|
24/05/2019 14:56:26|DEBUG   |subliminal_patch.subtitle       |Changing encoding: to utf-8, from windows-1250|

This is correct actually. But it doesn't write BOM, that confuses Samsung.

gszigethy picture gszigethy on 24 May 2019

Ok. I don't get why BOM isn't written while we use .encode("utf-8") which should do it.

morpheus65535 picture morpheus65535 on 24 May 2019

Hmm. Maybe we should always write a BOM. Or only for Hungarian? I'll test.

pannal picture pannal on 24 May 2019

This is weird, though. There is no BOM in windows-1250.

pannal picture pannal on 24 May 2019

Wait, are you sure this is a BOM issue? https://en.wikipedia.org/wiki/Windows-1250
Could it be that the file is actually Windows-1252 encoded?

pannal picture pannal on 24 May 2019

Sublime Text detect it as 1252.

morpheus65535 picture morpheus65535 on 24 May 2019

BOM is for UTF-8 to let devices know that is an UTF-8 file. When you push a stream to UTF-8, you may or may not add BOM at the beginnig of it. It is unrealted to the source encoding that is correctly identified. I've tested with iconv under linux and the same translation between Windows-1250 and utf-8 works the same way as in bazarr.

gszigethy picture gszigethy on 24 May 2019

Im downloading Czech and Slovak subs. And i havent any encoding issues.

halali picture halali on 24 May 2019

BOM is for UTF-8 to let devices know that is an UTF-8 file. When you push a stream to UTF-8, you may or may not add BOM at the beginnig of it. It is unrealted to the source encoding that is correctly identified. I've tested with iconv under linux and the same translation between Windows-1250 and utf-8 works the same way as in bazarr.

No, not exclusively. It also indicates the endianness of the text following.

pannal picture pannal on 24 May 2019

@pannal Should you add an elif specific for Hungarian?

            # Hungarian
            elif self.language.alpha3 in ("hun"):
                encodings.extend(['windows-1252'])
morpheus65535 picture morpheus65535 on 24 May 2019

Probably. @gszigethy can you try the attached file?
cherno_test_utf8.zip

@morpheus65535 if this works, then yes.

pannal picture pannal on 24 May 2019

Case study:

Chernobyl S01E01

24/05/2019 17:45:57|INFO    |subliminal_patch.subtitle       |Guessing encoding for language hu|
24/05/2019 17:45:57|DEBUG   |subliminal_patch.subtitle       |Trying encodings ['utf-8', 'windows-1250', 'iso-8859-2']|
24/05/2019 17:45:57|INFO    |subliminal_patch.subtitle       |Guessed encoding windows-1250|
24/05/2019 17:45:57|DEBUG   |subliminal_patch.subtitle       |Changing encoding: to utf-8, from windows-1250|
24/05/2019 17:45:57|DEBUG   |subliminal_patch.core           |All languages downloaded|
24/05/2019 17:45:57|INFO    |subliminal_patch.core           |Downloaded 1 subtitle(s)|
24/05/2019 17:45:57|DEBUG   |subliminal.core                 |Terminating initialized providers|
24/05/2019 17:45:57|INFO    |subliminal_patch.core           |Terminating provider supersubtitles|
24/05/2019 17:45:57|DEBUG   |subliminal_patch.core           |Subtitle formats requested: ('srt',)|
24/05/2019 17:45:57|DEBUG   |subliminal_patch.core           |Saving <SuperSubtitlesSubtitle u'Chernobyl (2019) S01E01 [WEBRip-ION10, WEBRip-TBS, WEBRip-AFG, WEBRip.720p-TBS, WEBRip.720p-RMTeam, INTERNAL.WEB.1080p-MEMENTO, INTERNAL.WEB.1080p-RMTeam]' [hu]> to u'/tv/data/sorozat/Chernobyl/Season 01/Chernobyl.S01E01.720p.WEBRip.x264-TBS.srt'|
24/05/2019 17:45:57|DEBUG   |root                            |BAZARR Subtitles file saved to disk: /tv/data/sorozat/Chernobyl/Season 01/Chernobyl.S01E01.720p.WEBRip.x264-TBS.srt|

Checking in Linux:

root@llamaduck:~# uchardet /srv/tomb/jbod/data/sorozat/Chernobyl/Season\ 01/Chernobyl.S01E01.720p.WEBRip.x264-TBS.srt 
UTF-8

Same subtitle downloaded manually from here

root@llamaduck:~# uchardet /srv/tomb/jbod/data/sorozat/Chernobyl/Season\ 01/chernobyl.s01e01.webrip.x264-tbs.srt.orig 
windows-1252

Then I recode it with iconv:
root@llamaduck:~# iconv -f windows-1252 -t utf-8

Result is, when manually downloaded and encoded from Windows-1252 to UTF-8, is the same. Wrong display on TV and BSPlayer.

Then I add the BOM with the above sed command and it works.

gszigethy picture gszigethy on 24 May 2019

Yeah, please check my subtitle I attached. It should work.

This is an encoding detection issue, and it should not have anything to do with BOM.

pannal picture pannal on 24 May 2019

The issue with Windows-1250 and 1252 is that they're compatible (readable vice-versa), but have slightly swapped character tables. I'm pretty sure that's the issue here and the fix is easy.

pannal picture pannal on 24 May 2019

@pannal No, the uploaded subtitle has the same issue unfortunately. If I add the BOM to yours, it also works.

gszigethy picture gszigethy on 24 May 2019

The question now is: Does including the BOM always destroy anything else?

pannal picture pannal on 24 May 2019

I think adding the flag for not normalizing is the better fix.

pannal picture pannal on 24 May 2019

Do I have to quote Murphy's law? ;-)

morpheus65535 picture morpheus65535 on 24 May 2019

I think adding the flag for not normalizing is the better fix.

I agree. Leaving the option enabled by default will not break backwards compatibility while people using strange characters can turn it off and use the raw subtitle.

gszigethy picture gszigethy on 24 May 2019

I've added the SZ_KEEP_ENCODING env var to the newest subliminal_patch, which is now merged into bazarr-develop. You can try setting that var to anything ("True" for example) and see if that fixes things until it's a flag in the GUI.

pannal picture pannal on 24 May 2019

@gszigethy were you able to try that fix out?

pannal picture pannal on 26 May 2019

@gszigethy were you able to try that fix out?

Ok, it's getting there :) What I did:

  1. Pulled linuxserver/bazarr:b909bd65-ls34 docker container from the hub. That includes your changes. https://github.com/linuxserver/docker-bazarr/releases/tag/b909bd65-ls34

  2. First I deployed the container with -e SZ_KEEP_ENCODING=True \ but then I realized the logic is actually reversed:
    if not os.environ.get("SZ_KEEP_ENCODING", False):
    If SZ_KEEP_ENCODING is not False (so anything basically) --> normalize. If False: don't do normalization.

  3. So I had to set to false to avoid normalizing, so I deployed the container with -e SZ_KEEP_ENCODING=False \.
    With this setting it seems to work almost properly, with the following comments:

  4. It still seem to do something with the subtitles, because the size of the original and the bazarr/subliminal processed file is different. However most subtitles work.
  5. Interestingly, Chernoby S01E01 subtitle is now identified incorrectly as UTF-16 and subliminal fails as a result. Debug log:

26/05/2019 10:10:16|DEBUG   |apscheduler.scheduler           |Looking for jobs to run|
26/05/2019 10:10:16|INFO    |apscheduler.executors.default   |Running job "Update series list from Sonarr (trigger: interval[0:01:00], next run at: 2019-05-26 10:10:16 CEST)" (scheduled at 2019-05-26 10:10:16.142615+02:00)|
26/05/2019 10:10:16|DEBUG   |urllib3.connectionpool          |Starting new HTTP connection (1): 192.***.***.***:8989|
26/05/2019 10:10:16|DEBUG   |apscheduler.scheduler           |Next wakeup is due at 2019-05-26 10:11:16.142615+02:00 (in 59.975510 seconds)|
26/05/2019 10:10:16|DEBUG   |subliminal_patch.http           |DNS: Falling back to default DNS or IP on 192.***.***.***|
26/05/2019 10:10:16|DEBUG   |urllib3.connectionpool          |http://192.***.***.***:8989 "GET /api/profile?apikey=********2a HTTP/1.1" 200 None|
26/05/2019 10:10:16|DEBUG   |urllib3.connectionpool          |Starting new HTTP connection (1): 192.***.***.***:8989|
26/05/2019 10:10:16|DEBUG   |subliminal_patch.http           |DNS: Falling back to default DNS or IP on 192.***.***.***|
26/05/2019 10:10:16|DEBUG   |urllib3.connectionpool          |http://192.***.***.***:8989 "GET /api/v3/languageprofile?apikey=********2a HTTP/1.1" 404 None|
26/05/2019 10:10:16|DEBUG   |urllib3.connectionpool          |Starting new HTTP connection (1): 192.***.***.***:8989|
26/05/2019 10:10:16|DEBUG   |subliminal_patch.http           |DNS: Falling back to default DNS or IP on 192.***.***.***|
26/05/2019 10:10:16|DEBUG   |urllib3.connectionpool          |http://192.***.***.***:8989 "GET /api/series?apikey=********2a HTTP/1.1" 200 None|
26/05/2019 10:10:16|INFO    |apscheduler.executors.default   |Job "Update series list from Sonarr (trigger: interval[0:01:00], next run at: 2019-05-26 10:11:16 CEST)" executed successfully|
26/05/2019 10:10:20|INFO    |apscheduler.scheduler           |Added job "manual_wanted_search_missing_subtitles" to job store "default"|
26/05/2019 10:10:20|DEBUG   |apscheduler.scheduler           |Looking for jobs to run|
26/05/2019 10:10:20|INFO    |apscheduler.scheduler           |Removed job manual_wanted_search_missing_subtitles|
26/05/2019 10:10:20|DEBUG   |apscheduler.scheduler           |Next wakeup is due at 2019-05-26 10:11:16.142615+02:00 (in 55.724674 seconds)|
26/05/2019 10:10:20|INFO    |apscheduler.executors.default   |Running job "manual_wanted_search_missing_subtitles (trigger: date[2019-05-26 10:10:20 CEST], next run at: 2019-05-26 10:10:20 CEST)" (scheduled at 2019-05-26 10:10:20.378917+02:00)|
26/05/2019 10:10:20|DEBUG   |root                            |BAZARR Searching subtitles for this file: /tv/data/sorozat/Chernobyl/Season 01/Chernobyl.S01E01.720p.WEBRip.x264-TBS.mkv|
26/05/2019 10:10:20|DEBUG   |subzero.video                   |Parsing video: Chernobyl.S01E01.720p.WEBRip.x264-TBS.mkv, hints: {'type': 'episode', 'title': u'Chernobyl'}|
26/05/2019 10:10:20|INFO    |subliminal_patch.core           |Determining basic video properties for u'Chernobyl.S01E01.720p.WEBRip.x264-TBS.mkv' in u'/tv/data/sorozat/Chernobyl/Season 01'|
26/05/2019 10:10:20|DEBUG   |subliminal_patch.core           |GuessIt found: {
    "title": "Chernobyl", 
    "season": 1, 
    "episode": 1, 
    "screen_size": "720p", 
    "format": "WEBRip", 
    "video_codec": "h264", 
    "release_group": "TBS", 
    "container": "mkv", 
    "type": "episode"
}|
26/05/2019 10:10:20|DEBUG   |root                            |BAZARR is using those video object properties: {'series': u'Chernobyl', 'original_path': u'/tv/data/sorozat/Chernobyl/Season 01/Chernobyl.S01E01.720p.WEBRip.x264-TBS.mkv', 'series_imdb_id': None, 'audio_codec': u'EAC3', 'year': 2019, 'size': None, 'title': u'1:23:45', 'original_name': u'Chernobyl.S01E01.720p.WEBRip.x264-TBS.mkv', 'release_group': u'TBS', 'original_series': True, 'plexapi_metadata': {}, 'format': u'WEBRip', 'season': 1, 'tvdb_id': None, 'imdb_id': None, 'external_subtitle_languages': set([]), 'series_tvdb_id': 360893, 'hashes': {}, 'hints': {'type': 'episode', 'expected_title': [u'Chernobyl'], 'single_value': True, 'title': u'Chernobyl'}, 'episode': 1, 'name': u'/tv/data/sorozat/Chernobyl/Season 01/Chernobyl.S01E01.720p.WEBRip.x264-TBS.mkv', 'subtitle_languages': set([]), 'used_scene_name': True, 'alternative_series': [], 'audio_languages': set([]), 'video_codec': u'h264', 'resolution': u'720p'}|
26/05/2019 10:10:20|INFO    |subliminal_patch.core           |Using 1 threads for 1 providers (['supersubtitles'])|
26/05/2019 10:10:20|INFO    |subliminal_patch.core           |Downloading best subtitles for <Episode [u'Chernobyl', 2019, 1x1]>|
26/05/2019 10:10:20|INFO    |subliminal_patch.core           |Listing subtitles with provider 'supersubtitles' and languages set([<Language [hu]>])|
26/05/2019 10:10:20|INFO    |subliminal_patch.core           |Initializing provider supersubtitles|
26/05/2019 10:10:20|INFO    |subliminal_patch.providers.supersubtitles|Get series id from URL https://www.feliratok.info/index.php?term=Chernobyl&nyelv=0&action=autoname|
26/05/2019 10:10:20|DEBUG   |urllib3.connectionpool          |Starting new HTTPS connection (1): www.feliratok.info:443|
26/05/2019 10:10:20|DEBUG   |subliminal_patch.http           |DNS: Falling back to default DNS or IP on www.feliratok.info|
26/05/2019 10:10:20|DEBUG   |urllib3.connectionpool          |https://www.feliratok.info:443 "GET /index.php?term=Chernobyl&nyelv=0&action=autoname HTTP/1.1" 200 None|
26/05/2019 10:10:20|INFO    |subliminal_patch.providers.supersubtitles|URL for subtitles https://www.feliratok.info/index.php?search=&soriSorszam=&nyelv=&sorozatnev=&sid=4934&complexsearch=true&knyelv=0&evad=1&epizod1=1&cimke=0&minoseg=0&rlsr=0&tab=all|
26/05/2019 10:10:20|DEBUG   |urllib3.connectionpool          |https://www.feliratok.info:443 "GET /index.php?search=&soriSorszam=&nyelv=&sorozatnev=&sid=4934&complexsearch=true&knyelv=0&evad=1&epizod1=1&cimke=0&minoseg=0&rlsr=0&tab=all HTTP/1.1" 200 None|
26/05/2019 10:10:21|INFO    |subliminal_patch.providers.supersubtitles|Get IMDB id from URL https://www.feliratok.info/index.php?tipus=adatlap&azon=a_1557245386|
26/05/2019 10:10:21|DEBUG   |urllib3.connectionpool          |https://www.feliratok.info:443 "GET /index.php?tipus=adatlap&azon=a_1557245386 HTTP/1.1" 200 None|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |utf-8  confidence = 0.99|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |SHIFT_JIS Japanese confidence = 0.01|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |EUC-JP Japanese confidence = 0.01|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |GB2312 Chinese confidence = 0.01|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |EUC-KR Korean confidence = 0.01|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |CP949 Korean confidence = 0.01|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |Big5 Chinese confidence = 0.01|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |EUC-TW Taiwan confidence = 0.01|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |windows-1251 Russian confidence = 0.01|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |KOI8-R Russian confidence = 0.01|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |ISO-8859-5 Russian confidence = 0.0|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |MacCyrillic Russian confidence = 0.01|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |IBM866 Russian confidence = 0.01|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |IBM855 Russian confidence = 0.01|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |ISO-8859-7 Greek confidence = 0.0|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |windows-1253 Greek confidence = 0.0|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |ISO-8859-5 Bulgairan confidence = 0.0|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |windows-1251 Bulgarian confidence = 0.0|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |TIS-620 Thai confidence = 0.153686348192|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |ISO-8859-9 Turkish confidence = 0.574258518913|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |windows-1255 Hebrew confidence = 0.0|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |windows-1255 Hebrew confidence = 0.0|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |windows-1255 Hebrew confidence = 0.0|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |utf-8  confidence = 0.99|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |SHIFT_JIS Japanese confidence = 0.01|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |EUC-JP Japanese confidence = 0.01|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |GB2312 Chinese confidence = 0.01|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |EUC-KR Korean confidence = 0.01|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |CP949 Korean confidence = 0.01|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |Big5 Chinese confidence = 0.01|
26/05/2019 10:10:21|DEBUG   |chardet.charsetprober           |EUC-TW Taiwan confidence = 0.01|
26/05/2019 10:10:21|DEBUG   |subliminal_patch.core           |<SuperSubtitlesSubtitle u'Chernobyl (2019) S01E01 [AMZN.WEBRip.720p-NTb, AMZN.WEB-DL.720p-NTb, WEBRip.720p-RMTeam, WEB-DL.720p-RMTeam, AMZN.WEBRip.1080p-NTb, AMZN.WEB-DL.1080p-NTb, WEBRip.1080p-RMTeam, WEB-DL.1080p-RMTeam]' [hu]>: Found matches set(['episode', 'format', 'series', 'year', 'season', 'resolution'])|
26/05/2019 10:10:21|INFO    |subliminal_patch.score          |<SuperSubtitlesSubtitle u'Chernobyl (2019) S01E01 [AMZN.WEBRip.720p-NTb, AMZN.WEB-DL.720p-NTb, WEBRip.720p-RMTeam, WEB-DL.720p-RMTeam, AMZN.WEBRip.1080p-NTb, AMZN.WEB-DL.1080p-NTb, WEBRip.1080p-RMTeam, WEB-DL.1080p-RMTeam]' [hu]>: Computing score for video <Episode [u'Chernobyl', 2019, 1x1]> with {'hearing_impaired': False}|
26/05/2019 10:10:21|DEBUG   |subliminal_patch.score          |Matched hearing_impaired|
26/05/2019 10:10:21|INFO    |subliminal_patch.score          |<SuperSubtitlesSubtitle u'Chernobyl (2019) S01E01 [AMZN.WEBRip.720p-NTb, AMZN.WEB-DL.720p-NTb, WEBRip.720p-RMTeam, WEB-DL.720p-RMTeam, AMZN.WEBRip.1080p-NTb, AMZN.WEB-DL.1080p-NTb, WEBRip.1080p-RMTeam, WEB-DL.1080p-RMTeam]' [hu]>: Computed score 340 with final matches set(['episode', 'format', 'series', 'year', 'season', 'resolution', 'hearing_impaired'])|
26/05/2019 10:10:21|DEBUG   |subliminal_patch.core           |<SuperSubtitlesSubtitle u'Chernobyl (2019) S01E01 [WEBRip-ION10, WEBRip-TBS, WEBRip-AFG, WEBRip.720p-TBS, WEBRip.720p-RMTeam, INTERNAL.WEB.1080p-MEMENTO, INTERNAL.WEB.1080p-RMTeam]' [hu]>: Found matches set(['episode', 'format', 'series', 'year', 'season', 'resolution', 'release_group'])|
26/05/2019 10:10:21|INFO    |subliminal_patch.score          |<SuperSubtitlesSubtitle u'Chernobyl (2019) S01E01 [WEBRip-ION10, WEBRip-TBS, WEBRip-AFG, WEBRip.720p-TBS, WEBRip.720p-RMTeam, INTERNAL.WEB.1080p-MEMENTO, INTERNAL.WEB.1080p-RMTeam]' [hu]>: Computing score for video <Episode [u'Chernobyl', 2019, 1x1]> with {'hearing_impaired': False}|
26/05/2019 10:10:21|DEBUG   |subliminal_patch.score          |Matched hearing_impaired|
26/05/2019 10:10:21|INFO    |subliminal_patch.score          |<SuperSubtitlesSubtitle u'Chernobyl (2019) S01E01 [WEBRip-ION10, WEBRip-TBS, WEBRip-AFG, WEBRip.720p-TBS, WEBRip.720p-RMTeam, INTERNAL.WEB.1080p-MEMENTO, INTERNAL.WEB.1080p-RMTeam]' [hu]>: Computed score 355 with final matches set(['episode', 'format', 'series', 'year', 'hearing_impaired', 'season', 'resolution', 'release_group'])|
26/05/2019 10:10:21|DEBUG   |subliminal_patch.core           |<SuperSubtitlesSubtitle u'Chernobyl (2019) S01E01 [AMZN.WEBRip.720p-NTb, AMZN.WEB-DL.720p-NTb, WEBRip.720p-RMTeam, WEB-DL.720p-RMTeam, AMZN.WEBRip.1080p-NTb, AMZN.WEB-DL.1080p-NTb, WEBRip.1080p-RMTeam, WEB-DL.1080p-RMTeam]' [en]>: Skipping, language not searched for|
26/05/2019 10:10:21|DEBUG   |subliminal_patch.core           |<SuperSubtitlesSubtitle u'Chernobyl (2019) S01E01 [WEBRip-ION10, WEBRip-TBS, WEBRip-AFG, WEBRip.720p-TBS, WEBRip.720p-RMTeam, INTERNAL.WEB.1080p-MEMENTO, INTERNAL.WEB.1080p-RMTeam]' [en]>: Skipping, language not searched for|
26/05/2019 10:10:21|DEBUG   |subliminal_patch.core           |<SuperSubtitlesSubtitle u'Chernobyl (2019) S01E01 [WEBRip-ION10, WEBRip-TBS, WEBRip-AFG, WEBRip.720p-TBS, WEBRip.720p-RMTeam, INTERNAL.WEB.1080p-MEMENTO, INTERNAL.WEB.1080p-RMTeam]' [hu]>: Trying to download subtitle with matches set(['episode', 'format', 'series', 'year', 'hearing_impaired', 'season', 'resolution', 'release_group']), score: 355; release(s): WEBRip-ION10, WEBRip-TBS, WEBRip-AFG, WEBRip.720p-TBS, WEBRip.720p-RMTeam, INTERNAL.WEB.1080p-MEMENTO, INTERNAL.WEB.1080p-RMTeam|
26/05/2019 10:10:21|INFO    |subliminal_patch.core           |Downloading subtitle <SuperSubtitlesSubtitle u'Chernobyl (2019) S01E01 [WEBRip-ION10, WEBRip-TBS, WEBRip-AFG, WEBRip.720p-TBS, WEBRip.720p-RMTeam, INTERNAL.WEB.1080p-MEMENTO, INTERNAL.WEB.1080p-RMTeam]' [hu]>|
26/05/2019 10:10:21|INFO    |subliminal_patch.providers.supersubtitles|Downloading subtitle '1557245369'|
26/05/2019 10:10:21|DEBUG   |urllib3.connectionpool          |https://www.feliratok.info:443 "GET /index.php?action=letolt&fnev=chernobyl.s01e01.webrip.x264-tbs.srt&felirat=1557245369 HTTP/1.1" 200 35457|
26/05/2019 10:10:21|INFO    |subliminal_patch.subtitle       |Guessing encoding for language hu|
26/05/2019 10:10:21|DEBUG   |subliminal_patch.subtitle       |Trying encodings ['utf-8', 'utf-16', 'windows-1250', 'iso-8859-2']|
26/05/2019 10:10:21|INFO    |subliminal_patch.subtitle       |Guessed encoding utf-16|
26/05/2019 10:10:21|ERROR   |subliminal_patch.subtitle       |PySRT-parsing failed, trying pysubs2|
26/05/2019 10:10:21|DEBUG   |subliminal_patch.subtitle       |Trying parsing with PySubs2|
26/05/2019 10:10:21|ERROR   |subliminal_patch.subtitle       |Couldn't convert subtitle Subtitle id: 1557245369 Series: Chernobyl Season: 1 Episode: 1 Version: WEBRip-ION10, WEBRip-TBS, WEBRip-AFG, WEBRip.720p-TBS, WEBRip.720p-RMTeam, INTERNAL.WEB.1080p-MEMENTO, INTERNAL.WEB.1080p-RMTeam Releases: ['WEBRip-ION10', 'WEBRip-TBS', 'WEBRip-AFG', 'WEBRip.720p-TBS', 'WEBRip.720p-RMTeam', 'INTERNAL.WEB.1080p-MEMENTO', 'INTERNAL.WEB.1080p-RMTeam'] DownloadLink: https://www.feliratok.info/index.php?action=letolt&fnev=chernobyl.s01e01.webrip.x264-tbs.srt&felirat=1557245369 Matches: set(['episode', 'format', 'series', 'year', 'hearing_impaired', 'season', 'resolution', 'release_group']) Year: 2019 to .srt format: Traceback (most recent call last):  File "/app/bazarr/bazarr/../libs/subliminal_patch/subtitle.py", line 242, in is_valid    subs = pysubs2.SSAFile.from_string(text)  File "/app/bazarr/bazarr/../libs/pysubs2/ssafile.py", line 118, in from_string    return cls.from_file(fp, format_, fps=fps, **kwargs)  File "/app/bazarr/bazarr/../libs/pysubs2/ssafile.py", line 145, in from_file    format_ = autodetect_format(fragment)  File "/app/bazarr/bazarr/../libs/pysubs2/formats.py", line 64, in autodetect_format    raise FormatAutodetectionError("No suitable formats")FormatAutodetectionError: No suitable formats|Traceback (most recent call last):  File "/app/bazarr/bazarr/../libs/subliminal_patch/subtitle.py", line 242, in is_valid    subs = pysubs2.SSAFile.from_string(text)  File "/app/bazarr/bazarr/../libs/pysubs2/ssafile.py", line 118, in from_string    return cls.from_file(fp, format_, fps=fps, **kwargs)  File "/app/bazarr/bazarr/../libs/pysubs2/ssafile.py", line 145, in from_file    format_ = autodetect_format(fragment)  File "/app/bazarr/bazarr/../libs/pysubs2/formats.py", line 64, in autodetect_format    raise FormatAutodetectionError("No suitable formats")FormatAutodetectionError: No suitable formats|
26/05/2019 10:10:21|ERROR   |subliminal_patch.core           |Invalid subtitle|
26/05/2019 10:10:21|DEBUG   |subliminal_patch.core           |<SuperSubtitlesSubtitle u'Chernobyl (2019) S01E01 [AMZN.WEBRip.720p-NTb, AMZN.WEB-DL.720p-NTb, WEBRip.720p-RMTeam, WEB-DL.720p-RMTeam, AMZN.WEBRip.1080p-NTb, AMZN.WEB-DL.1080p-NTb, WEBRip.1080p-RMTeam, WEB-DL.1080p-RMTeam]' [hu]>: Trying to download subtitle with matches set(['episode', 'format', 'series', 'year', 'season', 'resolution', 'hearing_impaired']), score: 340; release(s): AMZN.WEBRip.720p-NTb, AMZN.WEB-DL.720p-NTb, WEBRip.720p-RMTeam, WEB-DL.720p-RMTeam, AMZN.WEBRip.1080p-NTb, AMZN.WEB-DL.1080p-NTb, WEBRip.1080p-RMTeam, WEB-DL.1080p-RMTeam|
26/05/2019 10:10:21|INFO    |subliminal_patch.core           |Downloading subtitle <SuperSubtitlesSubtitle u'Chernobyl (2019) S01E01 [AMZN.WEBRip.720p-NTb, AMZN.WEB-DL.720p-NTb, WEBRip.720p-RMTeam, WEB-DL.720p-RMTeam, AMZN.WEBRip.1080p-NTb, AMZN.WEB-DL.1080p-NTb, WEBRip.1080p-RMTeam, WEB-DL.1080p-RMTeam]' [hu]>|
26/05/2019 10:10:21|INFO    |subliminal_patch.providers.supersubtitles|Downloading subtitle '1557245386'|
26/05/2019 10:10:21|DEBUG   |urllib3.connectionpool          |https://www.feliratok.info:443 "GET /index.php?action=letolt&fnev=Chernobyl.S01E01.***.***.***.720p.AMZN.WEB-DL.DDP5.1.H.264-NTb.srt&felirat=1557245386 HTTP/1.1" 200 35457|
26/05/2019 10:10:21|INFO    |subliminal_patch.subtitle       |Guessing encoding for language hu|
26/05/2019 10:10:21|DEBUG   |subliminal_patch.subtitle       |Trying encodings ['utf-8', 'utf-16', 'windows-1250', 'iso-8859-2']|
26/05/2019 10:10:21|INFO    |subliminal_patch.subtitle       |Guessed encoding utf-16|
26/05/2019 10:10:21|ERROR   |subliminal_patch.subtitle       |PySRT-parsing failed, trying pysubs2|
26/05/2019 10:10:21|DEBUG   |subliminal_patch.subtitle       |Trying parsing with PySubs2|
26/05/2019 10:10:21|ERROR   |subliminal_patch.subtitle       |Couldn't convert subtitle Subtitle id: 1557245386 Series: Chernobyl Season: 1 Episode: 1 Version: AMZN.WEBRip.720p-NTb, AMZN.WEB-DL.720p-NTb, WEBRip.720p-RMTeam, WEB-DL.720p-RMTeam, AMZN.WEBRip.1080p-NTb, AMZN.WEB-DL.1080p-NTb, WEBRip.1080p-RMTeam, WEB-DL.1080p-RMTeam Releases: ['AMZN.WEBRip.720p-NTb', 'AMZN.WEB-DL.720p-NTb', 'WEBRip.720p-RMTeam', 'WEB-DL.720p-RMTeam', 'AMZN.WEBRip.1080p-NTb', 'AMZN.WEB-DL.1080p-NTb', 'WEBRip.1080p-RMTeam', 'WEB-DL.1080p-RMTeam'] DownloadLink: https://www.feliratok.info/index.php?action=letolt&fnev=Chernobyl.S01E01.1.23.45.720p.AMZN.WEB-DL.DDP5.1.H.264-NTb.srt&felirat=1557245386 Matches: set(['episode', 'format', 'series', 'year', 'season', 'resolution', 'hearing_impaired']) Year: 2019 to .srt format: Traceback (most recent call last):  File "/app/bazarr/bazarr/../libs/subliminal_patch/subtitle.py", line 242, in is_valid    subs = pysubs2.SSAFile.from_string(text)  File "/app/bazarr/bazarr/../libs/pysubs2/ssafile.py", line 118, in from_string    return cls.from_file(fp, format_, fps=fps, **kwargs)  File "/app/bazarr/bazarr/../libs/pysubs2/ssafile.py", line 145, in from_file    format_ = autodetect_format(fragment)  File "/app/bazarr/bazarr/../libs/pysubs2/formats.py", line 64, in autodetect_format    raise FormatAutodetectionError("No suitable formats")FormatAutodetectionError: No suitable formats|Traceback (most recent call last):  File "/app/bazarr/bazarr/../libs/subliminal_patch/subtitle.py", line 242, in is_valid    subs = pysubs2.SSAFile.from_string(text)  File "/app/bazarr/bazarr/../libs/pysubs2/ssafile.py", line 118, in from_string    return cls.from_file(fp, format_, fps=fps, **kwargs)  File "/app/bazarr/bazarr/../libs/pysubs2/ssafile.py", line 145, in from_file    format_ = autodetect_format(fragment)  File "/app/bazarr/bazarr/../libs/pysubs2/formats.py", line 64, in autodetect_format    raise FormatAutodetectionError("No suitable formats")FormatAutodetectionError: No suitable formats|
26/05/2019 10:10:21|ERROR   |subliminal_patch.core           |Invalid subtitle|
26/05/2019 10:10:21|INFO    |subliminal_patch.core           |Downloaded 0 subtitle(s)|
26/05/2019 10:10:21|DEBUG   |subliminal.core                 |Terminating initialized providers|
26/05/2019 10:10:21|INFO    |subliminal_patch.core           |Terminating provider supersubtitles|
26/05/2019 10:10:21|DEBUG   |root                            |BAZARR No subtitles were found for this file: /tv/data/sorozat/Chernobyl/Season 01/Chernobyl.S01E01.720p.WEBRip.x264-TBS.mkv|
26/05/2019 10:10:21|INFO    |root                            |BAZARR Finished searching for missing subtitles. Check histories for more information.|
26/05/2019 10:10:21|INFO    |apscheduler.executors.default   |Job "manual_wanted_search_missing_subtitles (trigger: date[2019-05-26 10:10:20 CEST], next run at: 2019-05-26 10:10:20 CEST)" executed successfully|

Not sure if this is the result of the same change. With the stable version it is identified correctly as follows:

26/05/2019 10:30:07|INFO    |subliminal_patch.providers.supersubtitles|Downloading subtitle '1557245369'|
26/05/2019 10:30:07|DEBUG   |urllib3.connectionpool          |https://www.feliratok.info:443 "GET /index.php?action=letolt&fnev=chernobyl.s01e01.webrip.x264-tbs.srt&felirat=1557245369 HTTP/1.1" 200 35457|
26/05/2019 10:30:07|INFO    |subliminal_patch.subtitle       |Guessing encoding for language hu|
26/05/2019 10:30:07|DEBUG   |subliminal_patch.subtitle       |Trying encodings ['utf-8', 'windows-1250', 'iso-8859-2']|
26/05/2019 10:30:07|INFO    |subliminal_patch.subtitle       |Guessed encoding windows-1250|
26/05/2019 10:30:07|DEBUG   |subliminal_patch.subtitle       |Changing encoding: to utf-8, from windows-1250|
26/05/2019 10:30:07|DEBUG   |subliminal_patch.core           |All languages downloaded|
26/05/2019 10:30:07|INFO    |subliminal_patch.core           |Downloaded 1 subtitle(s)|
26/05/2019 10:30:07|DEBUG   |subliminal.core                 |Terminating initialized providers|
26/05/2019 10:30:07|INFO    |subliminal_patch.core           |Terminating provider supersubtitles|
26/05/2019 10:30:07|DEBUG   |subliminal_patch.core           |Subtitle formats requested: ('srt',)|
26/05/2019 10:30:07|DEBUG   |subliminal_patch.core           |Saving <SuperSubtitlesSubtitle u'Chernobyl (2019) S01E01 [WEBRip-ION10, WEBRip-TBS, WEBRip-AFG, WEBRip.720p-TBS, WEBRip.720p-RMTeam, INTERNAL.WEB.1080p-MEMENTO, INTERNAL.WEB.1080p-RMTeam]' [hu]> to u'/tv/data/sorozat/Chernobyl/Season 01/Chernobyl.S01E01.720p.WEBRip.x264-TBS.srt'|

I couldn't reproduce with other subtitles, but with this one it's reproducable.

gszigethy picture gszigethy on 26 May 2019
  • Interestingly, Chernoby S01E01 subtitle is now identified incorrectly as UTF-16 and subliminal fails as a result.

Ok, I checked and this is caused by https://github.com/morpheus65535/bazarr/commit/1ed4f11a6794d8a9b44043438cb95a7ed578f9df. If I roll back those 2 small changes in this commit, everything works flawlessly.

gszigethy picture gszigethy on 26 May 2019

Yeah I think the move to always try UTF-16 as well was bad. I'll revert that commit.

pannal picture pannal on 27 May 2019

Yeah I think the move to always try UTF-16 as well was bad. I'll revert that commit.

You are a rock star! Fixed my issue. Will send you a beer. Cheers.

gszigethy picture gszigethy on 27 May 2019

if not os.environ.get("SZ_KEEP_ENCODING", False):
If it's set to anything, but empty string, normalization will not occur. The False in there is just a fallback in case the variable isn't set at all (e.g.: The default case).

pannal picture pannal on 27 May 2019
👍1

if not os.environ.get("SZ_KEEP_ENCODING", False):
If it's set to _anything_, but empty string, normalization will not occur. The False in there is just a fallback in case the variable isn't set at all (e.g.: The default case).

        subtitle.normalize()
        if not os.environ.get("SZ_KEEP_ENCODING", False):
            subtitle.normalize()

Are you sure? If I read right, it says if system variable SZ_KEEP_ENCODING is NOT False, then normalize. Any other cases, don't normalize. I understand it works, but logically it's reversed.

It should be

        subtitle.normalize()
        if not os.environ.get("SZ_KEEP_ENCODING", True):
            subtitle.normalize()

Meaning, if _keep encoding_ is not equal to _True_, then normalize. It's just cosmetics, but I think this is logically correct.

gszigethy picture gszigethy on 27 May 2019

Are you sure?

Yes.

Again, os.environ.get("SZ_KEEP_ENCODING", False) equals to "SZ_KEEP_ENCODING is set to anything but a value that equals False (in this case anything but an empty string) == True; if it's not set, it's False.

not False is True, so the equation works by default. not True is False, so once it's set, the equation is negative and normalization doesn't occur.

It seems backwards, that's right.

pannal picture pannal on 27 May 2019

Again, the False here is a fallback value for when os.environ doesn't have any key named SZ_KEEP_ENCODING.

pannal picture pannal on 27 May 2019

One thing that might throw you off here is the fact that setting SZ_KEEP_ENCODING=False in your env actually results in os.environ.get("SZ_KEEP_ENCODING", False) returning "False" (a string, not a boolean, and bool("False")==True).

Edit: I'm explicitly checking for key existence in that equation, not using the value itself.

pannal picture pannal on 27 May 2019
👍1

OK, I believe it's a language barrier :) I understand the logic, but I interpreted the variable name KEEP_ENCODING that _we keep the current character coding of the file_. I believe your logic works when I interpret the variable name KEEP_ENCODING in a way that _let's continue encoding the file to UTF-8_.

gszigethy picture gszigethy on 27 May 2019

normalize() throws the original encoding out of the window, so SZ_KEEP_ENCODING equals "not normalize".

pannal picture pannal on 27 May 2019

This one have been fixed in bed98832ed2a5b596265712bb3c0ac77d28ed264.

morpheus65535 picture morpheus65535 on 20 Jun 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

hergonoway picture hergonoway  路  4Comments
Bresol picture Bresol  路  6Comments
ExtensiveJS picture ExtensiveJS  路  6Comments
hawkash picture hawkash  路  7Comments
ronajon picture ronajon  路  6Comments