Jellyfin-web: Provide an alternative to JavascriptSubtitlesOctopus

[sshockwave]

JavascriptSubtitlesOctopus was introduced to render SSA/ASS subtitles with embedded fonts(https://github.com/jellyfin/jellyfin-web/pull/547) and later became the only renderer (https://github.com/jellyfin/jellyfin-web/commit/0cb54feb539ef4e7e7192b1fc92fdb5de24b4815). SubtitlesOctopus is ported from libass and relies on font files. This breaks the display of CJK characters since Jellyfin doesn't provide any CJK fonts by default(https://github.com/jellyfin/jellyfin-web/issues/934; https://github.com/jellyfin/jellyfin/issues/2670).

People have managed to bypass this issue by providing the renderer with CJK noto fonts(https://github.com/jellyfin/jellyfin-web/issues/934#issuecomment-602145363; https://github.com/jellyfin/jellyfin-web/issues/934#issuecomment-609442805; https://github.com/jellyfin/jellyfin-web/pull/1636). But CJK font files are HUGE. A complete experience requires GBs of locally installed font resources and even minified versions take up to 10MB+. Transferring them over the Internet is slow, and not mobile-friendly. On the other hand, there's no API in the browser to import local font resources as far as I know.

There are some alternatives, such as ASS.js, which offload font issues to the browser. ASS.js has implemented most of SSA/ASS features. It might not be as featureful or efficient as SubtitlesOctopus(unconfirmed), but it's not the most pressing issue. CJK characters turning into tofu basically makes Jellyfin unusable for me, and also for most CJK users, I believe.

I came up with several ways to introduce the alternative:

• Providing an option to switch SSA/ASS renderers on a per-server/per-user/per-video basis. It complicates the user experience a little bit. To remember the preference, it probably needs some changes to the main repo, too.
• Use SubtitlesOctopus for media with embedded fonts, and ASS.js for anything else. This might break complex subtitles, but very unlikely. It works along with the original motivation which brings in SubtitlesOctopus according to https://github.com/jellyfin/jellyfin-web/pull/547.

I would like to hear more about it before submitting a PR.

[dkanada]

I figured #1636 was an excellent solution to this issue, is the data transfer that much of an issue? We wanted to avoid maintaining two different solutions for subtitle rendering, but it's always an option.

As an aside, lots of people aren't a fan of the asynchronous subtitle downloads, especially to use with foreign languages, since the subtitles are very important for those videos. If you have any interest in adding a toggle for synchronous downloads before the movie starts that would be an excellent feature as well.

[MrTimscampi]

From a dependency perspective, I'm a bit opposed to adding ASS.js anyway.

It looks to be very incomplete in its support for the syntax: https://github.com/weizhenye/ASS#todo and usage looks minimal:

With libass-wasm (Which is the package for Octopus), we know it's at least used by major platforms like Crunchyroll.

There's likely a better way to fix this down the line than #1636 , but that's all we have right now.

A solid workaround, for now, is disabling in-browser rendering of complex subs in the Subtitle settings, which will make Jellyfin burn the subtitles in on the the server.

[sshockwave]

I figured #1636 was an excellent solution to this issue, is the data transfer that much of an issue? We wanted to avoid maintaining two different solutions for subtitle rendering, but it's always an option.

It's rather a hack IMHO. A minimal subset of characters won't cost much(~10MB), but if I had to work with a single font and an incomplete character set, I wouldn't have used ASS subtitles in the first place. Probably convert them to SRT, etc.

As an aside, lots of people aren't a fan of the asynchronous subtitle downloads, especially to use with foreign languages, since the subtitles are very important for those videos. If you have any interest in adding a toggle for synchronous downloads before the movie starts that would be an excellent feature as well.

Sorry, I'm not quite following up. Is the preloading necessary because of large subtitle files?

[dkanada]

Preloading could be necessary because if the audio is a language you don't know, and the subtitles take forty seconds to load, you wouldn't understand what's going on during those forty seconds. I assume this also applies to those who need subtitles for hearing issues. Our usual recommendation is to pause the video while they load, but it's not as nice as a setting to load the subtitles before starting the video.

[dkanada]

@MrTimscampi speaking of which, maybe there's a way we could show nice loading indicators for subtitles, video, et cetera to improve the UX while people wait more than twenty seconds for videos to load on slow connections.

[MrTimscampi]

@MrTimscampi speaking of which, maybe there's a way we could show nice loading indicators for subtitles, video, et cetera to improve the UX while people wait more than twenty seconds for videos to load on slow connections.

For video, usually we have the spinning circle going while playback is not yet able to be started.

For subs, the fix would really be to extract them during scan instead of when playing them for the first time. After that, we can block playback until the subs are loaded, pretty much.

[dkanada]

The spinning circle is somewhat useful, but for longer actions like video loads or the book player, which can take ages in comparison since it has to load the entire file, determinate progress indicators would probably be better.

[MrTimscampi]

The spinning circle is somewhat useful, but for longer actions like video loads or the book player, which can take ages in comparison since it has to load the entire file, determinate progress indicators would probably be better.

Yep, for video loads, I don't know how much we can do. But for the book player, I had proposed making a progress bar as a loading indicator a while ago.

[sshockwave]

From a dependency perspective, I'm a bit opposed to adding ASS.js anyway.

I was searching for some more popular alternatives but unfortunately, not much came up :( I wonder if that's too much work for too few use cases.

It looks to be very incomplete in its support for the syntax: https://github.com/weizhenye/ASS#todo and usage looks minimal:

True, but it seems sufficient for daily usage. Complex effects might never be properly implemented without the glyphs, but we could always have libass-wasm as a fallback. In other sense, libass-wasm doesn't render the subtitles correctly if we had not provided enough font resources. The font glyphs, in contrast, are what we are looking at for most of the time during the playback.

For subs, the fix would really be to extract them during scan instead of when playing them for the first time. After that, we can block playback until the subs are loaded, pretty much.

I'm gonna agree with this. Clicking buttons is too much work and the process should be automated.

[dkanada]

I actually like the method in the existing pull request because it resolves the issue without requiring the server to store any fonts or adding dependencies to the web client.

[sshockwave]

Sorry that I might have misunderstood https://github.com/jellyfin/jellyfin-web/pull/1636. Does the dialog choose a font file from a local path rather than a path on the server? I noticed that there is a new API to get the fallback font file.

[Route("/FallbackFont/Font", "GET", Summary = "Gets the fallback font file")]

[dkanada]

It does pull the font from the server.

[sshockwave]

I actually like the method in the existing pull request because it resolves the issue without requiring the server to store any fonts or adding dependencies to the web client.

Then shouldn't the server store the font files?

Also, it has a file size limit of 10MB. It's not working with font sets any larger than that, and of course, anything larger than that will be really slow on the first load. It's just not friendly on slow connections.

[frankyifei]

#1636 seems only support one custom font. Is there a way to support multiple custom fonts at same time? As JavascriptSubtitlesOctopus supports a list of fallback fonts, I currently hardcoded 3 woff2 fonts on server side each supports C, J and K. Each font file is about 2mb, so it is good for bandwidth. If I have to use a large font file that contains 3 languages, that would be a about 7-8mb.

[sshockwave]

@frankyifei I think it's quite feasible, but it takes some changes. Better wait for updates in the original PR.

[nyanmisaka]

#1636 seems only support one custom font. Is there a way to support multiple custom fonts at same time? As JavascriptSubtitlesOctopus supports a list of fallback fonts, I currently hardcoded 3 woff2 fonts on server side each supports C, J and K. Each font file is about 2mb, so it is good for bandwidth. If I have to use a large font file that contains 3 languages, that would be a about 7-8mb.

Multiple fonts is also supported now.

[sshockwave]

Closing in favor of https://github.com/jellyfin/jellyfin-web/pull/1636. Welcome to reopen this issue if better alternatives come up.