Handbrake: PGS Subtitles are not scaled correctly when copied to the new encode if crop is active

[dia3olik]

Problem description:

PGS Subtitles are not scaled correctly when copied to the new encode if crop is active.
For example if the source is 4:3 content but inside a 16:9 stream with black bars left and right and you want to crop to 4:3 without bars and copy the PGS subtitle, the subtitle is squeezed horizontally instead of being cropped like the source video.

HandBrake version (e.g., 1.3.0):

1.3.2

Operating system and version (e.g., Ubuntu 19.10, macOS 10.15 Catalina, Windows 10 1909):

Mojave 10.14.6

Error message text and/or screenshot:

no error messages

HandBrake Activity Log:

HandBrake Activity Log for Session: 2020-05-07T09:33:06Z
Handbrake Version: 1.3.2 (2020050300)
Vivre sa vie.mkv
Preset: H265 10bit ReRip MKV
[11:33:06] macgui: QueueCore scanning specifically for title: 1
[11:33:06] CPU: Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz
[11:33:06] - Intel microarchitecture Kaby Lake
[11:33:06] - logical processor count: 16
[11:33:06] hb_scan: path=, title_index=1
bdj.c:290: Unable to read path from /usr/libexec/java_home
bdj.c:746: BD-J check: Failed to load JVM library
bdj.c:290: Unable to read path from /usr/libexec/java_home
bdj.c:746: BD-J check: Failed to load JVM library
[11:33:08] scan: BD has 13 title(s)
[11:33:08] bd: scanning title 1
[11:33:08] bd: playlist 00001.MPLS
[11:33:08] bd: duration is 01:23:50 (5030066 ms)
[11:33:08] bd: video id=0x1011, stream type=H.264, format 1080p
[11:33:08] bd: aspect = 16:9
[11:33:08] bd: audio id=0x1100, lang=Francais (BD LPCM), 3cc=fra
[11:33:08] bd: audio id=0x1101, lang=English (AC3), 3cc=eng
[11:33:08] bd: subtitle id=0x1200, lang=English [PGS], 3cc=eng
[11:33:08] bd: chap 1, 577868 ms
[11:33:08] bd: chap 2, 211878 ms
[11:33:08] bd: chap 3, 471262 ms
[11:33:08] bd: chap 4, 172881 ms
[11:33:08] bd: chap 5, 287370 ms
[11:33:08] bd: chap 6, 518976 ms
[11:33:08] bd: chap 7, 532782 ms
[11:33:08] bd: chap 8, 241699 ms
[11:33:08] bd: chap 9, 442233 ms
[11:33:08] bd: chap 10, 325116 ms
[11:33:08] bd: chap 11, 660368 ms
[11:33:08] bd: chap 12, 587378 ms
[11:33:08] bd: chap 13, 250 ms
[11:33:08] bd: title 1 has 13 chapters
[11:33:08] scan: decoding previews for title 1
[11:33:08] scan: title angle(s) 1
[11:33:08] scan: audio 0x1100: pcm_bluray, rate=48000Hz, bitrate=2304000 Francais (BD LPCM) (1.0 ch) (2304 kbps)
[11:33:08] scan: audio 0x1101: ac3, rate=48000Hz, bitrate=192000 English (AC3) (1.0 ch) (192 kbps)
[11:33:15] scan: 10 previews, 1920x1080, 23.976 fps, autocrop = 0/0/238/238, aspect 16:9, PAR 1:1
[11:33:15] stream: 9 good frames, 0 errors (0%)
[11:33:15] libhb: scan thread found 1 valid title(s)
[11:33:15] macgui: QueueCore scan done
[11:33:15] macgui: QueueCore started encoding Vivre sa vie.mkv
[11:33:15] macgui: QueueCore with preset H265 10bit ReRip MKV
[11:33:15] Starting work at: Thu May 7 11:33:15 2020

[11:33:15] 1 job(s) to process
[11:33:15] Starting Task: Encoding Pass
[11:33:15] work: only 1 chapter, disabling chapter markers
[11:33:15] job configuration:
[11:33:15] * source
[11:33:15] + /xxxx
[11:33:15] + title 1, chapter(s) 2 to 2
[11:33:15] * destination
[11:33:15] + /Vivre sa vie.mkv
[11:33:15] + container: Matroska (libavformat)
[11:33:15] * video track
[11:33:15] + decoder: h264
[11:33:15] + bitrate 200 kbps
[11:33:15] + filters
[11:33:15] + Comb Detect (mode=3:spatial-metric=2:motion-thresh=1:spatial-thresh=1:filter-mode=2:block-thresh=40:block-width=16:block-height=16)
[11:33:15] + Framerate Shaper (mode=1:rate=27000000/1126125)
[11:33:15] + frame rate: 23.976 fps -> constant 23.976 fps
[11:33:15] + Crop and Scale (width=1440:height=1080:crop-top=0:crop-bottom=0:crop-left=240:crop-right=240)
[11:33:15] + source: 1920 * 1080, crop (0/0/240/240): 1440 * 1080, scale: 1440 * 1080
[11:33:15] + Output geometry
[11:33:15] + storage dimensions: 1440 x 1080
[11:33:15] + pixel aspect ratio: 1 : 1
[11:33:15] + display dimensions: 1440 x 1080
[11:33:15] + encoder: H.265 (libavcodec)
[11:33:15] + preset: default
[11:33:15] + profile: auto
[11:33:15] + bitrate: 9000 kbps, pass: 0
[11:33:15] + color profile: 1-1-1
[11:33:15] * subtitle track 1, English [PGS] (track 0, id 0x1200, Picture) -> Passthrough
[11:33:15] * audio track 1
[11:33:15] + decoder: Francais (BD LPCM) (1.0 ch) (2304 kbps) (track 1, id 0x1100)
[11:33:15] + bitrate: 2304 kbps, samplerate: 48000 Hz
[11:33:15] + mixdown: Mono
[11:33:15] + dither: triangular
[11:33:15] + encoder: AAC (Apple AudioToolbox)
[11:33:15] + bitrate: 80 kbps, samplerate: 48000 Hz
bdj.c:290: Unable to read path from /usr/libexec/java_home
bdj.c:746: BD-J check: Failed to load JVM library
bdj.c:290: Unable to read path from /usr/libexec/java_home
bdj.c:746: BD-J check: Failed to load JVM library
[11:33:15] sync: expecting 5080 video frames
[11:33:15] encavcodecInit: H.265 (VideoToolbox)
[11:33:15] encavcodec: encoding with stored aspect 1/1
[11:33:16] sync: first pts audio 0x1100 is 94
[11:33:16] sync: first pts subtitle 0x1200 is 3754
[11:33:16] sync: first pts video is 0
[11:33:16] sync: "Chapter 2" (2) at frame 1 time 0
[hevc_videotoolbox @ 0x7ffb8b007400] Color range not set for yuv420p. Using MPEG range.
[11:34:14] reader: end of chapter 2 (media 2) reached at media chapter 3
[11:34:14] reader: done. 2 scr changes
[11:34:15] work: average encoding speed for job is 86.746567 fps
[11:34:16] comb detect: heavy 3 | light 158 | uncombed 4919 | total 5080
[11:34:16] vfr: 5080 frames output, 0 dropped and 0 duped for CFR/PFR
[11:34:16] vfr: lost time: 0 (0 frames)
[11:34:16] vfr: gained time: 0 (0 frames) (0 not accounted for)
[11:34:16] stream: 5081 good frames, 0 errors (0%)
[11:34:16] pcm_bluray-decoder done: 42376 frames, 0 decoder errors
[11:34:16] h264-decoder done: 5080 frames, 0 decoder errors
[11:34:16] sync: got 5080 frames, 5080 expected
[11:34:16] sync: framerate min 23.976 fps, max 23.976 fps, avg 23.976 fps
[11:34:16] mux: track 0, 5080 frames, 234694605 bytes, 8859.74 kbps, fifo 512
[11:34:16] mux: track 1, 9933 frames, 2169753 bytes, 81.91 kbps, fifo 1024
[11:34:16] mux: track 2, 56 frames, 720062 bytes, 27.18 kbps, fifo 16
[11:34:16] Finished work at: Thu May 7 11:34:16 2020

[11:34:16] libhb: work result = 0

[woodstockathbf]

[11:33:15] * subtitle track 1, English [PGS] (track 0, id 0x1200, Picture) -> Passthrough

OK, so, did you submit a bug report to your playback software vendor? It would appear that they aren't handling scaling properly.

If you burn in the subtitle, handbrake would do the scaling. But pass-through is left up to the player.

[dia3olik]

No no, it's on ALL the players tested, it has to do with how HandBrake deal with PGS when cropping the video.
Try it yourself...
Again, HandBrake should "crop" the PGS subtitles if you crop the video.
PGS ara basically images so if the user apply a 4:3 crop from 1920x1080 to 1440x1080 then HandBrake should crop the sides of the subs in the same way as the video track not just squeeze the PGS inside 1440x1080 as it's happening now...

Click here for some screenshots

[jstebbins]

This is a typical problem with players and there is nothing we can do about it (short of rewriting the PGS stream instead of passing it through).

The PGS subtitles have a "video window" definition that defines the original video dimensions. The player is independently scaling the x axis to fit the new video width and the y axis to fit the new video height and this is affecting the pixel aspect of the subtitles. PGS subtitles (per the PGS spec) by definition have the same pixel aspect as the source video. The player should be scaling the subtitles in such a way as to also maintain the original pixel aspect.

The real problem here is that there is nothing to tell the player explicitly what the pixel aspect should be. It's not defined in the stream itself, it's just supposed to be assumed to be the same as the video (per PGS spec).

[jstebbins]

FYI, I'm in the process of writing a PGS encoder. So at some point, when I get through the rather lengthy ffmpeg review process, we will have the option to decode and re-encode PGS subtitles. This shouldn't be required if you are not changing the pixel aspect of the video, but as you've seen players just aren't very smart. It is required if you are changing the pixel aspect of the video.

So when I'm done with this, we'll probably just re-encode PGS when "passthru" is selected any time the dimensions are aspect are changed.

[dia3olik]

Thanks John for the clear explanation and that's great!
I think it will be very useful and it's totally the right move since if all the players are bad something has to be done, also probably it's cleaner to have a new PGS sub raster size matching the video track he he

[woodstockathbf]

A problem with modifying the PGS stream is that a number of titles out there use the top/bottom letterbox areas for subtitles, so matching the crop by itself could clip the subtitles. Applying scaling alone could cause readability issues, and may place the result higher on the image, unless the vertical crop is zero.

[dia3olik]

Good point! In fact probably the right approach would be to have an x/y offset field to overcome these issues if some titles needs it, no?

[Dendraspis]

The right approach should be converting those subtitles to plain text (PGS » SRT) as long as there are no pictures involved, jumping subtitles that fit the movement of artists or something like that.

Cropping bottom/top offers less problems, because the player only needs to use a vertical offset.

[dia3olik]

yeah...if and when OCR will really improve (maybe aided by neural networks?)...until then if you have a ton of blurays you want to rip to enjoy on the move it's not really an option...I never found a title which could be converted cleanly without major errors so PGS copy has it's place and MKV was invented to support these as track too so I think it would be great if HB will be able to handle this well.

[woodstockathbf]

I have not seen a good implementation of SRT that rivals the graphic subtitle types. SSA is much better, but that takes a lot of work to do well.

As for adding an X/Y offset scheme, the players aren't properly implementing the existing PGS standard, and you expect them to adopt something else to help fix that oversight? Shirley, you jest!

[dia3olik]

he he I meant x/y offset settings in the PGS conversion that John was talking about, inside HandBrake ;-)

[Dendraspis]

Seems to depend on the player and/or the selected font and/or its size - I have no problems with SRT subtitles.

OCR is almost no problem. Sometimes a little bit messy, but overall it works pretty good.

@dia3olik
I think what you would need is a x/y expansion/compression factor - the offset is made by the player itself.

[dia3olik]

SRT is not PGS so not relevant at all to this thread... ;-)
OCR is messy, tried many times...it works relatively okayish only on boring movies with simple dialogues with common words and with no italics and no symbols and names etc
yes and no, so do you think the best way is having HandBrake create a 1400x1080 mkv movie with a wrong 1920x1080 PGS subtitle inside like it does now and then asking to EACH player on the planet to implement a scaling/cropping on PGS tracks????
Or maybe it would be better to have HandBrake create a 1440x1080 movie with a matched 1440x1080 PGS subtitle which then will work on EVERY player around as it should??

[jstebbins]

PGS subtitles have:

• A "video descriptor" that gives the dimensions of the video
• A "window segment" with x, y position that says where in the video frame the objects for a subtitle are to be rendered
• A "presentation segment" with a collection of "object descriptors" each of which also have an x, y position relative to the window.

I'm thinking for best results, the subtitles should not be touched if the video is not clipped and the aspect ratio is not changed. I.e. there is no point is scaling down the subtitles if, during playback, they are scaled back up to a larger screen.

Ideally, what needs to happen...

If the video is cropped:

• Modify window positions so windows are not cropped
• Rewrite the "video descriptor" with the cropped dimensions before scaling

If the pixel aspect changes:

• Scale "video descriptor" width, height to new aspect
• Scale window x,y, width, height to the new aspect
• Scale object x,y, width, height to new aspect
• Scale bitmaps to new aspect

When scaling for aspect, one of the dimensions will remain unchanged (i.e. one dimension will scale down, the other will remain unchanged)

[jstebbins]

I was just re-reading the above. I'm having second thoughts about doing anything for the crop case.

A smart player would recognize the the subtitle frame and video frame do not match and know that there may be subtitles "out of frame" of the encoded video (i.e. in the black bars). A smart player would be able to properly position the subtitles in the black bar that the player creates when playing the cropped video full screen. And when not in full screen mode would know that it may need to reposition some windows.

I need to retest. But I could have sworn that Kodi was a smart player

[dia3olik]

John, of course do as you think is more appropriate, I don't have all your technical knowledge, but I do know that now it's not working properly and I think that copying the PGS track as is without modifying its canvas reference size and hoping the player is "smart"enough and able to correct this error and mismatch is not the cleanest way, the mismatch can and should not be in the encode at all so the PGS processing should be handled by HandBrake which create the file.

[jstebbins]

"Clean" is relative. Subtitles that are positioned in the black bar should be rendered in the black bar. That's clean and correct. The information to do this correctly is available to the player.

I'm ok with working around player bugs when it doesn't affect properly functioning players. But in this case, making such a change would break a properly functioning player.

[dia3olik]

Many films are 1.87:1 or 1.78:1 or 1.66:1 or 1.33:1 so no top and bottom black bars...also distortion manifest itself basically all the times on the horizontal axis only, the problem I'm talking about is squeeze/distortion which happens when you crop, not vertical placement (which of course I agree with you it should be 100% handled by the player and not HandBrake)

[Nomis101]

FYI, I'm in the process of writing a PGS encoder. So at some point, when I get through the rather lengthy ffmpeg review process, we will have the option to decode and re-encode PGS subtitles.

I have the feeling they sometimes tend to forget about patches (I often see people pinging for review again and again).

[jstebbins]

Yes, we all get busy. I've been preoccupied with home automation software lately :grin: so haven't pinged them in a while. I'll probably give it a ping Monday.

[Nomis101]

I hope they consider your PGS encoder patches for 4.3, which will be released in a few days.
http://ffmpeg.org/pipermail/ffmpeg-devel/2020-June/264179.html

[bradleysepos]

I mentioned it on #ffmpeg-devel and nobody showed interest, sadly. 🤷‍♂️

[10:54:53] BradleyS: has anyone had a chance to look at the pgs bsfs patch
[10:55:14] BradleyS: http://ffmpeg.org/pipermail/ffmpeg-devel/2020-May/262318.html
[10:57:47] BradleyS: really, anything john (j45) has submitted is probably beneficial to handbrake v.Next and we would appreciate review for inclusion into 4.3 if possible
[10:57:50] BradleyS: https://patchwork.ffmpeg.org/project/ffmpeg/list/?submitter=433
[10:58:22] BradleyS: you can always see our local patches against ffmpeg at https://github.com/HandBrake/HandBrake/tree/master/contrib/ffmpeg
[10:58:42] BradleyS: (if we had none, we wouldn't have to ship static ffmpeg at all...)
[11:00:05] BradleyS: to be fair, those are against 4.2.x

[Nomis101]

What a pity. Is #ffmpeg-devel still in use? The archive somehow seems to stop in February 2020.
https://lists.ffmpeg.org/pipermail/ffmpeg-devel-irc/

[bradleysepos]

Yes, quite active actually.