Exoplayer: nVidia Shield: HDR Mode not triggered for cropped content

[ebr11]

Issue description

Videos that are HDR but have been cropped to non-full resolution (e.g. 2.35 aspect) are not triggering the HDR mode on the TV when played. Items that are not cropped do turn on HDR properly.

Reproduction steps

Play the following video on an HDR capable set.

Link to test content

https://drive.google.com/file/d/1uQGkgw8fkkrMFyKQ4fl9aOvi6RuhLQHX/view

Version of ExoPlayer being used

2.9.1

Device(s) and version(s) of Android being used

Tested on NVidia Shield running Android 8.0

A full bug report captured from the device

Unable to capture report at this time.

[ojw28]

@andrewlewis - I think we'll need to follow up with the platform team about this. I'm not sure how it's supposed to work?

[andrewlewis]

@ebr11 Could you give a bit more detail on how you are cropping the video?

[andrewlewis]

Talked to @ojw28 and now I think I understand the question so please ignore the comment above.

[andrewlewis]

@sureshcnvidia: are you aware of any issues combining HDR surfaces with non-HDR content on Shield?

[ebr11]

Hi. Any more info here?

I think this is likely something relatively simple as I believe there is at least one Exo fork that fixes whatever the problem is (not public though).

Thanks.

[ojw28]

Have you tried calling setColorMode explicitly on the window? I've no idea whether it makes a difference, but it might be worth a try.

[ebr11]

Unfortunately, that does not appear to work. Have you discovered why this isn't working normally?

[ojw28]

No. I think it's a complicated issue though. I think we need nVidia to chime in.

@sureshcnvidia - Can you comment?

[ebr11]

Hi. Can you point us in any direction here so that maybe we can investigate further? I have determined that the issue is definitely related to the size of the video rendering surface (ApectRatioFrameLayout). If we do not allow that surface to re-size to less than the size of the actual screen, HDR is triggered properly. As soon as we allow that surface to re-size though, the problem occurs.

Thanks.

[cayars]

Have you heard anything from nVidia on this?
With more and more content available in HDR this is becoming a bigger problem all the time.

[ojw28]

Not yet [Internal ref: 124665441]. I'd suggest you also report the problem through nVidia's developer forum.

[sureshcnvidia]

Sorry for really coming back so late on this. I need to check why this does not work.

Shield should be enabling HDR is any of the layers are HDR in the overall 'view stack'. Will check the video internally and get back on this.

[sureshcnvidia]

I checked the clip shared above. For me playback with the default "Photos and Video's" app on Shield does play HDR everytime.

Do, I need to simply play the clip using latest Exo to see the exact problem?

[ebr11]

Yes. Use their demo app.

Thanks.

[CBers]

Any updates please?

[ojw28]

@sureshcnvidia - Were you able to give this a try?

[FrostByteLXV]

@sureshcnvidia is nVidia actively looking into this issue? None of my HDR content can be triggered by the Shield until this is resolved.

[HRSCR]

A patch is there, ask to the moderator for it

[CBers]

@hrscr What are you talking about?

[FrostByteLXV]

@ebr11 are you aware of a patch?

[HRSCR]

@CBers there is a patch, ask to the moderator on _Nvidia forums_ about that patch. There's not a update with that patch yet, but the fix is there (Nvidia Forums). Ask for the patch there.

[HRSCR]

I'm using that Nvidia patch around a month ago.

[CBers]

@HRSCR See here: https://emby.media/community/index.php?/topic/64038-emby-for-android-shield-tv-not-playing-back-hdr-on-all-titles/page-3#entry734540

Please comment so that @ebr11 is aware.

[HRSCR]

@ebr11 ask if anybody with the patch had issues, i answer: "no problems" was fixed with the patch.
I have no issues with HDR content in any way after the patch.

[CBers]

When/where did he ask that?

[HRSCR]

The moderator sniper
https://forums.geforce.com/member/2075904/

[FrostByteLXV]

He says he's never heard of the problem and there is no fix. :(

[HRSCR]

I have no issues, nothing of my part to add on this, sorry.

[FrostByteLXV]

Including cropped HDR files? Anything fullscreen works fine here also. It's just cropped content which doesn't trigger HDR

[gthrift]

@sureshcnvidia this is still an issue for many users.

[FrostByteLXV]

This is still an issue for all users who use cropped files. HRSCR was confused with another issue. Hoping nVidia responds soon.

[HRSCR]

Some versions out there have a lot of errors in the encoded script that they use. They are trying to do something in the wrong way, that's why is not working.
Check this:
cpuid=1173503 / frame-threads=4 / numa-pools=16 / wpp / no-pmode / no-pme / no-psnr / no-ssim / log-level=2 / input-csp=1 / input-res=3840x1600 / interlace=0 / total-frames=217718 / level-idc=51 / high-tier=1 / uhd-bd=0 / ref=4 / no-allow-non-conformance / repeat-headers / annexb / aud / hrd / info / hash=0 / no-temporal-layers / no-open-gop / min-keyint=1 / keyint=24 / bframes=4 / b-adapt=2 / b-pyramid / bframe-bias=0 / rc-lookahead=25 / lookahead-slices=4 / scenecut=40 / no-intra-refresh / ctu=64 / min-cu-size=8 / rect / no-amp / max-tu-size=32 / tu-inter-depth=1 / tu-intra-depth=1 / limit-tu=0 / rdoq-level=2 / dynamic-rd=0.00 / no-ssim-rd / signhide / no-tskip / nr-intra=0 / nr-inter=0 / no-constrained-intra / no-strong-intra-smoothing / max-merge=3 / limit-refs=3 / limit-modes / me=3 / subme=3 / merange=57 / temporal-mvp / weightp / no-weightb / no-analyze-src-pics / deblock=-3:-3 / no-sao / no-sao-non-deblock / rd=4 / no-early-skip / rskip / no-fast-intra / no-tskip-fast / no-cu-lossless / no-b-intra / no-splitrd-skip / rdpenalty=0 / psy-rd=2.00 / psy-rdoq=1.00 / no-rd-refine / analysis-reuse-mode=0 / no-lossless / cbqpoffs=0 / crqpoffs=0 / rc=crf / crf=17.0 / qcomp=0.60 / qpstep=4 / stats-write=0 / stats-read=0 / vbv-maxrate=160000 / vbv-bufsize=160000 / vbv-init=0.9 / crf-max=0.0 / crf-min=0.0 / ipratio=1.40 / pbratio=1.30 / aq-mode=1 / aq-strength=1.00 / cutree / zone-count=0 / no-strict-cbr / qg-size=32 / no-rc-grain / qpmax=69 / qpmin=0 / no-const-vbv / sar=1 / overscan=0 / videoformat=5 / range=0 / colorprim=9 / transfer=16 / colormatrix=9 / chromaloc=1 / chromaloc-top=2 / chromaloc-bottom=2 / display-window=0 / master-display=G(13250,34500)B(7500,3000)R(34000,16000)WP(15635,16450)L(10000000,0) / max-cll=0,0 / min-luma=0 / max-luma=1023 / log2-max-poc-lsb=8 / vui-timing-info / vui-hrd-info / slices=1 / no-opt-qp-pps / no-opt-ref-list-length-pps / no-multi-pass-opt-rps / scenecut-bias=0.05 / no-opt-cu-delta-qp / no-aq-motion / hdr / hdr-opt / no-dhdr10-opt / analysis-reuse-level=5 / scale-factor=0 / refine-intra=0 / refine-inter=0 / refine-mv=0 / no-limit-sao / ctu-info=0 / no-lowpass-dct / refine-mv-type=0 / copy-pic=1

[FrostByteLXV]

HRSRC, please stay out of this thread.

We already know what the issue is. Ebr stated what the issue was very clearly and provided an example. This is an aspect ratio issue and an aspect ratio issue only. If you crop the black bars at the top and bottom of the video then exoplayer no longer triggers HDR when the aspect ratio is no longer 16:9.

Now please stay only, you have muddied this issue too much already.

[ojw28]

The latest status of this issue is that we're still waiting for an update from nVidia. I have pinged our internal bug to try and get an update.

[sureshcnvidia]

We have checked this issue with Exoplayer demo and is fixed in our next upcoming public release for Shield.

[FrostByteLXV]

Great, thank you sir

[CBers]

Closed?

It's not fixed until the release has been made GA and people can confirm it's fixed.

[ojw28]

It's not an ExoPlayer issue, so this isn't the right place to be tracking it.

[CBers]

OK, thanks.

[ebr11]

@sureshcnvidia It appears the Shield Experience 8.0 does NOT solve this issue...

[sureshcnvidia]

Well HDR mode comes everytime with latest (or even Exo v2.9.3) with below command:
adb shell am start -a com.google.android.exoplayer.demo.action.VIEW -d file:// and it works all the time.

Am unsure what is the issue still. If possible please share more details and we would look into it.

[ebr11]

Sorry I forgot to come back and update this. The problem is that none of the surfaces behind the video can have a background color. If it does, HDR will not trigger. Once we removed all backgrounds, the fix works.

Thanks.

[ebr11]

Uh, oh. Has this re-emerged?

[ojw28]

I suspect that what you came up with still works. nVidia asked us to unlock the issue so they can comment on it, so expect an update here soon.

[ebr11]

Excellent. Thanks.

[kulve]

Hi,

The earlier comment ("_The problem is that none of the surfaces behind the video can have a background color_") gives quite a good hint where to look for.

Would you mind again providing the repro steps? Which version of the ExoPlayer to use, how to install it and how to run with and without the background color set? Note that I'm not familiar with the ExoPlayer but I am familiar on how the HDR gets triggered on the display side.

Thanks.

[ebr11]

Hi. I haven't tested this specifically, but I think if you were to just add:

android:background="#00000000"

To the main FrameLayout here the problem should manifest.

[kulve]

Any chances I could get ready made apks with and without that change?

[kulve]

I'm not familiar on how to build it so I was hoping that if you are, maybe you could easily provide those for me to test with.

[ebr11]

I cannot easily build it into an APK for you because you need some content to reproduce it with and that content is defined in the app (you modify a json file).

It is fairly straightforward to make that edit and build the app with Android Studio and send it straight to a device connected via ADB.

[kulve]

I'm not that familiar with Android Studio either but it was indeed fairly straightforward. I'm able to run
the default "demo-ima" config on my Shield by just clicking the run button.

Suresh mentioned above that he used the following command to play any content:
adb shell am start -a com.google.android.exoplayer.demo.action.VIEW -d file://

Is that a valid repro command for this bug? If I need to modify the json file, which file is it?

[andrewlewis]

Rather than "demo-ima", could you try "demo" which is the main demo app?

There are instructions for playing content here. It's probably easiest to send an intent rather than modifying a JSON file. Thanks.

[kulve]

Thanks for the pointers.

I'm now running the "demo" app of ExoPlayer 2.10.4 on Shield 8.0.2 and I'm testing with the cropped 3840x1600 The Martian clip from the first post.

I select run in Android Studio and once the UI is up, I launch the playback with:

adb shell am start -a com.google.android.exoplayer.demo.action.VIEW -d file:///sdcard/Movies/The_Martian.mkv
I do get HDR triggered correctly on my tests but I don't think the background color layer is effective.

HWComposer sees the following layers (video + UI):

Display 0 HWC layers:
-------------------------------------------------------------------------------
 Layer name
           Z |  Comp Type |   Disp Frame (LTRB) |          Source Crop (LTRB)
-------------------------------------------------------------------------------
 SurfaceView - com.google.android.exo[...]oid.exoplayer2.demo.PlayerActivity#0
  rel     -2 |     Device |    0  280 3840 1878 |    0.0    0.0 3840.0 1600.0
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
 com.google.android.exoplayer2.demo/c[...]oid.exoplayer2.demo.PlayerActivity#0
  rel      0 |     Device |    0    0 3840 2160 |    0.0    0.0 1920.0 1080.0
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

I also see ColorLayer/Background in the dumpsys SurfaceFlinger but I'm not exactly sure how to interpret that. It doesn't seem to get down to the HWC.

+ ColorLayer (Background for -SurfaceView - com.google.android.exoplayer2.demo/com.google.android.exoplayer2.demo.PlayerActivity#0)
  Region TransparentRegion (this=2ad2864c08 count=1)
    [  0,   0,   0,   0]
  Region VisibleRegion (this=2ad2864a10 count=1)
    [  0,   0,   0,   0]
  Region SurfaceDamageRegion (this=2ad2864a88 count=1)
    [  0,   0,   0,   0]
      layerStack=   0, z=       -3, pos=(0,140), size=(1920, 799), crop=[  0,   0,  -1,  -1], finalCrop=[  0,   0,  -1,  -1], isOpaque=0, invalidate=0, dataspace=Default, defaultPixelFormat=Unknown/None, color=(0.000,0.000,0.000,1.000), flags=0x00000002, tr=[1.00, 0.00][0.00, 1.00]
      parent=com.google.android.exoplayer2.demo/com.google.android.exoplayer2.demo.PlayerActivity#0
      zOrderRelativeOf=none
      activeBuffer=[   0x   0:   0,Unknown/None], queued-frames=0, mRefreshPending=0, windowType=-1, appId=-1

Am I missing something in the repro steps?

[kulve]

Any hints on how to get this reproed?

[ebr11]

You gave the window a background color?

[kulve]

It seems to be 88000000 by default, i.e. set already?

[kulve]

Actually, I checked the LinearLayout but I guess the point was to add it directly to the FrameLayout? I'll test.

[kulve]

I added android:background="#88ff0000" directly to FrameLayout and I now I see a reddish semitransparent background. But there's still no explicit background layer from SurfaceFlinger going to HWComposer and I do get HDR enabled correctly on my TV.

[ebr11]

Okay. I am not sure then. We had a similar frame layout with a background color (black) and the problem still occurred for us. Once we removed the background, it worked.

This was several versions of Exo ago though...

[kulve]

Should we close the issue then, if there's no repro case on the new code base?

[ebr11]

Yes, I think that is safe but will leave to Google.