Exoplayer: H.264 codec with SurfaceView vs TextureView on certain KitKat devices

[trandroid]

On certain KitKat devices I am seeing weird video artifacts when using ExoPlayer to render H.264 with a TextureView.

I have a previous implementation of my rendering pipeline that used MediaCodec directly and it does not exhibit this issue. Also using ExoPlayer with SurfaceView does not exhibit this issue.
Also, I've only observed this on certain KitKat devices (see table below).

Screenshot of ExoPlayer with TextureView showing issue:

Screenshot of expected good video:

Here's a test matrix showing when issue occurs. FAIL means the video artifacts are seen.

| Device | Android Ver. | ExoPlayer or MediaCodec | View Type | PASS/FAIL |
| --- | --- | --- | --- | --- |
| Samsung S4 | 4.4.2 | ExoPlayer | TextureView | FAIL |
| Samsung S3 | 4.4.2 | ExoPlayer | TextureView | FAIL |
| Samsung S4 | 4.4.2 | ExoPlayer | SurfaceView | Pass |
| Samsung S3 | 4.4.2 | ExoPlayer | SurfaceView | Pass |
| Samsung S4 | 4.4.2 | MediaCodec | TextureView | Pass |
| Samsung S3 | 4.4.2 | MediaCodec | TextureView | Pass |
| Nexus 5 | 4.4.4 | ExoPlayer | SurfaceView | Pass |
| Nexus 5 | 4.4.4 | ExoPlayer | TextureView | Pass |
| Nexus 5 | 4.4.4 | MediaCodec | SurfaceView | Pass |
| Nexus 5 | 4.4.4 | MediaCodec | TextureView | Pass |

[ojw28]

Yes, we've seen this too; it appears to be a problem with certain devices. We're following up with OEMs to try and get this addressed.

[ptran-gpfw]

Followup question:
In general we'd like to stick with TextureView. In our testing we measure up to 3 frames of additional latency when using SurfaceView.
(Though in this article on SurfaceView vs TextureView, the author notes more latency with SurfaceView; but I guess that's a discussion for another time).

So do you know of a way to detect that this issue is happening?

I'd like to use TextureView for the common case and then fallback to using SurfaceView only when necessary.

[ojw28]

@trandroid It would be great if you could figure out what the difference is between your use of TextureView and ExoPlayer's? I don't think ExoPlayer does anything wrong, but it would be good to narrow down what we're doing that triggers the underlying issue. Are you using DASH/SmoothStreaming, or are you using FrameworkSampleSource?

I don't know of an easy way to detect when this is happening, unfortunately.

[trandroid]

My video source sends TS packed in UDP datagrams with a proprietary header so I have my own SampleSource implementation that demuxes TS, parses the timestamp from PES and parses width/height from H.264 ES.

I just noticed that in the version of my player that directly uses MediaCodec I only set KEY_MIME_TYPE, KEY_WIDTH and KEY_HEIGHT. I never set KEY_MAX_INPUT_SIZE.

I'll try commenting out in ExoPlayer the setting of the other config fields like KEY_MAX_INPUT_SIZE. I'll keep you posted.

[ojw28]

Thanks for the information. KEY_MAX_WIDTH and KEY_MAX_HEIGHT are also prime candidates to try removing, if you're setting them (most likely by calling maybeSetMaxDimensionsV16). Just to see what happens.

[ojw28]

I'm a bit confused by the patch. Doesn't maybeSetIntegerV16 perform the exact same checks that you're adding? Am I being stupid :)?

[trandroid]

Ack... no I'm the stupid one, heh.
You're right, the check is already being done.
My code was actually making a call to exoplayer.MediaFormat.setMaxVideoDimensions.
Commenting out that call in my code actually resolved it for me.

Do you have any guidelines for settings max width & height? Should it be max dimensions of the input or max dimensions of the Surface?

[ojw28]

If you're dealing with fixed resolution content then you don't need to set them at all, and the problem is solved for your use case. If you're doing ABR that can switch resolution during playback, then you should specify the maximum video (not surface) dimension that you'll be using during playback.

[ojw28]

The root cause was an issue with some Qualcomm chipsets (e.g. APQ8064, MSM8960, MSM8930). The issue was subsequently fixed, but devices will obviously need to be updated if they're to pick up the fix. Closing because there's nothing that can be done to work around the problem (aside from not setting the max dimensions or not using TextureView).

https://www.codeaurora.org/cgit/quic/la/platform/hardware/qcom/display/commit/libgralloc/mapper.cpp?id=43da51a308c8176723b61a2c1ffcfff485ab53a2
https://www.codeaurora.org/cgit/quic/la/platform/hardware/qcom/display/commit/libgralloc/gralloc_priv.h?id=3d01d8d75e80f5bee0d9c37cc7b8fc1a6d9b7884

[nschwermann]

I'm trying to better understand this issue. It seems I am going to have a trade off either way. I currently need a TextureView so the surface can be recycled when switching to and from fullscreen views when the Surface is briefly detached from the window to attach it to the appropriate container.

This operation interrupts playback on a SurfaceView but not a TextureView. I don't think I'm explicitly setting the max dimensions of the TextureView anywhere but I'm still seeing this problem. I assume Exoplayer must be setting the max dimensions implicitly somewhere.

Lastly, is there a known way to detect the troubled chipsets so we can favor one implementation over the other in certain cases?

[crossle]

Nexus 7 2013 use APQ8064 chipsets, I used TextureView, it's sometimes weird...
Dont set max dimensions can resolve it.
BTW, Android 5.0

[edwardfoux]

Is there any way for ExoPlayer to detect the black/white issue, so could be handled in the code?
Thanks in advance!

[nschwermann]

There doesn't seem to be a good way to handle this. Right now I just detect known chipsets that have the issue.

private boolean shouldUseWorkaround(){
    final String board = Build.BOARD;
    if("APQ8064".equals(board) || "MSM8960".equals(board) || "MSM8930".equals(board)){
        Log.w(TAG, "Defective board detected only using one stream.");
        return true;
    } else {
        return false;
    }
}

This sort of works but isn't really ideal as these boards could actually have the patch and work correctly, also there are probably others that need to be added to the list.

[edwardfoux]

@schwiz Thank you!
@ojw28 on October 27th you mentioned couple boards as the defected ones. Would it be possible to list more affected boards? Thank you!

[ojw28]

I don't have a more exhaustive list, I'm afraid. It might be that those three are the only ones affected; I'm not sure. If anyone else knows of any additional affected boards, I'd encourage them to post details here!

[jasells]

@trandroid You mentioned you are handling TS over UDP data source (H.264). i am trying the same using MediaCodec, and having issues. I was wondering if you had any tips? I can parse out the H.264 ES, but it only shows a grey screen with a few pixelated artifacts using my own extractor. Any help or tips greatly appreciated!

[sungerk]

i used textureView instead of surfaceview in the demo module ,but it's not play .and without error log.

@trandroid Would you like to share me the demo of textureView play online video,my email is [email protected]

[trandroid]

@sungerk Do you have proper set up for TextureView.SurfaceTextureListener?
I don't have my source readily available but I essentially do that same as this call to textureView.setSurfaceTextureListener.

[trandroid]

@jasells My extractor just makes sure that all the data for a single frame is produced as a single sample with its own unique timestamp. I.e., copy exactly one video frame to sampleHolder when ExoPlayer calls SampleSourceReader.readData.

[sungerk]

@trandroid i had run the project,but it's not play video yet, i add a issue in the project

[jasells]

@trandroid Thanks for the reply. I figured out my issue. It was a combination of :

1) clearing MediaCodec's input buffers before writing to them
2) properly extracting the data from all the Mpeg TS headers.

[jasells]

@schwiz

I know you are using the standard Java Android API, but, just fyi, I am able to resize using a surface view no problem using Xamarin forms as my UI. The back end is a standard Android surface view, but the way that Xamarin binds to the platform view (surfaceView) means that it doesn't detach from the screen when resizing. I don't know if that might give you some insight. It is just a little demo app, so it is not polished, but it does work.

My project DroidVid using Xamarin

Summary, it has a video view and a button. Clicking the button will swap the size/possiton of the button and the video view. The video view could be made full screen by getting the parent page size and resizing to those values.