Exoplayer: Investigate using MediaCodec in async mode

[ghost]

The preferred solution in https://developer.android.com/reference/android/media/MediaCodec.html is asynchronous processing buffer. But ExoPlayer is still using the synchronous solution. Why not
use asynchronous processing buffer solution? Do you google have some comparison or benchmark reports between these two solution?

Thanks

[botaydotcom]

Thanks for the suggestion.

This is something that has been on our list for some time, and we are looking into now. Will update this issue once we have more results available.

[ghost]

Hi @botaydotcom, How can we sync audio stream and video stream in async mode?

[botaydotcom]

Hi @ycinfinity, I haven't made any progress on this task because of some other urgent tasks.
The plan is to explore this mode, and compare it with sync mode, before introducing any potential changes. It should take some time, and I expect to have more to share about this on December.

[botaydotcom]

Just to give some updates on this issue:

• I have tested async MediaCodec vs sync MediaCodec API. Normally, they have roughly similar CPU usage overall (I think async MediaCodec have slightly better stats, but not too different).
• In certain cases, async MediaCodec shows much better results. There was a test with which we have a Pixel 1 device, with CPU locked to a very low frequency (~ 0.5 GHz) for all 4 cores, and let it play a Widevine DRM H264 stream. Synchronous MediaCodec mode usage results in a lot of dropped frames, while the other mode drops very few frames. Still haven't been able to reproduce this on other devices, so the reason might be device specific? However, I think it shows that async mode has potential to perform better for certain cases.
• However, this mode (async API) has certain drawbacks as well. First, it requires a higher API level (21+). Then, in several cases, for older devices, the playback crashes after ~20-30s of playback, while it plays smoothly in synchronous MediaCodec mode (it might be because of my implementation bugs, however). It doesn't tell the whole picture, but I believe the other mode (sync API) is more stable on older devices (21-22), and it's tested more thoroughly (ExoPlayer has been running for a long time using this mode). We may be able to release the version using async MediaCodec API in some forms, but it may take time before we can find out most of the platform-related / device-specific bugs with this mode to make it as stable as the other mode.

@ycinfinity: For your specific question about syncing audio stream/video stream: You may use different way, but I base my implementation on the existing MediaCodecRenderer. The async callbacks are used to dequeue and get the available input/output buffers, and store them to a temporary arrays, then use them in render() instead of synchronously dequeuing from MediaCodec. We can keep the overall structure of the player intact, and let player maintain the syncing between different renderers for you. If you want to dig deeper on how this synchronization is maintained, you can check out ExoPlayerImplInternal, with its mediaClock logic.

[ghost]

Hi, @botaydotcom Thanks for your information.
I've already implemented an async mediacodec solution. Here is a benchmark test result.

In some extreme situation, like 60 fps with high bitrate. My async solution shows better performance and stability on some device, especially on Samsung S8.

Async mediacodec give us the ability to accelerate the process of queuing buffer without much cpu usage increase.

[ragotiteb]

Hi @ycinfinity , quite interesting results. I'm working on similar type of high-end devices, and see this approach as having some very promising potential. Is there a possibility of you to open source your solution? Or, as an alternative, to provide any tip you can provide regarding how to implement?

[ghost]

Hi @ragotiteb, I can't open source it for now. The async mediacodec give us a new information, the specific time when the buffer is available for queuing. In sync mode we don't know this information, what we can do is keep trying until it's available. We should take advantage of this new information in async mode, queuing the buffer as soon as possible.

[ragotiteb]

@ycinfinity , ok, understood, i'll give it a shot at implementing on my side then. Thanks for info!
@botaydotcom , is there any update regarding whether you would implement/release something like this on the "main" exoplayer branch?

[botaydotcom]

@ragotiteb
I have an implementation finished, but we have some discussions internally on pros and cons of this approach. Due to I/O, we haven't been able to give this more attention. Now that I/O is over, I think we will have update on this soon.

[ragotiteb]

@botaydotcom thanks for the update, look forward to hearing news on your end (been giving it a shot myself, and i now acknowledge some more on the challenges in making this work right, hehehe)

[ghost]

Hulu's async mediacodec solution has already been released on production.

[LanderlYoung]

Thanks, guys. Looking forward to the performance boost!
BTW, @ycinfinity , I checked out the MediaCodecRenderer class of release r2.8.4, only to find it's still using the synchronized mode.
Is there any way that I can try the async mode? Thanks!

[rocqetQueen]

Any news on this? I'm also interested in trying the async version :)
Thanks!

[tonihei]

Sorry, no updates so far. There hasn't been much work on that issue recently. Marking it as low priority for now.

[ghost]

@LanderlYoung, How can you check out the buffer processing mode of MediaCodec ?

[LanderlYoung]

Am I wrong? I was trying to see if MediaCodecRenderer is using the Asynchronous (callback) way.
Like this:

 MediaCodec codec = MediaCodec.createByCodecName(name);
 MediaFormat mOutputFormat; // member variable
 codec.setCallback(new MediaCodec.Callback() {
       ...

[ghost]

How can you get source code of Hulu app? Decompilation?

[LanderlYoung]

Got it. Sorry, I misunderstood you by thinking that Hulu's code is already merged to ExoPlayer.
Is there a PR for this? Thanks!

[ghost]

I see. We have our own player which isn't based on ExoPlayer. I just want to collect some information about async mediacodec from Google. So I created this issue. No PR for now.

[SemihGk]

I see this ticket is a very low priority, but can you at least share your branch of async approach implementation? We are also having high frame drop rate in FireTV and so we are interested in implementing the async solution. If there is already a branch ready, we would love to test it. Thank you.

[rocqetQueen]

I see this ticket is a very low priority, but can you at least share your branch of async approach implementation? We are also having high frame drop rate in FireTV and so we are interested in implementing the async solution. If there is already a branch ready, we would love to test it. Thank you.

Yeah, sharing your progress even if it's not fully working would be highly appreaciated :)

[dustinkerstein]

@ycinfinity, would you be able to post any snippets of your async implementation? I'm trying to compare your benchmark results to our async implementation but we're seeing much more variable onInputBufferAvailable and onOutputBufferAvailable timing. Ie. With onInputBufferAvailable we see 1ms-8ms usually and then sometimes between 20ms-50ms (on a Pixel XL). Have you ever seen that behavior?

Update - Just wanted to provide some data. Here is what the timing in milliseconds looks like between onInputBufferAvailable callbacks: 1,12,0,1,0,5,0,12,0,1,0,5,1,12,2,0,8 and occasional jumps higher (20ms+).

Update 2 - Ah, I was looking at the 300 frame interval in your post above. The first 100 frame graph does appear to also show the same type of variance in the buffer timing, so I guess what I'm seeing is normal. Is the 300 frame interval showing averages, or just the exact value every 300 frames?

[ghost]

@dustinkerstein
Sorry for late reply, maybe the occasional jumps in your data are caused by rebuffer. The rebuffer time is an unstable factor which should be removed while collecting the onInputBufferAvailable duration time.

The 300 frames value is the original collected value.

I am working on a PR for Exoplayer.

[dustinkerstein]

@ycinfinity no worries at all on the delay. How do you mean "rebuffer"? And just to clarify, the 300 frame example is the raw time between two consecutive frames but sampled like: frame 0-1, 300-301, 600-601, 900-901, etc, correct?

[VOstopolets]

@ycinfinity @dustinkerstein
Actually, do you have any snippets/branch of correct async implementation? The syncing audio/video streams in async mode is still a problem for me. I can't use MediaSync class because should support Android 21+ and I don't want to go back to the sync mode.

[hoax-killer]

@ycinfinity @dustinkerstein

With an aim to improve decoding performance, I implemented two or say N<4 MediaCodec based async decoders, each running on separate threads. However, decoder performance did not improve (e.g. decoder FPS was around 250 FPS for 1080P - this is not the rendering FPS, but purely based on decoder's output). The rationale behind implementing multiple async decoders was to use multiple hardware decoders that are available in modern COTS. Could any confirm these numbers or shed light on using multiple threads of async decoders?