Exoplayer: Error playing video with poor audio/video sample interleaving

[danikula]

I used in my app 'decompressed' videos which contains a lot of key frames for smooth seeking. But ExoPlayer can't play it. Videos are not corrupted and can be played by native MediaPlayer or any another video player app. To reproduce just replace first link in Sample class (demo app) by this link and launch Misc-Dizzy sample. There is no any error in log, no unusual warning, just 'buffering' state and nothing also.

Reproduced on ExoPlayer 1.4.2, 1.4.5. Android 6.0, Nexus 5.

[ojw28]

In the sample file the video samples are placed in the byte range [3227,66370313], and the audio samples are placed at the end, in the byte range [64844208,66394879]. For any sane streaming use case you really need the audio and video samples to be interleaved evenly throughout the file.

If the samples aren't interleaved then a player needs to issue multiple separate network requests or maintain a huge buffer. ExoPlayer does actually play this sample correctly if you increase the buffer to hold the entire video (i.e. ~70MB), but the correct solution here is to encode the media in a more sensible way.

[danikula]

@ojw28 Thanks so much for your help. I rewrote video encoding and now ExoPlayer works well for my videos.

[cvoell-franklyinc]

@ojw28 Can you point me in the right direction to determining the byte ranges for video and audio samples in an mp4 file? You mentioned libav in another thread, but I can't figure out how to get sample byte range info with that. Thanks!

"In the sample file the video samples are placed in the byte range [3227,66370313], and the audio samples are placed at the end, in the byte range [64844208,66394879]. For any sane streaming use case you really need the audio and video samples to be interleaved evenly throughout the file."

[andrewlewis]

avprobe -show_packets file.mp4 will output information on each media packet in file.mp4, including the position and size of each sample.

[cvoell-franklyinc]

@andrewlewis @ojw28

I have found several pieces of information that lead me to believe that there are errors in the logic of the Exoplayer mp4 data extraction that goes beyond the conditions of having an extremely poorly interleaved file. This example mp4 file (http://23.239.4.36/test_audio/kwwl_20161114195742230AC.mp4) when played through the Exoplayer 2 demo app seems to fail because of the audio renderer falling behind the video, but the gaps in byte position between audio and video data never seem to be larger than ~50-100kb (viewed using "avprobe -show_packets" ).

I can improve playback by increasing the buffer size, and modifying the Mp4Extractor getTrackIndexOfEarliestCurrentSample() method to automatically choose the track whose timestamp is behind by more than a given threshold. But this seems like a bandaid fix, where as the root cause of the issue seems to be that unlike other media players (Android MediaPlayer, Chrome Browser's mp4 player etc), the Exoplayer mp4 playback seems to not utilize the tracks' sample table's timestampsUs values at all, and instead relies on track data always being loaded in the order it is stored in the byte data, and hoping that the shared buffer for video/audio does not fill up with video data without the corresponding timestamps' required audio data.

Perhaps the fix is to have separate buffers for each track?

One other bug my issue seems to reveal is that when multiple tracks(audio/video) are loaded to different timestamps, the buffer displays the maximum timestamp of the tracks that are loaded, when I believe it should be the minimum (if video is loaded to 45 sec, but audio is only loaded to 40, when I skip to 44 seconds all the data is not actually loaded yet even though the visual buffer indicated it was). I believe this is caused by ExtractorMediaPeriod.java getBufferedPositionUs() method referencing getLargestQueuedTimestampUs() method which doesn't make sense, and might be affecting the code elsewhere.

Let me know if you agree with my assessment. I will continue to debug the exoplayer code to see if there are improvements that may be suggested in a PR. Thanks!

[ojw28]

This example mp4 file (http://23.239.4.36/test_audio/kwwl_20161114195742230AC.mp4) when played through the Exoplayer 2 demo app seems to fail because of the audio renderer falling behind the video, but the gaps in byte position between audio and video data never seem to be larger than ~50-100kb (viewed using "avprobe -show_packets" ).

Playback of this file is fixed in dev-v2 (the relevant change is https://github.com/google/ExoPlayer/commit/488c2d82708fcd547a5a0b19bd56160914ed8567).

Perhaps the fix is to have separate buffers for each track?

We already use separate buffers for each track ;).

One other bug my issue seems to reveal is that when multiple tracks(audio/video) are loaded to different timestamps, the buffer displays the maximum timestamp of the tracks that are loaded, when I believe it should be the minimum (if video is loaded to 45 sec, but audio is only loaded to 40, when I skip to 44 seconds all the data is not actually loaded yet even though the visual buffer indicated it was). I believe this is caused by ExtractorMediaPeriod.java getBufferedPositionUs() method referencing getLargestQueuedTimestampUs() method which doesn't make sense, and might be affecting the code elsewhere.

Yes. This is what https://github.com/google/ExoPlayer/commit/488c2d82708fcd547a5a0b19bd56160914ed8567 fixed. It's a little more nuanced than just using the minimum because the file may also contain sparse tracks. Take a look at the change if you want to see the logic for handling this case correctly.

Let me know if you agree with my assessment. I will continue to debug the exoplayer code to see if there are improvements that may be suggested in a PR. Thanks!

I think the conclusion is that if you use dev-v2 you should find these problems are resolved.

[cvoell-franklyinc]

amazing! I'll make sure to check dev branches before future questions. You guys rock.