Exoplayer: Synchronizing Multiple ExoPlayers

[jtabor]

Hello,

I'm working on using ExoPlayer for a custom VR video streaming application. I want to be able to play media from multiple DASH sources to different SurfaceTextures, which will be put into an OpenGL shader to be warped and displayed in VR (I know how to do the OpenGL part already). Each DASH source corresponds to a different tile or quadrant of the VR video.

I've been able to play my tiles using separate SimpleExoPlayer instances, but there isn't a way to synchronize those so the tiles would be out of sync. I've tried a couple things to get this working:

• I setup a DashMediaSource and tried to read samples directly from the sample streams and put them into MediaCodec on my own. I did this by creating a MediaPeriod from the DashMediaSource, and once it was prepared, I selected a track from it. Whenever there is data available, I read it into the decoder and set the right flags. This didn't work. In fact, it was so bad that the phone would reset whenever I ran the code. I think there might have been a problem with the initialization of the MediaCodec, but the debugger wasn't too useful, so after a week of trying to debug it, I gave up. (I can paste some code, if that idea has promise, and it's just a matter of debugging)
• I tried modifying MediaCodecVideoRenderer to keep DecoderOutputBuffers around until I wanted to render them. This also didn't work.

What I want to know is, is there a way I can either synchronize SimpleExoPlayer (or ExoPlayer) instances? If I can't do that, I'd like to be able to get a sample stream that I can put into a MediaCodec instance myself, and handle the playback and synchronization myself.

Thanks in advance for any help!

[ojw28]

This is a pretty advanced use case :). We don't provide a way to synchronize SimpleExoPlayer (or ExoPlayer) instances. With a bit of work, however, you might manage to get all of the streams playing at once inside a single instance. You'll need to touch a few components, but at a high level you'll need to do something like:

1. Inject a custom RenderersFactory instance that builds as many video renderers as you need. You can probably extend DefaultRenderersFactory to do this, then override buildVideoRenderers to build N instances of MediaCodecVideoRenderer. You can see the implementation in DefaultRenderersFactory to see how it builds a MediaCodecVideoRenderer, and copy that.
2. If you have control over the source content, make it so that you have a single big manifest containing all of the adaptation sets that you need. You can then use DashMediaSource as normal. If this isn't possible, create all of the DashMediaSource instances and then merge them using MergingMediaSource.
3. At this point the remaining work is to ensure the TrackSelector selects the correct track(s) for each of the video renderers. DefaultTrackSelector wont do anything like this for you, so you'll probably need to implement your own TrackSelector from scratch.

Once that's all wired up, it might "just work", although note that many devices only have a limited number of video decoders available (and for low end devices, the limit might be 1).

[jtabor]

Thanks for the reply. I'll try it out, and let you know so people can reference this later if they need to do the same thing. (Unless it doesn't work, then I'll ask about that, hahaa).

As far as decoders, we're targeting a Google Pixel, which I'm told has 8 decoders.

[Endmaril]

Hi,

@jtabor, you managed to instantiate several SimpleExoplayerthen you should be able to instantiate one with several MediaCodecVideoRenderer.

I am currently (for 2 months now) using the solution described by @ojw28 and it is working great on Samaung Galaxy S7 for now with up to 9 decoders instantiated at the same time (didn't test with more).
Unfortunately I can't share the code for now, we may release it in a year or so which will be obviously too late for you sorry. But i thought it would be helpful to let you know that someone was able to make it work somehow.

I'll try to help on the subject if I can.

[jtabor]

@Endmaril Thanks! No worries about sharing the code! It's good to know that someone else got this working. I'll try it out and see if I can get it working too.