Exoplayer: Adding image filters to exoplayer video

[jmgirven]

I am interested in adding Instagram-like image filters into my exoplayer video implementation. I can think of two ways to approach this challenge, either to:

1) Override MediaCodecVideoTrackRenderer#processOutputBuffer and, similarly to what this gist does for modifying audio, modify the video ByteBuffer buffer to contain the processed filtered frames and return it to super. If this is the best method, I am wondering what the ByteBuffer actually contains, i.e. how far through the processing is it? If I were to pass it to a library like Android GPUImage, which utilises OpenGL to process images, does anyone have an experience how I could get the ByteBuffer into the OpenGL functions?

2) Allow Exoplayer to write the video frames to a Surface, extract the frames from the Surface, and pass them to an image processing library before writing back onto a Surface. Even better would of course be to intercept the frames just before writing to the Surface and pass those instead.

One example of an image processing library that produces the required effects is Android GPUImage. It requires a GLSurfaceView to write to. GLSurfaceView extends from SurfaceView, so there is no problem getting a Surface for exoplayer to write to. I tried in demo/PlayerActivity#onCreate to change the SurfaceView to a GLSurfaceView and include:

    mGPUImage = new GPUImage(this);
    mGPUImage.setGLSurfaceView(surfaceView);
    mGPUImage.setFilter(new GPUImageSepiaFilter());

but I get an exception when running:

ExoPlayerImplInternal: Internal track renderer error.
com.google.android.exoplayer.ExoPlaybackException: com.google.android.exoplayer.MediaCodecTrackRenderer$DecoderInitializationException: Decoder init failed: OMX.qcom.video.decoder.avc, MediaFormat(1, video/avc, -1, 336240, 1280, 720, 0, 1.0, -1, -1, null, 9994000, false, -1, -1)
    at com.google.android.exoplayer.MediaCodecTrackRenderer.notifyAndThrowDecoderInitError(MediaCodecTrackRenderer.java:389)
    at com.google.android.exoplayer.MediaCodecTrackRenderer.maybeInitCodec(MediaCodecTrackRenderer.java:375)
    at com.google.android.exoplayer.MediaCodecTrackRenderer.onInputFormatChanged(MediaCodecTrackRenderer.java:722)
    at com.google.android.exoplayer.MediaCodecVideoTrackRenderer.onInputFormatChanged(MediaCodecVideoTrackRenderer.java:333)
    at com.google.android.exoplayer.MediaCodecTrackRenderer.readFormat(MediaCodecTrackRenderer.java:497)
    at com.google.android.exoplayer.MediaCodecTrackRenderer.doSomeWork(MediaCodecTrackRenderer.java:480)
    at com.google.android.exoplayer.SampleSourceTrackRenderer.doSomeWork(SampleSourceTrackRenderer.java:129)
    at com.google.android.exoplayer.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:431)
    at com.google.android.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:213)
    at android.os.Handler.dispatchMessage(Handler.java:98)
    at android.os.Looper.loop(Looper.java:148)
    at android.os.HandlerThread.run(HandlerThread.java:61)
    at com.google.android.exoplayer.util.PriorityHandlerThread.run(PriorityHandlerThread.java:40)
 Caused by: com.google.android.exoplayer.MediaCodecTrackRenderer$DecoderInitializationException: Decoder init failed: OMX.qcom.video.decoder.avc, MediaFormat(1, video/avc, -1, 336240, 1280, 720, 0, 1.0, -1, -1, null, 9994000, false, -1, -1)
    at com.google.android.exoplayer.MediaCodecTrackRenderer.maybeInitCodec(MediaCodecTrackRenderer.java:375) 
    at com.google.android.exoplayer.MediaCodecTrackRenderer.onInputFormatChanged(MediaCodecTrackRenderer.java:722) 
    at com.google.android.exoplayer.MediaCodecVideoTrackRenderer.onInputFormatChanged(MediaCodecVideoTrackRenderer.java:333) 
    at com.google.android.exoplayer.MediaCodecTrackRenderer.readFormat(MediaCodecTrackRenderer.java:497) 
    at com.google.android.exoplayer.MediaCodecTrackRenderer.doSomeWork(MediaCodecTrackRenderer.java:480) 
    at com.google.android.exoplayer.SampleSourceTrackRenderer.doSomeWork(SampleSourceTrackRenderer.java:129) 
    at com.google.android.exoplayer.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:431) 
    at com.google.android.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:213) 
    at android.os.Handler.dispatchMessage(Handler.java:98) 
    at android.os.Looper.loop(Looper.java:148) 
    at android.os.HandlerThread.run(HandlerThread.java:61) 
    at com.google.android.exoplayer.util.PriorityHandlerThread.run(PriorityHandlerThread.java:40) 
 Caused by: java.lang.IllegalArgumentException
    at android.media.MediaCodec.native_configure(Native Method)
    at android.media.MediaCodec.configure(MediaCodec.java:1778)
    at com.google.android.exoplayer.MediaCodecVideoTrackRenderer.configureCodec(MediaCodecVideoTrackRenderer.java:327)
    at com.google.android.exoplayer.MediaCodecTrackRenderer.maybeInitCodec(MediaCodecTrackRenderer.java:364)
    at com.google.android.exoplayer.MediaCodecTrackRenderer.onInputFormatChanged(MediaCodecTrackRenderer.java:722) 
    at com.google.android.exoplayer.MediaCodecVideoTrackRenderer.onInputFormatChanged(MediaCodecVideoTrackRenderer.java:333) 
    at com.google.android.exoplayer.MediaCodecTrackRenderer.readFormat(MediaCodecTrackRenderer.java:497) 
    at com.google.android.exoplayer.MediaCodecTrackRenderer.doSomeWork(MediaCodecTrackRenderer.java:480) 
    at com.google.android.exoplayer.SampleSourceTrackRenderer.doSomeWork(SampleSourceTrackRenderer.java:129) 
    at com.google.android.exoplayer.ExoPlayerImplInternal.doSomeWork(ExoPlayerImplInternal.java:431) 
    at com.google.android.exoplayer.ExoPlayerImplInternal.handleMessage(ExoPlayerImplInternal.java:213) 
    at android.os.Handler.dispatchMessage(Handler.java:98) 
    at android.os.Looper.loop(Looper.java:148) 
    at android.os.HandlerThread.run(HandlerThread.java:61) 
    at com.google.android.exoplayer.util.PriorityHandlerThread.run(PriorityHandlerThread.java:40) 

I guess this comes from GPUImage trying to write to the Surface using GLSurfaceView.Renderer, whilst at the same time Exoplayer is trying to write to the Surface itself. Does anyone have any suggestions how I could get the two renderers to play nicely together?

Generally, which of the two methods do you think is more likely to work? I understand you may have never heard of GPUImage, so I am trying to keep my questions relevant to an Exoplayer implementation. Equally GPUImage is not the only library in the world, so I am open to suggestions.

[ojw28]

The question itself falls quite far out of scope for ExoPlayer. It's definitely possible to do what you're attempting, and this guide might be of use to you. It uses MediaPlayer, but you could pretty easily replace MediaPlayer with ExoPlayer in the sample. Beyond that, you'll probably have to work out how to get there yourself ;).

[Zo2m4bie]

@jmgirven I know it's kind of an old question. But did you find how to fix it? I want to cast video to chromecast, so I have to use GlSurfaceView for it.

[jmgirven]

@Zo2m4bie I was never particularly happy with the solution I came up with. The guide that ojw28 posted is pretty much it, and I just needed to bastardise GPUImage to be used in the same way. Let me know if you can do better!

[Zo2m4bie]

@jmgirven So, do you use simple MediaPlayer as in example?

[BattleShipPark]

@jmgirven did you solve the problem? I am looking for and example using image filter in ExoPlayer. Could you help me? Thanks.

[MasayukiSuda]

I release ExoPlayerFilter. This project can use Overlay filter Please try it.
https://github.com/MasayukiSuda/ExoPlayerFilter