Exoplayer: How to get all frame data while playing video?

[bebop7-dev]

Hi,
I'm looking for how to access all frame data while playing video using exoplayer 2.11.3.
I need the features both video playback and internal image processing for other purposes (not kind of filters) in realtime.
So I need to access all frame data while playing video.

I searched about it for a long time and I tried to make a subclass from "MediaCodecVideoRenderer" and override the "processOutputBuffer" method. but I couldn't get the image from buffer at all!

Part of codes are below:

interface OnProcessEventListener {
    void onProcess(MediaCodec codec, ByteBuffer buffer, int bufferIndex);
}

public class CustomMediaCodecVideoRenderer extends MediaCodecVideoRenderer {
    private OnProcessEventListener onProcessEventListener;

    ...

    @Override
    protected boolean processOutputBuffer(...) {
        if (onProcessEventListener != null) {
            onProcessEventListener.onProcess(codec, buffer, bufferIndex);
        }
        return super.processOutputBuffer(positionUs, elapsedRealtimeUs, codec, buffer, 
                bufferIndex, bufferFlags, bufferPresentationTimeUs, isDecodeOnlyBuffer, 
                isLastBuffer, format);
    }
}

in Activity:

@Override
public void onProcess(MediaCodec codec, ByteBuffer buffer, int bufferIndex) {
    if (mWidth > 0 && mHeight > 0) {
        final Bitmap bm = Bitmap.createBitmap(mWidth, mHeight, Bitmap.Config.ARGB_8888);
        bm.copyPixelsFromBuffer(buffer);

        runOnUiThread(new Runnable() {
            @Override
            public void run() {
                imageView.setImageBitmap(bm);
            }
        });
    }
}

And the exception message is below:

java.lang.RuntimeException: Buffer not large enough for pixels
      at android.graphics.Bitmap.copyPixelsFromBuffer(Bitmap.java:675)

In short, the questions are

1. My direction (using processOutputBuffer method) to get frame data is right?
2. If so, how can I get Bitmap object from buffer?

[andrewlewis]

ExoPlayer gets MediaCodec to output to a Surface by default, which means that the buffers containing output frames are not mapped/copied into byte buffers, improving codec performance. Please see also MediaCodec data types and MediaCodec using an output Surface.

Some options for this:

• If you have ExoPlayer output to a TextureView you can get the output as a bitmap via TextureView.getBitmap. That may be an easy but inefficient way to do this but is probably good enough if you just want to sample the output occasionally rather than processing every frame.
• If you need to get raw output in a byte buffer, you could try using ByteBuffer mode instead. It would be necessary to pass null as the Surface when configuring the decoder. If doing this with MediaCodecVideoRenderer I'd expect that some more changes would be required to remove the assumption of using a surface.
• You could try continuing to use Surface mode but use an ImageReader to read the output back. Last time I tried this the model was that you create an image reader and use the Surface returned by ImageReader.getSurface() as the codec's output surface. Then you attach a listener for Images becoming available and process them as desired.
• Finally, if your use case can be performed using GL operations (I guess not given your comment about not filtering) you can get the player to render to an off-screen SurfaceTexture so that you have the video frames in a GL texture. The gl demo may help with this. The video frames can be copied to the screen or processed with a shader, or you could even try reading back the data using glReadPixels though that may be slow.

Note: there isn't a way to do this for DRM protected content, in general. The approaches above will only work if the media is clear.

[shiv-u]

@tonihei Will the above approaches work if you are using L3 device with DRM protected content ?

[andrewlewis]

@shiv-u Please see https://github.com/google/ExoPlayer/issues/1033#issuecomment-279269696. If no secure output path is required then it may work (I haven't tried it though!).

[bebop7-dev]

@andrewlewis I'm sorry I checked late. Thank you for the comment. I'll try above approaches!