Cameraview: Recording normal video while processing frames

[GiacomoRandazzo]

Hi, I was trying to record a video normally while also doing frame processing (the idea is to add the processing result as a video overlay at a later point in time). By "normally" I mean without the results of the frame processing, using startCapturingVideo(File file).
This is not possible since after calling mCamera.unlock() for setting up the MediaRecorder, onPreviewFrame(byte[] data, Camera camera) does not fire anymore.

I was able to solve this by forking this repository and adding these two lines after mMediaRecorder.start():

...
initMediaRecorder();
                    try {
                        mMediaRecorder.prepare();
                        mMediaRecorder.start();
                        mCamera.reconnect();                    <---------
                        mCamera.setPreviewCallbackWithBuffer(Camera1.this);   <--------
                    } catch (Exception e) {
                        LOG.e("Error while starting MediaRecorder. Swallowing.", e);
                        mVideoFile = null;
                        mCamera.lock();
                        endVideoImmediately();
                    }
...

I took this solution from this StackOverflow answer.
I tested it on a Samsung Galaxy A5 2016 and a Xiaomi Redmi 5 Plus, it seems to work very well.

I'm no expert so I'm not sure this is the right way to do things, it would be awesome if you could add this functionality.

Thanks for the great work!

[igalanyvision]

Hi,

I've encountered the same issue, but the solution is not working for me. Did you do something else?

Thanks.

[GiacomoRandazzo]

Oh that sucks. Do you have any ideas that I can implement on how to tackle the problem?

[igalanyvision]

Looks like calling:

mCamera.reconnect();
mCamera.setPreviewCallbackWithBuffer(Camera1.this);

before mMediaRecorder.prepare() (but after initMediaRecorder()) does the job. It seems that once the mediaRecorder is on, you cannot add more callback listeners.

[igalanyvision]

Actually it's not really working. The frame that returns is the last one that was shown before starting to record.

Any ideas?

[GiacomoRandazzo]

In the coming week I'll work on the fork for recording and processing simultaneously, MediaCodec + OpenGL should do. Do you wanna help?

[igalanyvision]

Sure thing. What's the plan? Generally speaking, so I can read up (not an expert in these areas)

[GiacomoRandazzo]

Ping me [email protected]

[natario1]

Will be available in v2

[GiacomoRandazzo]

Awesome! How are you going to approach the problem?
At the end I just turned to Camera2, implemented my own solution based on shadercam so that I can process, record and draw on the preview and recorded video.

[natario1]

With a new preview based on GLSurfaceView and a new api called takeVideoSnapshot() that will pass the frames to a video encoder instead of using MediaRecorder APIs. It works well, records audio, and crops the result based on the view dimensions (so we finally have square videos!).

What's left in my mind is if we should maybe use open GL to read the drawn pixels and pass this buffer to the frame processors, instead of using Camera1 buffer callbacks. This would definitely have some good consequences but also some bad ones. 😕

[GiacomoRandazzo]

My implementation is much more similar with the first options, you basically forward the frames to two different destinations, video encoder and processor, right?
I also tried the second one but could not make it fast, I haven't put too much effort into it so there might be a way... Android does not support renderbuffer which, in my understanding (OpenGL beginner) would make our lives much easier.

[natario1]

Currently processor frames come from Camera1 internal APIs which we have no control.

In v2 with video snapshot, the OpenGL texture is drawn 2 times, on the preview surface and on the video encoder. This does not disturb the Camera1 processor frames so it keeps working.

What I was thinking about is get rid of this Camera1 dependency and draw/use the OpenGL texture 3 times, preview, encoder and processor. This is probably what you are doing! But I am not 100% sure if we want to go this way in the lib

[GiacomoRandazzo]

I'm doing processing on it's own Surface and OpenGL draws twice on preview and MediaRecorder (since api>21).

As I said I tried to make OpenGL draw three times but it turned out too slow, but it was probably my fault 😅

[natario1]

I see... we'll stick to Camera1 for now 👍

[GiacomoRandazzo]

Do you have a timeline for when v2 will be available?

[natario1]

In a few weeks!

[GiacomoRandazzo]

Great! I will definitely switch to your solution, it will certainly be more stable than anything I can come up with. Thanks for the hard work!

[bipinvaylu]

Hey, I am facing the same issue, As and when I start the recording, FrameProcessor callback stopped working. I have tried @RAN3000 workaround of adding 2 lines, but no success though. Here, I have shared all the details and demo app as well to reproduce issue.

[bipinvaylu]

@natario1 I have tried out v2 branch by cloning as well but seems frame processing and recording simultaneously is still under development. Right? If yes, It would be nice if you can provide a workaround to do it to me or some approx time to develop and release it in your mind.

[natario1]

It was released. You must record with takeVideoSnapshot instead of takeVideo for frames to keep coming. If it doesn't work please open a new issue!

[GiacomoRandazzo]

Hey @natario1, I will try v2 as soon as I have some free time, I really appreciate your hard work!
Do you think you will support drawing on the preview and on the snapshot?
In my case I would like to draw an overlay with the results of the frame processing.

The way I do it now would be equivalent to adding (and updating) an additional OpenGL texture after updateTexImage() in GlCameraPreview's onDrawFrame. The texture comes from a Bitmap which get redrawn after a processing call is completed.

[natario1]

The lib internals currently are all private and not meant to be used, but it would be a useful addition - create base interfaces, add protected methods, make classes public and let you override the drawing step and whatever else. If some of you plans to do this job let's open an issue so we can discuss !

[3akat]

So, this fix is working only for API 18+, am I right?

[SAFatemi]

With a new preview based on GLSurfaceView and a new api called takeVideoSnapshot() that will pass the frames to a video encoder instead of using MediaRecorder APIs. It works well, records audio, and crops the result based on the view dimensions (so we finally have square videos!).

What's left in my mind is if we should maybe use open GL to read the drawn pixels and pass this buffer to the frame processors, instead of using Camera1 buffer callbacks. This would definitely have some good consequences but also some bad ones. 😕

Please note about this Issue in Documentation, I has problem about frame processing while recording video and solve it with takeVideoSnapshot method.
I think this issue should be not in Video recording document.
Thanks.