Camerakit-android: Individual Frame Inspection when using CameraKit Preview or Video

[austinkettner]

Ability to get individual frames of a video and pick said frames

[GiacomoRandazzo]

Hi, I'm building an app that does some frame processing on a camera preview.

Right now I'm using Fotoapparat, I have a class that implements their FrameProcessor interface. The class gets a frame as a byte[] in NV21 color space, it gets passed to some NDK functions that convert the color space to RGB, resize and eventually rotate it before doing the actual processing. The frame processing happens synchronously.
To display the results of the processing on top of the camera preview I use a FrameLayout with a transparent view with a Canvas on top of the Fotoapparat's camera preview. This way the camera preview runs smoothly. This system works wonders for me, most of the credit is due to this repo: RubikDetector-Android, I recommend reading it's README, the process is explained more clearly.

I also need to record the video along with the processing results, to achieve that I employed an under performing solution. Basically I do some additional drawing directly on each frame on the C++ side after the processing. I then pass those output frames to Java where they are fed to a MediaRecorder. This produces low quality and low FPS videos, but at least it works.
The reason I'm looking for an alternative is that Fotoapparat (seems to) does not support video recording anymore. My idea is to record the video from the camera as it is, and at a later point merge it with the result of the processing as an overlay, using ffmpeg or other libraries.

This is my current use case, I'll tell you some improvements that IMO could be made to the frame processing approach adopted by Fotoapparat (and other libraries): an option to specify the color format, orientation support (right now I'm rotating each frame using OpenCV in portrait mode).

I'm no expert so I'm not sure this is the best approach to frame processing, but it is the only way I could make it work decently. I tried to be exhaustive with my use case without boring you with the details, if you have any question I'll be happy to answer.

[emersoncloud]

Hey @RAN3000 thanks for the detailed message with your use case. We appreciate it and will definitely take that into consideration when designing our frame processing. I'm sure we'll have more questions for you a bit further into the process. Thanks!

[XinyueZ]

I need frame processing as well, something like:

onFrame(buffer: ByteArray) {
}

[XinyueZ]

419 has the same issue.

[kleysonr]

Any update on this feature ?

Best Regards.

[Zaniyar]

I really appreciate your work with the CameraKit. It would be nice, to hear if you are still working on the frame feature or not anymore. Thank you! <3