Javacv: Recording through Camera2 Api stores only green frames.

[Iqra786]

Hi,
I am working on demo that records frames with Camera2 API. I am getting and storing frames successfully but all the frames are in green color(All video is in green color). Could you help or guide me what I am doing wrong. The frames are in ImageFormat.YUV_420_888 because ImagerReader does not support NV21 format. I am initializing the ImageReader as follow.
mImageReader = ImageReader.newInstance(320, 320, ImageFormat.YUV_420_888, 10);
and processing the frames as follow.

 @Override
    public void onImageAvailable(ImageReader reader) {
        Image image = reader.acquireNextImage();
        if (mPreviewStarted != null) {
            ByteBuffer buffer = image.getPlanes()[0].getBuffer();
          //  System.out.println("test test " + image.getFormat());
            byte[] bytes = new byte[buffer.remaining()];
            buffer.get(bytes);
// passing the frames to FFmpeg recorder for recording.
            mPreviewStarted.onPreviewFrame(bytes);
        }
        image.close();
    }

Thanks

[tomerhatav]

you are getting only the Y plane.
you need to get Y and V plane for creating a NV21 byte array.
try this function below.

 private byte[] convertYUV420ToNV21(Image imgYUV420) {
        byte[] rez;

        ByteBuffer buffer0 = imgYUV420.getPlanes()[0].getBuffer();
        ByteBuffer buffer2 = imgYUV420.getPlanes()[2].getBuffer();
        int buffer0_size = buffer0.remaining();
        int buffer2_size = buffer2.remaining();
        rez = new byte[buffer0_size + buffer2_size];

        buffer0.get(rez, 0, buffer0_size);
        buffer2.get(rez, buffer0_size, buffer2_size);

        return rez;
    }

[Iqra786]

Thanks, I tried same method from stack overflow early today but still the frames were in green. Have you tried above method? Please let me know so I'll try one more time.

[tomerhatav]

Yes, I tried above. can you paste here the code where you init the ffmpegrecorder and the code where you do .record(frame)?

[Iqra786]

Thanks, I am not front of the computer so I'll paste code in the morning. Few point. When you initialize imageReader.newInstace() what are u using as height , width and image format.

[tomerhatav]

640,480, ImageFormat.YUV_420_888

[Iqra786]

private void initRecorder() {
        Log.w(LOG_TAG, "init recorder");
        if (yuvImage == null) {
            yuvImage = new Frame(imageWidth, imageHeight, Frame.DEPTH_UBYTE, 2);
            Log.i(LOG_TAG, "create yuvImage");
        }
        String ffmpeg_link =CameraMethods.getVideoFile().toString();
        recorder = new FFmpegFrameRecorder(ffmpeg_link, 320, 320, 1);
        recorder.setFormat("mp4");
        recorder.setSampleRate(sampleAudioRateInHz);
        // Set in the surface changed methodr
        recorder.setFrameRate(frameRate);
//        recorder.setVideoCodec(avcodec.AV_CODEC_ID_H264);
        recorder.setPixelFormat(avutil.AV_PIX_FMT_NV21);
        int bitRate = recorder.getVideoBitrate();
//        recorder.setVideoBitrate(1967000);
        recorder.setVideoBitrate(bitRate);
        audioRecordRunnable = new AudioRecordRunnable();
        audioThread = new Thread(audioRecordRunnable);
        runAudioThread = true;
    }

// saving frames
if (recorder.getTimestamp() < endTime) {
                ((ByteBuffer) yuvImage.image[0].position(0)).put(data);
                try {
                        long t = 1000 * (System.currentTimeMillis() - startTime);
                        if (t > recorder.getTimestamp() && (t <= endTime)) {
                            recorder.setTimestamp(t);
if (runAudioThread) recorder.record(yuvImage);
                        }
                } catch (Exception e) {
                    e.printStackTrace();
                }
            }

[saudet]

FFmpeg supports pretty much any format, so simply specifying it on record like this should work: FFmpegFrameRecorder.record(frame, AV_PIX_FMT_YUV420P)

[Iqra786]

@saudet Thanks. You are saying following method is write. I don't need any conversion.

  @Override
    public void onImageAvailable(ImageReader reader) {
        Image image = reader.acquireNextImage();
        if (mPreviewStarted != null) {
            ByteBuffer buffer = image.getPlanes()[0].getBuffer();
          //  System.out.println("test test " + image.getFormat());
            byte[] bytes = new byte[buffer.remaining()];
            buffer.get(bytes);
// passing the frames to FFmpeg recorder for recording.
            mPreviewStarted.onPreviewFrame(bytes);
        }
        image.close();
    }

[saudet]

Well, like @tomerhatav points out, you're going to need to get the U and V planes as well, but yes otherwise that's looks alright.

[Iqra786]

@tomerhatav and @saudet Thanks.
How can I improve recorded frame quality. There is method called setVideoQuality(double videoQuality) where parameter videoQuality come from.

[saudet]

The value is used in two places in FFmpegFrameRecorder, which set the global_quality as well as the crf option:
https://github.com/bytedeco/javacv/blob/master/src/main/java/org/bytedeco/javacv/FFmpegFrameRecorder.java#L387
https://github.com/bytedeco/javacv/blob/master/src/main/java/org/bytedeco/javacv/FFmpegFrameRecorder.java#L520
You can check FFmpeg's documentation to understand what they mean given the codec you are using: https://ffmpeg.org/documentation.html

Of course, JavaCV itself could have better documentation. If you would like to make a contribution, it would be very welcome!

[saudet]

BTW, @lfdversluis has started to work on a sample using the new camera API: https://github.com/bytedeco/javacv/issues/163

Please consider taking part in that effort. I'm sure it will help a lot of other people like you! Thank you

[lfdversluis]

The google sample is a good way to start, but I found that it takes a lot of lines (1001 in this sample) to just get the camera up and running. If you got some additional information, please share it :)

[kmlx]

Hi,

First of all, the function convertYUV420ToNV21 is completely wrong. That is not how you transform YUV420 to NV21.

Transforming YUV_420_8888 to NV21 is kind of complicated, but there is a much easier way of passing frames to ffmpeg:

Use RGBA_8888, 8 bytes, 4 channels

yuvImage = new Frame(width, height, Frame.DEPTH_UBYTE, 4);

Initialise the ImageReader with PixelFormat.RGBA_8888, like so:

imageReader = ImageReader.newInstance(VIDEO_WIDTH, VIDEO_HEIGHT, PixelFormat.RGBA_8888, 1);
imageReader.setOnImageAvailableListener(onImageAvailableListener, null);

then on ImageReader.onImageAvailableListener get the bytes and process the image:

ImageReader.OnImageAvailableListener onImageAvailableListener = new ImageReader.OnImageAvailableListener() {
        @Override
        public void onImageAvailable(final ImageReader reader){
            mBackgroundHandler.post(new Runnable() {
                @Override
                public void run() {
                    Image img = reader.acquireNextImage();
                    final ByteBuffer buffer = img.getPlanes()[0].getBuffer();
                    byte[] bytes = new byte[buffer.remaining()];
                    buffer.get(bytes, 0, bytes.length);
                    img.close();
...
                     ((ByteBuffer) yuvImage.image[0].position(0)).put(bytes);
                     ffmpegRecorder.record(yuvImage);

...
                }
            });

[saudet]

@kmlx If you could update AndroidFrameConverter with that method, it would be great! Then please send a pull request.

[ghost]

@kmlx Thanks for the code samples. They helped lots!

On a Nexus 5 I'm getting the following error with RGBA_8888

04-30 12:43:54.599 26844-27561/live.fanstream.live E/ImageReader_JNI: Producer output buffer format: 0x22, ImageReader configured format: 0x1
04-30 12:43:54.608 26844-27561/live.fanstream.live E/AndroidRuntime: FATAL EXCEPTION: CameraBackground
Process: live.fanstream.live, PID: 26844
java.lang.UnsupportedOperationException: The producer output buffer format 0x22 doesn't match the ImageReader's configured buffer format 0x1.

YUV_420_888 runs OK so I'm assuming not all devices can deliver RGBA_8888. Would this be correct?
Or is there a way to configure CameraDevice or Surface to output RGBA_8888 to ImageReader?

[rahulsnitd1014]

Have anyone succeeded in recording video using camera 2 and FFmpegFrameRecorder , I am geeting green frames in my video . Thanks in advance.

[kmlx]

@fanstream-tv @rahulsnitd1014
Yep. Solution works for RGBA_8888 cameras.

For cameras that support only YUV_420_8888 conversion is needed, and that is costly. No solution for conversion at this moment (except maybe renderscript, if you can get it to work with javacv).

Also, when implementing, please take care with Camera API2 support levels, since not all manufacturers implemented full-blown Camera2. This further hampers the use of Camera2.

android.info.supportedHardwareLevel:
LEGACY. These devices expose capabilities to apps through the Camera API2 interfaces that are approximately the same capabilities as those exposed to apps through the Camera API1 interfaces. The legacy frameworks code conceptually translates Camera API2 calls into Camera API1 calls; legacy devices do not support Camera API2 features such as per-frame controls.
FULL. These devices support all of major capabilities of Camera API2 and must use Camera HAL 3.2 or later and Android 5.0 or later.
LIMITED. These devices support some Camera API2 capabilities (but not all) and must use Camera HAL 3.2 or later.

[rahulsnitd1014]

@kmlx Thanks for your reply.
My Camera supports RGBA_8888 thats for sure.
I am using RGBA_8888 format for imageReader .. this is my code how inisialized it.

mImageReader = ImageReader.newInstance(previewSize.width, previewSize.height,PixelFormat.RGBA_8888, 5);

and this is the code where I am trying to record the camera Buffer....

ImageReader.OnImageAvailableListener onImageListener = new ImageReader.OnImageAvailableListener() {

        @Override
        public void onImageAvailable(ImageReader reader) {


            if (image == null)
                return;
             Image image = reader.acquireLatestImage();
            Long timestamp = SystemClock.elapsedRealtime();
            ByteBuffer buffer = image.getPlanes()[0].getBuffer();
            if (RECORD_LENGTH > 0 && recording) {
                int i = imagesIndex++ % images.length;
                yuvImage = images[i];
                timestamps[i] = System.currentTimeMillis();
            byte[] toRecord = new byte[buffer.remaining()];
            buffer.get(toRecord, 0, toRecord.length);
            image.close();
            if (yuvImage != null && recording) {
                if (RECORD_LENGTH <= 0) {
                    ((ByteBuffer) yuvImage.image[0].position(0)).put(toRecord);
                    try {
                        Log.v(TAG, "Writing Frame");
                        long t = 1000 * (System.currentTimeMillis() - startTime);
                        if (t > recorder.getTimestamp()) {
                            recorder.setTimestamp(t);
                        }
                        recorder.record(yuvImage);
                    } catch (FFmpegFrameRecorder.Exception e) {
                        Log.v(TAG, e.getMessage());
                        e.printStackTrace();
                    }
                }

But I am getting green Frames in my mp4 recorded. Please help if you have working code.Thankyou

[kmlx]

@rahulsnitd1014 depends on what Android API you're running. I wouldn't use camera2 on anything below API23, it's simply too buggy.

But I would also try out renderscript to transform YUV_420_888 to RGB
check this link for how to do it: http://stackoverflow.com/questions/36212904/yuv-420-888-interpretation-on-samsung-galaxy-s7-camera2

by using YUV_420_888 you should be able to ensure top compatibility with devices (except for some which haven't implemented proper camera2 APIs; this can be detected using android.info.supportedHardwareLevel; check my previous comment for why this matters)

regarding code, start out here: https://github.com/googlesamples/android-Camera2Basic
git checkout the repo and test it on your device. If it doesn't work, no worries, a lot of devs have issues with camera2: https://github.com/googlesamples/android-Camera2Basic/issues

But if it does work, then look for this line: https://github.com/googlesamples/android-Camera2Basic/blob/master/Application/src/main/java/com/example/android/camera2basic/Camera2BasicFragment.java#L241

That's the one you need to replace with the code that you've provided in your comment. Of course, you would need to instantiate ffmpeg, something close to https://github.com/bytedeco/javacv/blob/master/samples/RecordActivity.java#L219

Hope I helped.

[rahulsnitd1014]

@kmlx Thanks for your response.

1.I have already ran "https://github.com/googlesamples/android-Camera2Basic" on my phone and its worked perfectly.

1. What is the use of using renderscript to transform YUV_420_888 to RGB for recording. Is it gona help me in any way for recording.
2. I have retrived the proper bitmap from image by using ARGB_8888 as imageReader Format.
   this is my code ....

 ImageReader.OnImageAvailableListener onImageListener = new ImageReader.OnImageAvailableListener() {

        @Override
        public void onImageAvailable(ImageReader reader) {
            if (reader != mImageReader)
                return;
            Image image = reader.acquireLatestImage();
            if (image == null)
                return;
            Long timestamp = SystemClock.elapsedRealtime();
            ByteBuffer buffer = image.getPlanes()[0].getBuffer();
            if (RECORD_LENGTH > 0 && recording) {
                int i = imagesIndex++ % images.length;
                yuvImage = images[i];
                timestamps[i] = System.currentTimeMillis();
            }
//boolStride is just a boolean variable .
            if (boolStride) {
                pixelStride = image.getPlanes()[0].getPixelStride();
                rowStride = image.getPlanes()[0].getRowStride();
                rowPadding = rowStride - pixelStride * 480;
                displayWidth = 480 + rowPadding / pixelStride;
                boolStride = false;
                bitmap = Bitmap.createBitmap(displayWidth, 340, Bitmap.Config.ARGB_8888);
                canvas = new Canvas(bitmap);
            }
            bitmap.copyPixelsFromBuffer(buffer);

Aactually what I am doing is I have made a streaming application using wifidirect and camera2 Api. Streaming is working perfectly fine.
This is how I send the stream , by compressing it in JPEG format.
canvas.drawBitmap(bmp, 30, 210, null);
bitmap.compress(Bitmap.CompressFormat.JPEG, 30, mJpegOutputStream);
//sendPreview used for sending the camera stream to other device.
sendPreviewFrame(timestamp, mJpegOutputStream);

I have used canvas to draw some images and text on my stream .Which is visible on the receiver side.

I want to record what I am sending to other device. Is there a way to record the bitmap as mp4 which I am sending to receiver side app.
Thanks is advance.

[ericli0625]

You can refer to the below link:
http://androidxref.com/6.0.1_r10/xref/packages/apps/Camera2/src/com/android/camera/processing/imagebackend/TaskJpegEncode.java#79

Function:
convertYUV420ImageToPackedNV21()

By the way, plane[1] & plane[2] They contents were almost same.

[saudet]

@ericli0625 Could you add this to AndroidFrameConverter.java and send a pull request? Thanks!

[kmlx]

FYI, TaskJpegEncode is really, really slow, and cannot produce at least 20fps.
TaskJpegEncode might be good for taking a picture, but when you have to broadcast video, that function doesn't help.

If you need 20fps one needs to implement renderscript. Java is simply too slow here.

Planes depend on the camera device and what format it produces.
My suggestion is to test this out on as many devices as possible because plane[1] & plane[2] They contents were almost same. depends on device. My suggestion is to test it out on the following devices: Galaxy Note devices, Galaxy Sx devices, Moto devices.
Test on as many as you can, but be prepared to be let down :)

[xdeop]

Hi,
I'm using camera1 api and with a Nexus 4 I'm also getting green frames with ffmpefframerecorder. I've followed the RecordActivity.java example and is working fine in a Samsung Galaxy Alpha. Why is not working in a Nexus 4? How can I solve the issue?

Thanks.

[kmlx]

@xdeop green frames means the resolution or the camera settings are not supported.
choose the right resolution/settings and it will work.
you can find how to choose the right resolution on stack overflow.

[xdeop]

Ok. Thanks. I'll give it a try.

Has someone accomplished migrating the RecordActivity example with the
Camera2 api?

Thanks.

2017-04-18 10:15 GMT+02:00 Adrian Stanescu [e-mail]:

@xdeop https://github.com/xdeop green frames means the resolution or
the camera settings are not supported.
choose the right resolution/settings and it will work.
you can find how to choose the right resolution on stack overflow.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/bytedeco/javacv/issues/298#issuecomment-294724315,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AE_Frpm3i4YMVPVq5jJZ5te59hkWB_fZks5rxHEegaJpZM4G_AYG
.

--
Xavi Deop

[xiandanin]

If the resolution is large, frequent buffer.get (bytes);will lead to preview sluggish, how to solve

[victor-lund]

I am also struggling with this. Anyone?

[kmlx]

@dengyuhan you need threaded camera access. buffer.get (bytes); will need to be called on a separate thread than the UI.

@victor-lund

green frames means the resolution or the camera settings are not supported.
choose the right resolution/settings and it will work.

[xiandanin]

@kmlx If buffer.get (bytes); is used in a child thread,although it can make the preview smooth,but the recorded data is actually dropped frames.

[pascalbaljet]

Any updates on this issue?

[kmlx]

@pascalbaljet yes. solution is to use correct resolution when opening camera.

green frames means the resolution or the camera settings are not supported.
choose the right resolution/settings and it will work.

to put things into perspective: i've tested this library on hundreds of phones and I never got a green frame.

[houyidg]

hi, boys and gays !!!!!!!!
try yuv_420 format convert to nv21, and then output ffmpeg engine !
https://www.polarxiong.com/archives/Android-YUV_420_888%E7%BC%96%E7%A0%81Image%E8%BD%AC%E6%8D%A2%E4%B8%BAI420%E5%92%8CNV21%E6%A0%BC%E5%BC%8Fbyte%E6%95%B0%E7%BB%84.html