Javacv: How to convert byte[] we get from android preview call back to Mat.

[sivadatta-igt]

I've been trying to implement this example in an android app. I've created a camera preview class and tried to send the byte[] I got from onPreviewFrame in to the facedetector class I modified so that a Mat object takes in the byte array and do processing on it.

public void processFrame(byte[] frame){
   Mat colorimg = new Mat(frames);
}

But when it is given to the detectanddraw() method from the sample it gives out an error saying the mat object is null and I get this error.

E/cv::error(): OpenCV(3.4.2) Error: Assertion failed (!ssize.empty()) in void cv::resize(cv::InputArray, cv::OutputArray, cv::Size, double, double, int), file /home/travis/build/javacpp-presets/opencv/cppbuild/android-x86/opencv-3.4.2/modules/imgproc/src/resize.cpp, line 4044

I found out that Mat object is being created without any size. So I need a way to convert the byte array I receive byte array to Mat with size equal to the image size.
I've tried the below code as well but still the same error.

colorimg = new Mat(width,height, CvType.CV_8UC3);
colorimg.data().put(frames);

But I got this error:

java.lang.NullPointerException: Attempt to invoke virtual method 'org.bytedeco.javacpp.BytePointer org.bytedeco.javacpp.BytePointer.put(byte[])' on a null object reference

[saudet]

What are the values of "width" and "height"?

[sivadatta-igt]

the width and height of camera

 int width = camera.getParameters().getPictureSize().width;
 int height = camera.getParameters().getPictureSize().height;

[saudet]

Right, so can you log them or something and check their values?

[saudet]

Also, what does your build file look like?

[sivadatta-igt]

Right, so can you log them or something and check their values?

Yes and I also tried to pass them to the Mat() object to create a Mat object with those sizes as I said in the initial comment.

Also, what does your build file look like?

apply plugin: 'com.android.application'

android {
    compileSdkVersion 28
    defaultConfig {
        applicationId "com.facerecognition.example.facerecognitionandroid"
        minSdkVersion 23
        targetSdkVersion 28
        versionCode 1
        versionName "1.0"
        testInstrumentationRunner "android.support.test.runner.AndroidJUnitRunner"
    }
    buildTypes {
        release {
            minifyEnabled false
            proguardFiles getDefaultProguardFile('proguard-android.txt'), 'proguard-rules.pro'
        }
    }
}

dependencies {
    implementation fileTree(include: ['*.jar'], dir: 'libs')
    implementation 'com.android.support:appcompat-v7:28.0.0-alpha1'
    implementation 'com.android.support.constraint:constraint-layout:1.1.2'
    testImplementation 'junit:junit:4.12'
    androidTestImplementation 'com.android.support.test:runner:1.0.2'
    androidTestImplementation 'com.android.support.test.espresso:espresso-core:3.0.2'
    implementation files('libs/ffmpeg.jar')
}

The JARS I've imported to the libs directory:

• ffmpeg

• ffmpeg-android-arm

• ffmpeg-android-arm64

• ffmpeg-android-x86

• ffmpeg-android-x86_64
• javacpp
• javacv
• javacv-platform
• opencv
• opencv-android-arm
• opencv-android-arm64
• opencv-android-x86
• opencv-android-x86_64

[saudet]

What are the versions of these dependencies? And are you using any other
dependencies?

[sivadatta-igt]

I'm using the dependencies from JavaCV 1.4.2.

[saudet]

Ok, can you reproduce the issue on the desktop? Or does it happen only on
Android?

[sivadatta-igt]

Only in android. I even made a standalone GUI program using that example and works fine as the mat object takes input directly from the VideoCapture class. So I tried to use it in android by giving the input to Mat object from preview call back method.

[saudet]

I mean, could you try to create a Mat using the same arguments and then
call Mat.data() and see what that gives on the desktop?

[sivadatta-igt]

No I haven't. I'll try it and post the results.

[sivadatta-igt]

So I've tried it on desktop. Code for it and got the same error.
So I did a imwrite and saw that the Mat object formed was a image with height of 1 and when I pass it to the DetecandDraw method I get a nullpointer exception.

[saudet]

imageMat = new Mat(data); -> that will give you a row vector, unless we also specify the height and width.

[saudet]

Yes, you'll need to use imdecode() for JPEG data.

[sivadatta-igt]

I tried creating with width and height as here.

and I got the output as here.

Am I creating the Mat object wrong?

[sivadatta-igt]

Ok the Mat Object is getting created properly but I am still getting the null pointer exception at

net.setInput(blob);

Code

[saudet]

"blob" is not an image, we need to call blobFromImage(image, ...) on the image to get one.

[saudet]

 private static opencv_dnn.Net net = null;

You'll also need to assign something else than null to the net variable.

[sivadatta-igt]

 private static opencv_dnn.Net net = new opencv_dnn.Net();

Tried initializing the net like above but got the following error

Width: 615 Height: 409
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ff846c23d75, pid=5128, tid=0x00000000000003dc
#
# JRE version: Java(TM) SE Runtime Environment (8.0_181-b13) (build 1.8.0_181-b13)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.181-b13 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# C  [opencv_dnn342.dll+0xd3d75]
#
# Failed to write core dump. Minidumps are not enabled by default on client versions of Windows
#
# An error report file with more information is saved as:
# D:\FaceRecognitionTest\hs_err_pid5128.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.

but then I ran it again and got the following error

Width: 615 Height: 409
Exception in thread "main" java.lang.RuntimeException: OpenCV(3.4.2) C:\projects\javacpp-presets\opencv\cppbuild\windows-x86_64\opencv-3.4.2\modules\dnn\src\dnn.cpp:431: error: (-215:Assertion failed) inputs.size() == requiredOutputs in function 'cv::dnn::experimental_dnn_v5::DataLayer::getMemoryShapes'

    at org.bytedeco.javacpp.opencv_dnn$Net.forward(Native Method)
    at facerecognitiontest.MatTestClass.detectAndDraw(MatTestClass.java:89)
    at facerecognitiontest.MatTestClass.main(MatTestClass.java:69)
C:\Users\Innogeecks\AppData\Local\NetBeans\Cache\8.2\executor-snippets\run.xml:53: Java returned: 1
BUILD FAILED (total time: 1 second)

[saudet]

You'll need to initialize it with a line like this:

net = readNetFromCaffe(PROTO_FILE, CAFFE_MODEL_FILE);

https://github.com/bytedeco/javacv/blob/master/samples/DeepLearningFaceDetection.java#L47

[sivadatta-igt]

Thank you. It's working now( on desktop). Mistake on my part. Now I have to try it in android.