Quickstart-android: ML Kit on Device Face detection API is Very Slow

[charlizesmith]

Step 1: Describe the environment

• Android device: Moto C Plus
• Android OS version: 7.0
• Google Play Services version: 'com.google.gms:google-services:4.2.0'
• Firebase/Play Services SDK version: com.google.firebase:firebase-core:17.0.1

Step 2:

ML Kit on Device Face detection API is Very Slow

I want to detect face from gallery Image. I tried with Mobile vision Api and detected face successfully. On website of mobile vision api, they have mentioned about Firebase MLKIT.
I also tried firebase ML Kit and detected face.

---

Steps to reproduce:

1. Select Image from gallery
2. call detect function with FirebaseVisionFaceDetector on bItmap Image
3. Takes long time to give result.

Observed Results:

• Too much time to process data.

Expected Results:

• Performance should be fast. As Mobile Vision API is faster as compare to Firebase ML Kit.
  
  
  
  • Can I used Mobile Vision API for my app instead of Firebase ML Kit?
  
  

Relevant Code:

```
FirebaseVisionFaceDetector detector = FirebaseVision.getInstance()
.getVisionFaceDetector(option);

detector.detectInImage(image).addOnSuccessListener(
        new OnSuccessListener<List<FirebaseVisionFace>>() {
            @Override
            public void onSuccess(List<FirebaseVisionFace> faces) {

}
```

[google-oss-bot]

This issue does not seem to follow the issue template. Make sure you provide all the required information.

[samtstern]

Duplicate of #929

Please don't post duplicate issues.

[jllarraz]

If you are using a bitmap Firebase is extremely slow around 800 ms for face detection, but if you transform the bitmap to nv21, it does the job in 5-20ms.

fun toNv21(inputWidth: Int, inputHeight: Int, scaled: Bitmap): ByteArray {

    val argb = IntArray(inputWidth * inputHeight)

    scaled.getPixels(argb, 0, inputWidth, 0, 0, inputWidth, inputHeight)
    val yuv = ByteArray(inputHeight * inputWidth + 2 * Math.ceil(inputHeight / 2.0).toInt() * Math.ceil(inputWidth / 2.0).toInt())
    //val yuv = ByteArray(inputWidth * inputHeight * 3 / 2)
    encodeYUV420SP(yuv, argb, inputWidth, inputHeight)

    scaled.recycle()

    return yuv
}

fun encodeYUV420SP(yuv420sp: ByteArray, argb: IntArray, width: Int, height: Int) {
    val frameSize = width * height
    var yIndex = 0
    var uvIndex = frameSize
    var a: Int
    var R: Int
    var G: Int
    var B: Int
    var Y: Int
    var U: Int
    var V: Int
    var index = 0
    for (j in 0 until height) {
        for (i in 0 until width) {
            a = argb[index] and -0x1000000 shr 24 // a is not used obviously
            R = argb[index] and 0xff0000 shr 16
            G = argb[index] and 0xff00 shr 8
            B = argb[index] and 0xff shr 0
            // well known RGB to YUV algorithm
            Y = (66 * R + 129 * G + 25 * B + 128 shr 8) + 16
            U = (-38 * R - 74 * G + 112 * B + 128 shr 8) + 128
            V = (112 * R - 94 * G - 18 * B + 128 shr 8) + 128

            // NV21 has a plane of Y and interleaved planes of VU each sampled by a factor of 2
            //    meaning for every 4 Y pixels there are 1 V and 1 U.  Note the sampling is every other
            //    pixel AND every other scanline.
            yuv420sp[yIndex++] = (if (Y < 0) 0 else if (Y > 255) 255 else Y).toByte()
            if (j % 2 == 0 && index % 2 == 0) {
                yuv420sp[uvIndex++] = (if (V < 0) 0 else if (V > 255) 255 else V).toByte()
                yuv420sp[uvIndex++] = (if (U < 0) 0 else if (U > 255) 255 else U).toByte()
            }
            index++
        }
    }
}

[fernando-clvm]

If you are using a bitmap Firebase is extremely slow around 800 ms for face detection, but if you transform the bitmap to nv21, it does the job in 5-20ms.

Wow, this actually increased face detection time 7x for me. Is this FirebaseVisionImage aspect documented somewhere?

[procmixin]

If you are using a bitmap Firebase is extremely slow around 800 ms for face detection, but if you transform the bitmap to nv21, it does the job in 5-20ms.

Wow, this actually increased face detection time 7x for me. Is this FirebaseVisionImage aspect documented somewhere?

It's provided in the ML Kit GitHub Sample.