Mediapipe: Deeplab v3 support

[PrinceP]

260

[mgyong]

@PrinceP Not sure what you are asking for?

[PrinceP]

I was trying to run the default model present for deeplab from https://www.tensorflow.org/lite/models/segmentation/overview
Following is the log from loading the tflite model with same setup of hair segmentation.

I/native: I0101 18:25:18.158473 19366 tflite_inference_calculator.cc:211] TfLiteInference:Process1
I/native: I0101 18:25:18.168609 19366 tflite_inference_calculator.cc:247] TfLiteInference:Process2
I/native: I0101 18:25:18.169961 19366 recolor_calculator.cc:131] RGB OUT Tagged
I/native: I0101 18:25:18.179634 19366 graph.cc:415] Start running the graph, waiting for inputs.
I/native: I0101 18:25:18.180408 19366 gl_context_egl.cc:158] Successfully initialized EGL. Major : 1 Minor: 5
I/native: I0101 18:25:18.182210 19440 gl_context.cc:233] GL version: 3.2 (OpenGL ES 3.2 [email protected] (GIT@bc00834, I609ab310b2) (Date:04/11/19))
I/native: I0101 18:25:18.186280 19434 ColorSliderCalculator.cc:77] Color Slider Calculator Runner0 0 0
D/ExternalTextureConv: Created output texture: 3 width: 1444 height: 1925
I/native: I0101 18:25:18.198865 19439 ColorSliderCalculator.cc:77] Color Slider Calculator Runner0 0 0
I/native: I0101 18:25:18.199147 19436 jni_util.cc:40] GetEnv: not attached
I/native: I0101 18:25:18.307039 19365 resource_util_android.cc:42] Successfully loaded: person_segmentation.tflite
I/tflite: Initialized TensorFlow Lite runtime.
I/tflite: Created TensorFlow Lite delegate for GPU.
I/tflite: Replacing 70 node(s) with delegate (TfLiteGpuDelegate) node, yielding 1 partitions.
I/native: I0101 18:25:19.411098 19365 tflite_inference_calculator.cc:301] TfLiteInference:Process3
I/native: I0101 18:25:19.464181 19365 tflite_inference_calculator.cc:308] TfLiteInference:Process4
    I0101 18:25:19.464304 19365 tflite_inference_calculator.cc:314] TfLiteInference:Process4.1
    I0101 18:25:19.464329 19365 tflite_inference_calculator.cc:317] TfLiteInference:Process4.1

The tflite inference calculator gets stuck.
By changing the max_num_channels to 3 the Process of Calculator tflite_inference_calculator is finished, but the close calculator is not getting called.

node {
  calculator: "TfLiteConverterCalculator"
  input_stream: "IMAGE_GPU:mask_embedded_input_video"
  output_stream: "TENSORS_GPU:image_tensor"
  node_options: {
    [type.googleapis.com/mediapipe.TfLiteConverterCalculatorOptions] {
      zero_center: false
      max_num_channels: 4
    }
  }
}

The latest edit is in link

Is there a model issue, or at the post processing end? Can you guide us in the understanding of the functionality?

[afsaredrisy]

Hey @PrinceP the model which you are using is 21 class while mediapipe segmentation current calculators supports 2 channel output only. This is mention in TfLiteTensorsToSegmentationCalculator like _Only 2 channel segmentation tensor currently supported_ . This is 2 class foreground , background segmentation.
For testing you can use this model it wont give you accuracy but it will run in mediapipe segmentation.
https://drive.google.com/open?id=1D5QpycBOI8L9ZXMkrbcCWKV9AieNv25Y
Changes in Graph (pbtxt file): Change input size from 512 to 128 in ImageTransformationCalculator in pbtxt file like this
node: { calculator: "ImageTransformationCalculator" input_stream: "IMAGE_GPU:throttled_input_video" output_stream: "IMAGE_GPU:transformed_input_video" #output_stream: "LETTERBOX_PADDING:letterbox_padding" node_options: { [type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] { output_width: 128 output_height: 128 scale_mode: FIT } }
Change number of channels from 4 to 3 in TfLiteConverterCalculator node.
you can remove SetAlphaCalculator node from graph (optional)
make sure to use appropriate file name in BUILD file of your project.

[PrinceP]

The application ran perfectly @afsaredrisy. Thank you for the guidance.
The latest edit is in link

Also, I am still trying to run the deeplab model by changing the channel numbers.
How the TfLiteTensorsToSegmentationCalculator can be modified for the same.

[poltavski]

@afsaredrisy thank you for providing the model, it works
Can you comment the process/ give pipeline of how you get such a model from a deeplab, preprocess it to foreground and background mask for specific class "people", how can it be changed to another class

@PrinceP if you also have any guidelines for this topic, please share your pipeline, because now it's unclear yet