Models: [tflite][quantization][deeplabv3] Constant array MobilenetV2/expanded_conv_7/depthwise/depthwise_weights lacks MinMax information

[kanul]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):no
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):Ubuntu 16.04
• TensorFlow installed from (source or binary):Source
• TensorFlow version (use command below):1.9.0
• Python version:2.7.12
• Bazel version (if compiling from source):0.12.0
• GCC/Compiler version (if compiling from source):5.4.0
• CUDA/cuDNN version:cuda-9.0/7.0
• GPU model and memory:GeForce GTX 1080/8105MiB
• Phone:xiaomi5 (Snapdragon 820)
• Exact command to reproduce:
  bazel run --config=opt //tensorflow/contrib/lite/toco:toco --
  --input_file=/external_home/data/model/deeplabv3_mnv2_pascal_train_aug/frozen_inference_graph.pb
  --output_file=/external_home/data/model/deeplabv3_mnv2_pascal_train_aug/kanul.tflite
  --inference_type=QUANTIZED_UINT8
  --input_shape=1,513,513,3
  --input_array=sub_7
  --output_array=ResizeBilinear_3

Describe the problem
I have tried to quantize MobileNetV2 for deeplabV3+ with TFlite. But I fail to convert the model.
From the following issue, I saw that the operations were not supported for the option of quantization.

https://github.com/tensorflow/models/blob/master/research/deeplab/g3doc/model_zoo.md
Checkpoint name: mobilenetv2_coco_voc_trainaug

As we can see graphs from the tensorboard, there is one big problem.

In "import/MobilenetV2/expanded_conv_7/depthwise/depthwise",

the operation of depthwise consists of the subgraph with 3 nodes: (depthwise) and BatchToSpaceND, SpaceToBatchND.

But, in "import/MobilenetV2/expanded_conv_6/depthwise/depthwise",

the operation of depthwise is DepthwiseConv2dNative itself.

From the difference, we can not quantize deeplabv3 based on mobilenetv2.

The one thing is that MobilenetV2/expanded_conv_7~16 does not have min/max value to be needed for quantization with tflite.

Although I implement the needed min/max value in hardcode_min_max.cc,

This model does not run well in mobile environments.

The ultimate problem is caused by the fact that depthwise_conv_7~16 consist of 3 nodes including BatchToSpaceND and SpaceToBatchND.

I request you to notify the method to resolve above issues.

Source code / logs

bazel run --config=opt //tensorflow/contrib/lite/toco:toco --
--input_file=/external_home/data/model/deeplabv3_mnv2_pascal_train_aug/frozen_inference_graph.pb
--output_file=/external_home/data/model/deeplabv3_mnv2_pascal_train_aug/kanul.tflite
--inference_type=QUANTIZED_UINT8
--input_shape=1,513,513,3
--input_array=sub_7
--output_array=ResizeBilinear_3

2018-07-11 04:40:01.330069: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before pre-quantization graph transformations: 166 operators, 340 arrays (1 quantized)
2018-07-11 04:40:01.330711: W tensorflow/contrib/lite/toco/graph_transformations/hardcode_min_max.cc:339] Tweaking the MinMax of array ResizeBilinear_1, which is an input to {Concatenation operator with output concat}, because we want all inputs and outputs of a Concatenation operator to have the same MinMax so that it can be implemented as a pure byte-copy, no arithmetic.
2018-07-11 04:40:01.332983: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] After pre-quantization graph transformations pass 1: 111 operators, 285 arrays (1 quantized)
2018-07-11 04:40:01.335731: I tensorflow/contrib/lite/toco/graph_transformations/graph_transformations.cc:39] Before quantization graph transformations: 111 operators, 285 arrays (1 quantized)
2018-07-11 04:40:01.337575: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_7/depthwise/depthwise_weights lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.337670: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_7/depthwise/BatchNorm/FusedBatchNorm_mul_0_param lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.337695: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_7/depthwise/BatchNorm/FusedBatchNorm_add_param lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.338553: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_8/depthwise/depthwise_weights lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.338711: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_8/depthwise/BatchNorm/FusedBatchNorm_mul_0_param lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.338786: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_8/depthwise/BatchNorm/FusedBatchNorm_add_param lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.339777: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_9/depthwise/depthwise_weights lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.339918: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_9/depthwise/BatchNorm/FusedBatchNorm_mul_0_param lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.339985: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_9/depthwise/BatchNorm/FusedBatchNorm_add_param lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.340933: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_10/depthwise/depthwise_weights lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.341034: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_10/depthwise/BatchNorm/FusedBatchNorm_mul_0_param lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.341059: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_10/depthwise/BatchNorm/FusedBatchNorm_add_param lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.342497: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_11/depthwise/depthwise_weights lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.342593: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_11/depthwise/BatchNorm/FusedBatchNorm_mul_0_param lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.342620: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_11/depthwise/BatchNorm/FusedBatchNorm_add_param lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.344311: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_12/depthwise/depthwise_weights lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.344422: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_12/depthwise/BatchNorm/FusedBatchNorm_mul_0_param lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.344452: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_12/depthwise/BatchNorm/FusedBatchNorm_add_param lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.345978: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_13/depthwise/depthwise_weights lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.346094: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_13/depthwise/BatchNorm/FusedBatchNorm_mul_0_param lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.346122: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_13/depthwise/BatchNorm/FusedBatchNorm_add_param lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.349163: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_14/depthwise/depthwise_weights lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.349318: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_14/depthwise/BatchNorm/FusedBatchNorm_mul_0_param lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.349351: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_14/depthwise/BatchNorm/FusedBatchNorm_add_param lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.353356: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_15/depthwise/depthwise_weights lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.353511: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_15/depthwise/BatchNorm/FusedBatchNorm_mul_0_param lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.353545: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_15/depthwise/BatchNorm/FusedBatchNorm_add_param lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.357264: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_16/depthwise/depthwise_weights lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.357400: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_16/depthwise/BatchNorm/FusedBatchNorm_mul_0_param lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.
2018-07-11 04:40:01.357432: W tensorflow/contrib/lite/toco/graph_transformations/quantize.cc:92] Constant array MobilenetV2/expanded_conv_16/depthwise/BatchNorm/FusedBatchNorm_add_param lacks MinMax information. To make up for that, we will now compute the MinMax from actual array elements. That will result in quantization parameters that probably do not match whichever arithmetic was used during training, and thus will probably be a cause of poor inference accuracy.

[tensorflowbutler]

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
What is the top-level directory of the model you are using

[kanul]

@parvizp

I know that you have a experience to quantize the mobilenetV2 for the task of image classificaiton with tflite.

Do you give me a help to resolve above issues?

[suharshs]

Hi @kanul ,

Are you able to convert to float tflite and see if the spacetobatch and batchtospace are fused?

[suharshs]

This should be resolved in the newest version of TF AFAIK.