Models: [TF2 Object Detection] Converting SSD models into .tflite uint8 format

[SukyoungCho]

Hi, I was wondering if anyone could help how to convert and quantize SSD models on TF2 Object Detection Model Zoo.
It seems like there's a difference in converting to .tflite in TF1 and TF2. To the best of my knowledge, in TF1, we first frozen the model using exporter and then quantized and converted it into .tflite. And, I had no problem in doing it in TF1.
The models I tried was

SSD MobileNet V2 FPNLite 320x320 | 22 | 22.2 | Boxes
-- | -- | -- | --
SSD MobileNet V2 FPNLite 640x640 | 39 | 28.2 | Boxes
SSD ResNet50 V1 FPN 640x640 (RetinaNet50) | 46 | 34.3 | Boxes

However, when I followed the guideline provided on the github repo 1.(https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_on_mobile_tf2.md#step-1-export-tflite-inference-graph) and 2. (https://www.tensorflow.org/lite/performance/post_training_quantization#full_integer_quantization). I was not able to convert them into .tflite.

Running "Step 1: Export TFLite inference graph", created saved_model.pb file in the given output dir {inside ./saved_model/}
However, it displayed the skeptic messages below while exporting them, and not sure if it's run properly.

Running "Step 2: Convert to TFLite", is the pain in the ass. I managed to convert the model generated in the step 1 into .tflite without any quantization following the given command, although I am not sure if it can be deployed on the mobile devices.

tflite_convert \
  --saved_model_dir=/tmp/mobilenet_saved_model \
  --output_file=/tmp/mobilenet.tflite

But, I am trying to deploy it on the board with the coral accelerator and need to conver the model into 'uint8' format. I thought the models provided on the model zoo are not QAT trained, and hence they require PTQ. Using the command line below,

import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
def representative_dataset_gen():
  for _ in range(num_calibration_steps):
    # Get sample input data as a numpy array in a method of your choosing.
    yield [input]
converter.representative_dataset = representative_dataset_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.int8  # or tf.uint8
converter.inference_output_type = tf.int8  # or tf.uint8
tflite_quant_model = converter.convert()

It shows the error message below, and i am not able to convert the model into .tflite format. I think the error occurs because something went wrong in the first step.

Below, I am attaching the sample script I used to run "Step 2". I have never train a model, and i am just trying to check if it is possible to convert SSD models on TF 2 OD API Model Zoo into Uint8 format .tflite. That is why, i dont have the sample data used to train the model, and just using MNIST data in Keras to save the time and cost to create data. (checkpoint CKPT = 0)

import tensorflow as tf

# Convert the model
converter = tf.lite.TFLiteConverter.from_saved_model("/root/ecomfort/tf2_model_zoo/tflite/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8/saved_model")
converter.optimizations = [tf.lite.Optimize.DEFAULT]
#def representative_dataset_gen():
#  for _ in range(num_calibration_steps):
#    # Get sample input data as a numpy array in a method of your choosing.
#    yield [input]
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train = x_train.reshape(60000, 28, 28, 1).astype('float32') / 255
x_test = x_test.reshape(10000, 28,28, 1).astype('float32') / 255
y_train = y_train.astype('float32')
y_test = y_test.astype('float32')

def create_represent_data(data):
  def data_gen():
    for i in data:
      yield [list([i])]
  return data_gen
#converter.representative_dataset = representative_dataset_gen
converter.representative_dataset = create_represent_data(x_train[:5000])
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8  # or tf.int8
converter.inference_output_type = tf.uint8  # or tf.int8
tflite_quant_model = converter.convert()

# Save the model.
with open('model.tflite', 'wb') as f:
  f.write(tflite_model)

The environment description.
CUDA = 10.1
Tensorflow = 2.3, 2.2 (both are tried)
TensorRT =
ii libnvinfer-plugin6 6.0.1-1+cuda10.1 amd64 TensorRT plugin libraries
ii libnvinfer6 6.0.1-1+cuda10.1 amd64 TensorRT runtime libraries

It would be appreciated if anyone could help to solve the issue, or provide a guideline. @srjoglekar246 Would you be able to provide the guideline or help me to convert models into uint8? Hope you documented while you were enabling SSD models to be converted into .tflite. Thank you so much.

**_

Update

I realized I dont need to do the first step, as I didnt do any training and the models are already in .pb file. But when I run the same python script I attached above, to convert to .tflite Uint8. It shows another error message below

---

[albertoisorna]

Same problem here,

triying now with different models of the model ZOO, for the moment impossible for me to convert to TFLITE a mobilenet V2 with TF2 Object detection API.

[srjoglekar246]

You cannot directly convert the SavedModels from the detection zoo to TFLite, since it contains training parts of the graph that are not relevant to TFLite. Can you post the command you used to export the TFLite-friendly SavedModel using export_tflite_graph_tf2?

Running "Step 2: Convert to TFLite", is the pain in the ass. I managed to convert the model generated in the step 1 into .tflite without any quantization following the given command, although I am not sure if it can be deployed on the mobile devices.

Why unsure? What is the size of the converted model? You can use a tool like Netron to check if the .tflite model is valid.

For quantization, you need to implement representative dataset gen in such a way that it mimics how you would typically pass image tensors to TFLite models. It usually boils down to something like this:

def representative_dataset_gen():
  for i in range(100):
    image = tf.io.read_file(os.path.join(data_dir, image_files[i % NUM_FILES]))
    image = tf.compat.v1.image.decode_jpeg(image)
    image = preprocess(image)

    yield [image]

For the preprocess function, look at this implementation.

[SukyoungCho]

@srjoglekar246 Thanks for your reply.
I tried to export TFLite-friendly SavedModel using following command line. I am confused that how "TFLite-friendly SavedModel" is different from the SavedModel from Model Zoo as they seem almost same to me. In addition, although you are saying I cannot directly convert the SavedModels from the detection Zoo, It was converted to TFLite somehow... although im not sure why.

CUDA_VISIBLE_DEVICES=1 python object_detection/export_tflite_graph_tf2.py \
    --pipeline_config_path /root/ecomfort/tf2_model_zoo/ssd_mobilenet_v2_fpnlite_640x640_coco17_tpu-8/pipeline.config \
    --trained_checkpoint_dir /root/ecomfort/tf2_model_zoo/ssd_mobilenet_v2_fpnlite_640x640_coco17_tpu-8/checkpoint \
    --output_directory /root/ecomfort/tf2_model_zoo/tflite/ssd_mobilenet_v2_fpnlite_640x640_coco17_tpu-8

For quantization, I thought it could be any data as long as it is image tensors, and thats why I used MNIST data which does not need to be downloaded. By the way, I think I saw somewhere that "representative_dataset_gen()" or TF2 takes images as numpy array, so could you let me know which is correct? I will try to do quantization following your suggested command line. I will update it here. Thanks.

[SukyoungCho]

@srjoglekar246 Hi, I got this error while quantizing. Wonder if you are familiar with this kind of error.

[SukyoungCho]

@srjoglekar246 Hi, I tried to train the model from a model zoo and convert it to .tflite and quantize it. However, there were 2 critical issues.

1. During the training, at certain point, the loss becomes NaN.

2. Converting and quantizing the trained model, it still shows ValueError: Failed to parse the model: pybind11::init(): factory function returned nullptr. error

[srjoglekar246]

Not sure if there is a bug with training. The ODAPI folks might have a better idea of any issues that occur before using export_tflite_graph_tf2.py.

Can you show your exact code for tflite conversion? Though the model should probably be trained well first.

[SukyoungCho]

@srjoglekar246 This is how I ran the step 1

CUDA_VISIBLE_DEVICES=1 tflite_convert \
  --saved_model_dir=/root/ecomfort/tf2_model_zoo/tflite/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8/saved_model \
  --output_file=/root/ecomfort/tf2_model_zoo/tflite/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8/test.tflite

and this is how I ran the step 2
CUDA_VISIBLE_DEVICES=1 python demo_convert.py

and this is demo_convert

import tensorflow as tf

# Convert the model
converter = tf.lite.TFLiteConverter.from_saved_model("/root/ecomfort/tf2_model_zoo/tflite/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8/saved_model")
converter.optimizations = [tf.lite.Optimize.DEFAULT]

def representative_dataset_gen():
  for i in range(100):
    #dir path where .jpg (image) files are
    image = tf.io.read_file(os.path.join("/root/ecomfort/data/valid_data/total/", image_files[i % NUM_FILES]))
    image = tf.compat.v1.image.decode_jpeg(image)
    image = preprocess(image)

    yield [image]

"""
(x_train, y_train), (x_test, y_test) = tf.keras.datasets.mnist.load_data()
x_train = x_train.reshape(60000, 28, 28, 1).astype('float32') / 255
x_test = x_test.reshape(10000, 28,28, 1).astype('float32') / 255
y_train = y_train.astype('float32')
y_test = y_test.astype('float32')

def create_represent_data(data):
  def data_gen():
    for i in data:
      yield [list([i])]
  return data_gen
converter.representative_dataset = create_represent_data(x_train[:5000])
"""

converter.representative_dataset = representative_dataset_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8  # or tf.int8
converter.inference_output_type = tf.uint8  # or tf.int8
tflite_quant_model = converter.convert()

# Save the model.
with open('model.tflite', 'wb') as f:
  f.write(tflite_model)

[srjoglekar246]

@SukyoungCho How did you run export_tflite_graph_tf2.py? Note that the saved model directly from the model zoo will not convert to TFLite in a performant way (and won't quantize either), since it is not optimized for on-device inference.

In your demo_convert, how are you implementing preprocess?

[SukyoungCho]

@srjoglekar246 sorry for confusing. this is how i ran export_tflite_grpah_tf.py

CUDA_VISIBLE_DEVICES=1 python object_detection/export_tflite_graph_tf2.py \
    --pipeline_config_path /root/ecomfort/tf2_model_zoo/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8/pipeline.config \
    --trained_checkpoint_dir /root/ecomfort/tf2_model_zoo/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8/checkpoint \
    --output_directory /root/ecomfort/tf2_model_zoo/tflite/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8

Yes, I ran about 12000 steps of training and tried converting it to tflite and quantize it.

For preprocessing, image = preprocess(image) this is all i have as I just copied and pasted the code you provided above. Should i modify it? thanks

[SukyoungCho]

@srjoglekar246 Should I implement preprocess within the script? like this?

import tensorflow as tf

# Convert the model
converter = tf.lite.TFLiteConverter.from_saved_model("/root/ecomfort/tf2_model_zoo/tflite/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8/saved_model")
converter.optimizations = [tf.lite.Optimize.DEFAULT]

def preprocess(image,
                        height,
                        width,
                        central_fraction=0.875,
                        scope=None,
                        central_crop=True,
                        use_grayscale=False):
  with tf.name_scope(scope, 'eval_image', [image, height, width]):
    if image.dtype != tf.float32:
      image = tf.image.convert_image_dtype(image, dtype=tf.float32)
    if use_grayscale:
      image = tf.image.rgb_to_grayscale(image)
    # Crop the central region of the image with an area containing 87.5% of
    # the original image.
    if central_crop and central_fraction:
      image = tf.image.central_crop(image, central_fraction=central_fraction)

    if height and width:
      # Resize the image to the specified height and width.
      image = tf.expand_dims(image, 0)
      image = tf.image.resize_bilinear(image, [height, width],
                                       align_corners=False)
      image = tf.squeeze(image, [0])
    image = tf.subtract(image, 0.5)
    image = tf.multiply(image, 2.0)
    return image

def representative_dataset_gen():
  for i in range(100):
    #dir path where .jpg (image) files are
    image = tf.io.read_file(os.path.join("/root/ecomfort/data/valid_data/total/", image_files[i % NUM_FILES]))
    image = tf.compat.v1.image.decode_jpeg(image)
    image = preprocess(image)

    yield [image]

converter.representative_dataset = representative_dataset_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8  # or tf.int8
converter.inference_output_type = tf.uint8  # or tf.int8
tflite_quant_model = converter.convert()

# Save the model.
with open('model.tflite', 'wb') as f:
  f.write(tflite_model)

[srjoglekar246]

@SukyoungCho Yes.

[SukyoungCho]

@srjoglekar246 When I try to quantize the model, using the script above. It still gives me an error message below..

this is the exact script

import tensorflow as tf

# Convert the model
converter = tf.lite.TFLiteConverter.from_saved_model("/root/ecomfort/tf2_model_zoo/tflite/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8/saved_model")
converter.optimizations = [tf.lite.Optimize.DEFAULT]

def preprocess(image,
                        height,
                        width,
                        central_fraction=0.875,
                        scope=None,
                        central_crop=True,
                        use_grayscale=False):
  with tf.name_scope(scope, 'eval_image', [image, height, width]):
    if image.dtype != tf.float32:
      image = tf.image.convert_image_dtype(image, dtype=tf.float32)
    if use_grayscale:
      image = tf.image.rgb_to_grayscale(image)
    # Crop the central region of the image with an area containing 87.5% of
    # the original image.
    if central_crop and central_fraction:
      image = tf.image.central_crop(image, central_fraction=central_fraction)

    if height and width:
      # Resize the image to the specified height and width.
      image = tf.expand_dims(image, 0)
      image = tf.image.resize_bilinear(image, [height, width],
                                       align_corners=False)
      image = tf.squeeze(image, [0])
    image = tf.subtract(image, 0.5)
    image = tf.multiply(image, 2.0)
    return image

def representative_dataset_gen():
  for i in range(100):
    #dir path where .jpg (image) files are
    image = tf.io.read_file(os.path.join("/root/ecomfort/data/valid_data/total/", image_files[i % NUM_FILES]))
    image = tf.compat.v1.image.decode_jpeg(image)
    image = preprocess(image)

    yield [image]

converter.representative_dataset = representative_dataset_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8  # or tf.int8
converter.inference_output_type = tf.uint8  # or tf.int8
tflite_quant_model = converter.convert()

# Save the model.
with open('model.tflite', 'wb') as f:
  f.write(tflite_model)

[srjoglekar246]

This seems like an error with post-training quantization. Adding @jianlijianli who might be able to help.

[srjoglekar246]

@SukyoungCho Can you confirm if conversion to float works?

[SukyoungCho]

@SukyoungCho Can you confirm if conversion to float works?

I remember it worked before. But, I will try it again and update it as soon as possible.

[SukyoungCho]

@srjoglekar246 Hi, when I try conversion to float 32 or float 16, the code does not show any error and gives out a .tflite model file, but it is suspicious. The model produced by step 1 was about 7mb. However, the .tflite file from step 2 is only about 560 bytes - less than 1 kb. Here I attach the model resulted from the step 1 (https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_on_mobile_tf2.md) and the result from the step 2 (.tflite conversion to float 32).

ssd_mobilenet_v2_fpnlite_320.zip

Step 1
Step 1: Export TFLite inference graph
This step generates an intermediate SavedModel that can be used with the TFLite Converter via commandline or Python API.

To use the script:

# From the tensorflow/models/research/ directory
python object_detection/export_tflite_graph_tf2.py \
    --pipeline_config_path path/to/ssd_model/pipeline.config \
    --trained_checkpoint_dir path/to/ssd_model/checkpoint \
    --output_directory path/to/exported_model_directory

[mpa74]

@srjoglekar246 Hi, when I try conversion to float 32 or float 16, the code does not show any error and gives out a .tflite model file, but it is suspicious. The model produced by step 1 was about 7mb. However, the .tflite file from step 2 is only about 560 bytes - less than 1 kb. Here I attach the model resulted from the step 1 (https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_on_mobile_tf2.md) and the result from the step 2 (.tflite conversion to float 32).

ssd_mobilenet_v2_fpnlite_320.zip

Step 1
Step 1: Export TFLite inference graph
This step generates an intermediate SavedModel that can be used with the TFLite Converter via commandline or Python API.

To use the script:

# From the tensorflow/models/research/ directory
python object_detection/export_tflite_graph_tf2.py \
    --pipeline_config_path path/to/ssd_model/pipeline.config \
    --trained_checkpoint_dir path/to/ssd_model/checkpoint \
    --output_directory path/to/exported_model_directory

Hi! Try this solution (https://github.com/tensorflow/models/issues/9394#issuecomment-713143622) It solve problem for me (and converting to INT too)

[SukyoungCho]

@srjoglekar246 Hi, when I try conversion to float 32 or float 16, the code does not show any error and gives out a .tflite model file, but it is suspicious. The model produced by step 1 was about 7mb. However, the .tflite file from step 2 is only about 560 bytes - less than 1 kb. Here I attach the model resulted from the step 1 (https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_on_mobile_tf2.md) and the result from the step 2 (.tflite conversion to float 32).
ssd_mobilenet_v2_fpnlite_320.zip
Step 1
Step 1: Export TFLite inference graph
This step generates an intermediate SavedModel that can be used with the TFLite Converter via commandline or Python API.
To use the script:

# From the tensorflow/models/research/ directory
python object_detection/export_tflite_graph_tf2.py \
    --pipeline_config_path path/to/ssd_model/pipeline.config \
    --trained_checkpoint_dir path/to/ssd_model/checkpoint \
    --output_directory path/to/exported_model_directory

Hi! Try this solution (#9394 (comment)) It solve problem for me (and converting to INT too)

Hi @mpa74, thank you so much for your help. I have tried the solution you offered and I was able to get .tflite file with seemingly legit size. However, I was not able to quantize it into uint8 format.

In addition, when I perform step 1 and step 2, i am getting lots of error messages although it is giving out the converted file. So, I am not sure if they are converted properly. Could you let me know if you have experienced the same error message? Thank you so much.

Error Message during step 1 (No Concrete functions found for untraced function)

W1022 17:38:15.026157 140288236451648 save.py:228] No concrete functions found for untraced function `projection_1_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.026237 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_fn` while saving. This function will not be callable after loading.
W1022 17:38:15.026307 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.026376 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.026456 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_fn` while saving. This function will not be callable after loading.
W1022 17:38:15.026524 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.026593 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.032178 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_2_layer_call_fn` while saving. This function will not be callable after loading.
W1022 17:38:15.032264 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_2_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.032336 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_2_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.037694 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_1_layer_call_fn` while saving. This function will not be callable after loading.
W1022 17:38:15.037776 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_1_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.037863 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_1_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
INFO:tensorflow:Unsupported signature for serialization: (([(<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec47b8>, TensorSpec(shape=(None, 40, 40, 32), dtype=tf.float32, name='image_features/0/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec48d0>, TensorSpec(shape=(None, 20, 20, 96), dtype=tf.float32, name='image_features/1/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec45c0>, TensorSpec(shape=(None, 10, 10, 1280), dtype=tf.float32, name='image_features/2/1'))], False), {}).
I1022 17:38:18.807447 140288236451648 def_function.py:1170] Unsupported signature for serialization: (([(<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec47b8>, TensorSpec(shape=(None, 40, 40, 32), dtype=tf.float32, name='image_features/0/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec48d0>, TensorSpec(shape=(None, 20, 20, 96), dtype=tf.float32, name='image_features/1/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec45c0>, TensorSpec(shape=(None, 10, 10, 1280), dtype=tf.float32, name='image_features/2/1'))], False), {}).
INFO:tensorflow:Unsupported signature for serialization: (([(<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300f32cf8>, TensorSpec(shape=(None, 40, 40, 32), dtype=tf.float32, name='image_features/0/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300f32d30>, TensorSpec(shape=(None, 20, 20, 96), dtype=tf.float32, name='image_features/1/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300f32eb8>, TensorSpec(shape=(None, 10, 10, 1280), dtype=tf.float32, name='image_features/2/1'))], True), {}).

Errors during the step 2

WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
2020-10-22 17:43:38.769464: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:316] Ignored output_format.
2020-10-22 17:43:38.769521: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:319] Ignored drop_control_dependency.
2020-10-22 17:43:38.769530: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:325] Ignored change_concat_input_ranges.
2020-10-22 17:43:38.770421: I tensorflow/cc/saved_model/reader.cc:32] Reading SavedModel from: ./saved_model
2020-10-22 17:43:38.838857: I tensorflow/cc/saved_model/reader.cc:55] Reading meta graph with tags { serve }
2020-10-22 17:43:38.838917: I tensorflow/cc/saved_model/reader.cc:93] Reading SavedModel debug info (if present) from: ./saved_model
2020-10-22 17:43:38.838986: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2020-10-22 17:43:38.839010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-10-22 17:43:38.839019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]
2020-10-22 17:43:39.042039: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2020-10-22 17:43:39.095908: I tensorflow/cc/saved_model/loader.cc:206] Restoring SavedModel bundle.
2020-10-22 17:43:39.154998: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 3591830000 Hz
2020-10-22 17:43:39.651706: I tensorflow/cc/saved_model/loader.cc:190] Running initialization op on SavedModel bundle at path: ./saved_model
2020-10-22 17:43:39.884801: I tensorflow/cc/saved_model/loader.cc:277] SavedModel load for tags { serve }; Status: success: OK. Took 1114382 microseconds.
2020-10-22 17:43:40.914707: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:194] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
2020-10-22 17:43:41.582975: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2020-10-22 17:43:41.585994: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:0a:00.0 name: TITAN X (Pascal) computeCapability: 6.1
coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s
2020-10-22 17:43:41.586061: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2020-10-22 17:43:41.586133: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2020-10-22 17:43:41.586188: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2020-10-22 17:43:41.586241: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2020-10-22 17:43:41.586310: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2020-10-22 17:43:41.586499: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2020-10-22 17:43:41.586580: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2020-10-22 17:43:41.586641: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2020-10-22 17:43:41.591077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2020-10-22 17:43:41.591123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-10-22 17:43:41.591139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
2020-10-22 17:43:41.591154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
2020-10-22 17:43:41.595432: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11217 MB memory) -> physical GPU (device: 0, name: TITAN X (Pascal), pci bus id: 0000:0a:00.0, compute capability: 6.1)
Exception ignored in: <bound method Buckets.__del__ of <tensorflow.python.eager.monitoring.ExponentialBuckets object at 0x7f6e7542dc48>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/monitoring.py", line 407, in __del__
AttributeError: 'NoneType' object has no attribute 'TFE_MonitoringDeleteBuckets'
root@9e19458b0d02:/workspace#

[mpa74]

@srjoglekar246 Hi, when I try conversion to float 32 or float 16, the code does not show any error and gives out a .tflite model file, but it is suspicious. The model produced by step 1 was about 7mb. However, the .tflite file from step 2 is only about 560 bytes - less than 1 kb. Here I attach the model resulted from the step 1 (https://github.com/tensorflow/models/blob/master/research/object_detection/g3doc/running_on_mobile_tf2.md) and the result from the step 2 (.tflite conversion to float 32).
ssd_mobilenet_v2_fpnlite_320.zip
Step 1
Step 1: Export TFLite inference graph
This step generates an intermediate SavedModel that can be used with the TFLite Converter via commandline or Python API.
To use the script:

# From the tensorflow/models/research/ directory
python object_detection/export_tflite_graph_tf2.py \
    --pipeline_config_path path/to/ssd_model/pipeline.config \
    --trained_checkpoint_dir path/to/ssd_model/checkpoint \
    --output_directory path/to/exported_model_directory

Hi! Try this solution (#9394 (comment)) It solve problem for me (and converting to INT too)

Hi @mpa74, thank you so much for your help. I have tried the solution you offered and I was able to get .tflite file with seemingly legit size. However, I was not able to quantize it into uint8 format.

In addition, when I perform step 1 and step 2, i am getting lots of error messages although it is giving out the converted file. So, I am not sure if they are converted properly. Could you let me know if you have experienced the same error message? Thank you so much.

Error Message during step 1 (No Concrete functions found for untraced function)

W1022 17:38:15.026157 140288236451648 save.py:228] No concrete functions found for untraced function `projection_1_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.026237 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_fn` while saving. This function will not be callable after loading.
W1022 17:38:15.026307 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.026376 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.026456 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_fn` while saving. This function will not be callable after loading.
W1022 17:38:15.026524 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.026593 140288236451648 save.py:228] No concrete functions found for untraced function `nearest_neighbor_upsampling_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.032178 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_2_layer_call_fn` while saving. This function will not be callable after loading.
W1022 17:38:15.032264 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_2_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.032336 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_2_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.037694 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_1_layer_call_fn` while saving. This function will not be callable after loading.
W1022 17:38:15.037776 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_1_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
W1022 17:38:15.037863 140288236451648 save.py:228] No concrete functions found for untraced function `smoothing_1_layer_call_and_return_conditional_losses` while saving. This function will not be callable after loading.
INFO:tensorflow:Unsupported signature for serialization: (([(<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec47b8>, TensorSpec(shape=(None, 40, 40, 32), dtype=tf.float32, name='image_features/0/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec48d0>, TensorSpec(shape=(None, 20, 20, 96), dtype=tf.float32, name='image_features/1/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec45c0>, TensorSpec(shape=(None, 10, 10, 1280), dtype=tf.float32, name='image_features/2/1'))], False), {}).
I1022 17:38:18.807447 140288236451648 def_function.py:1170] Unsupported signature for serialization: (([(<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec47b8>, TensorSpec(shape=(None, 40, 40, 32), dtype=tf.float32, name='image_features/0/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec48d0>, TensorSpec(shape=(None, 20, 20, 96), dtype=tf.float32, name='image_features/1/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300ec45c0>, TensorSpec(shape=(None, 10, 10, 1280), dtype=tf.float32, name='image_features/2/1'))], False), {}).
INFO:tensorflow:Unsupported signature for serialization: (([(<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300f32cf8>, TensorSpec(shape=(None, 40, 40, 32), dtype=tf.float32, name='image_features/0/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300f32d30>, TensorSpec(shape=(None, 20, 20, 96), dtype=tf.float32, name='image_features/1/1')), (<tensorflow.python.framework.func_graph.UnknownArgument object at 0x7f9300f32eb8>, TensorSpec(shape=(None, 10, 10, 1280), dtype=tf.float32, name='image_features/2/1'))], True), {}).

Errors during the step 2

WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
WARNING:absl:Could not find any concrete functions to restore for this SavedFunction object while loading. The function will not be callable.
2020-10-22 17:43:38.769464: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:316] Ignored output_format.
2020-10-22 17:43:38.769521: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:319] Ignored drop_control_dependency.
2020-10-22 17:43:38.769530: W tensorflow/compiler/mlir/lite/python/tf_tfl_flatbuffer_helpers.cc:325] Ignored change_concat_input_ranges.
2020-10-22 17:43:38.770421: I tensorflow/cc/saved_model/reader.cc:32] Reading SavedModel from: ./saved_model
2020-10-22 17:43:38.838857: I tensorflow/cc/saved_model/reader.cc:55] Reading meta graph with tags { serve }
2020-10-22 17:43:38.838917: I tensorflow/cc/saved_model/reader.cc:93] Reading SavedModel debug info (if present) from: ./saved_model
2020-10-22 17:43:38.838986: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2020-10-22 17:43:38.839010: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-10-22 17:43:38.839019: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]
2020-10-22 17:43:39.042039: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:196] None of the MLIR optimization passes are enabled (registered 0 passes)
2020-10-22 17:43:39.095908: I tensorflow/cc/saved_model/loader.cc:206] Restoring SavedModel bundle.
2020-10-22 17:43:39.154998: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 3591830000 Hz
2020-10-22 17:43:39.651706: I tensorflow/cc/saved_model/loader.cc:190] Running initialization op on SavedModel bundle at path: ./saved_model
2020-10-22 17:43:39.884801: I tensorflow/cc/saved_model/loader.cc:277] SavedModel load for tags { serve }; Status: success: OK. Took 1114382 microseconds.
2020-10-22 17:43:40.914707: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:194] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
2020-10-22 17:43:41.582975: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2020-10-22 17:43:41.585994: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties:
pciBusID: 0000:0a:00.0 name: TITAN X (Pascal) computeCapability: 6.1
coreClock: 1.531GHz coreCount: 28 deviceMemorySize: 11.91GiB deviceMemoryBandwidth: 447.48GiB/s
2020-10-22 17:43:41.586061: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2020-10-22 17:43:41.586133: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2020-10-22 17:43:41.586188: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2020-10-22 17:43:41.586241: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2020-10-22 17:43:41.586310: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2020-10-22 17:43:41.586499: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2020-10-22 17:43:41.586580: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11
2020-10-22 17:43:41.586641: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8
2020-10-22 17:43:41.591077: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1862] Adding visible gpu devices: 0
2020-10-22 17:43:41.591123: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-10-22 17:43:41.591139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      0
2020-10-22 17:43:41.591154: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1280] 0:   N
2020-10-22 17:43:41.595432: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1406] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 11217 MB memory) -> physical GPU (device: 0, name: TITAN X (Pascal), pci bus id: 0000:0a:00.0, compute capability: 6.1)
Exception ignored in: <bound method Buckets.__del__ of <tensorflow.python.eager.monitoring.ExponentialBuckets object at 0x7f6e7542dc48>>
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/eager/monitoring.py", line 407, in __del__
AttributeError: 'NoneType' object has no attribute 'TFE_MonitoringDeleteBuckets'
root@9e19458b0d02:/workspace#

Hi @SukyoungCho ! Looks like I have similar warnings on step 1, but error with ‘NoneType’ object I have never received. Sorry, I don’t know what to do with this error

[srjoglekar246]

@mpa74 If a SavedModel is getting generated by step 1, the logs just seem to be warnings based on APIs etc. You can ignore them

For step 2, you also need this for conversion:

converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8, tf.lite.OpsSet.TFLITE_BUILTINS]

The final post-processing op needs to run in float, there is no quantized version for it.

[SukyoungCho]

@srjoglekar246
Hi, I do appreciate for your help! It's been a while, but I'm still struggling from exporting a model into uint8 quantized .tflite model. In the meantime, I have tried to convert not only SSD models but also EfficientDet and CenterNet, but all of them failed.

If you don't mind, could you elaborate more on what do you mean by "The final post-processing op needs to run in float, there is no quantized version for it." this? It sounds confusing to me, since the models from TF1 OD API are eligible for uint8 quantization, tflite conversion, and EdgeTPU compile while they have post processing ops on them. Is it due to a difference in tflite conversion processes for TF1 and TF2?

To be honest, the reason why I was looking for EfficientDet uint8 was to deploy it on a board with coral accelerator. I have been using ssd mobiledet_dsp_coco from TF1 OD API, but its accuracy was not satisfactory (while i was only detecting 'person' the AP was about 38 and had way too many false positive). However, I found out that on the same image set, CenterNet and EfficientDet showed extremely low false positive rate (I ran an inference on the server for this not on the board).

So it would be very appreciated if you could manage to release uint8 quantization for EfficientDet or CenterNet so that I can EdgeTPU compile it and deploy it on a board! Or, if you could tell me if there are any models better than SSD Mobiledet that can be quantized into uint8, or EfficientDet for TF1. Thank you so much!

[srjoglekar246]

It looks like the Coral team has uncovered some issues with quantization of detection models. Will ping back once I have updates.

[SukyoungCho]

@mpa74 If a SavedModel is getting generated by step 1, the logs just seem to be warnings based on APIs etc. You can ignore them

For step 2, you also need this for conversion:

converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8, tf.lite.OpsSet.TFLITE_BUILTINS]

The final post-processing op needs to run in float, there is no quantized version for it.

@srjoglekar246 Thank you! and it would be appreciated if you could explain what do you mean by this!

[srjoglekar246]

@SukyoungCho So the basic idea is that multi-class Non-Max Suppression for SSD is written as a 'custom' op in TFLite. This is because if we try to directly convert TF's logic for multi-class NMS, its too complex to be efficient on mobile.
This custom op is what takes in boxes/anchors/etc & returns the final (post-processed) boxes, classes, scores, etc from the model.
The logic performed by this op isn't really implemented on most of our delegates (& might not even make sense on backends like the GPU), so it doesn't run on them. Even for quantized models from TF1, the heavy backbone of the model (All conv/depthwise conv ops etc) runs in quantized mode (possibly on GPU/DSP etc), but the post-processing runs on the CPU, since the custom SSD op is only implemented on CPU.

[PINTO0309]

The conversion was successfully completed in about 30 minutes without any problems. Unless you introduce tf-nightly, it cannot be made to work properly at present. There is no difference in behavior between tf.int8 and tf.uint8.

I hope this is helpful to you all.

### Uninstalling a legitimate version of TensorFlow
$ sudo pip3 uninstall tensorboard-plugin-wit tb-nightly tensorboard \
                      tf-estimator-nightly tensorflow-gpu \
                      tensorflow tf-nightly tensorflow_estimator

```bash

Installation of tf-nightly

$ sudo pip3 install tf-nightly

```bash
### export 
$ export PYTHONPATH=$PYTHONPATH:${HOME}/git/models
$ cd ${HOME}/git/models/research

$ python3 object_detection/export_tflite_graph_tf2.py \
  --pipeline_config_path ${HOME}/Downloads/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8/pipeline.config \
  --trained_checkpoint_dir ${HOME}/Downloads/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8/checkpoint \
  --output_directory ${HOME}/Downloads/ssd_mobilenet_v2_fpnlite_320x320_coco17_tpu-8/export_tflite_graph

### saved_model to tflite
### tf-nightly==2.5.0-dev20201128
import tensorflow as tf
import tensorflow_datasets as tfds
import numpy as np

def representative_dataset_gen():
  for data in raw_test_data.take(10):
    image = data['image'].numpy()
    image = tf.image.resize(image, (320, 320))
    image = image[np.newaxis,:,:,:]
    image = image - 127.5
    image = image * 0.007843
    yield [image]

raw_test_data, info = tfds.load(name="coco/2017", with_info=True, split="test", data_dir="~/TFDS", download=False)

converter = tf.lite.TFLiteConverter.from_saved_model('export_tflite_graph/saved_model')
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS]
tflite_model = converter.convert()
with open('{}/model_float32.tflite'.format('export_tflite_graph/saved_model'), 'wb') as w:
    w.write(tflite_model)
print("tflite convert complete! - {}/model_float32.tflite".format('export_tflite_graph/saved_model'))


converter = tf.lite.TFLiteConverter.from_saved_model('export_tflite_graph/saved_model')
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8, tf.lite.OpsSet.SELECT_TF_OPS]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8
# converter.inference_input_type = tf.uint8
# converter.inference_output_type = tf.uint8
converter.allow_custom_ops = True
converter.representative_dataset = representative_dataset_gen
tflite_model = converter.convert()
with open('{}/model_full_integer_quant.tflite'.format('export_tflite_graph/saved_model'), 'wb') as w:
    w.write(tflite_model)
print("tflite convert complete! - {}/model_full_integer_quant.tflite".format('export_tflite_graph/saved_model'))

$ edgetpu_compiler -s export_tflite_graph/saved_model/model_full_integer_quant.tflite

Edge TPU Compiler version 15.0.340273435
Input: export_tflite_graph/saved_model/model_full_integer_quant.tflite
Output: model_full_integer_quant_edgetpu.tflite

Operator                       Count      Status

PACK                           4          Tensor has unsupported rank (up to 3 innermost dimensions mapped)
RESHAPE                        6          Mapped to Edge TPU
RESHAPE                        4          More than one subgraph is not supported
RESHAPE                        2          Operation is otherwise supported, but not mapped due to some unspecified limitation
CONCATENATION                  1          Operation is otherwise supported, but not mapped due to some unspecified limitation
CONCATENATION                  1          More than one subgraph is not supported
LOGISTIC                       1          Operation is otherwise supported, but not mapped due to some unspecified limitation
CUSTOM                         1          Operation is working on an unsupported data type
ADD                            10         Mapped to Edge TPU
ADD                            2          More than one subgraph is not supported
CONV_2D                        14         More than one subgraph is not supported
CONV_2D                        58         Mapped to Edge TPU
DEQUANTIZE                     1          Operation is otherwise supported, but not mapped due to some unspecified limitation
DEQUANTIZE                     1          Operation is working on an unsupported data type
QUANTIZE                       4          More than one subgraph is not supported
QUANTIZE                       6          Mapped to Edge TPU
DEPTHWISE_CONV_2D              14         More than one subgraph is not supported
DEPTHWISE_CONV_2D              37         Mapped to Edge TPU

[PINTO0309]

Note that SSD MobileNet v2 320x320 needed to be fixed due to a strange configuration in pipeline.config.

• SSD MobileNet v2 320x320
  http://download.tensorflow.org/models/object_detection/tf2/20200711/ssd_mobilenet_v2_320x320_coco17_tpu-8.tar.gz

model {
  ssd {
    num_classes: 90
    image_resizer {
      fixed_shape_resizer {
        height: 300
        width: 300
      }
    }
    feature_extractor {
      type: "ssd_mobilenet_v2_keras"
      depth_multiplier: 1.0

to

model {
  ssd {
    num_classes: 90
    image_resizer {
      fixed_shape_resizer {
        height: 320
        width: 320
      }
    }
    feature_extractor {
      type: "ssd_mobilenet_v2_keras"
      depth_multiplier: 1.0

Model successfully compiled but not all operations are supported by the Edge TPU. A percentage of the model will instead run on the CPU, which is slower. If possible, consider updating your model to use only operations supported by the Edge TPU. For details, visit g.co/coral/model-reqs.
Number of operations that will run on Edge TPU: 109
Number of operations that will run on CPU: 3

Operator                       Count      Status

CUSTOM                         1          Operation is working on an unsupported data type
ADD                            10         Mapped to Edge TPU
CONCATENATION                  2          Mapped to Edge TPU
RESHAPE                        13         Mapped to Edge TPU
LOGISTIC                       1          Mapped to Edge TPU
DEPTHWISE_CONV_2D              17         Mapped to Edge TPU
QUANTIZE                       11         Mapped to Edge TPU
CONV_2D                        55         Mapped to Edge TPU
DEQUANTIZE                     2          Operation is working on an unsupported data type

[ChowderII]

@PINTO0309
Hi ! Thank you for the detailed post! I used the exact same model as you and my version of tensorflow is : tf_nightly-2.5.0.dev20201202.

But I get the following error when I convert with the quantization:

Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2020.1\plugins\python-ce\helpers\pydev\pydevd.py", line 1438, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm Community Edition 2020.1\plugins\python-ce\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/brackman/Documents/Tensorflow/models/research/object_detection/ssd_mobilenet_quant/convertion.py", line 39, in <module>
    tflite_model = converter.convert()
  File "C:\Users\brackman\Anaconda3\envs\tf-nightly\lib\site-packages\tensorflow\lite\python\lite.py", line 767, in convert
    result = _modify_model_io_type(result, **flags_modify_model_io_type)
  File "C:\Users\brackman\Anaconda3\envs\tf-nightly\lib\site-packages\tensorflow\lite\python\util.py", line 853, in modify_model_io_type
    model_object = _convert_model_from_bytearray_to_object(model)
  File "C:\Users\brackman\Anaconda3\envs\tf-nightly\lib\site-packages\tensorflow\lite\python\util.py", line 579, in _convert_model_from_bytearray_to_object
    model_object = schema_fb.Model.GetRootAsModel(model_bytearray, 0)
AttributeError: module 'tensorflow.lite.python.schema_py_generated' has no attribute 'Model'

When I look at this error it points me to #41846

Here is my code if you want to see if I did anything wrong

import tensorflow as tf
import numpy as np


data_dir = "C:/Users/brackman/Documents/Tensorflow/workspace/IVP_V4/data/test"
saved_model_dir="C:/Users/brackman/Documents/Tensorflow/models/research/object_detection/ssd_mobilenet_quant/tflite/saved_model"


def representative_dataset_gen():
    import glob
    from PIL import Image
    image_list = glob.glob("C:/Users/brackman/Documents/Tensorflow/workspace/IVP_V4/data/test/*.png")

    for data in image_list:
        image = np.asarray(Image.open(data))
        image = tf.image.resize(image, (320, 320))
        image = image[np.newaxis, :, :, :]
        image = image - 127.5
        image = image * 0.007843
        yield [image]


converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS, tf.lite.OpsSet.SELECT_TF_OPS]
tflite_model = converter.convert()

with open('{}/model_float32.tflite'.format('export_tflite_graph/saved_model'), 'wb') as w:
    w.write(tflite_model)
print("tflite convert complete! - {}/model_float32.tflite".format('export_tflite_graph/saved_model'))

converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8, tf.lite.OpsSet.SELECT_TF_OPS]
converter.inference_input_type = tf.int8
converter.inference_output_type = tf.int8
converter.allow_custom_ops = True
converter.representative_dataset = representative_dataset_gen
tflite_model = converter.convert()
with open('{}/model_full_integer_quant.tflite'.format('export_tflite_graph/saved_model'), 'wb') as w:
    w.write(tflite_model)
print("tflite convert complete! - {}/model_full_integer_quant.tflite".format('export_tflite_graph/saved_model'))

[NobuoTsukamoto]

@ChowderII
I created a notebook that converts v2 models to tflite model that works with Google Colab.
Here is link (Click ”Open in Colab” and run all)
https://gist.github.com/NobuoTsukamoto/f48df315be490dcf0c76375c2e04ddb1#file-export_tfv2_lite_models-ipynb

There were no errors in Colab.

(See below for uploading and downloading files to Colab.
https://colab.research.google.com/notebooks/io.ipynb)

[aeozyalcin]

@ChowderII can you check to see if "C:\Users\brackman\Anaconda3envstf-nightly\lib\site-packages\tensorflow\lite\python\schema_py_generated.py" file is empty?

I was getting the same error you were getting, and it turns out tensorflow has some sort of a bug, where schema_py_generated.py is empty with a pip installation.

[ChowderII]

@aeozyalcin Yes it is empty! How odd ...

Also, @NobuoTsukamoto thank you so much for this, I will upload my dataset and my model there first thing tomorrow and try it!

[aeozyalcin]

@ChowderII

See this post. I replaced my empty file with the one attached to this comment, and was able to convert quantized tflite models that way.

https://github.com/tensorflow/tensorflow/issues/44882#issuecomment-729274083

[PINTO0309]

It's definitely a problem that only reproduces in a Windows environment. If you look at the contents of .whl, it is obvious.

• Image above：tf_nightly-2.5.0.dev20201202-cp36-cp36m-manylinux2010_x86_64.whl
• Image below：tf_nightly-2.5.0.dev20201202-cp37-cp37m-win_amd64.whl

[ChowderII]

@aeozyalcin Your solution worked! Omg I'm so happy!
@NobuoTsukamoto Thanks for your solution, I'm sure yours is going to work as well but The other one was a lot simpler for my use case!

[tucachmo2202]

@ChowderII
I created a notebook that converts v2 models to tflite model that works with Google Colab.
Here is link (Click ”Open in Colab” and run all)
https://gist.github.com/NobuoTsukamoto/f48df315be490dcf0c76375c2e04ddb1#file-export_tfv2_lite_models-ipynb

There were no errors in Colab.

(See below for uploading and downloading files to Colab.
https://colab.research.google.com/notebooks/io.ipynb)

Thank for sharing. However, I have a question that I see you set converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8, tf.lite.OpsSet.TFLITE_BUILTINS], means when an op not supported for integer quantization implementation, the converter will use tflite-builtin op instead? So how does it run on device only using integer op??

[srjoglekar246]

@tucachmo2202 SSD models typically do not run only in integer format. The post-processing (NMS) step works best in float, so only the heavy Convolutional backbone (for eg, MobileNet) is run quantized.