Edgetpu: Quantize node not being converted

[mrharicot]

Hi,

I am trying to convert a simple Keras model from TF 2.0.

I follwed the "Full integer quantization of weights and activations" tutorial from the doc

def representative_dataset_gen():
    for i in range(100):
        yield [x_train[i, None]]

converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.optimizations = [tf.lite.Optimize.DEFAULT]
converter.representative_dataset = representative_dataset_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8

The conversion goes well and results in the following graph in Netron:

The edgetpu compilation is successful but states that the quantize nodes will be mapped to the CPU.

Edge TPU Compiler version 2.0.267685300
Input: /mnt/localtmp/model.tflite
Output: /mnt/localtmp/model_edgetpu.tflite

Operator                       Count      Status

SOFTMAX                        1          Mapped to Edge TPU
FULLY_CONNECTED                1          Mapped to Edge TPU
QUANTIZE                       1          Operation is otherwise supported, but not mapped due to some unspecified limitation
CONV_2D                        4          Mapped to Edge TPU
DEQUANTIZE                     1          Operation is working on an unsupported data type

However if I try to run this model in python it either throws Cannot cast array data from dtype('float32') to dtype('uint8') according to the rule 'safe' for float input or segfaults for uint8 input.

I see that all the test models do not have a quantization node but instead have a quantization information on their input nodes (seen in Netron) quantization: -1 ≤ 0.0078125 * (q - 128) ≤ 0.9921875.

Am I doing something wrong?
Is the edgetpu_compiler compatible with post training quantization from TF 2.0?

[Namburger]

@mrharicot it should be compatible, however, I know the previous version of the compiler has a little issue with keras models. Are you running it with the edgetpu api?
Could you attach your model, both pre/post compiled? We'll have to take a look at the model.

[DocDriven]

@mrharicot
With the release of TF 2.0, the attributes inference_input_type and inference_output_type have been removed (https://www.tensorflow.org/lite/convert/python_api). It seems tflite models can only take floats, and convert them to uint8s internally.

@Namburger
Maybe you can provide further details. Personally, I run into similar problems when using TF 2.0. It seems that your functions (especially RunInference in model_utils.cc) expect uint8 data, but as soon as they try to set the input tensors, an error is thrown due to the model expecting float data types.

If this is a known problem, I could create a pull request that addresses this issue.

[Namburger]

@DocDriven We do need the tensor to be int8 data type to be compatible with the tpu.
@mrharicot I just confirmed with the team, unfortunately TF2.0 is non compatible with our edgetpu as of now :/

[xadrianzetx]

@mrharicot, for now possible workaround for this issue is to use tf.compat.v1.lite.TFLiteConverter.from_keras_model_file()

I had success compiling models build with TF 2.0 Keras API to run fully on edgetpu with INT8 input and output.

[Namburger]

@xadrianzetx
did you have to revert back to tf1.15 to use from_keras_model_file()?
I thought it's deprecated. FYI:
https://gist.github.com/Namburger/4cc57ab3cbe4bf5abbc1d3e84536b75c

[xadrianzetx]

@Namburger no, I did not. Here's colab demo

[Namburger]

Nice, it's because of *.compat.v1.*
Thanks!

[mrharicot]

@xadrianzetx Thanks! I can't believe that simply using the v1 compat function but keeping the v2 semantics worked. I had tried one or the other, not mixing the two :D

[mpariente]

So, is this solved? I have a similar problem even if I work 100% in tf==1.15.

[feranick]

I am encountering a very related issue. Following the same example, but applied to my data have issues when I identify the input_data for the interpreter as float32 (below my data in R is in float32 already normalized between [0,10]):

input_data = np.array(R, dtype=np.float32)
interpreter.set_tensor(input_details[0]['index'], input_data)
interpreter.invoke()

When then I call the TFLite converter (using tf.compat.v1.lite.TFLiteConverter.from_keras_model_file in TF2.x or tf.lite.TFLiteConverter.from_keras_model_file in TF1.x) again following from th edocumentation:

converter.optimizations = [tf.lite.Optimize.OPTIMIZE_FOR_SIZE]
converter.representative_dataset = representative_dataset_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
tflite_quant_model = converter.convert()

The post-quantized tflite is correctly created (and when compiled for the TPU, all ops are correctly assigned). However, when I run a prediction, I get the following error:
ValueError: Cannot set tensor: Got value of type FLOAT32 but expected type UINT8 for input 16, name: conv2d_input

Essentially, the input_data = np.array(R, dtype=np.float32), needs to be set as uint8 for this to work. However, that breaks the model completely, as it casts all data between [0,1] into just zeros.

To be clear, this does NOT happen when using TF2.x, by calling the converter with tf.lite.TFLiteConverter.from_keras_model(). Using the input_data = np.array(R, dtype=np.float32), the post-training quantized tflite model is created and when used for predictions, it works as intended. However, when compiled, because I am using the TFLiteConverterV2 , the quantization/dequantization ops are not assigned to the TPU.

Any insight is greatly appreciated. Thanks in advance.

[feranick]

I managed to reproduce the same issue with a slight rearrangement of the code in the documentation. Please see the attachment (it's obviously python3 code, despite the txt extension to allow upload).

When line 52 is uncommented and line 55 is commented the code works.
When line 52 is commented and line 55 is uncommented, the code does not work and gives the same error message in the previous comment:

ValueError: Cannot set tensor: Got value of type FLOAT32 but expected type UINT8 for input 11, name: reshape_input

test_issue_13.txt

[feranick]

The only way I found this to work is to make sure the input during prediction is formatted as uint8.

[DLMasterCat]

@xadrianzetx
Do you mean that you train on tf2.0 and under tf-nightly 2.1.0-dev20191110, you convert it with compat.v1.? I saw your colab demo, and datatype is tf.uint8. Thus I want to know if tf.int8 is still OK.
Thanks!!

[xadrianzetx]

@DLMasterCat

Don't mind nightly version of tf, I had to use it at time, since ResizeNearestNeighbor op (used in tf.keras.layers.UpSampling2D) was missing from 2.0 release. Since then, ops set has been updated.

As for your question, Edge TPU docs states, that unsigned int I/O is required, but I tested it out with int8 and to my surprise, MobileNetV2 compiles without a problem. Did not test it on Edge TPU though, so I would not be surprised if model outputs did not make any sense.

[Namburger]

@xadrianzetx actually, maybe you could give the float i/o model a try?
ensure to apply the preprocessing for your inputs first before feeding it into the model, similar to this test.
To clarify, the compiler now supports float i/o, our library doesn't fully support it yet (only the c++) and none of our examples show it. But techniquely, you should be able to use it!

[xadrianzetx]

@Namburger

Gave it a quick try, model compiles without a problem with float i/o (Edge TPU Compiler version 2.1.302470888) which makes compat.v1 workaround obsolete. Also managed to run this model with tflite_runtime-2.1.0 and it works perfectly. Try it, I've prepared a demo.

[Namburger]

@xadrianzetx wow that's really kool, looks like you've set up a server on google doc for us to request the model from anywhere? (Haven't tested yet, just reading the code).
Just FYI: tf2.* should be expected to work with our library soon!

[xadrianzetx]

@Namburger Yeah, code pulls model and example .jpg from google drive. Both assets come from one of my projects, but model has been recompiled with float i/o. Should be platform independent too.