Edgetpu: Much larger output from edgetpu-compiler v.14.1 (compared to 2.x)
I have a tflite model:
https://www.dropbox.com/s/1oglpyv5gshcwz9/model_classifier_CNN.tflite?dl=0
that when converted for use with the edgetpu_compiler v. 2.0.291256449 produces a file with this memory profile:
Edge TPU Compiler version 2.0.291256449
Model compiled successfully in 39 ms.
Input model: model_classifier_CNN.tflite
Input size: 692.70KiB
Output model: model_classifier_CNN_edgetpu.tflite
Output size: 908.54KiB
On-chip memory available for caching model parameters: 5.95MiB
On-chip memory used for caching model parameters: 891.50KiB
Off-chip memory used for streaming uncached model parameters: 0.00B
Number of Edge TPU subgraphs: 1
Total number of operations: 13
Operation log: model_classifier_CNN_edgetpu.log
Operator Count Status
SOFTMAX 1 Mapped to Edge TPU
FULLY_CONNECTED 2 Mapped to Edge TPU
MAX_POOL_2D 4 Mapped to Edge TPU
QUANTIZE 2 Mapped to Edge TPU
CONV_2D 3 Mapped to Edge TPU
DEPTHWISE_CONV_2D 1 Mapped to Edge TPU
However, when using the edgetpu_compiler v. 14.1, the output size is much larger (7.77MiB vs 908.54KiB) with Off chip memory usage that goes from 0.00B to 6.90MiB. This leads to a significant degradation in performance as Off-chip board needs to be constantly accessed.
Edge TPU Compiler version 14.1.317412892
Model compiled successfully in 183 ms.
Input model: model_classifier_CNN.tflite
Input size: 692.70KiB
Output model: model_classifier_CNN_edgetpu.tflite
Output size: 7.77MiB
On-chip memory used for caching model parameters: 870.00KiB
On-chip memory remaining for caching model parameters: 5.10MiB
Off-chip memory used for streaming uncached model parameters: 6.90MiB
Number of Edge TPU subgraphs: 1
Total number of operations: 13
Operation log: model_classifier_CNN_edgetpu.log
Operator Count Status
QUANTIZE 2 Mapped to Edge TPU
CONV_2D 3 Mapped to Edge TPU
DEPTHWISE_CONV_2D 1 Mapped to Edge TPU
FULLY_CONNECTED 2 Mapped to Edge TPU
SOFTMAX 1 Mapped to Edge TPU
MAX_POOL_2D 4 Mapped to Edge TPU
feranick
All 22 comments
Running with compiler 2.1.302470888 seems to work as expected (i.e. like in 2.0.291256449):
Edge TPU Compiler version 2.1.302470888
Model compiled successfully in 44 ms.
Input model: model_classifier_CNN.tflite
Input size: 692.70KiB
Output model: model_classifier_CNN_edgetpu.tflite
Output size: 908.53KiB
On-chip memory used for caching model parameters: 891.50KiB
On-chip memory remaining for caching model parameters: 5.07MiB
Off-chip memory used for streaming uncached model parameters: 0.00B
Number of Edge TPU subgraphs: 1
Total number of operations: 13
Operation log: model_classifier_CNN_edgetpu.log
Operator Count Status
DEPTHWISE_CONV_2D 1 Mapped to Edge TPU
SOFTMAX 1 Mapped to Edge TPU
FULLY_CONNECTED 2 Mapped to Edge TPU
MAX_POOL_2D 4 Mapped to Edge TPU
QUANTIZE 2 Mapped to Edge TPU
CONV_2D 3 Mapped to Edge TPU
Some more context. Here's the method I use to crete the quantized tflite model where A is the
def makeQuantizedTFmodel(x_train, model, model_name):
import tensorflow as tf
print("\n Creating quantized TensorFlowLite Model...\n")
A2 = tf.cast(x_train, tf.float32)
A = tf.data.Dataset.from_tensor_slices((A2)).batch(1)
def representative_dataset_gen():
for input_value in A.take(100):
yield[input_value]
try:
converter = tf.compat.v1.lite.TFLiteConverter.from_keras_model_file(model_name) # TF2.x
#converter = tf.lite.TFLiteConverter.from_keras_model(model) # TF2.x only. Does not support EdgeTPU
except:
converter = tf.lite.TFLiteConverter.from_keras_model_file(model_name) # T1.x
converter.optimizations = [tf.lite.Optimize.OPTIMIZE_FOR_SIZE]
converter.representative_dataset = representative_dataset_gen
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
converter.inference_input_type = tf.uint8
converter.inference_output_type = tf.uint8
tflite_quant_model = converter.convert()
with open(os.path.splitext(model_name)[0]+'.tflite', 'wb') as o:
o.write(tflite_quant_model)
Note: I am still using the tf.compat.v1.lite.TFLiteConverter.from_keras_model_file call when using TF 2.x, since tf.lite.TFLiteConverter.from_keras_model still does not seem to convert all parts of the models to the TPU (i.e. they run on the CPU).
feranick
on 10 Jul 2020
@feranick
Thanks for reporting the issue, I was also able reproduced. Have you seen this behavior with other model as well?
I tried this on a number of different models but so far only yours have this issue which is very odd, I can definitely pass this long to get it check out.
For now, what tensorflow version are you using?
Note: I am still using the tf.compat.v1.lite.TFLiteConverter.from_keras_model_file call when using TF 2.x, since tf.lite.TFLiteConverter.from_keras_model still does not seem to convert all parts of the models to the TPU (i.e. they run on the CPU).
That is expected since tensorflow version 2.0 -> 2.2 did not supports float i/o tensors, our compiler adapted to it. But with tf-nightly (I believe as early as >tf2.3) the model should have uint8 tensors now although you'll need to turn off MLIR converter by adding this line:
converter.experimental_new_converter = False
Could you give that a try to see if the compiler does better?
Namburger
on 10 Jul 2020
The model I uploaded was generated based on TF 1.15. I am going to produce one made with a more recent version and use:
- TF 2.2 and the
tf.compat.v1.lite.TFLiteConverter.from_keras_model_file - later I will try 2ith TF 2.3 and the new experimental feature.
Thanks!
feranick
on 10 Jul 2020
Here's an update:
- The original (not quantized) model was trained with TF 1.15
- I converted it to a quantized tflite model using TF 2.2
- When compiled with edgetpu_compiler v.14.1, it still suffers from the issue.
feranick
on 10 Jul 2020
So I tried to take the initial not quantized model trained with TF1.15 and converted with TF2.3-rc0 using:
converter = tf.lite.TFLiteConverter.from_keras_model(model)
While now floats are handled correctly, when compiled with edgetpu_compiler v.14.1, the original issue with RAM is still present (there seems to be no difference while setting converter.experimental_new_converter = False)
feranick
on 10 Jul 2020
I am training now the original model (not quantized) using TF2.2. Will take a while, but I will report back once it's done.
feranick
on 10 Jul 2020
So I used a much smaller training set and a very low number of ephocs to get a quick model. The problem still persists. So if I train the initial model with TF1.15, TF2.2 or TF2.3, you get the inconsistency with RAM usage depending on the edgetpu_compiler version. Here's the logs;
Using TF1.15 trained model, converted to TFlite using TF1.15
Edge TPU Compiler version 2.1.302470888
Model compiled successfully in 26 ms.
Input model: model_classifier_CNN.tflite
Input size: 186.45KiB
Output model: model_classifier_CNN_edgetpu.tflite
Output size: 304.53KiB
On-chip memory used for caching model parameters: 289.50KiB
On-chip memory remaining for caching model parameters: 7.14MiB
Off-chip memory used for streaming uncached model parameters: 0.00B
Number of Edge TPU subgraphs: 1
Total number of operations: 11
Operation log: model_classifier_CNN_edgetpu.log
Operator Count Status
CONV_2D 1 Mapped to Edge TPU
DEPTHWISE_CONV_2D 1 Mapped to Edge TPU
SOFTMAX 1 Mapped to Edge TPU
FULLY_CONNECTED 4 Mapped to Edge TPU
MAX_POOL_2D 2 Mapped to Edge TPU
QUANTIZE 2 Mapped to Edge TPU
Edge TPU Compiler version 14.1.317412892
Model compiled successfully in 96 ms.
Input model: model_classifier_CNN.tflite
Input size: 186.45KiB
Output model: model_classifier_CNN_edgetpu.tflite
Output size: 3.23MiB
On-chip memory used for caching model parameters: 3.20MiB
On-chip memory remaining for caching model parameters: 4.31MiB
Off-chip memory used for streaming uncached model parameters: 0.00B
Number of Edge TPU subgraphs: 1
Total number of operations: 11
Operation log: model_classifier_CNN_edgetpu.log
Operator Count Status
DEPTHWISE_CONV_2D 1 Mapped to Edge TPU
QUANTIZE 2 Mapped to Edge TPU
CONV_2D 1 Mapped to Edge TPU
SOFTMAX 1 Mapped to Edge TPU
MAX_POOL_2D 2 Mapped to Edge TPU
FULLY_CONNECTED 4 Mapped to Edge TPU
Using the model trained with TF2.2 or TF2.3-rc0 (and converted with TF2.2, or TF2.3-rc0)
Edge TPU Compiler version 2.1.302470888
Model compiled successfully in 27 ms.
Input model: model_classifier_CNN.tflite
Input size: 187.62KiB
Output model: model_classifier_CNN_edgetpu.tflite
Output size: 304.53KiB
On-chip memory used for caching model parameters: 289.50KiB
On-chip memory remaining for caching model parameters: 7.14MiB
Off-chip memory used for streaming uncached model parameters: 0.00B
Number of Edge TPU subgraphs: 1
Total number of operations: 12
Operation log: model_classifier_CNN_edgetpu.log
Operator Count Status
SOFTMAX 1 Mapped to Edge TPU
FULLY_CONNECTED 4 Mapped to Edge TPU
MAX_POOL_2D 2 Mapped to Edge TPU
QUANTIZE 2 Mapped to Edge TPU
CONV_2D 2 Mapped to Edge TPU
RESHAPE 1 Mapped to Edge TPU
Edge TPU Compiler version 14.1.317412892
Model compiled successfully in 95 ms.
Input model: model_classifier_CNN.tflite
Input size: 187.62KiB
Output model: model_classifier_CNN_edgetpu.tflite
Output size: 3.23MiB
On-chip memory used for caching model parameters: 3.20MiB
On-chip memory remaining for caching model parameters: 4.31MiB
Off-chip memory used for streaming uncached model parameters: 0.00B
Number of Edge TPU subgraphs: 1
Total number of operations: 12
Operation log: model_classifier_CNN_edgetpu.log
Operator Count Status
QUANTIZE 2 Mapped to Edge TPU
CONV_2D 2 Mapped to Edge TPU
FULLY_CONNECTED 4 Mapped to Edge TPU
MAX_POOL_2D 2 Mapped to Edge TPU
RESHAPE 1 Mapped to Edge TPU
SOFTMAX 1 Mapped to Edge TPU
The model structure:
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 1, 784, 30) 1530
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 1, 130, 30) 0
_________________________________________________________________
dropout (Dropout) (None, 1, 130, 30) 0
_________________________________________________________________
conv2d_1 (Conv2D) (None, 1, 111, 15) 9015
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 1, 111, 15) 0
_________________________________________________________________
dropout_1 (Dropout) (None, 1, 111, 15) 0
_________________________________________________________________
flatten (Flatten) (None, 1665) 0
_________________________________________________________________
dense (Dense) (None, 100) 166600
_________________________________________________________________
dropout_2 (Dropout) (None, 100) 0
_________________________________________________________________
dense_1 (Dense) (None, 50) 5050
_________________________________________________________________
dropout_3 (Dropout) (None, 50) 0
_________________________________________________________________
dense_2 (Dense) (None, 20) 1020
_________________________________________________________________
dropout_4 (Dropout) (None, 20) 0
_________________________________________________________________
dense_3 (Dense) (None, 67) 1407
=================================================================
Total params: 184,622
Trainable params: 184,622
Non-trainable params: 0
@feranick thanks for the report, I didn't expected you to retrain the model with different tensorflow version, but now it's clear that there's no work around on this issue. I'll report this to our team shortly to see what changes has caused this, thanks
Namburger
on 10 Jul 2020
No problem at all! Please let me know if you need more testing. Thank you.
feranick
on 10 Jul 2020
@feranick
by means of an update, after discussing this issue with our team, we've opened an internal bug on this one!
I'll give you updates once we have a fix, thanks again for reporting and please also submits other models here if you're having the same issue.
Namburger
on 14 Jul 2020
I am doing more testing. Here's a model trained from a different dataset, using a similar network configuration as above. While the edgetpu model is fully contained in RAM, the output size is significantly larger when compiled with 14.1 vs 2.1.
The tfmodel can be found here: https://www.dropbox.com/s/320y824jahuvf2u/model_classifier_CNN_3.tflite?dl=0
Output from edgetpu_compiler 14.1:
Edge TPU Compiler version 14.1.317412892
Model compiled successfully in 102 ms.
Input model: model_classifier_CNN.tflite
Input size: 379.89KiB
Output model: model_classifier_CNN_edgetpu.tflite
Output size: 3.28MiB
On-chip memory used for caching model parameters: 3.27MiB
On-chip memory remaining for caching model parameters: 2.84MiB
Off-chip memory used for streaming uncached model parameters: 0.00B
Number of Edge TPU subgraphs: 1
Total number of operations: 14
Operation log: model_classifier_CNN_edgetpu.log
Operator Count Status
RESHAPE 1 Mapped to Edge TPU
MAX_POOL_2D 4 Mapped to Edge TPU
FULLY_CONNECTED 2 Mapped to Edge TPU
CONV_2D 4 Mapped to Edge TPU
QUANTIZE 2 Mapped to Edge TPU
SOFTMAX 1 Mapped to Edge TPU
Output from edgetpu_compiler v2.1:
Edge TPU Compiler version 2.1.302470888
Model compiled successfully in 35 ms.
Input model: model_classifier_CNN.tflite
Input size: 379.89KiB
Output model: model_classifier_CNN_edgetpu.tflite
Output size: 548.53KiB
On-chip memory used for caching model parameters: 542.50KiB
On-chip memory remaining for caching model parameters: 5.57MiB
Off-chip memory used for streaming uncached model parameters: 0.00B
Number of Edge TPU subgraphs: 1
Total number of operations: 14
Operation log: model_classifier_CNN_edgetpu.log
Operator Count Status
FULLY_CONNECTED 2 Mapped to Edge TPU
SOFTMAX 1 Mapped to Edge TPU
MAX_POOL_2D 4 Mapped to Edge TPU
QUANTIZE 2 Mapped to Edge TPU
CONV_2D 4 Mapped to Edge TPU
RESHAPE 1 Mapped to Edge TPU
Model structure:
Layer (type) Output Shape Param #
=================================================================
conv2d (Conv2D) (None, 1, 784, 60) 3060
_________________________________________________________________
max_pooling2d (MaxPooling2D) (None, 1, 392, 60) 0
_________________________________________________________________
dropout (Dropout) (None, 1, 392, 60) 0
_________________________________________________________________
conv2d_1 (Conv2D) (None, 1, 353, 50) 120050
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 1, 176, 50) 0
_________________________________________________________________
dropout_1 (Dropout) (None, 1, 176, 50) 0
_________________________________________________________________
conv2d_2 (Conv2D) (None, 1, 147, 40) 60040
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 1, 73, 40) 0
_________________________________________________________________
dropout_2 (Dropout) (None, 1, 73, 40) 0
_________________________________________________________________
conv2d_3 (Conv2D) (None, 1, 54, 30) 24030
_________________________________________________________________
max_pooling2d_3 (MaxPooling2 (None, 1, 54, 30) 0
_________________________________________________________________
dropout_3 (Dropout) (None, 1, 54, 30) 0
_________________________________________________________________
flatten (Flatten) (None, 1620) 0
_________________________________________________________________
dense (Dense) (None, 100) 162100
_________________________________________________________________
dropout_4 (Dropout) (None, 100) 0
_________________________________________________________________
dense_1 (Dense) (None, 67) 6767
=================================================================
Total params: 376,047
Trainable params: 376,047
Non-trainable params: 0
@feranick many thanks, we actually already found the regression, there should be a fix soon, but I'm not sure when it'll be release :/
Namburger
on 15 Jul 2020
Thanks!
feranick
on 15 Jul 2020
This seems to have been fixed with the new release of the compiler 15.1 and new drivers for Linux. However the drivers for MacOS and Windows seems to be still the old ones in this page, so effectively there are no working drivers available for MacOS or Windows....
feranick
on 7 Nov 2020
So I found the updated macOS and Windows drivers, here:
https://coral.ai/software/#edgetpu-runtime
All seems to work. I will report on possible performance regressions if any. However the link above should be updated with the correct URL for the drivers.
feranick
on 7 Nov 2020
@feranick yes, please get the latest one from here, I'll have our team update the link from the tutorial, thanks for reporting
Namburger
on 7 Nov 2020
@Namburger
Where I can find edgetpu compiler 15.1?
Current version in stable repo is 15.0 - the same as in unstable
Is compiler open sourced? I found only binary v14.1.317412892
marcin-kamionowski
on 12 Feb 2021
@feranick Do you have 15.1 compiler or this was you a typo?
marcin-kamionowski
on 22 Feb 2021
@marcin-kamionowski
Is compiler open sourced? I found only binary v14.1.317412892
unfortunately no
Namburger
on 24 Feb 2021
@feranick Do you have 15.1 compiler or this was you a typo?
Sorry it was a typo. 15.0.340273435 is the one I am referring to and it seems to work.
feranick
on 24 Feb 2021
Related issues
rdejana
路
6Comments
j-o-d-o
路
3Comments
Bartvelp
路
6Comments
tommiesatellite
路
6Comments
hansamann
路
5Comments