Deepspeech: Feature request: Full integer quantization for tflite: Coral edge TPU compatibility

[jacobjennings]

For support and discussions, please use our Discourse forums.

If you've found a bug, or have a feature request, then please create an issue with the following information:

• Have I written custom code (as opposed to running examples on an unmodified clone of the repository):
  No
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04):
  Kubuntu 18.04
• TensorFlow installed from (our builds, or upstream TensorFlow): N/A
• TensorFlow version (use command below):
• Python version:
• Bazel version (if compiling from source):
• GCC/Compiler version (if compiling from source):
• CUDA/cuDNN version:
• GPU model and memory:
• Exact command to reproduce:

Instructions from https://coral.withgoogle.com/docs/edgetpu/compiler/
Download pretrained model for DeepSpeech 0.5.1

edgetpu_compiler output_graph.tflite

Feature request

I picked up the Coral USB ML accelerator which can run inference on tflite models with additional restrictions:

https://coral.withgoogle.com/products/accelerator
https://coral.withgoogle.com/docs/edgetpu/models-intro/

"Note: Starting with our July 2019 release (v12 of the Edge TPU runtime), the Edge TPU supports models built with TensorFlow's post-training quantization, but only when using full integer quantization (you must use the TensorFlow 1.15 "nightly" build and set both the input and output type to uint8). Previously, we supported only quantization-aware training, which uses "fake" quantization nodes to simulate the effect of 8-bit values during training. So although you now have the option to use post-training quantization, keep in mind that quantization-aware training generally results in a higher accuracy model because it makes the model more tolerant of lower precision values."

Include any logs or source code that would be helpful to diagnose the problem. For larger logs, link to a Gist, not a screenshot. If including tracebacks, please include the full traceback. Try to provide a reproducible test case.

deepspeech/deepspeech-0.5.1-models$ edgetpu_compiler output_graph.tflite
Edge TPU Compiler version 2.0.258810407
INFO: Initialized TensorFlow Lite runtime.
Invalid model: output_graph.tflite
Model not quantized

[jacobjennings]

If it helps, I would be happy to fund a Coral USB for you to try. jacob.r.[email protected]
I think there's potential to cross the real-time barrier on a Pi with this thing. Would be fun for my DIY projects.

[lissyx]

I've already tried to get that working but the intersection between what is supported for EdgeTPU and our current model makes it incompatible. Please see existing threads on discourse and also NNAPI and GPU delegations issues on github.

[jacobjennings]

Unfortunate. Thanks for the info.

[lissyx]

Yeah, don't worry, I'd like to get it working so I'll keep testing on some spare cycles

[rhamnett]

Given we are now using TF 1.15 do you think it would be possible to try the quantization again? What is the current output type, is it uint8?

[lissyx]

Given we are now using TF 1.15 do you think it would be possible to try the quantization again? What is the current output type, is it uint8?

I used 1.15 during my previous experiments