Edgetpu: Compilation of tflite model shows that all operations will be on CPU rather than TPU even though all ops are supported.

[akssieg]

I converted and quantized the cpm_hand model provided in the repo https://github.com/timctho/convolutional-pose-machines-tensorflow to tflite.

The quantized tflite file can be found at https://drive.google.com/file/d/15aHHyR-iamsDDpooV-GaKE9rjRqzZsfe/view?usp=sharing.

When I executed edgetpu_compiler -s -m 10 hand_quant.tflite command in the terminal then it provided the following output:

Edge TPU Compiler version 14.1.317412892

Model compiled successfully in 70 ms.

Input model: tflite_model/hand_quant.tflite
Input size: 23.38MiB
Output model: hand_quant_edgetpu.tflite
Output size: 23.38MiB
On-chip memory used for caching model parameters: 0.00B
On-chip memory remaining for caching model parameters: 0.00B
Off-chip memory used for streaming uncached model parameters: 0.00B
Number of Edge TPU subgraphs: 0
Total number of operations: 42
Operation log: hand_quant_edgetpu.log

Model successfully compiled but not all operations are supported by the Edge TPU. A percentage of the model will instead run on the CPU, which is slower. If possible, consider updating your model to use only operations supported by the Edge TPU. For details, visit g.co/coral/model-reqs.
Number of operations that will run on Edge TPU: 0
Number of operations that will run on CPU: 42

Operator                       Count      Status

QUANTIZE                       6          Operation is otherwise supported, but not mapped due to some unspecified limitation
CONCATENATION                  2          Operation is working on an unsupported data type
CONV_2D                        31         Operation is working on an unsupported data type
MAX_POOL_2D                    3          Operation is working on an unsupported data type

As per the docs https://coral.ai/docs/edgetpu/models-intro/#ops-table CONCATENATION, CONV_2D and MAX_POOL_2D are supported by Edge TPU and should run on TPU but that is not the case here.

Please comment on this and any help would be appreciated.

[Namburger]

@akssieg apologies, it seems to be hitting a compiler bug that we've fixed. We're expecting a new release round mid Q4 along with this fix, for now, here is your model:
https://drive.google.com/file/d/13Lm9WrcJIyP-iIX13_piyef9YyGEhfYE/view?usp=sharing
FYI, that looks like a really kool project o_0

[akssieg]

@Namburger thanks for the reply, I tried unzipping the files you provided via https://drive.google.com/file/d/13Lm9WrcJIyP-iIX13_piyef9YyGEhfYE/view?usp=sharing using the command tar -xzf hand_quant.tar.gz but every time I get the following output:

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now

Secondly, I have a query regarding Coral TPU. Can I create my own custom ops for the TPU and then register it on the TPU compiler backend for its use?

[Namburger]

@akssieg oh, you should be able to decompress it with this tar -xvf hand_quant.tar.gz
Unfortunately, since the compiler isn't opensource, you won't be able to implement this support for custom ops

[akssieg]

@Namburger there was a little issue from my end regarding the decompression :| . Now its fine.

[akssieg]

@Namburger quantized tflite model which has float input and output produces the desired correct results but when I do full integer quantization then the inference produces wrong results.

Since full integer quantized tflite model is not working, TPU compiled model is also not working.

Is there any way around? Could graph surgery be a good option? Like executing first layer on the cpu and the rest on coral usb accelerator?

[Namburger]

@akssieg Oh, may I see how you perform the post processing ops?
Make sure you take the scale and zero points into consideration, similar to this: https://github.com/google-coral/tflite/blob/master/python/examples/classification/classify.py#L40-L41

Is there any way around? Could graph surgery be a good option? Like executing first layer on the cpu and the rest on coral usb accelerator?

The compiler should already take care of this for you :) When you compile the model and your i/o is still float, those ops wil automatically be on the CPU, fyi, quantize and dequantize is also a very lightweight ops, so I wouldn't worry too much about that in the first place if the float i/o model is working fine for you

[akssieg]

For the float Input/Output tflite model, I do the following:

interpreter.set_tensor(input_details[0]['index'], img.astype("float32"))
interpreter.invoke()
output_data = interpreter.get_tensor(output_details[0]['index'])

The float model could be found at https://drive.google.com/file/d/15s1PDuCBelEHU1p3vPYXXLkgxV1Z0OW_/view?usp=sharing.

And for the full integer quantized tflite model:

scale, zero_point = input_details[0]['quantization']
img_input = img / scale + zero_point
interpreter.set_tensor(input_details[0]['index'], np.uint8(img_input))
interpreter.invoke()
output_data = interpreter.get_tensor(output_details[0]['index'])
scale1, zero_point1 = output_details[0]['quantization']
output_data = scale1 * (output_data - zero_point1)

Fully quantized float model => https://drive.google.com/file/d/1HHeNbsYQnyYwMODYsz-2sYmFp7XklBe_/view?usp=sharing.

@Namburger Sorry for asking you again but could you please also compile the float tflite model https://drive.google.com/file/d/15s1PDuCBelEHU1p3vPYXXLkgxV1Z0OW_/view?usp=sharing for coral usb accelerator?

[Namburger]

@akssieg It is actually our fault that the compiler is being flaky :) Hopefully we'll have less bugs on the next release, here is the models you requested:
https://drive.google.com/file/d/1OGyBGKDcvnHO_iHt7R8ANscSUdHwvxVa/view?usp=sharing

[JavierGSanchez]

Hi Namburger,

My Project is experiencing the same issue regarding the compiled tflite model having 0 Edge TPU subgraphs. As we need the formfactor of the Pi Zero combined with an Edge TPU, we don't have another option and our project is completely blocked.

We look forward to your team's next release that will address this issue, but can you help us now by compiling our quantized tflite model for runtime version 10?

The model is online here: https://drive.google.com/file/d/1sVhXYgJeZVOjaSvkCPaGorYoUDv2ssSx/view?usp=sharing

This will unblock us and should keep us productive for weeks. We are huge fans of the Edge TPU and plan to use them at scale with our commercial product.

Please help!

[Namburger]

@JavierGSanchez sincerely apologize for this issue. As it turn out, even with our latest compiler, I can map everything to the edgetpu for the newest runtime version, but for runtime version 10
second_part.tar.gz

Since libedgetpu and tflite_runtime are both open source, would you like to try with a newer runtime version instead?
Here is a related query: https://github.com/google-coral/edgetpu/issues/229#issuecomment-704576309

[Namburger]

@JavierGSanchez See this answer for everything you need to get a very updated version of everything that works on a pi0:
https://github.com/google-coral/edgetpu/issues/229#issuecomment-708478704
Cheers!

[JavierGSanchez]

Hi @Namburger, this is invaluable information and we'll be working on this today as moving to the latest version of the runtime is perfect for our project.

Just to confirm, you are recommending that we cross-compile both the tflite_runtime (or use rhadnum's wheels) and the libedgetpu.so for the armv6 arch per you instructions on #229 (comment)?

And this would remove the need to use the edgetpu compiler correct, as the tflite_runtime would handle that for us?

[Namburger]

@JavierGSanchez Hello!
I definitely suggest using rhadnum's wheel (2.3.1) as he is nice enough to to provide it and we know for sure his wheel is working, then compile libedgetpu.so using my instruction.

And this would remove the need to use the edgetpu compiler correct, as the tflite_runtime would handle that for us?

Unfortunately, you'll still need to use the compiler, but you can use the latest runtime as opposed to the runtime version 10.

[JavierGSanchez]

Perfect! Will do!