Edgetpu: Decreased accuracy from AutoML Vision object detection model when compiled for edge TPU

[BotScutters]

Hello,

This is a follow-up to closed issue #38, previously addressed by @Namburger.

I've been working with models trained using GCP AutoML Vision Object Detection and I'm continuing to observe that there is a significant reduction in model accuracy between the model.tflite model exported from GCP and the model_edgetpu.tflite model that I get out of the edgetpu_compiler. i.e. the original detection model from GCP is generally functional with an average precision around 70%, but once compiled it's significantly below 30%.

There are also a significant number of operations (132/294) still not being mapped to the TPU, resulting in average inference times around 0.065 s.

Do you guys have any additional insights into why we're still seeing this hit to accuracy and also if more of these operations will be supported on the TPU any time soon?

Thanks for your help!

For reference:
Edge TPU Compiler version 2.1.302470888
Runtime version 13
The device I'm using is running Ubuntu 16.04 with a M.2 Accelerator A+E TPU installed.

I've confirmed this behavior using models generated with AutoML Model Types "Mobile Low Latency" and "Mobile Best Trade-Off"

The output during model compilation is as follows:

Edge TPU Compiler version 2.1.302470888
Input: model.tflite
Output: model_edgetpu.tflite

Operator                       Count      Status

MEAN                           8          Mapped to Edge TPU
LOGISTIC                       11         Mapped to Edge TPU
LOGISTIC                       6          More than one subgraph is not supported
MAX_POOL_2D                    9          More than one subgraph is not supported
MAX_POOL_2D                    6          Mapped to Edge TPU
MUL                            10         Mapped to Edge TPU
MUL                            6          More than one subgraph is not supported
REDUCE_MAX                     3          Mapped to Edge TPU
REDUCE_MAX                     5          More than one subgraph is not supported
PACK                           24         Tensor has unsupported rank (up to 3 innermost dimensions mapped)
RESHAPE                        32         More than one subgraph is not supported
RESHAPE                        12         Tensor has unsupported rank (up to 3 innermost dimensions mapped)
ADD                            12         Mapped to Edge TPU
ADD                            19         More than one subgraph is not supported
CUSTOM                         1          Operation is working on an unsupported data type
CONCATENATION                  2          More than one subgraph is not supported
CONV_2D                        63         Mapped to Edge TPU
CONV_2D                        23         More than one subgraph is not supported
DEPTHWISE_CONV_2D              19         Mapped to Edge TPU
DEPTHWISE_CONV_2D              23         More than one subgraph is not supported

[Namburger]

@BotScutters
Thanks for reaching out, I'm actually working with a couple other users and realized that the current compiler released isn't fully compatible with the AutoML's API....
Apologies for this, internally with the latest code of our compiler, things are working in harmony. I've raised this issue with our team and will update with you on this as soon as we can make a decision on what we can do.

P.S. Since we recently open sourced libedgetpu, my hope is that we can do the same for the compiler soon so that all these won't be an issue any more :)

[BotScutters]

Thanks @Namburger, looking forward to the update!

[Namburger]

@BotScutters

Hi all -
please send your model to [email protected] for now, make sure you mention this issue and I can help! Thanks

[gabrielterleski]

Hello @Namburger , I have this same problem, is there a forecast for this update?

[Namburger]

@gabrielterleski is it you that just sent over the model? I'll send you the model back soon. We are expecting the compiler release to be within this month!

[gabrielterleski]

thank you very much, I'm having another problem to solve, but how is it
possible to test this model.

Em seg., 6 de jul. de 2020 às 16:47, Nam Vu [e-mail]
escreveu:

@gabrielterleski https://github.com/gabrielterleski is it you that just
sent over the model? I'll send you the model back soon. We are expecting
the compiler release to be within this month!

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/google-coral/edgetpu/issues/144#issuecomment-654431287,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFHUCHB5VZD3TNU6G47MVJLR2IS3XANCNFSM4N3XTZNA
.

--
Gabriel Terleski

[Namburger]

@gabrielterleski here is a quick demo for using object detection model: https://gist.github.com/Namburger/f7e6c18af94ef3d6a70076a130eb1f7c

maybe this is a good start?

[gabrielterleski]

It worked, but this with high latency and little fps.
What do you recommend that I do?

fps: 1
fps: 2
fps: 3
fps: 3
fps: 3
fps: 3
fps: 3
fps: 3
fps: 3
fps: 3
fps: 3
fps: 3
fps: 3
fps: 3
fps: 3
fps: 3
fps: 3
fps: 3
fps: 3
fps: 3
fps: 3
fps: 3

Em ter., 7 de jul. de 2020 às 18:07, Nam Vu [e-mail]
escreveu:

@gabrielterleski https://github.com/gabrielterleski here is a quick
demo for using object detection model:
https://gist.github.com/Namburger/f7e6c18af94ef3d6a70076a130eb1f7c

maybe this is a good start?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/google-coral/edgetpu/issues/144#issuecomment-655134965,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AFHUCHCSVRLOML3LA5BHMPLR2OFADANCNFSM4N3XTZNA
.

--
Gabriel Terleski