Edgetpu: Decreased accuracy on usb accelerator when using an AutoML Vision trained model

[alemoreau]

Hello,

I am using the Coral usb accelerator for object detection. I trained a model on https://console.cloud.google.com/vision, optimized for latency, and successfully compiled the model with the edgetpu-compiler.

When I use the original (not compiled) .tflite, the objects are detected (~90% confidence) on each frame of the video.
But when I use the compiled .tflite, only background is detected (~60% confidence), on each frame.

Is such a decrease of accuracy expected ?

I'm using the same script for both .tflite : https://github.com/PINTO0309/TPU-MobilenetSSD/blob/master/MobileNet-SSD-TPU-sync.py, so i guess it is not a preprocessing issue.

Here is the output of the compilation
```Operator Count Status

MEAN 8 Mapped to Edge TPU
LOGISTIC 8 Mapped to Edge TPU
LOGISTIC 9 More than one subgraph is not supported
MAX_POOL_2D 6 Mapped to Edge TPU
MAX_POOL_2D 9 More than one subgraph is not supported
REDUCE_MAX 8 Operation is otherwise supported, but not mapped due to some unspecified limitation
MUL 8 Mapped to Edge TPU
MUL 8 More than one subgraph is not supported
PACK 24 Tensor has unsupported rank (up to 3 innermost dimensions mapped)
RESHAPE 32 More than one subgraph is not supported
RESHAPE 12 Tensor has unsupported rank (up to 3 innermost dimensions mapped)
ADD 10 Mapped to Edge TPU
ADD 21 More than one subgraph is not supported
CUSTOM 1 Operation is working on an unsupported data type
CONCATENATION 2 More than one subgraph is not supported
CONV_2D 61 Mapped to Edge TPU
CONV_2D 25 More than one subgraph is not supported
DEPTHWISE_CONV_2D 17 Mapped to Edge TPU
DEPTHWISE_CONV_2D 25 More than one subgraph is not supported
```

[Namburger]

@alemoreau hi Allen, edgetpu compiler currently only supports classification models from the AutoML vision library. There is a known bug with the AutoMl objection detection engine that we're working to fix on high priority!

[alemoreau]

@Namburger thank you for the quick response ! So I guess I will have to retrain the model once this is fixed.

[jveitchmichaelis]

@alemoreau did you also compare with a model trained using the Object Detection API? (e.g. a standard MobileNet SSD) I'm having a similar issue with an AutoML model here, so I was planning on trying that next.

(I'm guessing this bug still isn't squashed)

[alemoreau]

@jveitchmichaelis Yes I had good results with quantization-aware training SSD MobileNet V2 using Object Detection API

[Contruba]

@alemoreau hi Allen, edgetpu compiler currently only supports classification models from the AutoML vision library. There is a known bug with the AutoMl objection detection engine that we're working to fix on high priority!

@Namburger Is there an update on the progress towards fixing this bug? Thanks!

[Namburger]

@Contruba from what I last heard, the object detection models should be compatible with the compiler now, but inference is still a little slow, which is the next steps that we're trying to fix. I'm reaching out to another team member regarding this, I'll let you know once I hear back.

[Contruba]

@Namburger Thank you so much! I eagerly await to here what you find out from your team members.

This issue persists with the version currently in the repo (compiler version 2.0.291256449, updated 28 days ago). So I am curious to know if there is a fix, and if so, what the estimated turn-around time for it to be merged in Master.

Thanks again. We all really appreciate your team's hard work! :)

[Namburger]

@Contruba So I just heard back from the team, the current update is that inference will still be very slow after compilation. The speed up is from 1/2fps -> 4/5fps (on the dev board's CPU->TPU) which is not ideal, they're still working on this but I know their plate is quite full at the moment. Unfortunately, I can't give too much more info right now :/

[Contruba]

@Namburger Ok, thank you very much for chasing that down. Even though the accuracy issue has been fixed, I am guessing they won't push a new compiler version to the master branch until both the accuracy and speed is fixed?

[jveitchmichaelis]

Hi @Namburger the model I exported from AutoML ran with a latency of 135ms ish on a USB accelerator, so that's in line with 4-5fps (a bit faster in fact). That was a high accuracy edge morel I believe.

However there was still a big drop in accuracy, and the TPU compiled model misses a lot of boxes that the CPU one doesn't (@Contruba I'm not sure it is fixed, at least not in my case). I think that's probably a bigger concern at the moment, but it's good to know that some progress is being made. Thanks for the update!

I've switched to retraining an EdgeTPU Mobilenet quant model (the latest) which seems pretty good for now.

[BotScutters]

Hey @Namburger, I've been following this issue as well and am looking forward to seeing AutoML Object Detection fully supported on the edgetpu with the accuracy and speed issues addressed. Are there any more updates on this? Thanks!

[Namburger]

@BotScutters Hi there, yes, AutoML with object detection should be working now!
Closing this issue, if anyone else have any other problems with it, feel free to open a new issue as it will most likely be a bug.
[FYI] also climber here :)

[Namburger]

Hi all -
Seems that there are some works to be done, although this is fixed internally for us.
Please send your model to [email protected] for now, make sure you mention this issue and I can help! Thanks