Edgetpu: Internal compiler error. Aborting!

[prashant45]

I am facing an unusual problem. When I try to compile my model it throws me Internal compiler error. Aborting! error.

But I noticed sometimes I try running the same command again and again, it compiles after a few attempts.

********************:~/HOME$ edgetpu_compiler ModelDIR/model-quant.tflite -o ModelDIR/
Edge TPU Compiler version 14.1.317412892

Internal compiler error. Aborting! 
********************:~/HOME$ edgetpu_compiler ModelDIR/model-quant.tflite -o ModelDIR/
Edge TPU Compiler version 14.1.317412892

Internal compiler error. Aborting! 
********************:~/HOME$ edgetpu_compiler ModelDIR/model-quant.tflite -o ModelDIR/
Edge TPU Compiler version 14.1.317412892

Internal compiler error. Aborting! 
********************:~/HOME$ edgetpu_compiler ModelDIR/model-quant.tflite -o ModelDIR/
Edge TPU Compiler version 14.1.317412892

Internal compiler error. Aborting! 
********************:~/HOME$ edgetpu_compiler ModelDIR/model-quant.tflite -o ModelDIR/
Edge TPU Compiler version 14.1.317412892

Internal compiler error. Aborting! 
********************:~/HOME$ edgetpu_compiler ModelDIR/model-quant.tflite -o ModelDIR/
Edge TPU Compiler version 14.1.317412892

Internal compiler error. Aborting! 
********************:~/HOME$ edgetpu_compiler ModelDIR/model-quant.tflite -o ModelDIR/
Edge TPU Compiler version 14.1.317412892

Internal compiler error. Aborting! 
********************:~/HOME$ edgetpu_compiler ModelDIR/model-quant.tflite -o ModelDIR/
Edge TPU Compiler version 14.1.317412892

Internal compiler error. Aborting! 
********************:~/HOME$ edgetpu_compiler ModelDIR/model-quant.tflite -o ModelDIR/
Edge TPU Compiler version 14.1.317412892

Internal compiler error. Aborting! 
********************:~/HOME$ edgetpu_compiler ModelDIR/model-quant.tflite -o ModelDIR/
Edge TPU Compiler version 14.1.317412892

Internal compiler error. Aborting! 
********************:~/HOME$ edgetpu_compiler ModelDIR/model-quant.tflite -o ModelDIR/
Edge TPU Compiler version 14.1.317412892

Model compiled successfully in 70 ms.

Input model: ModelDIR/model-quant.tflite
Input size: 1.34MiB
Output model: ModelDIR/model-quant_edgetpu.tflite
Output size: 3.33MiB
On-chip memory used for caching model parameters: 224.00KiB
On-chip memory remaining for caching model parameters: 7.38MiB
Off-chip memory used for streaming uncached model parameters: 0.00B
Number of Edge TPU subgraphs: 1
Total number of operations: 34
Operation log: ModelDIR/model-quant_edgetpu.log

Model successfully compiled but not all operations are supported by the Edge TPU. A percentage of the model will instead run on the CPU, which is slower. If possible, consider updating your model to use only operations supported by the Edge TPU. For details, visit g.co/coral/model-reqs.
Number of operations that will run on Edge TPU: 5
Number of operations that will run on CPU: 29
See the operation log file for individual operation details.
********************:~/HOME$

model-details.txt

Can someone explain this behavior ?

[Namburger]

@prashant45
Hi! I've seen this before: https://github.com/google-coral/edgetpu/issues/173#issuecomment-659650552
It should be fixed internally, but could you attach model so I can take look?

[prashant45]

@Namburger

Really appreciate the quick response.

Model -
model-quant.tflite.tar.gz

I use an input shape [batch_size, 8000, 1] that is converted to [batch_size, 1, 8000, 1] by tflite converter.
Also the Conv1D operator is converted to Conv2D and there is an unnecessary Reshape op added in the graph.

[Namburger]

@prashant45 just checked, this is definitely a bug on our side, it should be fixed by next update for you, but the successfully compiled model model should works

[prashant45]

@Namburger can you comment regarding why most of the operations are pushed to CPU?

[Namburger]

@prashant45 I see some ops with really large tensors that is otherwise supported but has really large inputs that the edgetpu couldn't handle, could you try shrinking the inputs?

[michalrudowicz]

Hi,

I am having a similar problem, though I'm getting the internal compiler error consistently. For example, the following model can be compiled sucesfully: p_net_17_17_3.tflite.zip while a slightly bigger p_net_23_23_3.tflite.zip results in an internal compiler error. Is it the same issue and I should wait for the next update, or is more information required from my side?
Thanks in advance!

[Namburger]

@michalrudowicz thanks for reporting, this is definitely a bug with the current compiler version, but we have fixed in internally, here is your model:
pnet23x23.tar.gz

We are expecting a new compiler release around mid q4, sorry for the issue

[michalrudowicz]

Thanks a lot, good to know!

pt., 25 wrz 2020, 20:03 użytkownik Nam Vu [e-mail]
napisał:

@michalrudowicz https://github.com/michalrudowicz thanks for reporting,
this is definitely a bug with the current compiler version, but we have
fixed in internally, here is your model:
pnet23x23.tar.gz
https://github.com/google-coral/edgetpu/files/5284554/pnet23x23.tar.gz

We are expecting a new compiler release around mid q4, sorry for the issue

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/google-coral/edgetpu/issues/211#issuecomment-699072908,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AOGXKLLANRNYQDZEOZMWD2TSHTLQDANCNFSM4QQNAQYA
.

[ludgerh]

This is very similar to what prashant45 found:

ludger@debian-laptop:~/temp$ edgetpu_compiler cam-ai_model_quant.tflite
Edge TPU Compiler version 14.1.317412892

Internal compiler error. Aborting! 
ludger@debian-laptop:~/temp$ edgetpu_compiler cam-ai_model_quant.tflite
Edge TPU Compiler version 14.1.317412892

Internal compiler error. Aborting! 
ludger@debian-laptop:~/temp$ edgetpu_compiler cam-ai_model_quant.tflite
Edge TPU Compiler version 14.1.317412892

Model compiled successfully in 1626 ms.

Input model: cam-ai_model_quant.tflite
Input size: 46.29MiB
Output model: cam-ai_model_quant_edgetpu.tflite
Output size: 47.24MiB
On-chip memory used for caching model parameters: 5.99MiB
On-chip memory remaining for caching model parameters: 74.00KiB
Off-chip memory used for streaming uncached model parameters: 15.73MiB
Number of Edge TPU subgraphs: 1
Total number of operations: 108
Operation log: cam-ai_model_quant_edgetpu.log

Model successfully compiled but not all operations are supported by the Edge TPU. A percentage of the model will instead run on the CPU, which is slower. If possible, consider updating your model to use only operations supported by the Edge TPU. For details, visit g.co/coral/model-reqs.
Number of operations that will run on Edge TPU: 102
Number of operations that will run on CPU: 6
See the operation log file for individual operation details.

[ludgerh]

model.zip
Here is my model. Would it make sense to wait for the next compiler version?

[ludgerh]

For me it looks like this is fixed with compiler version 15.0.340273435...

[Namburger]

Hey, sorry I was out of town, we just had a release, please try compiling again!

[ludgerh]

Thank you guys, you fixed it. Compiler Version 15.0

[gdebrecz]

Hi !
I have very similar / same issues:

user@node:/hpegpu$ edgetpu_compiler -o /tmp/output.tflite.edgetpu ./frozen_inference_graph.tflite
Edge TPU Compiler version 15.0.340273435

Internal compiler error. Aborting!

Tried on many different platforms, same result everywhere. also tried with previous compiler (13)

I'm trying to compile this model:
frozen_inference_graph.tflite

Any help is appreciated, the meaningless error message is really frustrating.
thanks a lot in advance !

[gdebrecz]

Same error also for this model: deeplabv3_cityscapes_quantized_frozen_graph.tflite that has been produced following the official description here.

Thanks for any help,