Edgetpu: Edge TPU slower than CPU running inference

[Bartvelp]

Hi,

First off I do not know if this is the correct place to ask/report since I do not know if it's a bug or expected behavior. Please don't hesitate to close the issue if so.

I am using the Coral Dev Board and trying to run an extremely simple NN for the MNIST sample dataset on it. It's basically this:

def create_model():
  model = tf.keras.models.Sequential([ # Sequential model, easy mindmap
      tf.keras.layers.Flatten(input_shape=(28, 28)), # we need to flatten the matrix into a vector 
      tf.keras.layers.Dense(128, activation='relu'), # Rectified linear activator
      tf.keras.layers.Dense(10), # 10 digits, so one_hot output
      tf.keras.layers.Softmax() # Softmax the previous output scores for the loss function
  ])
  model.compile(
    optimizer='adam',
    loss='sparse_categorical_crossentropy',
    metrics=['accuracy'] # Log accuracy
  )
  return model

The problem is that when benchmarking this model with 50k images the Edge TPU is taking 1876 ms and the onboard CPU 700 ms. Is this a bug or does the allocating of Edge TPU just take a set amount of time? It is also very plausible I coded it wrong, please let me know.

I trained the model for 5 epochs on my laptop CPU and compiled it for the Edge TPU, this produces this log:

Edge TPU Compiler version 2.1.302470888
Input: converted_model_from_keras_8bit_all.tflite
Output: converted_model_from_keras_8bit_all_edgetpu.tflite

Operator                       Count      Status

SOFTMAX                        1          Mapped to Edge TPU
FULLY_CONNECTED                2          Mapped to Edge TPU
QUANTIZE                       2          Mapped to Edge TPU

Please see https://github.com/Bartvelp/edge_tpu_MNIST for the full code and steps to reproduce

[Namburger]

@Bartvelp Humnn, this doesn't sounds right at all o_0
Allocating tpu should only take a long time the first time you load the model and then faster after that. I was able to replicate your issue exactly on my dev board and it's quite puzzling.
Furthermore, I'm using your exact code and getting this result on a USB Accelerator:

$ python3 edge_use_number_tflite.py 
Starting inference rounds on CPU
0: 5000 images took 1694 ms
1: 5000 images took 1713 ms
2: 5000 images took 1694 ms
3: 5000 images took 1694 ms
4: 5000 images took 1687 ms
Starting inference rounds on TPU
0: 5000 images took 871 ms
1: 5000 images took 876 ms
2: 5000 images took 875 ms
3: 5000 images took 886 ms
4: 5000 images took 871 ms

This results is extremely odd because normally the USB are slower than the dev board due to data transfer speed limitations. Considering that I have a 12 cores processor @3.2Ghz I'm also iffy that it performs slower than the dev board's quadcore arm cortex. I wonder if this is either due to python caching is doing something behind the scene + the fact the tflite cpu model was optimized for arm processor?

[Bartvelp]

So I did some investigating and got the following results on the dev board.

Speed stats compared to amount of parameters

| Parameters | On-chip memory | off-chip memory | TPU (ms/500 img) | CPU (ms/500 img)|
| :------------ | :------------- | :-------------- | :--------------- | :-------------- |
| 20,587,010 | 7.82MiB | 12.24MiB | 16548 | 12751 |
| 12,585,010 | 7.83MiB | 4.54MiB | 6611 | 7837 |
| 7,999,550 | 7.84MiB | 0 | 279 | 5494 |
| 1,176,450 | 1.33MiB | 0 | 198 | 695 |
| 318,010 | 370KiB | 0 | 156 | 154 |
| 15,910 | 51KiB | 0 | 154 | 15 |

If the TPU needs to stream to much parameters from off-chip memory it becomes slower than the onboard CPU

Then there is a sweet spot where the TPU can use only on-chip memory and hugely outperforms the CPU.

But with less parameters than ~318k the CPU performance keeps getting better, but the TPU is hitting a plateau of 155 ms / 500 images.

According to google the mobilenet v2 NN has 3.4 Million parameters, which puts it in the sweet spot of the TPU. The Mobilenet NN is the model used in the example by Coral of python classification here.

Method

Tested using the following format of a model, by varing the size of all the dense layers, and sometimes removing or adding some to get the wanted amount of params.

def create_model():
    startingLayerSize = 784
    sizeDenseLayers = 200
    model = tf.keras.models.Sequential([ # Sequential model, easy mindmap
            tf.keras.layers.Dense(startingLayerSize, activation='relu', input_shape=(784,)), # Rectified linear activator
            tf.keras.layers.Dense(sizeDenseLayers, activation='relu'),
            tf.keras.layers.Dense(sizeDenseLayers, activation='relu'),
            tf.keras.layers.Dense(sizeDenseLayers, activation='relu'),
            tf.keras.layers.Dense(10, activation='relu'),
            tf.keras.layers.Softmax()   # Softmax the previous output scores for the loss function
    ])
    model.compile(
        optimizer='sgd', # stochastic gradient descent method that just works well
        # easiest to use loss function sparse_categorical_crossentropy, easiest to understand mean_squared_error
        loss='mean_squared_error',
        metrics=['accuracy'] # Log accuracy
    )
    return model

In this model the images are flattened before the benchmark, but I think it doesn't really matter anyways.

Can somebody reproduce these results?

[iwatake2222]

@Bartvelp
I did a similar experiment before. In my case, I changed the number of Dense layers.
I got the same result as you. Model with dense layers is slower on Edge TPU.
I think it's because Edge TPU is specialized in convolution operation. and I think it's right strategy because Conv occupies a majority of inference time in most models.

The below is my result. model is generated this code (https://github.com/iwatake2222/EdgeTPU_Analysis/blob/master/Ops/)
Model with convolution ops is super fast, while model with dense ops is slow on Edge TPU

• Dense only model

| Layer Num | Model Size [MiB] | | | Time [msec] | |
|-----------|------------------|---------------------|----------------------|-------------|---------|
| | filesize | On-chip memory used | Off-chip memory used | CPU | EdgeTPU |
| 100 | 1.458 | 0.993 | 0.268 | 0.8 | 1.4 |
| 200 | 2.911 | 0.993 | 1.540 | 1.8 | 5.8 |
| 300 | 4.368 | 0.993 | 2.810 | 2.4 | 9.4 |
| 400 | 5.825 | 0.993 | 4.080 | 2.9 | 13.1 |

• Conv only model

| Layer Num | Model Size [MiB] | | | Time [msec] | |
|-----------|------------------|---------------------|----------------------|-------------|---------|
| | filesize | On-chip memory used | Off-chip memory used | CPU | EdgeTPU |
| 10 | 0.493 | 0.876 | 0.000 | 1200.5 | 7.8 |
| 25 | 1.240 | 2.320 | 0.000 | 3165.0 | 20.1 |
| 50 | 2.481 | 4.710 | 0.000 | 6448.1 | 40.7 |
| 75 | 3.730 | 5.000 | 1.060 | 9688.7 | 61.6 |
| 100 | 4.981 | 5.000 | 2.250 | 12991.6 | 82.7 |

[Bartvelp]

Thanks for your results, I guess that makes sense with the design of the Edge TPU. I can confirm that I'm seeing similar results.
I am also seeing results like @Namburger, where the dev board CPU is way faster than my quadcore i7, weird.

[iwatake2222]

This is just my guess but I think TensorFlow has lots of optimization for ARM CPU (like using NEON) rather than x86_64. But I'm not sure why there is difference of TPU time.

[jk78346]

I also observe the similar result with @iwatake2222's report. However, it's not intuitive for me that why edgetpu is only optimized for convolution operation and not for dense operation. I assume that conv. is fundamentally transformed into matrix multiplication as well.