Edgetpu: FULLY_CONNECTED op is not converted

[vmarkovtsev]

I am converting the following file with edgetpu_compiler: model.zip
It consists of a single matrix multiplication op. The inputs and outputs are UINT8.

I get

Edge TPU Compiler version 2.0.267685300
...
Operator                       Count      Status

FULLY_CONNECTED                1          Filter, bias, or other param is not constant at compile-time

[vmarkovtsev]

I have a working hypothesis. "Filter" is the second matrix in the matrix multiplication op. It does not like that it is taken as an input, hence "not constant". I'll check it.

[Namburger]

@vmarkovtsev
Are you still using the from_concrete_function api from tensorflow2.0? Can you possibly attach some snippets for creating this model?

[vmarkovtsev]

I am, and you will definitely not like the code: I am patching flatbuffers JSON. However, I want to understand how everything works inside and hack it as much as I can. It should not really matter how I generate the model after all. Not having the source code of the compiler does not help, and I am feeling like a reverse engineer.

Code

import json
import os
from subprocess import run, PIPE, STDOUT

import tensorflow as tf


# !wget https://github.com/tensorflow/tensorflow/raw/master/tensorflow/lite/schema/schema.fbs

def echo_run(*cmd):
    p = run(list(cmd), stdout=PIPE, stderr=STDOUT)
    output = p.stdout.decode()
    if output:
        print(output)
    p.check_returncode()


def test(size):
    @tf.function(input_signature=[tf.TensorSpec([size] * 2, tf.int32)] * 2)
    def bench_func(a, b):
        x = tf.linalg.matmul(a, b, transpose_b=True)  # b is not transposed, but this is a benchmark, whatever
        return x  # tf.reduce_sum(x) fails some internal assertion, see later

    converter = tf.lite.TFLiteConverter.from_concrete_functions([bench_func.get_concrete_function()])
    tflite_model = converter.convert()
    fn = "bench_model_%d.tflite" % size
    with open(fn, "wb") as fout:
        fout.write(tflite_model)
    echo_run("flatc", "-t", "--strict-json", "--defaults-json", "schema.fbs", "--", fn)
    fn_json = fn.split(".")[0] + ".json"
    with open(fn_json) as fin:
        model_json = json.load(fin)
    for t in model_json["subgraphs"][0]["tensors"]:
        if t["name"] == "MatMul_bias":
            t["type"] = "INT32"
            continue
        t["type"] = "UINT8"
        try:
            data = model_json["buffers"][t["buffer"]]["data"]
        except KeyError:
            continue
        model_json["buffers"][t["buffer"]]["data"] = data[::4]
    with open(fn_json, "w") as fout:
        json.dump(model_json, fout, indent=4, sort_keys=True)
    echo_run("flatc", "-b", "schema.fbs", fn_json)
    # os.remove(fn_json)
    echo_run("edgetpu_compiler", "-s", fn)
    os.remove(fn.split(".")[0] + "_edgetpu.log")
    return
    os.remove(fn)
    fn = fn.split(".")[0] + "_edgetpu.tflite"

test(16)

[vmarkovtsev]

I'll try a different strategy tomorrow: taking a sample tflite model which is known to be compiled, convert it to JSON and methodically strip away everything but the final fully connected layer. I will be able to find the bloody problem and share it here.

[Namburger]

@vmarkovtsev it looks like you're creating a tflite model, but it's not fully quantized, so you take advantage of flatc to turn the model into it's json representation, rewrite over some values and then use flatc to convert it back to a tflite model.
We never documented any type of support similar to this. As for the issue, your model is not fully quantize which is probably why it's not mapped. Please review this again, you'll need representative data gen and TFLITE_BUILTINS_INT8 and please refrain from tf2.0 for now if you want it to be compatible with our compiler.

P.S. My apologies on the compiler not being open source

[vmarkovtsev]

So I understood everything and found the root cause of my problems here and in the neighbor issues.
I had to patch flatbuffers to avoid https://github.com/tensorflow/tensorflow/issues/34523 and ensure the lossless tflite<>JSON roundabout, but that's minor.

There is an unfortunate limitation that was actually documented in https://coral.withgoogle.com/docs/edgetpu/models-intro/#model-requirements as "Output tensor is one-dimensional.". Both my inputs were 2D. As soon as I changed one input to be 1D, it successfully compiled. I will still be able to implement matrix multiplication by running N 1D MatMul ops in parallel and concatenating the result. Hacky, but that's the whole point.

I would leave all my recent issues open nevertheless because the misleading compiler status messages could definitely be improved.

[TaWeiYeh]

@vmarkovtsev May I ask how did you run N 1D MatMul ops in parallel on Edge TPU? My model has only one "matmul" operation layer with two 1D inputs. Before converting to the TensorFlow Lite, the model would output a correct matrix multiplication result. However, after converting to the TensorFlow Lite model, the lite model would output an incorrect matrix multiplication result.

matrix1_input = keras.Input(shape=(16,), name="matrix1", dtype=tf.float32)
matrix2_input = keras.Input(shape=(16,), name="matrix2", dtype=tf.float32)
matrix_output = tf.matmul(matrix1_input, matrix2_input, name="matmul")[0]
model = keras.Model(inputs=[matrix1_input, matrix2_input], outputs=matrix_output,
name="model")

[vmarkovtsev]

Ufff, that was some time ago, I don't remember... This blog post could probably shed the light.