One: [luci-interpreter] Testing of luci-interpreter

[jinevening]

luci-interpreter has been introduced (#598) 👏 . Currently, luci-interpreter only has the skeleton codes with no operator support. But operators will be added soon, and they need to be tested.

This issue is about how to implement the test framework for luci-interpreter. The overall test process will proceed as follows

Step 1: Generate tflite and circle files from TFLite recipe
"recipe" -> tflchef -> "tflite file" -> tflite2circle -> "circle file"

Step 2: Run TFLite interpreter and luci-interpreter for tflite and circle, respectively
circle -> luci-interpreter -------> Execution result 1
tflite -> TFLite interpreter -----> Execution result 2

Step 3: Compare execution result 1 and 2. The result must be the same.

Pros & Cons of this approach
Pros: We can use existing test resources (TFLite recipes).
Cons: Only TFLite-supported Ops can be tested. We may need another method to test non-TFLite-supported Ops.

[s-barannikov]

@jinevening
I must have misunderstood you earlier.

But operators will be added soon, and they need to be tested.

They should be tested with _unit_ tests, and I'm going to add these tests along with kernels.

What you are suggesting is _integration_ testing, because it tests integration of several components, including tflchef, tflite2circle, luci importer _and_ luci interpreter. If a test fails it does _not_ mean that the error is in the interpreter component, it does mean that the components are not integrated well (assuming they have no errors, as should have already been verified by their respective _unit_ tests).

The interpreter was first introduced as a _testing tool_ for _system_ testing, e.g. to test the full pipeline tflite2circle->importer->optimizer->quantizer->exporter.
Of course it can be used in integration testing as well.

Everything I said above does not imply that you are doing something wrong, it only clarifies terminology 🙂
That said, I don't mind if the testing system you are implementing is located luci_interpreter/tests directory (as you've shown in the draft PR), but I think it is not the most appropriate place, because it tests integration of several equally important components. I would put it into luci/integration_tests directory if we had _all_ circle-related components inside luci directory. Unfortunately we don't (and it is one of the things I don't understand about the project structure).

[jinevening]

They should be tested with unit tests, and I'm going to add these tests along with kernels. What you are suggesting is integration testing, because it tests integration of several components, including tflchef, tflite2circle, luci importer and luci interpreter.

I agree that this is integration testing. I just followed the naming convention of similar tests, such as luci_unit_readtest and luci_unit_writetest, when naming luci_unit_evaltest. (luci_unit_readtest and luci_unit_writetest are also using several components for testing).

I think the important thing is that we need a testing framework for incoming kernels. This integration testing can at least assure that the execution result of each kernel is correct (if the kernel has passed the test).

I would put it into luci/integration_tests directory if we had all circle-related components inside luci directory.

@seanshpark wants to make the test directory outside of luci, under the compiler directory (talked offline). Do you have any suggestion on the directory name? @seanshpark recommended luci-integration-test and luci-value-test (see #848).

[seanshpark]

I think the important thing is that we need a testing framework for incoming kernels.

They should be tested with unit tests, and I'm going to add these tests along with kernels.

@jinevening , I see as _luci-interpreter_ will have its own unit test and this looks fine.
So, I think we can focus on integration test.

[s-barannikov]

Do you have any suggestion on the directory name? @seanshpark recommended luci-integration-test and luci-value-test (see #848).

I like both.

[jinevening]

luci-value-test was added under the compiler directory. Close this PR.