One: [luci-value-tests] failed items for existing kernels

[seanshpark]

These items failed where I could see kernel files in compiler/luci-interpreter/src/kernels folder

• ArgMax_000
• ArgMax_001
• ArgMax_002
• ArgMax_003
• ELU_000
• FullyConnected_002 : no bias recipe
• If_000
• If_001
• LeakyRelu_000
• LocalResponseNormalization_000
• SpaceToDepth_000
• Split_000
• Unpack_000
• Unpack_001
• Unpack_002

For quantized

• ArgMax_U8_000
• ArgMax_U8_001
• ArgMax_U8_002
• ArgMax_U8_003
• DepthwiseConv2D_U8_001

Kernel not supported yet

• FullyConnected_U8_000
• Softmax_U8_000

[seanshpark]

Related #3457

[struss]

ArgMax, ArgMax_U :

• rank down error. = update after version check. fix argmax kernel shape inference.
• output value type error. = update luci_eval_verifier.py
• output value not match. = somewhat version difference make shape difference.

FullyConnected_002 :

LeakyRelu_000 :

• output value not match. TFlite mplementation change in 1.15

LocalResponseNormalization_000 :

~- because Param not passed, output value not match.~

SpaceToDepth_000 :

~- Param not passed.~

FullyConnected_U8, Softmax_U8 :

• currently not supported yet.

ELU, IF :

• Tensorflow v1.13 version not supported. ELU added in 1.14, If add in 1.15

Split, Unpack :

• Multiple output not supported in luci_eval_verifier.py

DepthwiseConv2D_U8_001 :

• Channel wise quantize example. current TFlite Error ValueError: QuantizationParam has 4 scale values (only 1 is supported).

[jinevening]

LeakyRelu_000 :
output value not match. TFlite mplementation change in 1.15

ELU, IF :
Tensorflow v1.13 version not supported. ELU added in 1.14, If add in 1.15

3561 may help to resolve the above issues.

DepthwiseConv2D_U8_001 :
Channel wise quantize example. current TFlite Error ValueError: QuantizationParam has 4 scale values (only 1 is supported).

FYI, upgrading TF to 2.3.0-rc0 can resolve this issue, but the test will still fail because luci-interpreter currently does not support per-channel quantized DepthwiseConv2D kernel.

[struss]

LeakyRelu_000 :
output value not match. TFlite mplementation change in 1.15

ELU, IF :
Tensorflow v1.13 version not supported. ELU added in 1.14, If add in 1.15

3561 may help to resolve the above issues.

Everything works okay but, Mean U8 kernel have an error around 1.(tf2.1.0 vs tf2.3.0-rc0)
This comes from TFlite::round, TFlite::round, TFlite::round and std::round,std::min,std::max.(only difference in code)

[struss]

DepthwiseConv2D_U8_001 :
Channel wise quantize example. current TFlite Error ValueError: QuantizationParam has 4 scale values (only 1 is supported).

FYI, upgrading TF to 2.3.0-rc0 can resolve this issue, but the test will still fail because luci-interpreter currently does not support per-channel quantized DepthwiseConv2D kernel.

In TFlite V2.3.0-rc0, only Channel Wise Quantized input Type is int8 or int16.
https://github.com/tensorflow/tensorflow/blob/99fea8da0d98fb271b60b58cfa5755f2bd430079/tensorflow/lite/kernels/kernel_util.cc#L71

[jinevening]

In TFlite V2.3.0-rc0, only Channel Wise Quantized input Type is int8 or int16.
https://github.com/tensorflow/tensorflow/blob/99fea8da0d98fb271b60b58cfa5755f2bd430079/tensorflow/lite/kernels/kernel_util.cc#L71

We do not have a plan to bring the kernels for uint8 channel-wise quantized models, so let's exclude DetpwiseConv2D_U8_001 from the test list. It would be better if you leave some comments in test.lst for other people, for example

#addeval(DepthwiseConv2D_U8_001) # UINT8 quantized model, not supported by luci-interpreter

[struss]

So far, kernels belows are not test in luci-value-test

Result value not matching from version mismatching

- Mean_U8_000

Kernel Function spec not matching

- DepthwiseConv2D_U8_001

Kernel not supported yet

- FullyConnected_U8_000 -> hybrid example. so, this is need to be modified.
- Softmax_U8_000

[struss]

only version up : #3785 Done.
Uint8 with CWQ kernel addition : Will discuss later on......
and More Uint8 LWQ kernel : Softmax will be ready, but FullyConnected is hybrid. so new res will be added and current 000 is comment out.

[struss]

Close. Uint8 with CWQ kernel addition will be done later.