One: [onert] NYI CPU backend kernel - int8 quantization

Created on 19 Oct 2020 · 6Comments · Source: Samsung/ONE

Prerequest

[x] Introduce data type for int8 quantization

NYI cpu backend kernel: int8 quantization type

[x] Add #5509
[x] ArgMax
[x] ArgMin
[x] AvgPool2D #5593
[ ] BatchMatmul
[ ] BatchToSpaceND
[ ] BiDirectionalSequenceLSTM
[ ] BiDirectionalSequenceRNN
[ ] Ceil
[x] Concat #5203
[x] Conv2D #5619
[x] DepthToSpace #5181
[ ] Densify
[x] DepthwiseConv2D
[x] Dequantize #4657
[ ] Div
[ ] EmbeddingLookup
[ ] Equal
[ ] Exp
[x] ExpandDims
[ ] FullyConnected
[ ] Gather
[ ] GatherNd
[ ] Greater
[ ] GreaterEqual
[ ] HardSwish
[ ] HashtableLookup
[ ] L2Normalization
[ ] LeakyRelu
[ ] Less
[ ] LessEqual
[ ] LocalResponseNormalize
[ ] LogicalAnd
[ ] LogicalOr
[ ] Logistic
[ ] LogSoftmax
[ ] LSHProjection
[ ] MatrixDiag
[ ] MatrixSetDiag
[ ] Maximum
[x] MaxPool2D
[ ] Mean
[ ] Minimum
[ ] MirrorPad
[x] Mul #5476, #5485, #5488, #5495
[ ] NotEqual
[ ] OneHot
[ ] Pack
[x] Pad #5542
[x] PadV2 #5542
[ ] Pow
[ ] PReLU
[x] Quantize #5446, #5470
[x] Rank #4756
[ ] ReduceMax(Max)
[ ] ReduceMin(Min)
[ ] ReduceProd
[ ] ReduceSum(Sum)
[ ] ReLU
[ ] RELU_N1_TO_1
[ ] ReLU6
[x] Reshape
[x] ResizeBilinear #5463
[ ] ResizeNearestneighbor
[ ] RNN
[ ] ScatterND
[ ] Select
[ ] SelectV2
[x] Shape #4756
[ ] Slice
[x] Softmax #5511
[ ] SpaceToBatchND
[ ] SpaceToDepth
[ ] SparseToDense
[ ] Split
[ ] SplitV
[x] Squeeze
[ ] StridedSlice
[x] Sub #5512
[ ] Svdf
[ ] Tanh
[ ] TopKV2
[ ] Transpose
[ ] TransposeConv
[ ] UniDirectionalSequenceLSTM
[ ] UniDirectionalSequenceRNN
[ ] Unique
[ ] Unpack(Unstack)

Source

hseok-oh

All 6 comments

@hseok-oh As you already know, some operators need to be implemented in priority because they are used in one of our target model.

[x] ADD
[x] AVERAGE_POOL_2D
[x] CONV_2D (CWQ)
[x] DEPTHWISE_CONV_2D (CWQ)
[x] MUL
[x] PAD
[x] RESIZE_BILINEAR
[x] SOFTMAX
[x] SUB

Note that Conv2D and DepthwiseConv2D use channel-wise quantization.

Also we need to take a look

[x] QUANTIZE (UINT8 → INT8 and INT8 → UINT8)

because it got segmentation fault during running QUANTIZE only nnpackage.

$ LD_LIBRARY_PATH=lib Product/out/bin/nnpackage_run --nnpackage quantize 
Package Filename quantize
Segmentation fault

(ADDED)
I will take a look QUANTIZE, then get started with MUL and SUB.

(ADDED)
I've found there is problem selecting quantize from our target model by select_operator.py.
I've tried to fix it, however, it is another problem. I will make another issue with current investigation.
I've continued with workaround (#5353). Now, it shows:

$ LD_LIBRARY_PATH=lib Product/out/bin/nnpackage_run --nnpackage quantize
Package Filename quantize
Error during model loading : OperationValidator failed at line 288

Quantize does not handle INT8 input.

I will start to enable INT8 input type for Quantize since Quantize is the first operator of our target model.