One: Compiler Frontend: Operators to enable(3)

[seanshpark]

Next step of #277 / #873 operators to enable.

Please read #277 (comment) for steps

[seanshpark]

Operator | Status
-- | --
batch_normlization | --> (Mul + Add) done
fused_batch_norm | --> (Mul + Add) done
Conv2DBackpropInput(TransposeConv) | @d-krylov (#1666) done
Deconvolution (= TransposeConv ?) | (ignore this for now)
DepthwiseConv2D | @d-krylov done (add dilation: #1797)
DepthToSpace | @d-krylov (#1386) done
DilatedConvolution | (discussion #1673) @d-krylov (#1927) done
Elu | @d-krylov (#1190) done
Flatten | (discussion #1674) --> postpone
Floor | @struss (#1392) done
FloorDiv | @d-krylov (#432)(#1056) done
instance_norm | @seanshpark (#1741) done , TODO: #1804
L2Normalize | @AShedko (#1208 ) done
L2Pool | @seanshpark (#1689) done
LeakyRelu | @toomuchsalt (#1140) done
LessEqual | @kvochko (#1062) done
LocalResponseNormalize | @kvochko (#1222) done
LSTM | @d-krylov (#1477) --> postpone
MatMul | (discussion #1675) TODO: #1854
MaxPoolWithArgMax | (discussion #1676) --> postpone
Permute | (discussion #1677) --> postpone
PReLU |@AShedko (#1478) done
RealDiv (--> Div) | @kvochko (#1650) done
ReluN | (from pytorch. Ignore this for now) --> postpone
ResizeBilinear | @kvochko (#1476) done
ResizeNearestneighbor | @toomuchsalt (#1357) done
RNN | @seanshpark (#1712) --> postpone
SpaceToDepth | @kvochko (#1126) done

Finish parital implemented

Operator | Status
-- | --
AvgPool2D | @kvochko (#1851) done
Conv2D | @d-krylov (add dilation #1927 / #1848) done (there is no TF example)
Sqrt | @kvochko (#1849) done
DepthwiseConv2D | @d-krylov (add dilation: #1797) done

[binarman]

@seanshpark
What is RELUN operator?
I thought it is RELU_N1_TO_1 in #873 and #1032

[seanshpark]

What is RELUN operator?

Good question :)
I think it's from pytorch that we cannot author from TensorFlow. I'll add a comment

[binarman]

@seanshpark
Looks like DepthwiseConv2D already implemented, but not all attributes present in luci IR.
There are also no python example for this operator.

I suppose we need to add absent attributes and example, is it right?

[seanshpark]

I suppose we need to add absent attributes and example, is it right?

@binarman , thanks for checking! Yes, we need to fill other codes :)

[seanshpark]

I think DepthwiseConv2D came from compiler/exo when luci was first forked.
There are several Ops that came together.

[ko-dimi]

@seanshpark, again I have suggestion on optimizing operator support process. Now we review draft with all commits, and after that we cut draft into PRs per component (AFAIK this is formal rule we have), this means that we can do all luci changes in one PR: luci/lang, luci/import, luci/service, luci/logex, luci/export, luci/tests. Do you have motivated objections against doing all luci changes in one PR?

[binarman]

I have no objections against this proposal, if other reviewers are ok with it.

[seanshpark]

this means that we can do all luci changes in one PR

Already added my opinion about this: https://github.com/Samsung/ONE/issues/277#issuecomment-629866592

[binarman]

@seanshpark
I experimented with operators from the list and have some suggestions.
If you want, take a look at code I experimented with: examples.zip

batch_normlization
I tried to convert operator tf.nn.batch_normalization and toco converts it to mul + add sequence. We support both these operators.
Should we mark it as already enabled?

fused_batch_norm
I did not find this operator in main TF API, but it exists in raw ops: tf.raw_ops.FusedBatchNorm.
Looks like it is same as simple batch norm, but represented in TF graph as one operator.
Unfortunately toco crashed when I tried to convert this operator.
I don't see this operator in any of our target networks, maybe postpone it until we find example?

DilatedConvolution
Dilation is a part of conv 2d operator(it is not supported in current version of luci).
Is it all we need to implement for this task?

Flatten
I was looking at (https://www.tensorflow.org/api_docs/python/tf/compat/v1/layers/Flatten).
toco converts it to reshape (already supported).
Should we mark it as already enabled?

instance_norm
@d-krylov @kvochko @AShedko and everyone who is interested
Just want to note that this is circle specific operator, so we need to support import and other stuff that is not implemented yet.
There are nothing to do with tflite related components.

MatMul
I suppose this is about tf.linalg.matmul operator
Some forms of Matmul can be converted to FullyConnected(already supported).
But others are not. so matmul is a custom op. Example of such form is in attached archive.
Should we support MatMul custom op?

MaxPoolWithArgMax
I was looking at tf.nn.max_pool_with_argmax
This operator is converted to custom operator.
Should we support MaxPoolWithArgMax custom op?

Permute
I didn't find Permute operator in core TF, but it is present in keras: tf.keras.layers.Permute
It is converted to Transpose, that is already supported.
Should we mark Permute as already enabled?

RealDiv
As @kvochko mentioned in #1650 this operator is converted in ordinary div.
Should we mark RealDiv as already enabled?

[binarman]

I also want to clarify, what to do with MatMul, because we already have enabled BatchMatMul (tf.raw_ops.BatchMatMul), should we update it or create a new operator type that looks like tf.raw_ops.MatMul?

[seanshpark]

@binarman , I've added your questions(?) into separate issues for better discussion.

[seanshpark]

@binarman , I'll work on instance_norm

[binarman]

@seanshpark
FYI DepthToSpace is functional, but dumpers are missing (tfl/circle dumpers and logex), I think we will finish them tomorrow, but don't hurry closing this issue =)

[seanshpark]

but dumpers are missing

Thanks! I've remove done mark. I'll add it again after these lands :)

[seanshpark]

@binarman , I've posted #1804 for InstanceNorm, it's about adding transformation pass.
Will there be any one interested with this?

[binarman]

@seanshpark
Looks like everything is done here