One: Compiler FE: NCHW to NHWC conversion with ONNX models

[seanshpark]

There's an option convert_nchw_to_nhwc in _circle2circle_ which converts NCHW data format to NHWC for some (NOT all) operators for ONNX model conversion.

This issue is to share the status or any problems of this functionality.

[seanshpark]

CC @lemmaa , @jinevening

[seanshpark]

@jinevening , could you please add a link if exist, or some desctiption about how ConvertNCHWToNHWCPass works?

• how NCHW to NHWC conversion is done by Ops
• currently working Ops or unsupported Ops
• any conditions for conversion
• etc

This is not urgent but would be nice to leave some documents.

[jinevening]

could you please add a link if exist, or some desctiption about how ConvertNCHWToNHWCPass works?

https://github.com/Samsung/ONE/issues/5401 describes the overall idea why we need to convert NCHW to NHWC and why we need to do that.

how NCHW to NHWC conversion is done by Ops

Basic flow (Iterate through operators)

1. Change the shape of ifm/ofm from NCHW to NHWC.
2. Change the operator-specific information, e.g., axis of Concat, paddings of Pad.
3. Insert Transpose Ops properly.
4. Remove the Transpose Ops

1, 2, 3 are done by ConvertNCHWToNHWCPass and 4 is done by RemoveRedundantTransposePass.

currently working Ops or unsupported Ops

Working ops

Input (we can choose not to convert by setting --nchw_to_nhwc_preserve_input_shape in one-optimize)
Output (we can choose not to convert by setting --nchw_to_nhwc_preserve_output_shape in one-optimize)
Add
Concat
Leaky relu
Mul
Neg
Pad
Relu
Relu6

They were added whenever necessary.

any conditions for conversion

We cannot identify the layout of operators. We just assume the operators listed above are NCHW. Limitations for each operator is described in the source code.

[seanshpark]

@jinevening , thanks for the explanation!

Just curious, what about Conv2D and Pool2D type ops? They are not listed in the code(in your list) but we have those in ONNX models...

[jinevening]

what about Conv2D and Pool2D type ops?

They are already converted to NHWC by onnx-tf.

What convert_nchw_to_nhwc does is converting NCHW operators (which were not converted by onnx-tf) to NHWC.

[seanshpark]

They are already converted to NHWC by onnx-tf.

Ah... do you know the op list converted by onnx-tf itself? sorry for asking as I did enable the conversion but did not have much awareness... -_-;

[jinevening]

do you know the op list converted by onnx-tf itself?

Unfortunately, I don't know the whole list (I think that information should be written somewhere in onnx-tf, but I couldn't find the clear list).

One way to find such an Op is to see the converted model (Transpose Ops are generated before/after the converted Op).

[seanshpark]

Transpose Ops are generated before/after the converted Op

Thanks for the hint :) I'll dig in the onnx-tf code sometime :)

[seanshpark]

From https://github.com/Samsung/ONE/issues/6784#issuecomment-840293139

Below Ops are currently surrounded with tf.transpose to handle NCHW to NHWC conversion.

Ops from Mixin

• ConvMixin
  
  
  
  • Conv, ConvTranspose, QLinearConv, ConvInteger
  
  
• PoolMixin
  
  
  
  • AveragePool, LpPool, MaxPool
  
  
• GatherAndScatterMixin
  
  
  
  • GatherElements, GatherND, Gather, ScatterElements, ScatterND
  
  
• ScanMixin
  
  
  
  • Scan
  
  

Ops

• DepthToSpace
• GatherElements
• Gemm
• GRU
• Hardmax
• LSTM
• NonMaxSuppression
• Resize
• RNN
• RoiAlign
• SpaceToDepth
• TfIdfVectorizer
• TopK

[seanshpark]

closing as there seems no further activities to do.
please reopen or add new issue if we need more information or items to do.

[lemmaa]

@seanshpark , @jinevening , thanks for the detailed explanation and discussion.

It's a slightly different perspective, can we change the model designed in NCHW to NHWC as we like? If we have to change the model for inference performance, backend compatibility or other reasons, the layout of the input/out of the model should not change from the model user's point of view. Considering that, it is necessary to organize the conversion options well.

/cc @dongju-chae , @parjong

[seanshpark]

the layout of the input/out of the model should not change from the model user's point of view.

There is an option for this. As desctribed from above @jinevening ,

Input (we can choose not to convert by setting --nchw_to_nhwc_preserve_input_shape in one-optimize)
Output (we can choose not to convert by setting --nchw_to_nhwc_preserve_output_shape in one-optimize)

These two options ONLY apply when --convert_nchw_to_nhwc is given and will NOT convert input/output, that is input should be NCHW and output will be NCHW too.

[seanshpark]

@lemmaa , if you meant for other thing, please describe more, if not, you can close this issue or notifiy me so that I'll close it again.

[lemmaa]

@seanshpark , this is a slightly high level story. Maybe UX? In short, it's about keeping it up unless we explicitly give the option to change the model's input format at compile time. In addition, it must be well-prepared not only in compilation, but also throughout the whole inference process.

In other words, when the compiler compiles a model designed with NCHW input, it creates a model that receives NCHW input by default, and the backend system in charge of inference, that is, runtime, system service, device driver, etc., understands its characteristics well while inference.

As an example of a compiler perspective, when ONNX models designed to receive NCHW input are compiled, a circle model that receives NCHW input comes out.

I think the compiler already well equipped with the necessary basic options. So, it only needs a little touch in terms of UX. In contrast, in the case of the inference chain, there is a part that I do not know yet, so I think I need to look at it further. :)

[seanshpark]

Um.. it's a bit hard to understand what you want. I'll open a new issue for you request.