Coremltools: Onnx to mlmodel conversion fails to generate .mlmodel file

[malemo]

I'm trying to convert Pytorch model to MLModel with Onnx.

My code:

```from model import BiSeNet
import torch.onnx
import torch
from onnx_coreml import convert
import coremltools

net = BiSeNet(19)
net.cuda()
net.load_state_dict(torch.load('model.pth'))

net.eval()

dummy = torch.rand(1,3,512,512).cuda()

torch.onnx.export(net, dummy, "Model.onnx", input_names=["image"], output_names=["output"], opset_version=11)

finalModel = convert(model='Model.onnx', minimum_ios_deployment_target='12')
finalModel.save('ModelML.mlmodel')
```

After the code runs Model.onnx is generated, however, .mlmodel file is not generated. There're no errors in the console.

What could be the issue?

[leovinus2001]

Could be anything, but help us narrow it down :)
First, it is worth a try to use
minimum_ios_deployment_target='13'
instead of '12'.
Secondly, please attach the .onnx model as your code does not run as class BiSeNet() is missing. Or make the test case "standalone".

[malemo]

setting the target to 13 makes no difference
Here is the model Model.onnx

[malemo]

if it helps, this is the console output:

2020-04-15 21:49:32.367179: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
WARNING:root:TensorFlow version 2.2.0-rc2 detected. Last version known to be fully compatible is 1.14.0 .
WARNING:root:Keras version 2.3.1 detected. Last version known to be fully compatible of Keras is 2.2.4 .
1.4.0
/content/drive/My Drive/Collab/fp/model.py:116: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  size_array = [int(s) for s in feat32.size()[2:]]
/content/drive/My Drive/Collab/fp/model.py:80: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  size_array = [int(s) for s in feat.size()[2:]]
/content/drive/My Drive/Collab/fp/model.py:211: TracerWarning: Converting a tensor to a Python integer might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  size_array = [int(s) for s in feat.size()[2:]]

[leovinus2001]

Hi, Thanks for attaching the Model.onnx. That helps.

With a 3 line python script like this, you can get more insight into your model.

onnx_model = onnx.load("Model.onnx")
onnx.checker.check_model(onnx_model)
onnx.helper.printable_graph(onnx_model.graph)

First, the check_model() line complains about your attribute "strides".

onnx.onnx_cpp2py_export.checker.ValidationError: Attribute 'strides' is expected to have field 'ints'

==> Context: Bad node spec: input: "283" output: "284" op_type: "AveragePool" attribute { name: "ceil_mode" i: 0 type: INT } attribute { name: "kernel_shape" ints: 16 ints: 16 type: INTS } attribute { name: "pads" ints: 0 ints: 0 ints: 0 ints: 0 type: INTS } attribute { name: "strides" type: INTS }

This may or may not be related to your console warnings like

size_array = [int(s) for s in feat32.size()[2:]]

Seems, ONNX wants INTS and it is not finding them. Change your Python code to remove the warnings, and double check
https://pytorch.org/docs/master/onnx.html#torch.onnx.export

Second, digging a bit more, the "printable_graph" ONNX line above prints the whole model, a resnet-like model

While the beginning looks good,

ONNX model description :graph torch-jit-export (
%image[FLOAT, 1x3x512x512]
) initializers (

and you provide the correct input size in the "CoreML" converter, the issue is about these nodes,
in particular %283 and %284. You can see that node %284 struggles with

strides = Unknown ](%283)

%280 = Constantvalue =
%281 = Gatheraxis = 0
%282 = Constantvalue =
%283 = Padmode = 'constant'
%284 = AveragePoolceil_mode = 0, kernel_shape = [16, 16], pads = [0, 0, 0, 0], strides = Unknown
%285 = Convdilations = [1, 1], group = 1, kernel_shape = [1, 1], pads = [0, 0, 0, 0], strides = [1, 1]
%286 = BatchNormalizationepsilon = 9.99999974737875e-06, momentum = 0.899999976158142
%287 = Relu(%286)

In other words, i would interprete this as that
your python model in Torch needs a tweak to make the type of "strides", maybe the "padding" as well. INT or anything that ONNX understands.

Also, doing this tweak, then repeating export to ONNX and printONNX model should get you to the point that you can see in ONNX model that "strides" and "pad" are handled correctly, and that the checker accept the ONNX model.

After that, convert to CoreML :)

Finally, I disabled the checker to see what console output would be
on a conversion of this ONNX model to CoreML and we get

onnx_coreml/_graph.py", line 48, in _convertAttributeProto
raise ValueError("Unsupported ONNX attribute: {}".format(onnx_arg))
ValueError: Unsupported ONNX attribute: name: "strides"
type: INTS

In other words, the CoreML converter cannot handle "strides" which is exactly what the ONNX checker complains about.

Hope that helps.

[malemo]

@leovinus2001 Thank you for your detailed answer.
I found this original Pytorch model on Github. Unfortunately, I know nothing about how it was built.
I was hoping with ONNX it would be a straight line conversion. Does this mean it cannot be converted then?

[leovinus2001]

I found this original Pytorch model on Github. Unfortunately, I know nothing about how it works or how it was built.

Thanks for the pointer to the original model. That is a nice app.
https://github.com/zllrunning/face-parsing.PyTorch

I was hoping with ONNX it would be a straight line conversion.

There is always hope :) but getting it right by (model) design is better.

Does this mean it cannot be converted then?

No, not necessarily. Based on what we have seen, with a little work, it can be converted to CoreML.
You have a few options here and with all of them you can learn.

Firstly, you could look at, e.g.,
https://developer.apple.com/machine-learning/models/
whether someone has already converted it to CoreML. I mean, the route via PyTorch to iOS via ONNX is a "normal" route and other developers run into similar issues.

Secondly, I would encourage you to dig deeper.

a) As Pytorch and ONNX are OpenSource, you could grab the PyTorch code, and look where exactly (and why ) the onnx converter fails. Then, you can make the PyTorch converter core better or more general, get a correct ONNX model, and convert to CoreML.

b) In my experience, these PyTorch to ONNX conversion issues pop up on new/non-standard/fancy code and models. When someone builds a PyTorch model and did not look how to deploy it in ONNX or CoreML, then you get this.

Therefore, looking at the FaceParse project GitHub source code you mentioned above, search for code on "stride" and "pad" as that is what gives the errors in PyTorch to ONNX conversion. Then, find model part/layer and code that is non-standard (and maps to ONNX nodes 283/284).

No guarantees but as example, after a 1 minute look at the code, this "def _global_pooling"

https://github.com/zllrunning/face-parsing.PyTorch/blob/814d8547319552088b08cf7890e34a738da3e380/modules/deeplab.py
looks non-standard to me. As an example, if that code is not correctly converted to ONNX (due to 'something' like dynamic axis, dynamic values) then all you need to do is to replace it with something that is functionally equivalent (newer?) and it CAN be converted. Which is where you come in ;)

If not functionally equivalent then retrain the model with an architecture that you KNOW can be converted to ONNX.
Finally, lots of interesting stuff to be learned. Dig deeper and give it a try!

[srikris]

@bhushan23 Tagging to make sure this can be addressed in the 4.0 release.

[bhushan23]

Thanks @malemo for reporting this at coremltools
Did you get chance to try re-installing onnx-coreml?
I am getting attribute error which is expected for version 11 model.
Also, Could you please try exporting model with opset=10 from pytorch? (https://pytorch.org/docs/stable/onnx.html#torch.onnx.export)

ref: https://github.com/onnx/onnx-coreml/issues/564#issuecomment-617336460