Yolov3: Failing to convert model to CoreML

Created on 24 Nov 2019 · 4Comments · Source: ultralytics/yolov3

Describe the bug
I'm trying to convert the Ultralytics model to CoreML so I can use it in a mobile app, however I get an error while converting from ONNX to CoreML.

Do you have any code available that does the conversion?

To Reproduce
Steps to reproduce the behavior:

import onnx
import onnx_coreml
import torch.onnx

PYTORCH_WEIGHTS_PATH = 'yolov3/weights/ultralytics49.pt'
ONNX_MODEL_PATH = 'model.onnx'
COREML_MODEL_PATH = 'model.mlmodel'

dummy_input = torch.randn(1, 3, 416, 416)
model = models.Darknet('yolov3/cfg/yolov3-spp.cfg')
model.load_state_dict(torch.load(PYTORCH_WEIGHTS_PATH, map_location='cpu')['model'])

torch.onnx.export(model, dummy_input, ONNX_MODEL_PATH)

model = onnx.load(ONNX_MODEL_PATH)
cml = onnx_coreml.convert(model, target_ios='13')
cml.save(COREML_MODEL_PATH)

I get the following error:

yolov3/models.py:154: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  if (self.nx, self.ny) != (nx, ny):
yolov3/models.py:286: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  self.img_size = max(img_size)
yolov3/models.py:287: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  self.stride = self.img_size / max(ng)
yolov3/models.py:296: TracerWarning: torch.Tensor results are registered as constants in the trace. You can safely ignore this warning if you use this function to create tensors out of constant variables that would be the same every time you call this function. In any other case, this might cause the trace to be incorrect.
  self.ng = torch.Tensor(ng).to(device)
yolov3/models.py:296: TracerWarning: Converting a tensor to a Python float might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs!
  self.ng = torch.Tensor(ng).to(device)
yolov3/models.py:191: TracerWarning: There are 2 live references to the data region being modified when tracing in-place operator copy_ (possibly due to an assignment). This might cause the trace to be incorrect, because all other views that also reference this data will not reflect this change in the trace! On the other hand, if all other views use the same memory chunk, but are disjoint (e.g. are outputs of torch.split), this might still be safe.
  io[..., 0:2] = torch.sigmoid(io[..., 0:2]) + self.grid_xy  # xy
yolov3/models.py:192: TracerWarning: There are 2 live references to the data region being modified when tracing in-place operator copy_ (possibly due to an assignment). This might cause the trace to be incorrect, because all other views that also reference this data will not reflect this change in the trace! On the other hand, if all other views use the same memory chunk, but are disjoint (e.g. are outputs of torch.split), this might still be safe.
  io[..., 2:4] = torch.exp(io[..., 2:4]) * self.anchor_wh  # wh yolo method
yolov3/models.py:194: TracerWarning: There are 2 live references to the data region being modified when tracing in-place operator mul_. This might cause the trace to be incorrect, because all other views that also reference this data will not reflect this change in the trace! On the other hand, if all other views use the same memory chunk, but are disjoint (e.g. are outputs of torch.split), this might still be safe.
  io[..., :4] *= self.stride
yolov3/models.py:194: TracerWarning: There are 4 live references to the data region being modified when tracing in-place operator copy_ (possibly due to an assignment). This might cause the trace to be incorrect, because all other views that also reference this data will not reflect this change in the trace! On the other hand, if all other views use the same memory chunk, but are disjoint (e.g. are outputs of torch.split), this might still be safe.
  io[..., :4] *= self.stride
yolov3/models.py:197: TracerWarning: There are 2 live references to the data region being modified when tracing in-place operator sigmoid_. This might cause the trace to be incorrect, because all other views that also reference this data will not reflect this change in the trace! On the other hand, if all other views use the same memory chunk, but are disjoint (e.g. are outputs of torch.split), this might still be safe.
  torch.sigmoid_(io[..., 4:])
/usr/local/lib/python3.5/dist-packages/torch/onnx/symbolic_helper.py:198: UserWarning: You are trying to export the model with onnx:Upsample for ONNX opset version 9. This operator might cause results to not match the expected results by PyTorch.
ONNX's Upsample/Resize operator did not match Pytorch's Interpolation until opset 11. Attributes to determine how to transform the input were added in onnx:Resize in opset 11 to support Pytorch's behavior (like coordinate_transformation_mode and nearest_mode).
We recommend using opset 11 and above for models using this operator. 
  "" + str(_export_onnx_opset_version) + ". "

1/341: Converting Node Type Conv
2/341: Converting Node Type BatchNormalization
3/341: Converting Node Type LeakyRelu
4/341: Converting Node Type Conv
5/341: Converting Node Type BatchNormalization
6/341: Converting Node Type LeakyRelu
7/341: Converting Node Type Conv
8/341: Converting Node Type BatchNormalization
9/341: Converting Node Type LeakyRelu
10/341: Converting Node Type Conv
11/341: Converting Node Type BatchNormalization
12/341: Converting Node Type LeakyRelu
13/341: Converting Node Type Add
14/341: Converting Node Type Conv
15/341: Converting Node Type BatchNormalization
16/341: Converting Node Type LeakyRelu
17/341: Converting Node Type Conv
18/341: Converting Node Type BatchNormalization
19/341: Converting Node Type LeakyRelu
20/341: Converting Node Type Conv
21/341: Converting Node Type BatchNormalization
22/341: Converting Node Type LeakyRelu
23/341: Converting Node Type Add
24/341: Converting Node Type Conv
25/341: Converting Node Type BatchNormalization
26/341: Converting Node Type LeakyRelu
27/341: Converting Node Type Conv
28/341: Converting Node Type BatchNormalization
29/341: Converting Node Type LeakyRelu
30/341: Converting Node Type Add
31/341: Converting Node Type Conv
32/341: Converting Node Type BatchNormalization
33/341: Converting Node Type LeakyRelu
34/341: Converting Node Type Conv
35/341: Converting Node Type BatchNormalization
36/341: Converting Node Type LeakyRelu
37/341: Converting Node Type Conv
38/341: Converting Node Type BatchNormalization
39/341: Converting Node Type LeakyRelu
40/341: Converting Node Type Add
41/341: Converting Node Type Conv
42/341: Converting Node Type BatchNormalization
43/341: Converting Node Type LeakyRelu
44/341: Converting Node Type Conv
45/341: Converting Node Type BatchNormalization
46/341: Converting Node Type LeakyRelu
47/341: Converting Node Type Add
48/341: Converting Node Type Conv
49/341: Converting Node Type BatchNormalization
50/341: Converting Node Type LeakyRelu
51/341: Converting Node Type Conv
52/341: Converting Node Type BatchNormalization
53/341: Converting Node Type LeakyRelu
54/341: Converting Node Type Add
55/341: Converting Node Type Conv
56/341: Converting Node Type BatchNormalization
57/341: Converting Node Type LeakyRelu
58/341: Converting Node Type Conv
59/341: Converting Node Type BatchNormalization
60/341: Converting Node Type LeakyRelu
61/341: Converting Node Type Add
62/341: Converting Node Type Conv
63/341: Converting Node Type BatchNormalization
64/341: Converting Node Type LeakyRelu
65/341: Converting Node Type Conv
66/341: Converting Node Type BatchNormalization
67/341: Converting Node Type LeakyRelu
68/341: Converting Node Type Add
69/341: Converting Node Type Conv
70/341: Converting Node Type BatchNormalization
71/341: Converting Node Type LeakyRelu
72/341: Converting Node Type Conv
73/341: Converting Node Type BatchNormalization
74/341: Converting Node Type LeakyRelu
75/341: Converting Node Type Add
76/341: Converting Node Type Conv
77/341: Converting Node Type BatchNormalization
78/341: Converting Node Type LeakyRelu
79/341: Converting Node Type Conv
80/341: Converting Node Type BatchNormalization
81/341: Converting Node Type LeakyRelu
82/341: Converting Node Type Add
83/341: Converting Node Type Conv
84/341: Converting Node Type BatchNormalization
85/341: Converting Node Type LeakyRelu
86/341: Converting Node Type Conv
87/341: Converting Node Type BatchNormalization
88/341: Converting Node Type LeakyRelu
89/341: Converting Node Type Add
90/341: Converting Node Type Conv
91/341: Converting Node Type BatchNormalization
92/341: Converting Node Type LeakyRelu
93/341: Converting Node Type Conv
94/341: Converting Node Type BatchNormalization
95/341: Converting Node Type LeakyRelu
96/341: Converting Node Type Conv
97/341: Converting Node Type BatchNormalization
98/341: Converting Node Type LeakyRelu
99/341: Converting Node Type Add
100/341: Converting Node Type Conv
101/341: Converting Node Type BatchNormalization
102/341: Converting Node Type LeakyRelu
103/341: Converting Node Type Conv
104/341: Converting Node Type BatchNormalization
105/341: Converting Node Type LeakyRelu
106/341: Converting Node Type Add
107/341: Converting Node Type Conv
108/341: Converting Node Type BatchNormalization
109/341: Converting Node Type LeakyRelu
110/341: Converting Node Type Conv
111/341: Converting Node Type BatchNormalization
112/341: Converting Node Type LeakyRelu
113/341: Converting Node Type Add
114/341: Converting Node Type Conv
115/341: Converting Node Type BatchNormalization
116/341: Converting Node Type LeakyRelu
117/341: Converting Node Type Conv
118/341: Converting Node Type BatchNormalization
119/341: Converting Node Type LeakyRelu
120/341: Converting Node Type Add
121/341: Converting Node Type Conv
122/341: Converting Node Type BatchNormalization
123/341: Converting Node Type LeakyRelu
124/341: Converting Node Type Conv
125/341: Converting Node Type BatchNormalization
126/341: Converting Node Type LeakyRelu
127/341: Converting Node Type Add
128/341: Converting Node Type Conv
129/341: Converting Node Type BatchNormalization
130/341: Converting Node Type LeakyRelu
131/341: Converting Node Type Conv
132/341: Converting Node Type BatchNormalization
133/341: Converting Node Type LeakyRelu
134/341: Converting Node Type Add
135/341: Converting Node Type Conv
136/341: Converting Node Type BatchNormalization
137/341: Converting Node Type LeakyRelu
138/341: Converting Node Type Conv
139/341: Converting Node Type BatchNormalization
140/341: Converting Node Type LeakyRelu
141/341: Converting Node Type Add
142/341: Converting Node Type Conv
143/341: Converting Node Type BatchNormalization
144/341: Converting Node Type LeakyRelu
145/341: Converting Node Type Conv
146/341: Converting Node Type BatchNormalization
147/341: Converting Node Type LeakyRelu
148/341: Converting Node Type Add
149/341: Converting Node Type Conv
150/341: Converting Node Type BatchNormalization
151/341: Converting Node Type LeakyRelu
152/341: Converting Node Type Conv
153/341: Converting Node Type BatchNormalization
154/341: Converting Node Type LeakyRelu
155/341: Converting Node Type Conv
156/341: Converting Node Type BatchNormalization
157/341: Converting Node Type LeakyRelu
158/341: Converting Node Type Add
159/341: Converting Node Type Conv
160/341: Converting Node Type BatchNormalization
161/341: Converting Node Type LeakyRelu
162/341: Converting Node Type Conv
163/341: Converting Node Type BatchNormalization
164/341: Converting Node Type LeakyRelu
165/341: Converting Node Type Add
166/341: Converting Node Type Conv
167/341: Converting Node Type BatchNormalization
168/341: Converting Node Type LeakyRelu
169/341: Converting Node Type Conv
170/341: Converting Node Type BatchNormalization
171/341: Converting Node Type LeakyRelu
172/341: Converting Node Type Add
173/341: Converting Node Type Conv
174/341: Converting Node Type BatchNormalization
175/341: Converting Node Type LeakyRelu
176/341: Converting Node Type Conv
177/341: Converting Node Type BatchNormalization
178/341: Converting Node Type LeakyRelu
179/341: Converting Node Type Add
180/341: Converting Node Type Conv
181/341: Converting Node Type BatchNormalization
182/341: Converting Node Type LeakyRelu
183/341: Converting Node Type Conv
184/341: Converting Node Type BatchNormalization
185/341: Converting Node Type LeakyRelu
186/341: Converting Node Type Conv
187/341: Converting Node Type BatchNormalization
188/341: Converting Node Type LeakyRelu
189/341: Converting Node Type MaxPool
190/341: Converting Node Type MaxPool
191/341: Converting Node Type MaxPool
192/341: Converting Node Type Concat
193/341: Converting Node Type Conv
194/341: Converting Node Type BatchNormalization
195/341: Converting Node Type LeakyRelu
196/341: Converting Node Type Conv
197/341: Converting Node Type BatchNormalization
198/341: Converting Node Type LeakyRelu
199/341: Converting Node Type Conv
200/341: Converting Node Type BatchNormalization
201/341: Converting Node Type LeakyRelu
202/341: Converting Node Type Conv
203/341: Converting Node Type BatchNormalization
204/341: Converting Node Type LeakyRelu
205/341: Converting Node Type Conv
206/341: Converting Node Type Shape
207/341: Converting Node Type Gather
208/341: Converting Node Type Shape
209/341: Converting Node Type Gather
210/341: Converting Node Type Shape
211/341: Converting Node Type Gather
212/341: Converting Node Type Unsqueeze
213/341: Converting Node Type Unsqueeze
214/341: Converting Node Type Unsqueeze
215/341: Converting Node Type Concat
216/341: Converting Node Type Reshape
217/341: Converting Node Type Transpose
218/341: Converting Node Type Unsqueeze
219/341: Converting Node Type Concat
220/341: Converting Node Type Reshape
221/341: Converting Node Type Conv
222/341: Converting Node Type BatchNormalization
223/341: Converting Node Type LeakyRelu
224/341: Converting Node Type Shape
225/341: Converting Node Type Gather
226/341: Converting Node Type Cast
227/341: Converting Node Type Mul
228/341: Converting Node Type Cast
229/341: Converting Node Type Floor
230/341: Converting Node Type Shape
231/341: Converting Node Type Gather
232/341: Converting Node Type Cast
233/341: Converting Node Type Mul
234/341: Converting Node Type Cast
235/341: Converting Node Type Floor
236/341: Converting Node Type Unsqueeze
237/341: Converting Node Type Unsqueeze
238/341: Converting Node Type Concat
239/341: Converting Node Type Cast
240/341: Converting Node Type Shape
241/341: Converting Node Type Slice
242/341: Converting Node Type Cast
243/341: Converting Node Type Div
244/341: Converting Node Type Concat
245/341: Converting Node Type Upsample

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-1adf01fa2fc8> in <module>
     14 
     15 model = onnx.load(ONNX_MODEL_PATH)
---> 16 cml = onnx_coreml.convert(model, target_ios='13')
     17 cml.save(COREML_MODEL_PATH)

/usr/local/lib/python3.5/dist-packages/onnx_coreml/converter.py in convert(model, mode, image_input_names, preprocessing_args, image_output_names, deprocessing_args, class_labels, predicted_feature_name, add_custom_layers, custom_conversion_functions, onnx_coreml_input_shape_map, target_ios)
    623         print("%d/%d: Converting Node Type %s" %(i+1, len(graph.nodes), node.op_type))
    624         if disable_coreml_rank5_mapping:
--> 625             _convert_node_nd(builder, node, graph, err)
    626         else:
    627             _add_const_inputs_if_required(builder, node, graph, err)

/usr/local/lib/python3.5/dist-packages/onnx_coreml/_operators_nd.py in _convert_node_nd(builder, node, graph, err)
   2368 def _convert_node_nd(builder, node, graph, err):  # type: (NeuralNetworkBuilder, Node, Graph, ErrorHandling) -> None
   2369     converter_fn = _get_node_converter_fn(builder, node, err)
-> 2370     return converter_fn(builder, node, graph, err)
   2371 

/usr/local/lib/python3.5/dist-packages/onnx_coreml/_operators.py in _convert_upsample(builder, node, graph, err)
   1617         if len(node.inputs) > 1:
   1618             return err.unsupported_op_configuration(builder, node, graph,
-> 1619                                                     "This ONNX upsample layer has 'scales' provided as an input. CoreML upsample requires 'scales' as an attribute of the layer.")
   1620         height_scale = int(node.attrs.get('height_scale', 1))
   1621         width_scale = int(node.attrs.get('width_scale', 1))

/usr/local/lib/python3.5/dist-packages/onnx_coreml/_error_utils.py in unsupported_op_configuration(self, builder, node, graph, err_message)
     58         raise TypeError(
     59           "Error while converting op of type: {}. Error message: {} {}\n".format(node.op_type, err_message,
---> 60                                                                                   self.rerun_suggestion)
     61         )
     62 

TypeError: Error while converting op of type: Upsample. Error message: This ONNX upsample layer has 'scales' provided as an input. CoreML upsample requires 'scales' as an attribute of the layer. 
 Please try converting with higher target_ios.
You can also provide custom function/layer to convert the model.

Expected behavior
Successful conversion.

Screenshots
Not applicable.

Desktop (please complete the following information):

OS: Ubuntu
Version 16.04.5 LTS

Smartphone (please complete the following information):
Not applicable yet.

Additional context
Equivalent code works for alexnet model.

$ pip freeze

absl-py==0.6.1
astor==0.7.1
attrs==19.3.0
backcall==0.1.0
bcolz==1.2.1
bleach==3.0.2
blis==0.4.1
boto==2.49.0
boto3==1.10.26
botocore==1.13.26
catalogue==0.0.8
certifi==2019.9.11
chardet==3.0.4
Click==7.0
coremltools==3.0
cycler==0.10.0
cymem==2.0.3
decorator==4.3.0
defusedxml==0.5.0
docutils==0.15.2
en-core-web-sm==2.2.5
entrypoints==0.2.3
gast==0.2.0
gensim==3.8.1
GPUtil==1.4.0
graphviz==0.8.4
grpcio==1.16.0
h5py==2.8.0
idna==2.8
importlib-metadata==0.23
ipykernel==5.1.0
ipython==7.1.1
ipython-genutils==0.2.0
ipywidgets==7.4.2
jedi==0.13.1
Jinja2==2.10
jmespath==0.9.4
json5==0.8.5
jsonschema==3.2.0
jupyter==1.0.0
jupyter-client==5.2.3
jupyter-console==6.0.0
jupyter-contrib-core==0.3.3
jupyter-contrib-nbextensions==0.5.1
jupyter-core==4.4.0
jupyter-highlight-selected-word==0.2.0
jupyter-latex-envs==1.4.6
jupyter-nbextensions-configurator==0.4.1
jupyterlab==1.2.3
jupyterlab-server==1.0.6
jupyterthemes==0.20.0
Keras==2.3.1
Keras-Applications==1.0.6
Keras-Preprocessing==1.0.5
kiwisolver==1.0.1
lesscpy==0.13.0
llvmlite==0.30.0
lxml==4.4.1
Markdown==3.0.1
MarkupSafe==1.1.0
matplotlib==3.0.1
mistune==0.8.4
mock==2.0.0
more-itertools==7.2.0
mpmath==1.1.0
murmurhash==1.0.2
mxnet==1.1.0.post0
nbconvert==5.4.0
nbformat==4.4.0
notebook==5.7.0
numba==0.46.0
numpy==1.16.4
onnx==1.5.0
onnx-coreml==1.0
opencv-python==4.1.2.30
pandas==0.23.4
pandocfilters==1.4.2
parso==0.3.1
pbr==5.1.0
pexpect==4.6.0
pickleshare==0.7.5
Pillow==5.3.0
Pillow-SIMD==6.0.0.post0
plac==1.1.3
ply==3.11
preshed==3.0.2
prettytable==0.7.2
prometheus-client==0.4.2
prompt-toolkit==2.0.7
protobuf==3.6.1
ptyprocess==0.6.0
pycurl==7.43.0
Pygments==2.2.0
pygobject==3.20.0
pyparsing==2.3.0
pyrsistent==0.15.5
python-apt==1.1.0b1+ubuntu0.16.4.2
python-dateutil==2.7.5
pytz==2018.7
PyYAML==5.1.2
pyzmq==17.1.2
qtconsole==4.4.2
requests==2.22.0
resampy==0.2.1
s3transfer==0.2.1
scikit-learn==0.20.0
scipy==1.1.0
seaborn==0.9.0
Send2Trash==1.5.0
six==1.11.0
sklearn==0.0
smart-open==1.9.0
spacy==2.2.3
srsly==0.2.0
sympy==1.4
tensorboard==1.12.0
tensorflow==1.12.0
termcolor==1.1.0
terminado==0.8.1
testpath==0.4.2
thinc==7.3.1
torch==1.3.1
torchtext==0.4.0
torchvision==0.4.2
tornado==5.1.1
tqdm==4.39.0
traitlets==4.3.2
turicreate==5.8
typing==3.7.4.1
typing-extensions==3.7.4.1
urllib3==1.25.7
wasabi==0.4.0
wcwidth==0.1.7
webencodings==0.5.1
Werkzeug==0.14.1
widgetsnbextension==3.4.2
zipp==0.6.0

Do you have the code used to convert the model to your mobile app?

Stale bug iOS App

Source

acburigo

👍2

All 4 comments

@acburigo yes, the export pipeline is currently not working, due to incompatibilities in PyTorch's upsample layer, as you've seen. The only available option seems to be to wait for new PyTorch releases which may target more recent ONNX opset versions.

glenn-jocher on 30 Nov 2019

I have created a custom model using yolov3-tiny.cfg and got the pt file. Converted the pt file to .weights file and from there keras .h5 model as well. At the end of the model there is some Reshape_2 layer which is not allowing me to convert .h5 model to .mlmodel (coreml model).