Coremltools: Error converting from PyTorch to CoreML

[lp55]

I'm trying to convert a UNet model from pytorch to coreml and I'm getting the following error:

Traceback (most recent call last):
  File "convert_coreml.py", line 24, in <module>
    ctModel = ct.convert(trace,
  File "C:\Miniconda3\envs\lines\lib\site-packages\coremltools\converters\_converters_entry.py", line 292, in convert
    proto_spec = _convert(
  File "C:\Miniconda3\envs\lines\lib\site-packages\coremltools\converters\mil\converter.py", line 120, in _convert
    prog = frontend_converter(model, **kwargs)
  File "C:\Miniconda3\envs\lines\lib\site-packages\coremltools\converters\mil\converter.py", line 62, in __call__
    return load(*args, **kwargs)
  File "C:\Miniconda3\envs\lines\lib\site-packages\coremltools\converters\mil\frontend\torch\load.py", line 73, in load
    converter = TorchConverter(torchscript, inputs, outputs, cut_at_symbols)
  File "C:\Miniconda3\envs\lines\lib\site-packages\coremltools\converters\mil\frontend\torch\converter.py", line 140, in __init__
    raw_graph, params_dict = self._expand_and_optimize_ir(self.torchscript)
  File "C:\Miniconda3\envs\lines\lib\site-packages\coremltools\converters\mil\frontend\torch\converter.py", line 354, in _expand_and_optimize_ir
    _torch._C._jit_pass_canonicalize_ops(graph)
AttributeError: module 'torch._C' has no attribute '_jit_pass_canonicalize_ops'

I'm using pytorch nightly and coremltools 4.0b1 on Windows. Here's a simple code to test this:

import torch
import coremltools as ct

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = torch.hub.load('mateuszbuda/brain-segmentation-pytorch', 'unet',
    in_channels=3, out_channels=1, init_features=32, pretrained=True)
model = model.to(device)

model.eval()
dummy = torch.randn(1, 3, 512, 512).to(device)
trace = torch.jit.trace(model, dummy)

ctModel = ct.convert(trace, 
                     inputs=[ct.ImageType(name="input", shape=dummy.shape)#, 
                     #outputs=[ct.ImageType(name="output", shape=ct.Shape(shape=(1, 512, 512)))])

ctModel.save('C:\\unet.coreml')

Any ideas why this code gets that error? There are no special layers, and UNet ops are pretty standard.
Oh yeah, if I try to set the outputs parameters I get this exception: ValueError: outputs must not be specified for PyTorch. Any idea when this will be enabled?
I appreciate any help.

[DawerG]

Thanks @lp55 for reporting the issue. We are looking into it.

In the meanwhile, can you please try converting this model with torch==1.5.0?

[lp55]

Hi,

On the example code I gave, using pytorch==1.5 it worked (well I didn't actually tested the produced mode, but the convertion process was concluded), but on my trained network I got the following error:

RuntimeError: PyTorch convert function for op upsample_nearest2d not implemented

What upsample op is avaliable for pytorch conversion? I can change my training code to match it.
Thanks.

[lp55]

I read the coremltools convertion code and it seems bilinear upsampling is supported. I'll train with that and report the results afterwards.

[DawerG]

@ip55 This PR https://github.com/apple/coremltools/pull/758 was recently merged in the coremltools/master. It adds support for upsample_nearest2d. Can you try converting your model with this change?

Note: Core ML only supports upsample_nearest2d when scaling factors are integers.

[lp55]

Hi @DawerG

So here's the output after using the current master of coremltools:

Converting Frontend ==> MIL Ops: 24%|██████████▋ | 183/750 [00:00<00:01, 435.77 ops/s]WARNING:root:Saving value type of float16 into a builtin type of i8, might lose precision!
Converting Frontend ==> MIL Ops: 29%|████████████▋ | 216/750 [00:00<00:01, 394.56 ops/s]WARNING:root:Saving value type of float16 into a builtin type of i8, might lose precision!
Converting Frontend ==> MIL Ops: 41%|██████████████████▏ | 310/750 [00:00<00:01, 304.28 ops/s]WARNING:root:Saving value type of float16 into a builtin type of i8, might lose precision!
Converting Frontend ==> MIL Ops: 45%|███████████████████▉ | 339/750 [00:00<00:01, 281.43 ops/s]WARNING:root:Saving value type of float16 into a builtin type of i8, might lose precision!
Converting Frontend ==> MIL Ops: 54%|███████████████████████▉ | 408/750 [00:01<00:01, 315.19 ops/s]
Traceback (most recent call last):
File "convert_coreml.py", line 25, in
ctModel = ct.convert(trace,
File "C:\Miniconda3\envs\pytorch15\lib\site-packages\coremltools\converters_converters_entry.py", line 292, in convert
proto_spec = _convert(
File "C:\Miniconda3\envs\pytorch15\lib\site-packages\coremltools\converters\mil\converter.py", line 120, in _convert
prog = frontend_converter(model, *kwargs)
File "C:\Miniconda3\envs\pytorch15\lib\site-packages\coremltools\converters\mil\converter.py", line 62, in __call__
return load(args, *kwargs)
File "C:\Miniconda3\envs\pytorch15\lib\site-packages\coremltools\converters\mil\frontend\torch\load.py", line 86, in load
raise e
File "C:\Miniconda3\envs\pytorch15\lib\site-packages\coremltools\converters\mil\frontend\torch\load.py", line 76, in load
prog = converter.convert()
File "C:\Miniconda3\envs\pytorch15\lib\site-packages\coremltools\converters\mil\frontend\torch\converter.py", line 302, in convert
convert_nodes(self.context, self.graph)
File "C:\Miniconda3\envs\pytorch15\lib\site-packages\coremltools\converters\mil\frontend\torch\ops.py", line 55, in convert_nodes
_add_op(context, node)
File "C:\Miniconda3\envs\pytorch15\lib\site-packages\coremltools\converters\mil\frontend\torch\ops.py", line 301, in add
add_node = mb.add(x=add_inputs[0], y=add_inputs[1], name=node.name)
File "C:\Miniconda3\envs\pytorch15\lib\site-packages\coremltools\converters\mil\mil\ops\registry.py", line 62, in add_op
return cls._add_op(op_cls, *kwargs)
File "C:\Miniconda3\envs\pytorch15\lib\site-packages\coremltools\converters\mil\mil\builder.py", line 191, in _add_op
new_op.type_value_inference()
File "C:\Miniconda3\envs\pytorch15\lib\site-packages\coremltools\converters\mil\mil\operation.py", line 181, in type_value_inference
output_types = self.type_inference()
File "C:\Miniconda3\envs\pytorch15\lib\site-packages\coremltools\converters\mil\mil\ops\defs\elementwise_binary.py", line 43, in type_inference
ret_shape = broadcast_shapes(shapea, shapeb)
File "C:\Miniconda3\envs\pytorch15\lib\site-packages\coremltools\converters\mil\mil\ops\defs_utils.py", line 42, in broadcast_shapes
raise ValueError(
ValueError: Incompatible dim 2 in shapes (1, 128, -128, -128) vs. (1, 128, 128, 128)

I used float16 when possible during training to enable larger batch sizes. But I don't know why it's trying to convert float16 to i8 during this conversion process. Also why is producing this incompatible shapes??
Thanks.

[alwc]

I'm trying to convert a U-net-like model and I'm getting both ValueError: Incompatible dim 2 in shapes ... and Saving value type of float16 into a builtin type of i8, might lose precision! problems

[JacopoMangiavacchi]

I've a similar error converting a PyTorch CNN/GAN model (https://github.com/SystemErrorWang/FacialCartoonization) even using 1.5.1.

Using tracing I've basically the same ValueError: Incompatible dim 2 in shapes (1, 32, -128, -128) vs. (1, 32, 128, 128) on Converting Frontend to MIL.

Using scripting I'm experiencing instead this other RuntimeError:
temporary: the only valid use of a module is looking up an attribute but found = prim::SetAttrname="num_batches_tracked"

[leovinus2001]

In the context of scripting, prim::SetAttr pops up regularly in error logs but not sure what to do about it. For example, #817 or #802

[JacopoMangiavacchi]

Just tested with PyTorch 1.6 and CoreMLTools 4.0.b3 and I got exactly same errors above with both tracing and scripting.

[lp55]

Same thing for me. PyTorch 1.6 and CoreMLTools 4.0.b3 and I still got the same error.

[lp55]

When I use the original UNet I get a different error. Here's an example code:

import torch
import coremltools as ct

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = torch.hub.load('milesial/Pytorch-UNet', 'unet_carvana')
model = model.to(device)

model.eval()
dummy = torch.randn(1, 3, 512, 512).to(device)
trace = torch.jit.trace(model, dummy).to(device)

ctModel = ct.convert(trace.cpu(), inputs=[ct.ImageType(name="input", shape=dummy.shape)])

At first I got an error complaning about numpy.intc type. To fix that I changed coremltools\converters\mil\mil\types\type_mapping.py:201 from this:
elif np.issubclass_(nptype, np.int) or nptype == int:
to this:
elif np.issubclass_(nptype, np.int) or np.issubclass_(nptype, np.intc) or nptype == int:
After that I run again and got this error:

RuntimeError: PyTorch convert function for op 'constant_pad_nd' not implemented.

[likedan]

same issue. any fix?

[1duo]

same issue. any fix?

Are you facing the JIT pass issue? PyTorch 1.6 has been supported since coremltools==4.0b3, can you give it another try?

[1duo]

PyTorch 1.6 has been supported in coremltools==4.0b3.

[ZackPashkin]

Have the same issue PyTorch convert function for op 'constant_pad_nd' not implemented convertion effnetlite

pytorch 1.6.0
coremltools 4.0b4

Converting Frontend ==> MIL Ops:   2%|▏         | 12/564 [00:00<00:00, 3677.60 ops/s]
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-6-48baae8c011e> in <module>
     26 
     27 # Convert to Core ML using the Unified Conversion API
---> 28 model = ct.convert(
     29     traced_model,
     30     inputs=[ct.ImageType(name="input", shape=example_input.shape)], #name "input_1" is used in 'quickstart'

~/opt/anaconda3/envs/torch/lib/python3.8/site-packages/coremltools/converters/_converters_entry.py in convert(model, source, inputs, outputs, classifier_config, minimum_deployment_target, **kwargs)
    301             raise ValueError("outputs must not be specified for PyTorch")
    302 
--> 303         proto_spec = _convert(
    304             model,
    305             convert_from="torch",

~/opt/anaconda3/envs/torch/lib/python3.8/site-packages/coremltools/converters/mil/converter.py in _convert(model, convert_from, convert_to, converter_registry, **kwargs)
    132     frontend_converter = frontend_converter_type()
    133 
--> 134     prog = frontend_converter(model, **kwargs)
    135     common_pass(prog)
    136 

~/opt/anaconda3/envs/torch/lib/python3.8/site-packages/coremltools/converters/mil/converter.py in __call__(self, *args, **kwargs)
     82         from .frontend.torch import load
     83 
---> 84         return load(*args, **kwargs)
     85 
     86 

~/opt/anaconda3/envs/torch/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/load.py in load(model_spec, debug, **kwargs)
     82             print("the following model ops are MISSING:")
     83             print("\n".join(["  " + str(x) for x in sorted(missing)]))
---> 84         raise e
     85     except Exception as e:
     86         raise e

~/opt/anaconda3/envs/torch/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/load.py in load(model_spec, debug, **kwargs)
     74 
     75     try:
---> 76         prog = converter.convert()
     77     except RuntimeError as e:
     78         if debug and "convert function" in str(e):

~/opt/anaconda3/envs/torch/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/converter.py in convert(self)
    222 
    223             # Add the rest of the operations
--> 224             convert_nodes(self.context, self.graph)
    225 
    226             graph_outputs = [self.context[name] for name in self.graph.outputs]

~/opt/anaconda3/envs/torch/lib/python3.8/site-packages/coremltools/converters/mil/frontend/torch/ops.py in convert_nodes(context, graph)
     49         _logging.info("Converting op {} : {}".format(node.name, node.kind))
     50         if _add_op is None:
---> 51             raise RuntimeError(
     52                 "PyTorch convert function for op '{}' not implemented.".format(node.kind)
     53             )

RuntimeError: PyTorch convert function for op 'constant_pad_nd' not implemented.