Coremltools: Error reporting that upsample_nearest2d is not yet implemented for PyTorch conversion

[dlawrences]

Hi team

I am converting an YOLO network to CoreML through Torchscript and the Unified Conversion API.

During conversion, the following error is thrown:

RuntimeError: PyTorch convert function for op upsample_nearest2d not implemented

I guess this is similar to: https://github.com/apple/coremltools/issues/736

Is this going to be added to the next beta release? In the meantime, could I use any custom functions that I can pass to the converted to replace the upsample_nearest2d (some docs would help me a lot, I am having a hard time finding the right things it seems).

Cheers

[dlawrences]

So I guess in the meantime, if we do not want to go away from LeakyReLU and UpSample Nearest 2D, we would have to implement them as composite operators based on this: https://coremltools.readme.io/docs/composite-operators

Is there any way we would be able to point in this change directly to the MIL implemention operations?

• coremltools.converters.mil.mil.ops.defs.activation.leaky_relu
• coremltools.converters.mil.mil.ops.defs.image_resizing.upsample_nearest_neighbor

[aseemw]

Thank you for reporting the issue.
Yes, they will be added in the coming beta releases.
And yes you are right, using the composite op is a great way to integrate them in the conversion pipeline.

[HanClinto]

@dlawrences Great idea about using composite operators and just referencing the base implementations! Here is my clumsy attempt at what you suggested. It compiles and gets through most of the conversion, but gets hung up a little further down the line ("Error reading protobuf spec. validator error: Layer 'input' produces an output named 'input' which is also an output produced by the layer '__input'."). I'm guessing that's related to another issue of how I exported my original model or something -- I really don't know.

Regardless, at least it's importing these layers correctly now. Hopefully this is helpful to you or someone else who is struggling with the same thing.

Caveat: I don't know if I have the inputs to leaky_relu_ in the correct order -- the parameters of x=inputs[0] and alpha=inputs[1] may need to be swapped, but I figured it was worth a shot.

I put this near the top of my conversion script so that it registers these new pytorch operations when attempting to do conversion.

```from coremltools.converters.mil import Builder as mb
from coremltools.converters.mil import register_torch_op
from coremltools.converters.mil.frontend.torch.ops import _get_inputs
from coremltools.converters.mil.mil.var import Var, ListVar

@register_torch_op
def upsample_nearest2d(context, node):
inputs = _get_inputs(context, node)
_input = inputs[0]
output_size = inputs[1]

print("Upsample keys:")
print(*inputs)

if len(inputs) == 5:
    # For torch==1.5.0, upsample_bilinear2d (and presumably upsample_nearest2d) has 5 inputs.
    scales_h = inputs[3]
    scales_w = inputs[4]

if output_size is not None:
    # @output_size will be a list if scales was provided or a
    # single var if output size was provided
    if isinstance(output_size, list):
        output_size = [output_size[0].val, output_size[1].val]
    if isinstance(output_size, Var):
        output_size = [output_size.val[0], output_size.val[1]]

    # output size is computed using the formula
    # floor (scale * input_size) in Core ML (and PyTorch)
    # Thus, when computing the scales from the output size,
    # add a small positive constant to the output size,
    # to make sure that the floor formula results in the correct output
    # size and not 1 unit smaller, due to float precision issues
    # e.g. if output size = 34 and input size = 2, then scale will be
    # 17, which can get represented as 16.9999, resulting in an output size of 33
    # instead of 34, without this correction.
    scales_h = (output_size[0] + 1e-4) / float(_input.shape[-2])
    scales_w = (output_size[1] + 1e-4) / float(_input.shape[-1])

upsample_nn = mb.upsample_nearest_neighbor(
    x=_input,
    upscale_factor_height=(int)(scales_h),
    upscale_factor_width=(int)(scales_w),
    name=node.name)

context.add(upsample_nn)

@register_torch_op()
def leaky_relu_(context, node):
inputs = _get_inputs(context, node, expected=2)
res = mb.leaky_relu(x=inputs[0], alpha=inputs[1], name=node.name)
context.add(res)
```

[dlawrences]

Thanks @HanClinto , that's great. I'll give it a go later today and see how it goes.

In terms of your problem, I could help you if you provided a Netron file of the PyTorch model.

Cheers

[HanClinto]

@dlawrences I suspect we're using very similar models -- it sounds like you're using the Ultralytics YOLOv5 model, and I'm currently using the Ultralytics YOLOv3 model. I'm not sure how to export a file from Netron in any format other than PNG / SVG -- is that what you're asking for?

Thank you VERY much for the offer of help!

[dlawrences]

@dlawrences I suspect we're using very similar models -- it sounds like you're using the Ultralytics YOLOv5 model, and I'm currently using the Ultralytics YOLOv3 model. I'm not sure how to export a file from Netron in any format other than PNG / SVG -- is that what you're asking for?

Thank you VERY much for the offer of help!

Sorry, it's really early for me, kinda sleepy. Yea, I am using YOLOv5. Please send me the .pt file if that's something you can provide, otherwise an image would do, albeit it would be harder digging through all the ops.

[HanClinto]

@dlawrences That would be very much appreciated, thank you! I e-mailed the link to your gmail address listed in your Github profile.

[DawerG]

Hello @dlawrences @HanClinto

Can you try converting the model using this PR? https://github.com/apple/coremltools/pull/758

[dlawrences]

Hi @DawerG

I'll give it a go tomorrow. Thank you for the swift turnaround.

Cheers

[dlawrences]

Hi @DawerG

I've seen you have merged the PR by this point. I have installed the latest version of coremltools by doing:

pip install git+https://github.com/apple/coremltools.git

Conversion works well having this package. No issue for upsample_nearest2d or leaky_relu anymore.

Cheers!