Coremltools: Error reporting that leaky ReLU is not yet implemented for PyTorch conversion

[jbmaxwell]

I was trying to convert a ClusterGAN from PyTorch to CoreML yesterday and got an error about leaky_relu not being implemented yet. Is there a projected timeline for this? It seems to me leaky ReLU is a very popular activation function.
As a workaround, is there a reasonably simple way to write my own for the converter to use?

Any thoughts appreciated.

[aseemw]

Hi @jbmaxwell thanks for reporting the issue!
Yes it does seem the leaky relu mapping function is not implemented yet. I see the one for relu but not for leaky relu.

It should be added in the upcoming beta releases of coremltools.

As for the timeline, generally coremltools is aligned with the Core ML framework which is aligned with the rest of the Apple OS software.

[jbmaxwell]

Okay, fingers crossed that we'll see it in the next beta—and that it will drop relatively soon! :)

I have the model running using LibTorch for now, but I'm curious about performance and have a feeling that the CoreML version will be better optimized for the hardware (correct me if I'm wrong).

[srikris]

@jbmaxwell To clarify, this implementation is a good reference right? (https://github.com/zhampel/clusterGAN)

[jbmaxwell]

Yes, that's the one.

[leovinus2001]

Hitting the same issue
RuntimeError: PyTorch convert function for op leaky_relu_ not implemented
with a non GAN model.
Hence +1 for please add it in the next beta

[leovinus2001]

Here a small test case for your convenience. @aseemw @srikris
Change bool 'showBug' for testing on/off.
testLeakyRelu.jun25.txt

[dlawrences]

Running in the same issue when converting YOLO Object Detection models to CoreML.

I am using ReLU as a work-around for now, could we also supply custom activation functions instead of LeakyReLU?

Thanks

[HanClinto]

@dlawrences Were you able to switch ReLU out for LeakyReLU after training, or did you need to re-train the model from scratch after you did that? If so, how did you change this?

[dlawrences]

@dlawrences Were you able to switch ReLU out for LeakyReLU after training, or did you need to re-train the model from scratch after you did that? If so, how did you change this?

I have changed the model architecture to use nn.ReLU (https://pytorch.org/docs/master/generated/torch.nn.ReLU.html) instead of nn.LeakyReLU (https://pytorch.org/docs/master/generated/torch.nn.LeakyReLU.html). I don't think the weights trained for the LeakyReLU activation function would have worked that well on plain ReLU tbh - I would expect some gradients would go to zero really fast for the latter one.

[DawerG]

This PR #758 adds leaky_relu and upsample_nearest2d layers. Can you please try converting this model using the change in this PR?

[leovinus2001]

Pulled coremltools, rebuilt coremltools from TOT, reran the testcase I provided above, and that passes now. Seems like LeakyRelu works. Thanks.