Coremltools: Convolution kernel uses BNNS, not H11ANE on A12X. How can I fix it?

[y-ich]

❓Question

Hi.

I am still working about #485.
The converted model works correctly but slowly on A12X.
Regardless of a value of usesCPUOnly for prediction, the speeds are same.
And Instruments' Profiler shows BNNS is used for convolution kernel, not H11ANE.
How can I let Core ML to use H11ANE?

Thanks.

System Information

• iPad Pro 11-inch
• iPadOS 13.2

[y-ich]

I have another similar question.

I have a CNN weight which ran mainly using H11ANE on iOS 12.
It runs slowly on iOS 13 using MPS.

I guess that currently broadcasting of elementwise ops prevents Core ML from using ANE on iOS/iPadOS 13.
I will appreciate if you tell me conditions at which CNN models run on H11ANE.

[y-ich]

Now I may begin to understand the issue at high level.
I guess that Core ML 2's model compiler folded Conv2D + BatchNorm but Core ML 3's one doesn't.
That is the reason why the speeds are different between iOS 12 and 13.
Is my understanding right?

[y-ich]

I became confused further.

I have another mlmodel, which was converted for iOS 12 and runs using Neural Engine both on iOS 12 and 13.
I inspected the mlmodel and found that it includes batchnorms even though the original model have no batchnorms by folding.
I saw https://github.com/tf-coreml/tf-coreml/issues/288 and understood what happened.

Then I converted the original model to mlmodel for iOS 13.
The below is a comparison between iOS 12 mlmodel(left) and iOS 13 mlmodel(right).

I confirmed that biasAdds in the original model were converted to bias of previous convolutions now.
(Actually, adds in right diagram were "addBroadcastable", but since I suspected if broadcastable ops let Core ML not to use Neural Engine, I patched conversion)

I ran the iOS 13 mlmodel on iPad Pro 11, and was so surprised that it does not run on Neural Engine!
Now I conclude that tf-coreml/coremltools with "minimum_ios_deployment_target='13'" does not support Neural Engine.
I don't believe my conclusion.
Please let me know my misunderstanding.

Since I am developing AlphaZero-type AIs, the speed of neural networks is critical. Help me!

Thank you.

[y-ich]

About the model in the previous post, the model converted with minimum_ios_deployment_target='13' had batchedMatMul and addBroadcastable near output.
I made a patch extending fuse_bias_add for add to remove addBroadcastable.
The resulting model runs on Neural Engine.

About the model which slowed down on iOS 13,
I converted the original one with minimum_ios_deployment_target='13'.
The converter converts scalar multiply and add to multiplyBroadcastable and addBroadcastable respectedly.
I made a patch to convert them to mutiply and add with alpha parameter.
The resulting model runs on Neural Engine.

I become more convinced that currently broadcasting of elementwise ops prevents Core ML from using ANE on iOS/iPadOS 13.

[aseemw]

Hi @y-ich can you please share the original Tensorflow (.pb) model(s) on which you observed the change of behavior between minimum_ios_deployment_target = '12' and '13' values.

[y-ich]

@aseemw san,

The link below is the one.
https://www.dropbox.com/s/u4j1n6srflybyt4/matmul_add.pb?dl=0

[drjinying]

Hi @y-ich and @aseemw I have a similar question. My model runs much slower after iPhone upgraded from iOS 12 to iOS 13. The PB to CoreML conversion tool does have a different behavior because the conversion failed for me after switching to the latest tfcoreml 3.1. I guess we have something in common here, have you guys solved this issue?

[y-ich]

Hello, @shonejin san.

I had made several patches and had succeeded the conversions.
I think that it is better idea for you to create another issue since this issue is for performance.
About performance, I had resolved for some of my models by my patches and fails for some other models.
What I can tell you confidently is that addBroadcastable, which is the default add op on coremltools converter, slows down models to prevent Core ML to use Neural Engine.
Replacing addBroadcastable to "add" op may restore the speed of some models.

Cheers.

[y-ich]

After many trials and errors, I managed to let my weight run on Neural Engine.

[glenn-jocher]

I've seen drastic speed reductions in our YOLOv3 model (https://github.com/ultralytics/yolov3) exported with CoreML 2 vs 3, and have verified with Xcode Instruments that the fast model has no calls to Metal or MTL, whereas the new slower model uses 50% of time on Metal calls. Using Netron viewer I see two key differences:

• add replaced by addBroadcastable
• concat replaced by concatND

If @y-ich is correct, the add > addBroadcastable operations may be performed now on Metal rather than ANE. The effect is a terrible speed reduction, from 30 FPS before to only 6 FPS now.

[glenn-jocher]

Yes, I can confirm that removing the broadcastable operations from the coreml model moved the operations from Metal to ANE, and the speed returned to normal.