Ncnn: Speed issues

[quocnhat]

Hi, thank you for sharing this code.
Is that correct if the model converted from the Pytorch is slower than from the MXnet?
I try to re-convert mobile0.25 retinaface PyTorch model to ncnn, But the speed is much slower than the one you provided (., ie: 1000ms compare to 200ms). I strickly set one core of CPU running.
Any comment is really appropriated, thanks.

[nihui]

What is the hardware / platform / operating system etc .. ?

[quocnhat]

My system is Intel(R) Core(TM) i5-7500 CPU @ 3.40GHz. Ubuntu 18.04
Your model: [https://github.com/nihui/ncnn-assets/tree/master/models] is very fast.
my model: train retinaface mobilenet0.25 pytorch [https://github.com/biubug6/Pytorch_Retinaface] -> port to ncnn ( instruction from wiki) -> using retinaface.cpp ( edit Mat in, extract string ,... ). but the speed is so much slower.
Note: Using one core CPU.
thank you.

[Timen]

I would suggest you to download Netron this will allow you to open the .param files of NCNN models and expect the model structure. What I expect is happening is that the structure of the model you converted is probably not as optimized as the one from the ncnn-assets repo. The first thing to watch out for is that the model you converted is actually a completely different architecture than the one from ncnn-assets. This can be small details like input size, number of channels or large difference in how output processing is implemented. I expect this is the case and you'll have to look into what parts are so much less efficient. A good way to do this is to use add your model to the benchmark script and build NCNN with -DNCNN_BENCHMARK=ON. This will print per layer inference time and will probably point you in the right direction of why the performance is so much worse.

However if the architecture is not so different than common issues for slow downs are:

1. Convolution and bias add not one operator, some DL frameworks don't define the bias addition as part of the convolution but instead make it an independent operator following the convolution. Splitting these ops greatly reduces efficiency and will result in a significant slowdown.
2. You didn't run ncnnoptimize on the .param file, this script can merge the activation functions and do some other optimizations that will help your inference speed.
3. Your version maybe uses layers which aren't optimized for AVX/NEON etc. there are still a large number of layers of which only the c++ reference implementation is used which is often a lot slower than the AVX/NEON version.
4. Finally, it may be that your model uses a lot of layers which number of output channels are not a multiple or 8 or even 4. The reason this is an issue is that the AVX implementation is predominantly optimized for layers which output channels are a multiple of 8, as these channels fit exactly in the 8-lane data processing supported by AVX. I won't go into to much details but making sure the layers are a multiple of 8 will increase performance by about 2X. Unfortunately this will often requiring retraining a model to make sure the channel multiple 8 constraint is respected and may be out of the scope of whatever it is you are working on.

I hope this helps, in general I suspect the model you converted is either a completely different architecture (a lot heavier) or the conversion method you used didn't produce a NCNN friendly model due to some combination of pytorch operations used in the model. Either way if you post the .param file of the model you converted it will be easier to tell exactly what is the issue.

[quocnhat]

Thank you for your very detailed explanation,
After carefully follow your guide, I have successfully executed my ncnn pytorch with the same inference time as yours.

• Using Netron, my .param model is totally different structure ( layer names, number of layers,... ) from the provided.
• Re-built another retinaface.cpp to load and extract features from this new structure.
  Now I can use what Net I want to build ncnn for my specific task.
  Big appreciated, gonna close the issue.

[paklau99988]

@quocnhat is have same probelm. Can share your code? Thanks you.,

[quocnhat]

if you use pytorch to train retinaFace, follow this git may help. (https://github.com/biubug6/Face-Detector-1MB-with-landmark/tree/master/Face_Detector_ncnn)

[paklau99988]

Thanks you, is very help.