Incubator-mxnet: the inference speed using C++ API with mxnet of higher version is slower than lower mxnet

[IvyGongoogle]

Hello, I have a problem:
I use libmxnet.so compiled with mxnetv0.8 comparison to mxnetv1.0(or v1.3 and v1.4) to run my code to infer a batch images using C++ API, I find the inference speed with mxnet of higher version is slower than mxnetv0.8. What causes this problem? How to fix it? anyone can give some advises?

envs: cuda8/cudnn5.1.10/nvidia-driver384.81

[mxnet-label-bot]

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: Performance

[pengzhao-intel]

Could you try the CPU inference which is significantly improved since 0.7?

https://mxnet.incubator.apache.org/versions/master/tutorials/mkldnn/MKLDNN_README.html

[wkcn]

The issue may be similar to
https://github.com/apache/incubator-mxnet/issues/13928

However, I do not have any machine with NVIDIA GPU to test it : (

[IvyGongoogle]

@wkcn I dont know which op in my model is slower in newer mxnet. no dilated convolutional layers in my model which is a ocr recognition model with simple cnn and rnn. Can give some detailed test advises?

[wkcn]

@IvyGongoogle Could you please use the profiler to measure the execution time?
http://mxnet.incubator.apache.org/versions/master/tutorials/python/profiler.html?highlight=profiler

How much time in your model among different MXNet?
Since MXNet supports large tensors now, it may drop a little performance.

[IvyGongoogle]

@wkcn sorry, I can not find some C++ codes about how to use profiler.

[wkcn]

@IvyGongoogle Could you please provide the forward time between different versions of MXNet? Is it obvious?

[IvyGongoogle]

@wkcn thanks for your reply,

mxnet v0.8:
 initialization time(that is `MXPredCreat` function time): 69906ms(the first run)/1416ms/1419ms/1471ms, my model inference time:
218ms/209.3ms/207.6ms/207.4ms
mxnet v1.0.0:
 initialization time(that is `MXPredCreat` function time): 170572ms(the first run)/2779ms/2794ms/2756ms, my model inference time:
218.5ms/218.9ms/215.5ms/224.8ms

as we can see that the initialization time of higher mxnet is nearly double in contrast to mxnet v0.8, and my model inference time is also slower.

[sandeep-krishnamurthy]

@leleamol - Can you please take a look at this issue and suggest why it may be high initialization time with latest MXNet?

@IvyGongoogle - Is it possible to share a minimum reproducible example?

[sandeep-krishnamurthy]

Also, it may be related to this issue - https://github.com/apache/incubator-mxnet/issues/14569
where we have changed few things, like moving to Int64, to support large tensors.