Yet-another-efficientdet-pytorch: inference time of B0

[lampsonSong]

Thanks for the great work. The analysis of other versions is excellent. I tired your model. It runs precisely. My question is about the model inference time. The D0 version runs inference with 268ms in average with GTX1070. It is different with what paper claims. Do you get the same performance on speed ? Thanks again.

[zylo117]

hi, the first time is always slow, because cuda needs to initiate and the network/input needs to load into gpu.

And the anchor will be generated during the first input or when input_size has changed.

So all in all, the first time is slow.

I ran coco_eval with D0 on 2080Ti with batchsize 1, I got 30 FPS including image preprocessing and result postprocessing.

[lampsonSong]

hi, the first time is always slow, because cuda needs to initiate and the network/input needs to load into gpu.

And the anchor will be generated during the first input or when input_size has changed.

So all in all, the first time is slow.

I ran coco_eval with D0 on 2080Ti with batchsize 1, I got 30 FPS including image preprocessing and result postprocessing.

Hi, you are right. I run your efficientdet_test.py for 100 times and get the average inference time. I found the time consuming code is :

40 x = torch.tensor(framed_imgs).cuda()

Use from_numpy is faster. Since I test with only one image, I change this line to :+1:

x = torch.from_numpy(framed_imgs[0]).cuda()

The speed is normal.

[zylo117]

hi, the first time is always slow, because cuda needs to initiate and the network/input needs to load into gpu.
And the anchor will be generated during the first input or when input_size has changed.
So all in all, the first time is slow.
I ran coco_eval with D0 on 2080Ti with batchsize 1, I got 30 FPS including image preprocessing and result postprocessing.

Hi, you are right. I run your efficientdet_test.py for 100 times and get the average inference time. I found the time consuming code is :

40 x = torch.tensor(framed_imgs).cuda()

Use from_numpy is faster. Since I test with only one image, I change this line to

x = torch.from_numpy(framed_imgs[0]).cuda()

The speed is normal.

yes, torch.tensor is very slow indeed, I guess it'd infer the data type and then vaildate size or so on, from_numpy is much faster.

I used this in my latest code,

x = torch.stack([torch.from_numpy(fi).cuda() for fi in framed_imgs], 0)

this is useful and fast when dealing with a list (not numpy array) of images with the same size

[claire-s11]

Hi @zylo117 , thank you for your great repo!
I'm wondering why the inference time is so different from the official release? It took around 0.2 secs for batchsize == 1 for EfficientDet-D1. Did you test it?

Also, I tried to scan the Alipay code, but it says payment is unsuccessful because I'm in the U.S. I do wanna donate to express my thanks to you so can you provide another code or add me on WeChat? My wechat id is 747148072
Thank you!

[zylo117]

Hi @zylo117 , thank you for your great repo!
I'm wondering why the inference time is so different from the official release? It took around 0.2 secs for batchsize == 1 for EfficientDet-D1. Did you test it?

Also, I tried to scan the Alipay code, but it says payment is unsuccessful because I'm in the U.S. I do wanna donate to express my thanks to you so can you provide another code or add me on WeChat? My wechat id is 747148072
Thank you!

Thank you for your generosity. It's alright if alipay fails, I really appreciated.

About the speed, the first time running is always slow, like I mentioned here, https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch/issues/4#issuecomment-610728580

After the first time inference, it will be much faster.

Also, the efficientdet_test.py has a param named 'force_input_size', I set it to 1920 to boost the precision by a lot, the side effect is that it will be slower than 640 (D1 official input size). Set it to None if you care about speed instead of extra precision.

[claire-s11]

Hi @zylo117 , thank you for your reply!
For the speed issue, my question isn't about your model's inference time but the inference time of the model from the official release (https://github.com/google/automl/tree/master/efficientdet). Did you try inference using those models? It took around 0.2 secs for batchsize == 1 for EfficientDet-D1 (I ran it multiple times so it's not the first-time-runing issue). I'm just so confused why the inference time is so different.

[zylo117]

Hi @zylo117 , thank you for your reply!
For the speed issue, my question isn't about your model's inference time but the inference time of the model from the official release (https://github.com/google/automl/tree/master/efficientdet). Did you try inference using those models? It took around 0.2 secs for batchsize == 1 for EfficientDet-D1 (I ran it multiple times so it's not the first-time-runing issue). I'm just so confused why the inference time is so different.

I try running my D1 with coco_eval, my 2080Ti can do 20 FPS with batchsize 1, so 0.05 secs/image.

AFAIK, automl use numpy function to do postprocessing, including nms, which is super slow.

check out here, https://github.com/google/automl/blob/master/efficientdet/anchors.py#L575

I guess I know why they use numpy function, because I tried run it with the tf version of postprocessing. Its width/height, x/y are inverted, but I forked and fixed it.
https://github.com/zylo117/automl/commit/0f96135a00f3fbb918cade96aab6ef2a15dc9935

you should also see if 'disable_pyfun' is False, if False, it uses numpy to do postprocessing.

You can try using the tf version to infer, see if it is faster

[claire-s11]

I try running my D1 with coco_eval, my 2080Ti can do 20 FPS with batchsize 1, so 0.05 secs/image.

AFAIK, automl use numpy function to do postprocessing, including nms, which is super slow.

check out here, https://github.com/google/automl/blob/master/efficientdet/anchors.py#L575

I guess I know why they use numpy function, because I tried run it with the tf version of postprocessing. Its width/height, x/y are inverted, but I forked and fixed it.
zylo117/automl@0f96135

you should also see if 'disable_pyfun' is False, if False, it uses numpy to do postprocessing.

You can try using the tf version to infer, see if it is faster

Hi @zylo117 Thank you for your explanation!
Did you test it using the inference.py in the repo? I timed the buid_model and det_post_process part and they both took a long time with 1080Ti (around 0.2 secs/image). Could you tell me which part you time? Thank you so much for your help!

[zylo117]

I try running my D1 with coco_eval, my 2080Ti can do 20 FPS with batchsize 1, so 0.05 secs/image.
AFAIK, automl use numpy function to do postprocessing, including nms, which is super slow.
check out here, https://github.com/google/automl/blob/master/efficientdet/anchors.py#L575
I guess I know why they use numpy function, because I tried run it with the tf version of postprocessing. Its width/height, x/y are inverted, but I forked and fixed it.
zylo117/automl@0f96135
you should also see if 'disable_pyfun' is False, if False, it uses numpy to do postprocessing.
You can try using the tf version to infer, see if it is faster

Hi @zylo117 Thank you for your explanation!
Did you test it using the inference.py in the repo? I timed the buid_model and det_post_process part and they both took a long time with 1080Ti (around 0.2 secs/image). Could you tell me which part you time? Thank you so much for your help!

How did you run the model? If by default, it should be static graph on tensorflow v1, not eager execution.

Unlike pytorch which runs line by line, TF builds the mode before inference. It's not fair to time
the buid_model and det_post_process part, instead, you should time 'sess.run', that's the real inference.

[claire-s11]

Hi @zylo117 , thank you so much! Here's what I did to time the inference:

1. I run the model_inspect.py and time driver = inference.InferenceDriver(self.model_name, self.ckpt_path, self.image_size), it took about 1 min. I realized it includes the time of postprocess and visualization, so I want to just time the model part.
2. So I timed the sess.run in inference.py, it took 0.2-0.3 secs/image. Here's the layer-to-layer profiling using from tensorflow.python.client import timeline
   
   You can see the inference time is about 0.2 secs/image
3. I also timed the buid_model and det_post_process, but as you said, I shouldn't time the model-building part.

[zylo117]

Hi @zylo117 , thank you so much! Here's what I did to time the inference:

1. I run the model_inspect.py and time driver = inference.InferenceDriver(self.model_name, self.ckpt_path, self.image_size), it took about 1 min. I realized it includes the time of postprocess and visualization, so I want to just time the model part.
2. So I timed the sess.run in inference.py, it took 0.2-0.3 secs/image. Here's the layer-to-layer profiling using from tensorflow.python.client import timeline
   
   You can see the inference time is about 0.2 secs/image
3. I also timed the buid_model and det_post_process, but as you said, I shouldn't time the model-building part.

emmmmm... It seems fusedconv2d, topkv2 and pyfunc take the most of the time, but even if they optimize it, the other part is still slow.

[claire-s11]

@zylo117 Yes, exactly. Just to confirm the inference time 0.05 sec/image you tested is from your PyTorch version D1 model right? I'm wondering why the inference time has such a difference with the same model architecture.

[zylo117]

yes, coco_eval on D1 can achieve 20FPS, 0.05s.

In official efficientdet, the image input is already a TF tensor before sess.run.You can eval the image input before sess.run to confirm that. However in my coco_eval, the 0.05s includes the image file reading/resizing/totensor/tocuda, so the pytorch version is way faster than the official one.

[CanyonWind]

In official efficientdet, the image input is already a TF tensor before sess.run.You can eval the image input before sess.run to confirm that. However in my coco_eval, the 0.05s includes the image file reading/resizing/totensor/tocuda, so the pytorch version is way faster than the official one.

Obviously the pytorch implementation's speed surpasses the TF version's by a huge margin, which is super impressive. But could you please share some thoughts on what causes it? Since the TF version feeds a tensor directly, it shouldn't be the IO problem. Is it caused by different frameworks themselves? How could a static graph lose on speed wise to a dynamic graph...

[zylo117]

you're right. static execution should be faster, but it's not, not this time.

There is a timing here.

https://github.com/zylo117/Yet-Another-EfficientDet-Pytorch/issues/4#issuecomment-611337716

The pyfunc and the fused conv2d are what confuse me.

Why is there a fusedconv2d? It should not be there.

And pyfunc is a numpy post process including nms. So it should be slower than TF nms, right? But it is not, as I show in readme. TF gpu nms is slower!

And then I tried some simpler image containing 1 or 2 objects.

Now that the gap between the TF and pytorch is narrowed, but pytorch is still 5 times faster. Also, even in this case, numpy nms is still faster.

I did trible check the TF code, I don't see any part that causes performance loss. In fact, I even referred a lot of code from it. However running on pytorch is totally fine.

It might be cudnn or cuda issue as well since TF doesn't static compile them.

I'd like someone else do the testing again to confirm the speed.

[dschoerk]

Hi @zylo117 , thank you for your reply!
For the speed issue, my question isn't about your model's inference time but the inference time of the model from the official release (https://github.com/google/automl/tree/master/efficientdet). Did you try inference using those models? It took around 0.2 secs for batchsize == 1 for EfficientDet-D1 (I ran it multiple times so it's not the first-time-runing issue). I'm just so confused why the inference time is so different.

takes 200ms on a 2080Ti for me too. did you find out more meanwhile?

could linux vs windows be the reasons for that huge difference?

[zylo117]

running on windows is slower. and it's faster after the first inference.