Yolov5: convert yolov5 to openvino

[linhaoqi027]

Before You Start

train your yolov5 model on your own dataset following Train Custom Data

• change Activation Function to Relu or Leaky ReLU, because Hardswish is not suported in onnx and openvino.Pre-trained model can be loaded ignoring which Activation Function u use.
  https://github.com/ultralytics/yolov5/blob/5e0b90de8f7782b3803fa2886bb824c2336358d0/models/common.py#L26

export model to onnx，following #251

before run export.py:

• changeto torch==1.5.1,torchvision==0.6.1
• change https://github.com/ultralytics/yolov5/blob/5e0b90de8f7782b3803fa2886bb824c2336358d0/models/yolo.py#L49-L53

                c=(y[..., 0:2] * 2. - 0.5 + self.grid[i].to(x[i].device)) * self.stride[i]  # xy
                d=(y[..., 2:4] * 2) ** 2 * self.anchor_grid[i]  # wh
                e=y[..., 4:]
                f=torch.cat((c,d,e),4)
                z.append(f.view(bs, -1, self.no))

        return x if self.training else torch.cat(z, 1)

• change https://github.com/ultralytics/yolov5/blob/5e0b90de8f7782b3803fa2886bb824c2336358d0/models/export.py#L31

model.model[-1].export = False

• set https://github.com/ultralytics/yolov5/blob/5e0b90de8f7782b3803fa2886bb824c2336358d0/models/export.py#L51-L52
  to opset_version=10.Because only opset=10 support resize ops.

then you can run export.py to export onnx model.

install OPENVINO2020R4

you can install following https://bbs.cvmart.net/topics/3117

convert onnx to openvino through

python3 /opt/intel/openvino_2020.4.287/deployment_tools/model_optimizer/mo.py \
--input_model {input_dir}/yolov5s.onnx \
--output_dir {output_dir}  \
--input_shape [1,3,480,480]

then you can get openvino model .bin and .xml.

WARNING:NMS is not included in openvino model.

AS for how to use openvino to inference ,please click my profile or https://github.com/linhaoqi027/yolov5_openvino_sdk

[0806gcx]

When I changed the py file as you said, this error occurred. After I change the yolo.py file, do I need to retrain the networkmodel?

[linhaoqi027]

When I changed the py file as you said, this error occurred. After I change the yolo.py file, do I need to retrain the networkmodel?

NO,u don't need to retrain the model. mabe you can change e=y[...,4:6] to e=y[..., 4:] .I write e=y[...,4:6] because I just have one class. Please tell me if it works so i can solve the bug.

[0806gcx]

When I changed the py file as you said, this error occurred. After I change the yolo.py file, do I need to retrain the networkmodel?

NO,u don't need to retrain the model. mabe you can change e=y[...,4:6] to e=y[..., 4:] .I write e=y[...,4:6] because I just have one class. Please tell me if it works so i can solve the bug.

well,thank you well ,I have successfully converted, but will you encounter this warning?

[linhaoqi027]

When I changed the py file as you said, this error occurred. After I change the yolo.py file, do I need to retrain the networkmodel?

NO,u don't need to retrain the model. mabe you can change e=y[...,4:6] to e=y[..., 4:] .I write e=y[...,4:6] because I just have one class. Please tell me if it works so i can solve the bug.

well,thank you well ,I have successfully converted, but will you encounter this warning?

ignore it

[jinfagang]

Have u guys tried:

python -m onnxsim a.onnx a_sim.onnx

to simplify your onnx model? This is really helpful to eliminate unsupported ops and simplify whole onnx model structure.

If you guys can simplify your onnx model pls let me know.

[0806gcx]

Excuse me ? how do i change Activation Function to Relu or Leaky ReLU, I can't find the corresponding function

[linhaoqi027]

打扰一下 如何将激活功能更改为Relu或Leluy ReLU，我找不到对应的功能

it is in https://github.com/ultralytics/yolov5/blob/5e0b90de8f7782b3803fa2886bb824c2336358d0/models/common.py#L26.
change it before train. Then using pre-trained model or self-train directly.

[0806gcx]

I did it your way, trained a new network, but encountered new problems when I converted onnx to vino

[jinfagang]

@0806gcx As I mentioned above, the CPU mode traced ONNX model somewhat hard to convert to other framework such as TensorRT and OpenVINO. It can not even pass test in onnxruntime. @glenn-jocher

Simply verify it with onnx-simplifier.

[0806gcx]

@jinfagang
But this blogger successfully launched on openvino，I need to do it well.

[jinfagang]

@0806gcx It says model have a concat error in a Cocat node. dimension not match.

[jinfagang]

@0806gcx I think @glenn-jocher Add something and break this model, I previous using v1.0 and v2.0 can also sucessfully export onnx and convert to tensorrt very quickly, and now it breaks. Since this repo's author doesn't care much about cuda side (he always export in cpu mode) so that he doesn't realize this bug. But indeed, v3.0 exported onnx doesn't work any longer, it has some wrong Concat connnection and I can not even export it successfully in CUDA mode.

[linhaoqi027]

I did it your way, trained a new network, but encountered new problems when I converted onnx to vino

I met similar bug ever. DId you use openvino2020R4?

[FrzWang]

I met similar bug too. using openvino2020R4+LeakyRelu()

[linhaoqi027]

I met similar bug too. using openvino2020R4+LeakyRelu()

Did your version is torch==1.5.1,torchvision==0.6.1 when convert torch to onnx？Higher version may cause problem(This problem will cause error when convert onnx to openvino)

[glenn-jocher]

@linhaoqi027 @0806gcx @jinfagang unfortunately I don't have any openvino experience so I can't help here, but I think it would be worth raising this issue directly on the openvino repository since the errors are generated there.

The scope of the YOLOv5 repository as it stands now is limited to basic export to ONNX, Torchscript and CoreML. These 3 exports are currently working correctly when run in verified environments that meet all requirements.txt dependencies using the official models. 3rd party implementations that import exported models are not under our control and are beyond the scope of our support.

[jinfagang]

@glenn-jocher I solved the problem on cuda export, and it now also correct deploy to tensorrt.

[feolcn]

I met similar bug too. using openvino2020R4
Model Optimizer arguments:
Common parameters:
- Path to the Input Model: /data/aitest/yolov5/yolov5s.onnx
- Path for generated IR: /home/feol/intel/openvino_2020.4.287/deployment_tools/model_optimizer/.
- IR output name: yolov5s
- Log level: ERROR
- Batch: Not specified, inherited from the model
- Input layers: Not specified, inherited from the model
- Output layers: Not specified, inherited from the model
- Input shapes: [1,416,416,3]
- Mean values: Not specified
- Scale values: Not specified
- Scale factor: Not specified
- Precision of IR: FP32
- Enable fusing: True
- Enable grouped convolutions fusing: True
- Move mean values to preprocess section: False
- Reverse input channels: False
ONNX specific parameters:
Model Optimizer version:
[ ERROR ] Concat input shapes do not match
[ ERROR ] Shape is not defined for output 0 of "Concat_40".
[ ERROR ] Cannot infer shapes or values for node "Concat_40".
[ ERROR ] Not all output shapes were inferred or fully defined for node "Concat_40".
For more information please refer to Model Optimizer FAQ (https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html), question #40.
[ ERROR ]
[ ERROR ] It can happen due to bug in custom shape infer function .
[ ERROR ] Or because the node inputs have incorrect values/shapes.
[ ERROR ] Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
[ ERROR ] Run Model Optimizer with --log_level=DEBUG for more information.
[ ERROR ] Exception occurred during running replacer "REPLACEMENT_ID" (): Stopped shape/value propagation at "Concat_40" node.
For more information please refer to Model Optimizer FAQ (https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html), question #38.

[linhaoqi027]

I met similar bug too. using openvino2020R4
Model Optimizer arguments:
Common parameters:

• Path to the Input Model: /data/aitest/yolov5/yolov5s.onnx
• Path for generated IR: /home/feol/intel/openvino_2020.4.287/deployment_tools/model_optimizer/.
• IR output name: yolov5s
• Log level: ERROR
• Batch: Not specified, inherited from the model
• Input layers: Not specified, inherited from the model
• Output layers: Not specified, inherited from the model
• Input shapes: [1,416,416,3]
• Mean values: Not specified
• Scale values: Not specified
• Scale factor: Not specified
• Precision of IR: FP32
• Enable fusing: True
• Enable grouped convolutions fusing: True
• Move mean values to preprocess section: False
• Reverse input channels: False
  ONNX specific parameters:
  Model Optimizer version:
  [ ERROR ] Concat input shapes do not match
  [ ERROR ] Shape is not defined for output 0 of "Concat_40".
  [ ERROR ] Cannot infer shapes or values for node "Concat_40".
  [ ERROR ] Not all output shapes were inferred or fully defined for node "Concat_40".
  For more information please refer to Model Optimizer FAQ (https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html), question #40.
  [ ERROR ]
  [ ERROR ] It can happen due to bug in custom shape infer function .
  [ ERROR ] Or because the node inputs have incorrect values/shapes.
  [ ERROR ] Or because input shapes are incorrect (embedded to the model or passed via --input_shape).
  [ ERROR ] Run Model Optimizer with --log_level=DEBUG for more information.
  [ ERROR ] Exception occurred during running replacer "REPLACEMENT_ID" (): Stopped shape/value propagation at "Concat_40" node.
  For more information please refer to Model Optimizer FAQ (https://docs.openvinotoolkit.org/latest/_docs_MO_DG_prepare_model_Model_Optimizer_FAQ.html), question #38.

Did your version is torch==1.5.1,torchvision==0.6.1 ?

[feolcn]

是的

[linhaoqi027]

是的

I didn't meet this bug.May be it is cause by yolov5 version.I used yolov5_3

[FrzWang]

I met similar bug too. using openvino2020R4+LeakyRelu()

Did your version is torch==1.5.1,torchvision==0.6.1 when convert torch to onnx？Higher version may cause problem(This problem will cause error when convert onnx to openvino)

Thanks for your reply. It works when i change torch==1.5.1,torchvision==0.6.1.
Nice work.

[makaveli10]

@glenn-jocher I solved the problem on cuda export, and it now also correct deploy to tensorrt.

@jinfagang what exactly did you do to convert v3.0 to tensorrt. Can you please throw some light on that?

[jinfagang]

@makaveli10 I am simply convert to onnx, not inference via tensorrt, however seems here is an out-of-box example: http://manaai.cn/aisolution_detail.html?id=5

[makaveli10]

@jinfagang so you didnt convert v3.0 to tensorrt. but able to convert to onnx?

[masouduut94]

@linhaoqi027
@jinfagang

I successfully converted the onnx model to ir model. while I was using your code on this link I noticed on this line that the outputs are far from expected.

I trained the model on 2 classes so I expected network values like [210, 85 , 266, 175, 0.78, 1], but I got [-0.58, .25, 1.14, 12, -0.53].

What I mean is that I expect the last result to be 0 or 1 (for two classes) and the 4th value to be between 0 to 1 for confidence.

How can I solve this issue?
Is it because I did not consider arguments scale_values and mean_values when using model_optimizer conversion on openvino?

Please explain in details.
Thank you

[linhaoqi027]

@ linhaoqi027 @金发
港

我成功地将onnx模型转换为ir模型。当我在此链接上使用您的代码时，我在此行上注意到输出远远超出了预期。

我在2个班级上训练了模型，因此我希望[210, 85 , 266, 175, 0.78, 1]获得类似的网络价值，但是我得到了[-0.58, .25, 1.14, 12, -0.53]。

我的意思是，我希望最后的结果是0或1（对于两个类），而第4个值的可信度在0到1之间。

我该如何解决这个问题？
是因为我没有考虑参数scale_values，还是mean_values在openvino上使用model_optimizer转换时？

请详细说明。
谢谢
This output is the model output,NMS is not applied.You should use the output after NMS. The NMS code is included in my project. You should observe the output after NMS in https://github.com/linhaoqi027/yolov5_openvino_sdk/blob/3cd625264171a6bc5a545932ca69191979dbc7a9/ji.py#L219

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[JoshChristie]

The above steps worked for me when I revert back to commit c889267

[adityap27]

@JoshChristie @linhaoqi027
Yes these steps and given commit works for me.
But have you compared inference latency ?
It seems that there is no speed-up....or I am missing something.

[JoshChristie]

Yes, I see a noticeable decrease in inference time after converting the model to openvino for use on a CPU.

[adityap27]

@JoshChristie
If there is no speed-up then there's not much use of this conversion to openvino.
Let me know any solution.

[JoshChristie]

@adityap27
To be clear, here is what I get:
CPU inference time = 30ms. NMS time added additional 15ms
CPU with openvino inference time = 17ms. NMS not attempted. Openvino version 2020.4.287. With ONNX simplifier.

CPU info i7-8750H CPU @ 2.20GHz

Model:

# parameters
nc: 1  # number of classes
depth_multiple: 0.2  # model depth multiple
width_multiple: 0.2  # layer channel multiple

~ 1.2mill parameters

[adityap27]

@JoshChristie
Can you elaborate ? because 30ms to 17ms is a good speedup.
30ms is for pytorch model?
are these without NMS or with NMS?
which openvino version you are using ?
also which CPU model ?

I am getting same latency for both pytorch and openvino.
I need this conversion working with speedup asap.
It would be great if you elaborate about the above details I have asked for.

[JoshChristie]

@adityap27 Updated my previous comment.
Also, I used onnx simplifier but I don't know how much that changes things.

[adityap27]

@JoshChristie
Thanks for the details.
I haven't tried with onnx-simplifier. I will try that.

• Also are these default or have you changed ?

depth_multiple: 0.2  # model depth multiple
width_multiple: 0.2  # layer channel multiple

• Also were the input shapes same for both inference and what 480 or 640 ?

• Are you measuring this part ? in detect.py
  

[linhaoqi027]

@JoshChristie
如果没有加速，那么这种转换到openvino的用途就很少。
让我知道任何解决方案。

[linhaoqi027]

@adityap27 Updated my previous comment.
Also, I used onnx simplifier but I don't know how much that changes things.

The nms is not included in openvino.So i think the main time waste is in nms. I convert output(numpy) to pytorch than use tensor to do nms.So I think that cause main time waste.

[adityap27]

@linhaoqi027
Thanks for confirmation.
Can you tell me

• what's the purpose to change torch1.5.1 and not to use 1.6 ?? (before export.py)
• also have you noticed inference speedup(with/without nms) in openvino as compared to pytorch ?

[adityap27]

@JoshChristie
Thanks for the details.
I haven't tried with onnx-simplifier. I will try that.

* Also are these default or have you changed ?

depth_multiple: 0.2  # model depth multiple
width_multiple: 0.2  # layer channel multiple

* Also were the input shapes same for both inference and what 480 or 640 ?

* Are you measuring this part ? in detect.py
  ![image](https://user-images.githubusercontent.com/30194525/96680253-c79c6300-1392-11eb-9554-43049c72fd2b.png)

@JoshChristie can you help with these info ?

[linhaoqi027]

@JoshChristie
感谢您提供详细信息。
我没有尝试过onnx-simplifier。我会尝试的。

* Also are these default or have you changed ?

depth_multiple: 0.2  # model depth multiple
width_multiple: 0.2  # layer channel multiple

* Also were the input shapes same for both inference and what 480 or 640 ?

* Are you measuring this part ? in detect.py
  ![image](https://user-images.githubusercontent.com/30194525/96680253-c79c6300-1392-11eb-9554-43049c72fd2b.png)

@JoshChristie您可以提供这些信息吗？
you can refer to https://github.com/linhaoqi027/yolov5_openvino_sdk.
if i use 1.6,it will occur an error （what's the purpose to change torch1.5.1 and not to use 1.6 ?? (before export.py)）
And sorry i dont notice the inference speedup(with/without nms) in openvino as compared to pytorch

[manumenon]

having same size issue when trying to convert from onnx to ie

[linhaoqi027]

尝试从onnx转换为ie时遇到相同的大小问题

having same size issue when trying to convert from onnx to ie

you can refer to https://github.com/linhaoqi027/yolov5_openvino_sdk.

[Belinda-great]

@adityap27
To be clear, here is what I get:
CPU inference time = 30ms. NMS time added additional 15ms
CPU with openvino inference time = 17ms. NMS not attempted. Openvino version 2020.4.287. With ONNX simplifier.

CPU info i7-8750H CPU @ 2.20GHz

Model:

# parameters
nc: 1  # number of classes
depth_multiple: 0.2  # model depth multiple
width_multiple: 0.2  # layer channel multiple

~ 1.2mill parameters

Hi, very hanppy for your information! I use yolov5s, v3, img size is 640*640, one thread, test openvino(2020R4) time is 1.28s, and pytorch one thread time is 1.2s, it's too slow.
platform: E5-2699 V3 @2.30GHz.
Is there anything wrong with my test? and what img size did you use for input?

[JoshChristie]

With one thread your results make sense to me. My image size was 288x518.

[violet17]

Please refer the document in https://cdrdv2.intel.com/v1/dl/getContent/633562
https://www.intel.com/content/www/us/en/secure/design/internal/content-details.html?DocID=633562

Thanks.