Tensorrt: Mobilenet onnx model parsing fails with TensorRT 7

[svnoesis]

Description

I am trying to run inference with Mobilenetv1 using onnx model file. But the trt.OnnxParser call fails on TensorRT 7.0.0.11 with the following error. The same model works fine with TensorRT 6.0.1.

Below is the error I see
```raise RuntimeError("MobileNet onnx model parsing failed! Error: {:}".format(parser.get_error(0).desc()))

RuntimeError: MobileNet onnx model parsing failed!
Error: Assertion failed: !_importer_ctx.network()->hasImplicitBatchDimension() &&
"This version of the ONNX parser only supports TensorRT INetworkDefinitions
with an explicit batch dimension.

Please ensure the network was created using the EXPLICIT_BATCH NetworkDefinitionCreationFlag."
```

Is there a way to avoid the explicit_batch definition ?

Also I noticed that if I use onnx2trt to parse the model, it succeeds without any errors.

Can anyone please explain this ?

Environment

TensorRT Version: 7.0.0.11
GPU Type: T4
Nvidia Driver Version: 440.33.01
CUDA Version: 10.2
CUDNN Version: 7.6.5
Operating System + Version: RHEL 7.6
Python Version (if applicable): Python 3.7
TensorFlow Version (if applicable):
PyTorch Version (if applicable):
Baremetal or Container (if container which image + tag):

Relevant Files

I used the model file mobilenet_sym_no_bn.onnx
The inference code was from here.

[rmccorm4]

Hi @svnoesis,

As the error says, TensorRT 7 requires that you create the network with the EXPLICIT_BATCH flag when parsing ONNX models. If you wanted to edit the existing code you referenced, it's a simple one line add: https://github.com/mlperf/inference_results_v0.5/blob/master/closed/NVIDIA/code/mobilenet/tensorrt/MobileNet.py#L57-L58

            network_creation_flag = 1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_PRECISION)
            network_creation_flag |= 1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)
            self.network = self.builder.create_network(network_creation_flag)

If you wanted the code to work for both TRT6 and TRT7, you could add an if statement based on the version:

            network_creation_flag = 1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_PRECISION)
            major, minor, patch = trt.__version__.split('.')
            if int(major) >= 7:            
                network_creation_flag |= 1 << int(trt.NetworkDefinitionCreationFlag.EXPLICIT_BATCH)
            self.network = self.builder.create_network(network_creation_flag)

---

Re: onnx2trt, the ONNX parser had a release shortly after 6.0 called 6.0-full-dims to support dynamic shapes, which added the EXPLICIT_BATCH flag when parsing the onnx model. This is now default in TRT 7.0, all ONNX models are interpreted as having explicit batch, which is why you need to use the explicit_batch flag. You can see the commit that changed this in onnx2trt's main.cpp here: https://github.com/onnx/onnx-tensorrt/commit/dbd5585a223a8b844e616761ad3a83242fb9a30e#diff-118fcbaaba162ba17933c7893247df3a

[svnoesis]

@rmccorm4 Thanks for the clarification.
I had edited the code to use the EXPLICIT_BATCH flag while creating the network. That got past the ONNX parsing step, but landed into another error during calibration phase when builder.build_engine is invoked. Hence was wondering if it is due to the explicit batch definition and wanted to see if there is a way to avoid that flag.

After parsing the onnx model, there are some changes made to the network to add a new layer called "fc_replaced". Then build_engine is invoked and that is where TensorRT 7 throws the below errors. The same code was working fine and the engine was built successfully with TensorRT 6.
As per the error below TRT is expecting a different weights count, but since nothing has changed in the network, not sure why this error in TRT 7. Any pointers on this would be very helpful.

[TensorRT] ERROR: fc_replaced: kernel weights has count 1024000 but 1048576000 was expected
[TensorRT] ERROR: fc_replaced: count of 1024000 weights in kernel, but kernel dimensions (1,1) with 1024 input channels, 1024000 output channels and 1 groups were specified. Expected Weights count is 1024 * 1*1 * 1024000 / 1 = 1048576000
[TensorRT] ERROR: fc_replaced: kernel weights has count 1024000 but 1048576000 was expected
[TensorRT] ERROR: fc_replaced: count of 1024000 weights in kernel, but kernel dimensions (1,1) with 1024 input channels, 1024000 output channels and 1 groups were specified. Expected Weights count is 1024 * 1*1 * 1024000 / 1 = 1048576000
[TensorRT] ERROR: Layer fc_replaced failed validation
[TensorRT] ERROR: Network validation failed.