Tensorrt: ONNX Parser INVALID_GRAPH with Assertion Failed

[briliantnugraha]

Hi guys, I have an ONNX model (converted from Pytorch directly without issue). But when I am trying to use the onnx parser to TensorRT, I encounter this:

ERROR: Failed to parse the ONNX file.
In node 1511 (parseGraph): INVALID_GRAPH: Assertion failed: ctx->tensors().count(inputName)
Traceback (most recent call last):
File "embedmask_engine.py", line 82, in
save_engine(trt_engine, trt_engine_path)
File "onnx_parser_engine.py", line 49, in save_engine
buf = engine.serialize()
AttributeError: 'NoneType' object has no attribute 'serialize'

My step to reproduce this is:

python3 onnx_parser_engine.py

Node 1511 structure:
Image

I see that there is an error in before adding the layer (because The Reshape ID is 1507) or in Reshape Output (because output id of Reshape is 1511). I am not sure why it is not supported, because I see in the TensorRT docs that both "Reshape" and "Add" functions are supported.

Do you guys know why?
Thanks for your help in advance~

@chiehpower

==========================================
TensorRT Version: 7.0
GPU Type: GTX1060
Nvidia Driver Version: 440.33.01
CUDA Version: 10.0
CUDNN Version: 7.6.5
Operating System + Version: Ubuntu 18.04 LTS
Python Version (if applicable): 3.6.9
TensorFlow Version (if applicable): 1.14
PyTorch Version (if applicable): 1.3.1
Baremetal or Container (if container which image + tag):

[chiehpower]

As @briliantnugraha mentioned, his case was using the onnx parser to build an TRT engine.
In addition, if we used the onnx2trt tool directly to convert the onnx model, we got this error messages below:

Command:

$ onnx2trt model.onnx -o model.trt                    

Output:

Input filename:   model.onnx
ONNX IR version:  0.0.4
Opset version:    11
Producer name:    pytorch
Producer version: 1.3
Domain:           
Model version:    0
Doc string:       
----------------------------------------------------------------
Parsing model
[2020-03-11 09:11:30 WARNING] /home/chieh/github/onnx-tensorrt/onnx2trt_utils.cpp:235: Your ONNX model has been generated with INT64 weights, while TensorRT does not natively support INT64. Attempting to cast down to INT32.
[2020-03-11 09:11:30 WARNING] Tensor DataType is determined at build time for tensors not marked as input or output.
[2020-03-11 09:11:30 WARNING] Tensor DataType is determined at build time for tensors not marked as input or output.
[2020-03-11 09:11:30 WARNING] Tensor DataType is determined at build time for tensors not marked as input or output.
[2020-03-11 09:11:30 WARNING] Tensor DataType is determined at build time for tensors not marked as input or output.
[2020-03-11 09:11:30 WARNING] Tensor DataType is determined at build time for tensors not marked as input or output.
[2020-03-11 09:11:30 WARNING] Tensor DataType is determined at build time for tensors not marked as input or output.
[2020-03-11 09:11:30 WARNING] Tensor DataType is determined at build time for tensors not marked as input or output.
[2020-03-11 09:11:30 WARNING] Tensor DataType is determined at build time for tensors not marked as input or output.
[2020-03-11 09:11:30 WARNING] Tensor DataType is determined at build time for tensors not marked as input or output.
[2020-03-11 09:11:30 WARNING] Tensor DataType is determined at build time for tensors not marked as input or output.
[2020-03-11 09:11:30 WARNING] Tensor DataType is determined at build time for tensors not marked as input or output.
[2020-03-11 09:11:30 WARNING] Tensor DataType is determined at build time for tensors not marked as input or output.
[2020-03-11 09:11:30 WARNING] Tensor DataType is determined at build time for tensors not marked as input or output.
[2020-03-11 09:11:30 WARNING] Tensor DataType is determined at build time for tensors not marked as input or output.
Building TensorRT engine, FP16 available:0
    Max batch size:     32
    Max workspace size: 1024 MiB
[2020-03-11 09:11:30   ERROR] Layer: (Unnamed Layer* 1818) [Unary]'s output can not be used as shape tensor.
[2020-03-11 09:11:30   ERROR] Network validation failed.
terminate called after throwing an instance of 'std::runtime_error'
  what():  Failed to create object
[1]    26323 abort (core dumped)  onnx2trt model.onnx -o model.trt

I have no idea about this issue.
Is there any idea about this situation?
Thanks a lot!

[rmccorm4]

Hi @briliantnugraha - Please try to parse model after building OSS components, see comment below.

Also @chiehpower , onnx2trt is generally an older tool and isn't kept up to date as much. trtexec is preferred. It can be built to support latest ONNX parser using the steps in this (NVIDIA/TensorRT) repository's README. In fact, this is what's recommended here in the onnx-tensorrt README: https://github.com/onnx/onnx-tensorrt#building. So trtexec will generally be better for you to use.

[rmccorm4]

FYI, it's likely the same solution as this comment: https://github.com/NVIDIA/TensorRT/issues/284#issuecomment-572835659

Try building the TensorRT OSS components and using trtexec again:

#  (optional) Start TensorRT 7 container
docker run --runtime=nvidia -v $PWD:/mnt --workdir=/mnt -it nvcr.io/nvidia/tensorrt:20.02-py3

# Build OSS Components from https://github.com/NVIDIA/TensorRT
wget https://raw.githubusercontent.com/rmccorm4/tensorrt-utils/master/OSS/build_OSS.sh
source build_OSS.sh

# Parse model with new OSS trtexec (or re-run your python script)
trtexec --explicitBatch --onnx=model.onnx

[briliantnugraha]

Hi @rmccorm4 Thanks for your suggestion.
In the end, my friend @chiehpower points me to the solution.
The reason why there is this error is because some layer has unconvertible interpretation by TensorRT.

[2020-03-11 09:11:30 ERROR] Layer: (Unnamed Layer* 1818) [Unary]'s output can not be used as shape tensor.
[2020-03-11 09:11:30 ERROR] Network validation failed.

Hence, he points out by using this https://github.com/daquexian/onnx-simplifier. With this, we could remove the unnecessary operation (to be honest, I am not sure what do they delete/simplify from my model graph, but anyway it works for me).
By simplifying my model with this:
python3 -m onnxsim my_onnx_model_name output_my_onnx_model_name
I could get a prediction output.

However, we encounter another issue where the output without the simplify model gives different output tensor values. Anyway, I will close this issue, thanks for the help~

[rmccorm4]

You can more closely examine the differences between the original and simplified models interactively using Netron: https://lutzroeder.github.io/netron/

[chiehpower]

Also @chiehpower , onnx2trt is generally an older tool and isn't kept up to date as much. trtexec is preferred. It can be built to support latest ONNX parser using the steps in this (NVIDIA/TensorRT) repository's README. In fact, this is what's recommended here in the onnx-tensorrt README: https://github.com/onnx/onnx-tensorrt#building. So trtexec will generally be better for you to use.

Hi @rmccorm4 ,

As you mentioned onnx2trt which is an older tool, how about the onnx parser of TensorRT?
Because I preferred to use onnx parser rather than onnx2trt, I wonder whether you would recommend to use onnx parser? (Is it kept up to date?)

I knew trtexec this tool, but I haven't tried it. LOL
I will try trtexec later. Thanks for your information!!

You can more closely examine the differences between the original and simplified models interactively using Netron: https://lutzroeder.github.io/netron/
This is really an awesome visualization tool!

[rmccorm4]

I believe they both use the ONNX Parser API. I'm not sure of the specific differences between trtexec and onnx2trt. I just know the trtexec is preferred.

You can always write python/c++ to use the API as well. The tools above are just faster for a quick parse check.

[chiehpower]

Got it!
Thanks for your opinion! :)