Onnx-tensorrt: "Assertion failed: axis >= 0 && axis < nbDims" in convert_axis()

[lampsonSong]

When converting a onnx model to trt model, one node is Gather. The conversion would get an error : Assertion failed: axis >= 0 && axis < nbDims.

The conversion code is in builtin_op_importers.cpp, the code is :

if NV_TENSORRT_MAJOR >= 4

DEFINE_BUILTIN_OP_IMPORTER(Gather) {

nvinfer1::ITensor& data = convertToTensor(inputs.at(0), ctx);

nvinfer1::ITensor& indices = convertToTensor(inputs.at(1), ctx);

OnnxAttrs attrs(node);

int axis = attrs.get("axis", 0);

int nbDims = inputs.at(0).shape().nbDims;

TRT_CHECK(convert_axis(axis, nbDims));

RETURN_FIRST_OUTPUT(ctx->network()->addGather(data, indices, axis));

}

endif // NV_TENSORRT_MAJOR >= 4

We can find a convert_axis() method in the above code. In onnx2trt_utils.hpp, I found the source code

// Convert an ONNX axis into a TRT axis

inline Status convert_axis(int& axis, int nbDims)

{

// Support negative indexing

if (axis < 0)

{

axis += nbDims;

}

// If axis was positive, subtract 1 to strip batch dimension

else

{

axis = axis - 1;

}

ASSERT(axis >= 0 && axis < nbDims, ErrorCode::kUNSUPPORTED_NODE);

return Status::success();

}

However, my input here is axis = 0, nbDims = 1, then axis = axis -1 = -1. The assertion fails.

The question is how to handle the situation when axis = 0.

I am not clear about the axis transformation between onnx and tensorRT.
Thanks.

[Eddiesyn]

I also met the exact same problem. Is there already a workaround ?
I use TensorRT version 4 and the onnx model is exported using the latest Pytorch ngc container

[KorovkoAlexander]

I`m also looking forward for the solution too!

[adriacabeza]

I am experiencing the same issue too.

[yinxiaochuan]

me too

[lijx-nutonomy]

me too. Thanks.

[liuchang8am]

any updates?

[oftenliu]

any updates now?

[aidonchuk]

me too

[RockWenJJ]

Seems like the issue is the same as #16908. The problem is caused by codes like
x=x.review(x.size()[0], -1), and solution is to explicitly reshape like x=x.review(1, 400)

[xiaoLiuxiaoLiuxiaoLiu]

I have fix it by changing this x=x.view(x.size(),-1) as x=x.flatten(1).

[zimenglan-sysu-512]

i meet this problem.
any updates now?

[Zanderzt]

I have fix it by changing this x=x.view(x.size(),-1) as x=x.flatten(1).

Hello,where is x=x.view(x.size(),-1) ?

[xiaoLiuxiaoLiuxiaoLiu]

@Zanderzt Every torchvision classification model， such as resnet.py. Check the files carefully and you will find that code.

[xiaoLiuxiaoLiuxiaoLiu]

@Zanderzt 兄嘚 ，在每个分类模型都有一个平铺的过程，官方实现都是x=x.view(x.size(),-1)。你要做的是将这一句改成x=x.flatten(1), 导出来的onnx自然没问题。

[Zanderzt]

@Zanderzt 兄嘚 ，在每个分类模型都有一个平铺的过程，官方实现都是x=x.view(x.size(),-1)。你要做的是将这一句改成x=x.flatten(1), 导出来的onnx自然没问题。

好的，谢谢您！

[tibistrat]

I had a similar issue with with models containing upsampling exported from pytorch.
I explain a workaround here:
https://github.com/onnx/onnx-tensorrt/issues/192#issuecomment-526182296

[jinfagang]

Same issue here, my problem is not about this operation:

out.view(out.shape[0], -1, 2)

which convert a shape [1, 70, 70, 4] into a [1, 4900, 4] shape.

If written like this, then onnx to trt would throw this error.

The root reason for this, is that, trt can not actually get the correct first dimension when converting. so just avoid access the first dimension which is batch and everything else would work.

[iban-rodriguez]

Same issue here. Any advances on this? I'm use TensorRT version 5 with and the onnx model is exported using tensorflow-onnx and opset 9.

[sleepingidea]

Same issue here, my problem is not about this operation:

out.view(out.shape[0], -1, 2)

which convert a shape [1, 70, 70, 4] into a [1, 4900, 4] shape.

If written like this, then onnx to trt would throw this error.

The root reason for this, is that, trt can not actually get the correct first dimension when converting. so just avoid access the first dimension which is batch and everything else would work.
Because the trt can not get the shape of the tensor. So when transfer the onnx to trt, you can set the out.shape[0] to a const number, such the batch size as 1.

[jfkkf123]

Same issue here, my problem is not about this operation:

out.view(out.shape[0], -1, 2)

which convert a shape [1, 70, 70, 4] into a [1, 4900, 4] shape.
If written like this, then onnx to trt would throw this error.
The root reason for this, is that, trt can not actually get the correct first dimension when converting. so just avoid access the first dimension which is batch and everything else would work.
Because the trt can not get the shape of the tensor. So when transfer the onnx to trt, you can set the out.shape[0] to a const number, such the batch size as 1.

# output = inputs.contiguous().view(inputs.shape[0], inputs.shape[1], 
                   inputs.shape[2],self.num_anchors,self.num_classes)
   output = inputs.contiguous().view(1, inputs.shape[1], inputs.shape[2], 
                   self.num_anchors,self.num_classes)

I set the output.shape[0] to 1 ,but it still throw this error.
is there other way to solve this problem?
thanks

[depthwise]

This is a pretty bad bug that will affect literally everyone who wants to reshape tensors. The workaround suggested by @tibistrat does not allow one to create ONNX models with dynamic batch size (and PyTorch warns of it, as well).

When will this be fixed? This is pretty basic stuff.

[rmccorm4]

Can someone share a simple PyTorch script that creates a model that reproduces this error and exports it to ONNX? I can take a look when I get a chance if so.

[depthwise]

Here you go:

#!/usr/bin/env python3

import torch
from torch import onnx

input = torch.rand(2,64,4)

class Model(torch.nn.Module):
    def __init__(self):
        super().__init__()

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        n, hw, c = x.shape
        # Split the hw dimension into 8*8
        h = 8
        w = hw // h
        x = x.view(n, h, w, c)
        # NHWC -> NCHW is pretty common.
        return x.permute(0, 3, 1, 2)

# Opset version 9 is required by Jetson Xavier, which has TRT6.
torch.onnx.export(Model(), input, "/tmp/barf.onnx", opset_version=9)

As I said, pretty basic stuff that's needed all the time. I have not tried this with TRT7. My use case requires Jetson Xavier, and TRT7 is not available there.

Note also that to run the resulting ONNX file on the Jetson, you will need to run this program with PyTorch 1.2.0.

[rmccorm4]

Thanks @depthwise, here's a brief breakdown for this model.

TensorRT 7 and PyTorch 1.4 (x86)

# Start TensorRT 7 container
nvidia-docker run  -it -v $PWD:/mnt nvcr.io/nvidia/tensorrt:20.01-py3
pip install torch==1.4

# Running the code above for both opset 9 and opset 11
python repro.py --opset 9
python repro.py --opset 11

# Parse the opset9 model
trtexec --explicitBatch --onnx=permute.opset9.onnx
...
# &&&& PASSED TensorRT.trtexec # trtexec --explicitBatch --onnx=permute.opset9.onnx

# Parse the opset11 model
trtexec --explicitBatch --onnx=permute.opset11.onnx
...
# &&&& PASSED TensorRT.trtexec # trtexec --explicitBatch --onnx=permute.opset11.onnx

Once TensorRT 7 is released for Jetson, that should work for you.

---

However, as you mentioned, since it's not currently supported, you're stuck with TensorRT 6 at the moment. So let's look into this:

TensorRT 6 and PyTorch 1.2 fails as mentioned above

$ nvidia-docker run  -it -v $PWD:/mnt nvcr.io/nvidia/tensorrt:19.12-py3
$ trtexec --onnx=permute.opset9.onnx
...
ERROR: onnx2trt_utils.hpp:347 In function convert_axis:
[8] Assertion failed: axis >= 0 && axis < nbDims
[01/22/2020-08:17:03] [E] Failed to parse onnx file
[01/22/2020-08:17:03] [E] Parsing model failed
[01/22/2020-08:17:03] [E] Engine could not be created
&&&& FAILED TensorRT.trtexec # trtexec --onnx=permute.opset9.onnx

TensorRT 6 + TensorRT OSS 19.12 branch, fails with a different error:

$ nvidia-docker run  -it -v $PWD:/mnt nvcr.io/nvidia/tensorrt:19.12-py3
# Build/Install OSS 19.12 components 
$ bash /opt/tensorrt/install_opensource.sh
$ trtexec --explicitBatch --onnx=pt1.2.permute.opset9.onnx
...
ERROR: /opt/tensorrt/TensorRT/parsers/onnx/builtin_op_importers.cpp:689 In function importGather:
[8] Assertion failed: !(data->getType() == nvinfer1::DataType::kINT32 && nbDims == 1 && inputs.at(0).is_tensor()) && "Cannot perform gather on a shape tensor!"
[01/22/2020-08:23:28] [E] Failed to parse onnx file
[01/22/2020-08:23:28] [E] Parsing model failed
[01/22/2020-08:23:28] [E] Engine could not be created
&&&& FAILED TensorRT.trtexec # trtexec --explicitBatch --onnx=pt1.2.permute.opset9.onnx

---

And if you search this error, you'll find this issue: https://github.com/onnx/onnx-tensorrt/issues/283#issuecomment-545703178, which suggests to try onnx-simplifier in this case, which worked for me on this dummy model:

TensorRT 6 + OSS 19.12 + onnx-simplifier works:

$ nvidia-docker run  -it -v $PWD:/mnt nvcr.io/nvidia/tensorrt:19.12-py3
# Build/Install OSS 19.12 components 
$ bash /opt/tensorrt/install_opensource.sh
# Install and run onnx-simplifier
$ pip install onnx-simplifier
$ python -m onnxsim pt1.2.permute.opset9.onnx pt1.2.permute.opset9.simple.onnx
# Parse new simplified graph
$ trtexec --explicitBatch --onnx=pt1.2.permute.opset9.simple.onnx
...
----------------------------------------------------------------
Input filename:   pt1.2.permute.opset9.simple.onnx
ONNX IR version:  0.0.4
Opset version:    9
Producer name:    pytorch
Producer version: 1.2
Domain:
Model version:    0
Doc string:
----------------------------------------------------------------
[01/22/2020-08:24:12] [I] [TRT] 18:Reshape -> (2, 8, 8, 4)
[01/22/2020-08:24:12] [I] [TRT] 19:Transpose -> (2, 4, 8, 8)
 ----- Parsing of ONNX model pt1.2.permute.opset9.simple.onnx is Done ----
[01/22/2020-08:24:14] [I] [TRT] Detected 1 inputs and 1 output network tensors.
...

&&&& PASSED TensorRT.trtexec # trtexec --explicitBatch --onnx=pt1.2.permute.opset9.simple.onnx

So perhaps onnx-simplifier (or perhaps your own custom script that makes use of onnx.optimizer roughly like so: https://github.com/onnx/onnx/blob/master/docs/PythonAPIOverview.md#optimizing-an-onnx-model) can be a workaround for you in these cases until Jetson supports TensorRT 7 .

Looking at the original and simplified versions of the ONNX model in Netron, it seems like PyTorch creates a rather complex graph for such a simple model, whereas onnx-simplifier condenses it into a single Reshape and Transpose op, which aligns more accurately with the PyTorch code.

---

As a side note, you can also try torch2trt: https://github.com/NVIDIA-AI-IOT/torch2trt, though I haven't tried it myself yet. I believe that maps PyTorch ops to the TensorRT API directly and cuts out the ONNX middle man, so it would likely interpret the PyTorch model more closely to the result of onnx-simplifier on our original ONNX model.

Hope this helps.

[depthwise]

Sadly, the simplifier barfs on my full model (a fixed-size RetinaNet variant, with FPN modified to be compatible with TRT6). Something to do with the graph not being in SSA form after "simplification".

Regarding torch2trt, it doesn't seem like it'd work for my use case: our code is in C++, so we use TRT's C++ API.

The workaround with first converting the shape to tensor should be sufficient for us in the meantime though, even if the model is not dynamic in the batch dimension: we can simply fix the batch dimension.

I'm looking forward to TRT7 on the Xavier. Thanks for the thorough investigation.

[rmccorm4]

Sadly, the simplifier barfs on my full model (a fixed-size RetinaNet variant, with FPN modified to be compatible with TRT6). Something to do with the graph not being in SSA form after "simplification".

If you are able to share your model (publicly, or privately via email), or create a simplified model that reproduces the error with onnx-simplifier, maybe @daquexian could help with that.

Regarding torch2trt, it doesn't seem like it'd work for my use case: our code is in C++, so we use TRT's C++ API.

Do you mean your model doesn't actually originate from PyTorch? Or just that you're parsing ONNX and inferencing in C++? If the latter, then can you parse/convert and serialize your model with torch2trt , and then load/deserialize the engine in your C++ code base for inference?

[depthwise]

Do you need a trained model, or are random weights OK? I've noticed the simplifier does a forward pass, but I haven't checked to what end. If it just does e.g. shape inference, then I suppose a random model should be fine. If it checks the output, a random model probably won't suffice. I can share a random model, of course. A trained one would need to be cleared with my client.

We load ONNX in C++ using the bundled ONNX parser. ONNX itself does originate in PyTorch, but object detection is only a part of what we're doing on that Xavier. It's an agricultural robot, essentially.

[rmccorm4]

I would think the weights don't matter, but I haven't looked deep into the source - that would be up to @daquexian's expertise :slightly_smiling_face:

[rmccorm4]

I'd also be curious how the result of running onnx.utils.polish_model on your model may differ from onnx-simplifier: https://github.com/onnx/onnx/blob/master/docs/PythonAPIOverview.md#polishing-the-model

[daquexian]

@depthwise Sorry for the late reply.

Do you need a trained model, or are random weights OK?

Random weights are ok. Could you please send the model to [email protected] so that I can check the problem? Thanks!

[enesmsahin]

Hi @rmccorm4 ,

I am having similar issues. I was able to parse the model in TensorRT 7 but I got the same error with TensorRT 6.

My question is this:
Does this different behaviour between different versions come from onnx-tensorrt version or directly from TensorRT versions?

As I see, TensorRT 6 uses onnx @ 400df73 as submodule and TensorRT 7 uses onnx @ 84b5be1 as submodule. Would I be able to parse my onnx model in TensorRT 6 if I update libnvonnxparser.so library by rebuilding the onnx @ 84b5be1 snapshot?

[rmccorm4]

Hi @eesweng,

Short answer is no that probably won't work.

Long answer is that it might if you're lucky. It probably depends on if the changes to the ONNX parser you're referring to also reely on implementation changes in the TensorRT API that are specific to TensorRT 7, or if they are just tweakd/fixes to how the ops were being processed at the ONNX API/graph level.

Feel free to give it a shot and share your findings, I'd be interested to see.

[enesmsahin]

Hi @rmccorm4 ,
I guess it doesn't work. I tried to build onnx-tensorrt that is used in TensorRT7 but I got cmake errors. Then I made some googling and found this comment of you:
https://github.com/onnx/onnx-tensorrt/issues/355#issuecomment-568831810

I guess, I will have to wait TensorRT 7 release for Jetson devices.

[kevinch-nv]

@menessahin TensorRT 7 has been released for a while now, are you still having issues? Can this issue be closed?