Onnx-tensorrt: pytorch-onnx model can not be convertted to trt model of tensorrt

[perrywu1989]

envs:
Pytorch & Caffe2 (v1.0) form conda
Onnx(1.3.0) from pip
Onnx-tensorrt(0.1.0) from source

I can convert pytorch model to .onnx and .onnx to .pb of Caffe2, all of them can run successfully.

But when i convert the same .onnx model to .trt model of tensorrt, it is wrong. Said:

Input filename:   ocr_api.onnx
ONNX IR version:  0.0.3
Opset version:    9
Producer name:    pytorch
Producer version: 0.4
Domain:
Model version:    0
Doc string:
----------------------------------------------------------------
Parsing model
While parsing node number 70 [MatMul -> "194"]:
ERROR: /home/wfy/Downloads/onnx-tensorrt/builtin_op_importers.cpp:1066 In function importMatMul:
[8] Assertion failed: dims.nbDims == 3

if I use the onnx model from your webs, it works.

Thanks!

[perrywu1989]

I found the reason is that onnx-tensorrt does not support fc layer with input 3 dims. For example, input size [batch, length, channel1] pass through FC layer to bacome output size [batch, length, channel2]. Also it does not support Pytorch reshape function RESHAPE(torch.view), even pytorch can be convertted to .onnx, but pytorch-onnx can not be convertted to .trt. It would output error :not support Gather.

[yinghai]

@houseroad Any suggestions?

[perrywu1989]

@yinghai @houseroad I thought out 3 ways to make input size [b,w,c1] --->FC---> output size [b,w,c1], but on one works.
One way is to make input size [b * w,c1] --->FC---> output size [b * w,c2] , but when I reshape size [b,w,c1] to output size [b * w,c1], it said I cannot change input-output dims.

Assertion failed: get_shape_size(new_shape) == get_shape_size(tensor.getDimensions())

meanwhile,when I want to use tensor.size() to get shape of tensor in pytorch, it said not support.

No importer registered for op: Gather

Another way is that I slice the tensor [b,w,c1] to w * tensor [b, c1], but it said not support Slice.

No importer registered for op: Slice

Another way is that I konw batch_size=1, so I can unsqueeze batch dim to make [b,w,c1] to [w,c1], but it said first dim ==1.

Assertion failed: axis != BATCH_DIM

Oh,No.

[sidnav]

@perrywu1989 Have you figured out a solution for this? I have a similar issue where I cannot use a FC layer with 3 input dimensions.

[thefacetakt]

Hi everyone. Had a similar issue, my workaround is to replace fc layer with 1x1 conv2d. Apparently, TRT has some issues with reshapes, but nothing against squeeze/unsquese.

An example: (I use tf -> tf2onnx -> onnx2trt)
Instead of

    a = tf.placeholder(shape=(10, 200, 3), dtype=tf.float32, name="input")
    b = tf.layers.dense(b, 4, activation=None)
    d = tf.identity(b, name="d")

or

    a = tf.placeholder(shape=(10, 200, 3), dtype=tf.float32, name="input")
    b = tf.reshape(a, [10 * 200, 3])
    c = tf.layers.dense(b, 4)
    d = tf.reshape(c, [10, 200, 4], name="d")

I use now

    a = tf.placeholder(shape=(10, 200, 3), dtype=tf.float32, name="input")
    with tf.variable_scope("fully_connected"):
        b = tf.expand_dims(a, 1)
        kernel = tf.get_variable("kernel", dtype=tf.float32, shape=[3, 4])
        kernel = tf.expand_dims(kernel, 0)
        kernel = tf.expand_dims(kernel, 0)
        bias = tf.get_variable("bias", dtype=tf.float32, shape=[4])

        b = tf.nn.conv2d(
            b,
            kernel,
            strides=[1, 1, 1, 1],
            padding="SAME"
        )

        b = b + bias

        b = tf.squeeze(b, [1])
    d = tf.identity(b, name="d")

and it seems to work.

(In case you are wondering -- first two methods somehow produce quite different frozen graphs, and second one is _much_ simpler)

[kevinch-nv]

We didn't have the best op support in previous TensorRT versions, have you tried converting these models with the latest TensorRT version (7.2.1)?

[kevinch-nv]

Closing due to inactivity - if you are still having issues with the latest version of onnx-tensorrt feel free to open a new issue.