Tensorrt: [Question] How can I surgery INetworkDefinition.

[dhkim0225]

Thank you for developing&maintaining TensorRT.

I'm currently trying to deploy my model which is trained with pytorch 1.3.1.
So, I made onnx file with (opset 7, ir_version 0.0.4) and made trt engine.

However, I found that my Unet shaped model's trt_engine makes shifted images.
And now I know why it happened.
It's because of the difference of bilinear upsample functions.

So I will create a custom layer using the ATen library. I want to replace the existing IResizeLayer with the custom plugin layer that I created (actually, I will try to make it).

I test with following code to confirm possibility of my scenario, but if I write the following code, an error appears.

with open(onnx_file_path, 'rb') as model:
    parser.parse(model.read())

for i in range(len(network)):
    if network[i].type == trt.LayerType.RESIZE:
        network[i] = trt.IResizeLayer(align_corners=False,
                                      resize_mode=trt.ResizeMode.LINEAR,
                                      scales=2,
                                      shape=network[i].num_outputs)

TypeError: tensorrt.tensorrt.IResizeLayer: No constructor defined!

Any Ideas?

Should I have to define all my networks one by one with python API ? (Please.... My model has a lot of skip connections since the backbone of network is resnet...)

Environment

TensorRT Version: 6.0.1.8
GPU Type: T4
Nvidia Driver Version: 440.33.01
CUDA Version: 10.2
CUDNN Version: 7.6.5
Operating System + Version: host: centos 7, docker: Ubuntu 18.04
Python Version (if applicable): 3.6.9
TensorFlow Version (if applicable): 1.4.0
PyTorch Version (if applicable): 1.3.1

[rmccorm4]

Hi @dhkim0225,

In the Python API, I think you can achieve that by setting the attributes after initializing the object, like so:

for i in range(len(network)):
    if network[i].type == trt.LayerType.RESIZE:
        resize_layer = trt.IResizeLayer()
        resize_layer.align_corners = False
        resize_layer.resize_mode = trt.ResizeMode.LINEAR,
        resize_layer.scales = 2
        resize_layer.shape = network[i].num_outputs
        network[i] = resize_layer

Let me know if that works for you or not. Also make sure the types of the attributes are correct.

[dhkim0225]

Thank you for reply. @rmccorm4
Actually, trt.IResizeLayer() makes following error.

TypeError: tensorrt.tensorrt.IResizeLayer: No constructor defined!

And for now, I solved a problem with [ Nvidia-iot / torch2trt ] package.
As you know, this package has an implementation of bilinear upsample custom plugin with ATen package.
Converted model with torch2trt needs more VRAM, but much faster than (pytorch => onnx(ir4_opset7) => trt) model.

I'm really looking forward for new TensorRT version with onnx opset11 support. (Yup, I know. TensorRT supports IresizeLayer with align_corners=False. Please update parser onnx.)

[rmccorm4]

Hi @dhkim0225,

Sorry instead of

# Wrong
resize_layer = trt.IResizeLayer()

I think it should be add_resize() from docs: https://docs.nvidia.com/deeplearning/sdk/tensorrt-api/python_api/infer/Graph/Network.html#tensorrt.INetworkDefinition.add_resize

# Correct
resize_layer = network.add_resize(...)

[rmccorm4]

Also, TensorRT 7 should be released within the next few days with added RNN support and expanded ONNX op support. Check out https://developer.nvidia.com/tensorrt for more info, there's a TRT7 description at the bottom of the page.

Hopefully this will solve your issues! If not, please open a new issue, thanks!

[rmccorm4]

Update, it was released today 🙂

https://docs.nvidia.com/deeplearning/sdk/tensorrt-release-notes/tensorrt-7.html#rel_7-0-0

https://developer.nvidia.com/nvidia-tensorrt-download

[dhkim0225]

Thanks!! I'll try it