Onnx-tensorrt: Fail to convert the fp16 onnx.

Created on 22 Aug 2019 · 3Comments · Source: onnx/onnx-tensorrt

When I convert float16 onnx of resne18 to tensorRT engine. Error takes place.
but vgg16() does not make any error.

codes is the below.

import os
import torch
import torchvision

model = torchvision.models.resnet18(pretrained=True).eval().cuda().half()
dummy_input = torch.randn(1, 3, 224, 224, device='cuda').half()

torch.onnx.export(model, dummy_input, "test.onnx", verbose=False)

os.system('onnx2trt -w 12000000000 -b 1 -d 16 test.onnx -o test.engine')

error message

While parsing node number 1 [BatchNormalization -> "124"]:
ERROR: /kakao/TensorRT-5.1.5.0/onnx-tensorrt/builtin_op_importers.cpp:628 In function importBatchNormalization:
[8] Assertion failed: scale_weights.type == ::ONNX_NAMESPACE::TensorProto::FLOAT && bias_weights.type == ::ONNX_NAMESPACE::TensorProto::FLOAT && mean_weights.type == ::ONNX_NAMESPACE::TensorProto::FLOAT && variance_weights.type == ::ONNX_NAMESPACE::TensorProto::FLOAT

GPU: V100
CUDA: 10.0, cudnn7.5
tensorrt: TensorRT-5.1.5.0.Ubuntu-16.04.5.x86_64-gnu.cuda-10.0.cudnn7.5.tar.gz
python 3.7.0
pytorch 1.2.0

Source

andrew-yang0722

Most helpful comment

The BatchNorm layers need parameters in single precision (FP32, not FP16). You can use this to convert your model to half precision instead in a BatchNorm safe way:

def network_to_half(model):
    """
    Convert model to half precision in a batchnorm-safe way.
    """
    def bn_to_float(module):
        """
        BatchNorm layers need parameters in single precision. Find all layers and convert
        them back to float.
        """
        if isinstance(module, torch.nn.modules.batchnorm._BatchNorm):
            module.float()
        for child in module.children():
            bn_to_float(child)
        return module
    return bn_to_float(model.half())

# Convert model to have
model = network_to_half(model)

laggui on 22 Aug 2019

👍2

All 3 comments

The BatchNorm layers need parameters in single precision (FP32, not FP16). You can use this to convert your model to half precision instead in a BatchNorm safe way:

def network_to_half(model):
    """
    Convert model to half precision in a batchnorm-safe way.
    """
    def bn_to_float(module):
        """
        BatchNorm layers need parameters in single precision. Find all layers and convert
        them back to float.
        """
        if isinstance(module, torch.nn.modules.batchnorm._BatchNorm):
            module.float()
        for child in module.children():
            bn_to_float(child)
        return module
    return bn_to_float(model.half())

# Convert model to have
model = network_to_half(model)

laggui on 22 Aug 2019

👍2

It works. Thank you.

andrew-yang0722 on 23 Aug 2019

👍1

hi,
for me, it is showing

RuntimeError: Input type (torch.cuda.HalfTensor) and weight type (torch.cuda.FloatTensor) should be the same

Linux: 16.04
cuda: 10.0
Pytorch: 1.3.1
GPU: RTX 2080Ti