Tensorrt: Int8 calibrate failed while using a new IBuilderConfig

[SiyuanWei]

Hi, I.m trying to apply a mobilenetv2 onnx model and use int8 calibrator to get an int8 engine.
Meanwhile,because I would like to use a dynamic batch input shape, I create a new IBuilderConfig and build engine by "builder.build_engine(network,config)".
But the results show that codes had not excuted the int calibrator.
The function that I used to build engine is as following

[ShawnNew]

Can you print some log in your calibrator, so that you can see whether trt has started calibration or not.

[rmccorm4]

Hi @SiyuanWei,

Sorry for the slow reply.

If using IBuilderConfig, i.e. build_engine(network, config), then you should set all build attributes on the IBuilderConfig object instead of the IBuilder object.

For example, the old way might look like this:

builder.fp16_mode = True
builder.int8_mode = True
builder.int8_calibrator = calibrator
builder.build_cuda_engine(network)

But, the new way might look like this:

config.set_flag(trt.BuilderFlag.FP16)
config.set_flag(trt.BuilderFlag.INT8)
config.int8_calibrator = calibrator
builder.build_engine(network, config)

I think the only except for the parity of flags between builder and builder_config is the builder.max_batch_size flag, which only applies to implicit batch networks, which you probably wouldn't need to be using IBuilderConfig for anyway.

At a glance, your script looks more or less correct, except you're setting the int8 flag on the builder instead of the config object. Use this instead:

config.set_flag(trt.BuilderFlag.INT8)

You can refer to this code snippet as a rough reference for do several things with the IBuilderConfig (setting flags, creating optimization profiles, etc.), but it's mainly just for quick testing: https://github.com/rmccorm4/tensorrt-utils/blob/master/classification/imagenet/onnx_to_tensorrt.py