Detectron2: Do you support batch inference?

[zhaoxuyan]

🚀 Feature

How to easily take advantage of batch processing during inference?

Motivation

https://github.com/facebookresearch/Detectron/issues/84

[ppwwyyxx]

You just need to call model with a batch of inputs (https://detectron2.readthedocs.io/tutorials/models.html#model-input-format).

[zhaoxuyan]

You just need to call model with a batch of inputs (https://detectron2.readthedocs.io/tutorials/models.html#model-input-format).

@ppwwyyxx Thank you very much for your reply~
Can you tell me how to set up a batch of pictures in /demo/demo.py

img = read_image(path, format="BGR")
demo.run_on_image(img)

It can only infer one image at a time, how to infer a batch of images?
Because I have a lot of remaining gpu memory when infer one image at a time, and I want to get the results of multiple video streams(a batch of images) by doing only one inference

[ppwwyyxx]

demo.py does not support batch inference.

[zhaoxuyan]

demo.py does not support batch inference.

So how should I implement it please?

[ppwwyyxx]

https://detectron2.readthedocs.io/tutorials/models.html#use-models explains how to use a model.

[keko950]

You just need to call model with a batch of inputs (https://detectron2.readthedocs.io/tutorials/models.html#model-input-format).

That link doesn't refers in any way to batch inference, would be nice a more detailed answer since the api doesn't go deeper.

[ppwwyyxx]

The link explains that the input to the model is a list[{"image": ...}, ...]. As long as the length of the list > 1, you're doing batch inference.

[keko950]

I was thinking in DefaultPredictor, not model class itself, my bad!
Working like a charm, thank you!

[zhaoxuyan]

The link explains that the input to the model is a list[{"image": ...}, ...]. As long as the length of the list > 1, you're doing batch inference.

@keko950 @ppwwyyxx
Thank you for your reply. Another question:
if my list[{"image": ...}, ...] has 8 images, how to specify multi gpu_num = 4 in DefaultPredictor and each gpu infer with 8/4=2 images?

[ppwwyyxx]

DefaultPredictor uses 1 GPU.

[darkAlert]

I create a predictor as follows:

    cfg = get_cfg()
    cfg.merge_from_file(path_to_cfg)
    cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5 
    cfg.MODEL.WEIGHTS = path_to_weights
    predictor = DefaultPredictor(cfg)

        img = cv2.imread(img_path)
        img_list = [{"image":img}]
        predictor(img_list)

And then I get the following error:

File "/home/darkalert/builds/pytorch/build/lib.linux-x86_64-3.6/torch/autograd/grad_mode.py", line 49, in decorate_no_grad  return func(*args, **kwargs)
File "/home/darkalert/builds/detectron2/detectron2/engine/defaults.py", line 172, in __call__height, width = original_image.shape[:2] AttributeError: 'list' object has no attribute 'shape'

[ppwwyyxx]

@darkAlert as said above, you need to call a model, not predictor, with batch of inputs.
As the docs (https://detectron2.readthedocs.io/modules/engine.html#detectron2.engine.defaults.DefaultPredictor) said the DefaultPredictor does not support batch of inputs.

[jiangkansg]

I tried to follow the instructions, and this is the code:

cfg = get_cfg()
cfg.merge_from_file("./detectron2_repo/configs/COCO-Keypoints/keypoint_rcnn_R_50_FPN_3x.yaml")
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7 # set threshold for this model
cfg.MODEL.WEIGHTS = "detectron2://COCO-Keypoints/keypoint_rcnn_R_50_FPN_3x/137849621/model_final_a6e10b.pkl"

from detectron2.modeling import build_model
model = build_model(cfg) # returns a torch.nn.Module

from detectron2.checkpoint import DetectionCheckpointer
DetectionCheckpointer(model).load("detectron2://COCO-Keypoints/keypoint_rcnn_R_50_FPN_3x/137849621/model_final_a6e10b.pkl")

img = cv2.imread("./input.jpg")
img = np.transpose(img,(2,0,1)) # change from HWC to CHW
img_tensor = torch.from_numpy(img)
inputs = [{"image":img_tensor}, {"image":img_tensor}]
outputs = model(inputs)

The error message is:
/content/detectron2_repo/detectron2/modeling/proposal_generator/rpn_outputs.py in _get_ground_truth(self)
260 # Concatenate anchors from all feature maps into a single Boxes per image
261 anchors = [Boxes.cat(anchors_i) for anchors_i in self.anchors]
--> 262 for image_size_i, anchors_i, gt_boxes_i in zip(self.image_sizes, anchors, self.gt_boxes):
263 """
264 image_size_i: (h, w) for the i-th image

TypeError: zip argument #3 must support iteration

Did I miss anything? Thank you.

[ppwwyyxx]

To do inference you need to set a pytorch module to inference mode.

[jiangkansg]

Thank you. I tried by adding model.train(False). However, batch inference of keypoints "sometimes" (roughly 1 in 5 times) causes error. This is really strange.

Here is the complete code:
cfg = get_cfg()
cfg.merge_from_file("./detectron2_repo/configs/COCO-Keypoints/keypoint_rcnn_R_50_FPN_3x.yaml")
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7 # set threshold for this model
cfg.MODEL.WEIGHTS = "detectron2://COCO-Keypoints/keypoint_rcnn_R_50_FPN_3x/137849621/model_final_a6e10b.pkl"
from detectron2.modeling import build_model
model = build_model(cfg) # returns a torch.nn.Module
model.train(False)
img = cv2.imread("./input.jpg")
img = np.transpose(img,(2,0,1))
img_tensor = torch.from_numpy(img)
inputs = [{"image":img_tensor}, {"image":img_tensor}]
outputs = model(inputs) #error may happen here

/content/detectron2_repo/detectron2/structures/keypoints.py in heatmaps_to_keypoints(maps, rois)
193 assert (
194 roi_map_probs[keypoints_idx, y_int, x_int]
--> 195 == roi_map_probs.view(num_keypoints, -1).max(1)[0]
196 ).all()
197

AssertionError:

[ppwwyyxx]

Could you provide the image, or even better a colab notebook, that reproduces this error?

[jiangkansg]

Sure. Here is the colab notebook modified based on your tutorial.
https://colab.research.google.com/drive/1K9-r0tmYk5q_Zfp-gc5a8wvHZJ9jdIj6
Thank you.

[ppwwyyxx]

The model has to be loaded following https://detectron2.readthedocs.io/tutorials/models.html.
When it's not loaded, it sometimes produces some NaN values that eventually leads to the assertion. I'll see if NaN can be handled better

[jiangkansg]

I was using config to load the weights. After you latest comments, I now use DetectionCheckpointer(model).load(file_path) to load the weights. It is working now. Lessons learnt! Thank you.

To summaries, this is a working example:
cfg = get_cfg()
cfg.merge_from_file("./detectron2_repo/configs/COCO-Keypoints/keypoint_rcnn_R_50_FPN_3x.yaml")
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7 # set threshold for this model

model = build_model(cfg) # returns a torch.nn.Module
DetectionCheckpointer(model).load('./some_model.pkl') # must load weights this way, can't use cfg.MODEL.WEIGHTS = "..."
model.train(False) # inference mode

img = cv2.imread("./input.jpg")
img = np.transpose(img,(2,0,1))
img_tensor = torch.from_numpy(img)
inputs = [{"image":img_tensor}, {"image":img_tensor}] # inputs is ready

outputs = model(inputs)

[wiamadaya]

I was using config to load the weights. After you latest comments, I now use DetectionCheckpointer(model).load(file_path) to load the weights. It is working now. Lessons learnt! Thank you.

To summaries, this is a working example:
cfg = get_cfg()
cfg.merge_from_file("./detectron2_repo/configs/COCO-Keypoints/keypoint_rcnn_R_50_FPN_3x.yaml")
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7 # set threshold for this model

model = build_model(cfg) # returns a torch.nn.Module
DetectionCheckpointer(model).load('./some_model.pkl') # must load weights this way, can't use cfg.MODEL.WEIGHTS = "..."
model.train(False) # inference mode

img = cv2.imread("./input.jpg")
img = np.transpose(img,(2,0,1))
img_tensor = torch.from_numpy(img)
inputs = [{"image":img_tensor}, {"image":img_tensor}] # inputs is ready

outputs = model(inputs)

@jiangkansg can you help me understand this part of your code, rearrange the channel order to BGR?

img = np.transpose(img,(2,0,1))
img_tensor = torch.from_numpy(img)

i have a color image with image_height=352, image_width=640

Additionally i observe an anomaly, if i infer my image using DefaultPredictor the prediction output is accurate, but if i am using model one of the object is not detected (less accurate), i have set the cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5 accordingly, so i might be missing some finer detail here

[jiangkansg]

@wiamadaya , numpy image is arranged as (H, W, C), ie the color channels are located at the last axis. In pytorch, it is (C, H, W). That is why we need to re-arrange the axises.

[scploeger]

I was using config to load the weights. After you latest comments, I now use DetectionCheckpointer(model).load(file_path) to load the weights. It is working now. Lessons learnt! Thank you.

To summaries, this is a working example:
cfg = get_cfg()
cfg.merge_from_file("./detectron2_repo/configs/COCO-Keypoints/keypoint_rcnn_R_50_FPN_3x.yaml")
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.7 # set threshold for this model

model = build_model(cfg) # returns a torch.nn.Module
DetectionCheckpointer(model).load('./some_model.pkl') # must load weights this way, can't use cfg.MODEL.WEIGHTS = "..."
model.train(False) # inference mode

img = cv2.imread("./input.jpg")
img = np.transpose(img,(2,0,1))
img_tensor = torch.from_numpy(img)
inputs = [{"image":img_tensor}, {"image":img_tensor}] # inputs is ready

outputs = model(inputs)

@jiangkansg did you ever use this implementation and go further to visualize the output, ensuring that it worked as expected?

[sandeepsign]

Using .pth for batch inference

from detectron2.modeling import build_model

img = cv2.imread(image_path)

cfg = get_cfg()
cfg.merge_from_file("model_config.yaml")
cfg.MODEL.WEIGHTS = "model_final.pth"
cfg.MODEL.SCORE_THRESH_TEST = 0.5
cfg.MODEL.DEVICE='cpu'

Setup predictor

model = build_model(cfg)

model_dict = torch.load("model_final.pth", map_location=torch.device('cpu'))
model.load_state_dict(model_dict['model'] )

model.train(False)

img = np.transpose(img,(2,0,1))
img_tensor = torch.from_numpy(img)

inputs = [{"image":img_tensor}, {"image":img_tensor}, {"image":img_tensor}, {"image":img_tensor}, {"image":img_tensor}, {"image":img_tensor}, {"image":img_tensor}, {"image":img_tensor}, {"image":img_tensor}, {"image":img_tensor}]

outputs = model(inputs)

[cbasavaraj]

Hi, we got batch inference to work, but inference time grows linearly with batch size. Any idea why?