Detectron2: Visualization of rotated bounding boxes

[kurapan]

Hi!
I'm working with rotated bounding boxes now and I found out that the visualizer cannot display the bounding box annotation for rotated boxes. The reason it cannot display rotated bounding boxes for a dataset by using draw_dataset_dict is simply because the code expects a BoxMode to convert the dataset coordinates to the variant which is used for visualization. However, rotated bounding boxes currently don't have any BoxMode.
I made a small fix that let me skip the coordinate conversion and visualize the rotated bounding boxes. I was considering submitting a pull request, but I think that a cleaner solution would be to add an additional BoxMode value for rotated bounding boxes. What is your opinion?

This is my makeshift solution. I replaced line https://github.com/facebookresearch/detectron2/blob/706a3185d187ea912b324eb0b5eac064ca0a62ce/detectron2/utils/visualizer.py#L489 with this code.

if "bbox" in annos[0] and len(annos[0]["bbox"]) == 4:
    boxes = [BoxMode.convert(x["bbox"], x["bbox_mode"], BoxMode.XYXY_ABS) for x in annos]
else:
    boxes = [x["bbox"] for x in annos]

If you think this is okay, I will submit a pull request. If you think that an additional BoxMode value would be better, I can try adding one and submit a pull request then.
I'm looking forward to your reply.

[ppwwyyxx]

We have an internal commit that's going to add a box mode for rotated boxes soon.

[kurapan]

I see. Thanks for the reply!

[ppwwyyxx]

added in b83f35d30a016841041038caa4d04b4ae7feffc0

[ppwwyyxx]

Though the rotated box mode is added, draw_dataset_dict will now convert rotated boxes to axis-aligned boxes and draw them like axis-align boxes. However it should be able to draw them using overlay_rotated_instances. This is still TBD.

[ssaru]

Hi! now i trying visualize rotated bbox in detectron2.
but it's not visualize well

i build own get_dataset_dict as here

def get_rbox_detection_dicts():
    annotations = read_dataset()
    detection_anno = DetectionAnnotations(annotations)
    classes_list = collect_class_info(detection_anno)

    dataset_dicts = []
    for idx, FILE in enumerate(detection_anno.FILES):
        record = {}

        filename = FILE.FILEPATH
        height = FILE.IMAGE_HEIGHT
        width = FILE.IMAGE_WIDTH

        record["file_name"] = filename
        record["image_id"] = idx
        record["height"] = height
        record["width"] = width

        objs = []

        for OBJ in FILE.OBJECTS:
            xmin = OBJ.XMIN
            ymin = OBJ.YMIN
            xmax = OBJ.XMAX
            ymax = OBJ.YMAX
            theta = -OBJ.THETA

            classes = OBJ.CLASS

            cx, cy, bbox_w, bbox_h, angle = bbox_convert(xmin, ymin, xmax, ymax, theta)
            objs.append({"bbox": [cx, cy, bbox_w, bbox_h, angle],
                         "bbox_mode": BoxMode.XYWHA_ABS,
                         "category_id": classes_list.index(classes),
                         "iscrowd": 0})

            record["annotations"] = objs

        dataset_dicts.append(record)

    return dataset_dicts

after that i try visualize rotated box like here

DatasetCatalog.register("Rbox-Detection", get_rbox_detection_dicts)
MetadataCatalog.get("Rbox-Detection").set(thing_classes=["A", "B", "C", "D"])
rbox_detection_metadata = MetadataCatalog.get("Rbox-Detection")

dataset_dicts = get_rbox_detection_dicts()

import matplotlib.pyplot as plt

for d in random.sample(dataset_dicts, 3):
    img = cv2.imread(d["file_name"])
    visualizer = Visualizer(img[:, :, ::-1], metadata=rbox_detection_metadata, scale=1)

    vis = visualizer.draw_dataset_dict(d)
    test_img = vis.get_image()[:, :, ::-1]

    plt.figure()
    plt.imshow(test_img)
    plt.show()

but image display like this

so i fixed this line. like here

# Origin
boxes = [BoxMode.convert(x["bbox"], x["bbox_mode"], BoxMode.XYXY_ABS) for x in annos]

# Fixed
boxes = [BoxMode.convert(x["bbox"], x["bbox_mode"], BoxMode.XYWHA_ABS)
                     if x["bbox_mode"] is BoxMode.XYWHA_ABS else
                     BoxMode.convert(x["bbox"], x["bbox_mode"], BoxMode.XYXY_ABS) for x in annos]

https://github.com/facebookresearch/detectron2/blob/706a3185d187ea912b324eb0b5eac064ca0a62ce/detectron2/utils/visualizer.py#L489

as result it work well

[kurapan]

That's a way to bypass it and visualize rotated boxes with the current code, but hopefully some following updates will enable visualization of rotated boxes and training with rotated boxes by default.

[TemitopeOladokun]

Hello everyone,
Can someone explain why I am having this error message?

TypeError Traceback (most recent call last)
in ()
6 for d in random.sample(dataset_dicts, 3):
7 img = cv2.imread(d["file_name"])
----> 8 visualizer = Visualizer(img[:, :, ::-1], metadata=brands_metadata, scale=0.9)
9 vis = visualizer.draw_dataset_dict(d)
10 #cv2_imshow(vis.get_image()[:, :, ::-1])

TypeError: 'NoneType' object is not subscriptable`

[MrParosk]

@ppwwyyxx Is there any update on this? Otherwise I'm willing to fix it.

However, I'm a bit unsure how to approach it. One way would be to set the box-mode in the MetadataCatalog, i.e.

MetadataCatalog.get("my_dataset").set(box_mode=BoxMode.XYWHA_ABS) # Similar to thing_classes

Then later reuse it:

to_box_mode = metadata.get("bbox_mode", None)

# If None use XYXY_ABS
boxes = [BoxMode.convert(x["bbox"], x["bbox_mode"], to_box_mode) for x in annos] 

Or maybe hard-code it, i.e if box_mode is not XYWHA_ABS convert it to XYXY_ABS. If it is XYWHA_ABS, do nothing.

What do you think?

[ppwwyyxx]

Or maybe hard-code it, i.e if box_mode is not XYWHA_ABS convert it to XYXY_ABS. If it is XYWHA_ABS, do nothing

Yes this should be done by draw_dataset_dict. Using the length of the box may be better