Detectron2: How to display subset of classes during inference?

[kHarshit]

❓ Questions and Help

I want to output masks of a particular class in instance segmentation say person (id: 0 according to COCO). In order to do so (on the same image in colab notebook), I first find the indexes of non-zero classes then remove the corresponding tensors from pred_classses, scores, etc. as follows:

cls = outputs['instances'].pred_classes
scores = outputs["instances"].scores
masks = outputs['instances'].pred_masks
print(cls)
# tensor([17,  0,  0,  0,  0,  0,  0,  0, 25,  0, 25, 25,  0,  0, 24], device='cuda:0')

# non-zero elements (non-person class categories)
indx_to_remove = cls.nonzero().flatten().tolist()
print(indx_to_remove)
# [0, 8, 10, 11, 14]

# delete corresponding arrays
cls = np.delete(cls.cpu().numpy(), indx_to_remove)
scores = np.delete(scores.cpu().numpy(), indx_to_remove)
masks = np.delete(masks.cpu().numpy(), indx_to_remove, axis=0)

# convert back to tensor and move to cuda
cls = torch.tensor(cls).to('cuda:0')
scores = torch.tensor(scores).to('cuda:0')
masks = torch.tensor(masks).to('cuda:0')
print(cls)  # similar to as it was before except unwanted classes removed
# tensor([0, 0, 0, 0, 0, 0, 0, 0, 0, 0], device='cuda:0')

# not interested in boxes
outputs['instances'].remove('pred_boxes')

outputs['instances'].pred_classes = cls
outputs["instances"].scores = scores
outputs['instances'].pred_masks = masks

It gives me error:

AssertionError                            Traceback (most recent call last)

<ipython-input-47-695b068ba737> in <module>()
     15 masks = torch.tensor(masks).to('cuda:0')
     16 
---> 17 outputs['instances'].pred_classes = cls
     18 outputs["instances"].pred_boxes = None
     19 outputs["instances"].scores = scores

1 frames

/content/detectron2_repo/detectron2/structures/instances.py in set(self, name, value)
     69             assert (
     70                 len(self) == data_len
---> 71             ), "Adding a field of length {} to a Instances of length {}".format(data_len, len(self))
     72         self._fields[name] = value
     73 

AssertionError: Adding a field of length 10 to a Instances of length 15

Is it because the instances structure has a fixed label 15 in it {'instances': Instances(num_instances=15, image_height=480, image_width=640, fields=[... ?

How do I solve this problem, and is there any better method to do so?

[ppwwyyxx]

You need to create a new Instances object and then set its field.

You can also use instances = instances[instances.pred_classes == 0] as the documentation describes.

[kHarshit]

Thanks, solved it by creating new Instances object as follows:

# create new instance obj and set its fields
obj = detectron2.structures.Instances(image_size=(480, 640))
obj.set('pred_classes', cls)
obj.set('scores', scores)
obj.set('pred_masks', masks)

# now, pass the obj to visualize fn

[monajalal]

@kHarshit how do you convert the cls to actual object names in vgg?

[kHarshit]

@monajalal Use COCO mapping

or use this to get classes from your cfg's training datatset (also check https://detectron2.readthedocs.io/tutorials/datasets.html#metadata-for-datasets)

from detectron2.data import MetadataCatalog
for i, name in enumerate(MetadataCatalog.get(cfg.DATASETS.TRAIN[0]).thing_classes):
    print(i, name)
# output # for 80 classes (81st class is background during training)
0 person
1 bicycle
2 car
3 motorcycle
4 airplane
5 bus
6 train
7 truck
8 boat
9 traffic light
10 fire hydrant
11 stop sign
12 parking meter
13 bench
14 bird
15 cat
16 dog
17 horse
18 sheep
19 cow
20 elephant
21 bear
22 zebra
23 giraffe
24 backpack
25 umbrella
26 handbag
27 tie
28 suitcase
29 frisbee
30 skis
31 snowboard
32 sports ball
33 kite
34 baseball bat
35 baseball glove
36 skateboard
37 surfboard
38 tennis racket
39 bottle
40 wine glass
41 cup
42 fork
43 knife
44 spoon
45 bowl
46 banana
47 apple
48 sandwich
49 orange
50 broccoli
51 carrot
52 hot dog
53 pizza
54 donut
55 cake
56 chair
57 couch
58 potted plant
59 bed
60 dining table
61 toilet
62 tv
63 laptop
64 mouse
65 remote
66 keyboard
67 cell phone
68 microwave
69 oven
70 toaster
71 sink
72 refrigerator
73 book
74 clock
75 vase
76 scissors
77 teddy bear
78 hair drier
79 toothbrush

[SharifElfouly]

Thanks, solved it by creating new Instances object as follows:

# create new instance obj and set its fields
obj = detectron2.structures.Instances(image_size=(480, 640))
obj.set('pred_classes', cls)
obj.set('scores', scores)
obj.set('pred_masks', masks)

# now, pass the obj to visualize fn

I get the following error. It seems that the visualization needs the bboxes.

_DetectedInstance(classes[i], boxes[i], mask_rle=None, color=None, ttl=8)
TypeError: 'NoneType' object is not subscriptable