Detectron2: Training Mask R-CNN using RLE bitmasks error

[alono88]

I am following the Mask R-CNN tutorial and changed the dataset_dict to support segmentation maps in bitmap format using RLE instead of polygons. I confirmed the data is processed properly using detectron2 visualization tool. When trying to train the network, I'm getting an error regarding polygons.

File "/home/ubuntu/detectron2/detectron2/data/detection_utils.py", line 149, in transform_instance_annotations polygons = [np.asarray(p).reshape(-1, 2) for p in annotation["segmentation"]] File "/home/ubuntu/detectron2/detectron2/data/detection_utils.py", line 149, in <listcomp> polygons = [np.asarray(p).reshape(-1, 2) for p in annotation["segmentation"]] ValueError: cannot reshape array of size 1 into shape (2)

It seems there is no support for RLE format during training although the visualization works. Is there any way to train using bitmasks?

[ppwwyyxx]

The model does support training with bitmasks, if instances.gt_masks = BitMasks(..). See the BitMasks class at https://detectron2.readthedocs.io/modules/structures.html#detectron2.structures.BitMasks

But the data loader does not recognize RLE, and therefore does not convert RLE to bitmasks.

[ppwwyyxx]

You can use a different dataloader following https://detectron2.readthedocs.io/tutorials/data_loading.html

[alono88]

Thanks!

[adriaciurana]

I'm with the same problem. To solve it I have edited the following files.

In structures/masks.py BitMasks class add the following method:

@staticmethod
    def from_rle(
        rles
    ) -> "BitMasks":
        """
        Args:
            rles
            height, width (int)
        """
        masks = [mask_utils.decode(rle).astype(np.bool) for rle in rles]
        return BitMasks(torch.stack([torch.from_numpy(x) for x in masks]))

In data/detection_utils.py change:

if len(annos) and "segmentation" in annos[0]:
        polygons = [obj["segmentation"] for obj in annos]
        if mask_format == "polygon":
            masks = PolygonMasks(polygons)
        else:
            assert mask_format == "bitmask", mask_format
            masks = BitMasks.from_polygon_masks(polygons, *image_size)
        target.gt_masks = masks

by:

if len(annos) and "segmentation" in annos[0]:
        polygons = [obj["segmentation"] for obj in annos]
        if mask_format == "polygon":
            masks = PolygonMasks(polygons)
        elif mask_format == "bitmask":
            masks = BitMasks.from_polygon_masks(polygons, *image_size)
        elif mask_format == "rle":
            masks = BitMasks.from_rle(polygons)

        else:
            assert mask_format
        target.gt_masks = masks

In also dataset_mapper.py change:

# USER: Implement additional transformations if you have other types of data
            annos = [
                utils.transform_instance_annotations(
                    obj, transforms, image_shape, keypoint_hflip_indices=self.keypoint_hflip_indices
                )
                for obj in dataset_dict.pop("annotations")
                if obj.get("iscrowd", 0) == 0
            ]

by:

annos = [
                obj
                for obj in dataset_dict.pop("annotations")
                if obj.get("iscrowd", 0) == 0
            ]

In your cfg change cfg.INPUT.MASK_FORMAT = 'rle'

I think I have not had to apply more changes.

[ppwwyyxx]

Now the default dataloader can work with RLE formats inside your dataset.
All you need is:

1. use RLE format (documented at https://detectron2.readthedocs.io/tutorials/datasets.html#standard-dataset-dicts) in your dataset
2. set INPUT.MASK_FORMAT='bitmask'.

[pvtien96]

Hi @ppwwyyxx
I'm trying to train a instance segmentaion model with 1 class X using detectron2's tutorial, but with a binary mask dataset.
With your instructions, I use RLE format (documented at https://detectron2.readthedocs.io/tutorials/datasets.html#standard-dataset-dicts).
In each "annotations", my "segmention" is a dict "represents the per-pixel segmentation mask in COCO’s RLE format. The dict should have keys “size” and “counts”. You can convert a uint8 segmentation mask of 0s and 1s into RLE format by pycocotools.mask.encode(np.asarray(mask, order="F"))." (I verify that bit-mask is successfully converted to RLE format).
So in this case, can you explain some of my questions, please:

1. What exactly is the "segmentation"?
   For example:
   
   left: original binary mask of 0s and 1s
   center: binary mask of the 1st instance
   right: binary mask of the 2nd instance
   => "segmentation" = pycocotools.mask.encode(np.asarray(left, order="F")) OR if I'm not wrong, in each instance "segmentation" = pycocotools.mask.encode(np.asarray(center/right , order="F")), right?
2. I trained the model with the boilerplate code in your getting started collab notebook, every thing is ok. But when I try to evaluate the model's performance using AP metric implemented in COCO API as in your example, I got the error:
   WARNING [03/04 10:32:52 d2.evaluation.coco_evaluation]: json_file was not found in MetaDataCatalog for 'hand10_val'. Trying to convert it to COCO format ... Traceback (most recent call last): File "/home/n/detectron2/datasets/EgteaGaze+/hand10/test.py", line 132, in <module> evaluator = COCOEvaluator("hand10_val", cfg, False, output_dir="./output/") File "/home/n/detectron2/detectron2/evaluation/coco_evaluation.py", line 71, in __init__ convert_to_coco_json(dataset_name, cache_path) File "/home/n/detectron2/detectron2/data/datasets/coco.py", line 416, in convert_to_coco_json coco_dict = convert_to_coco_dict(dataset_name) File "/home/n/detectron2/detectron2/data/datasets/coco.py", line 334, in convert_to_coco_dict polygons = PolygonMasks([segmentation]) File "/home/n/detectron2/detectron2/structures/masks.py", line 271, in __init__ process_polygons(polygons_per_instance) for polygons_per_instance in polygons File "/home/n/detectron2/detectron2/structures/masks.py", line 271, in <listcomp> process_polygons(polygons_per_instance) for polygons_per_instance in polygons File "/home/n/detectron2/detectron2/structures/masks.py", line 262, in process_polygons "Got '{}' instead.".format(type(polygons_per_instance)) AssertionError: Cannot create polygons: Expect a list of polygons per instance. Got '<class 'dict'>' instead.

---

I guess that it must have to set INPUT.MASK_FORMAT='bitmask' (this is in the case of training config), so in the case of evaluating, what need to be set?
I also tried to print("cfg.__dict__= ", cfg.__dict__) but only got
type(cfg)= <class 'detectron2.config.config.CfgNode'> cfg.__dict__= {'__immutable__': False, '__deprecated_keys__': set(), '__renamed_keys__': {}, '__new_allowed__': False}
How can I see all the atribute of the cfg?

1. Can you note all the things relate in case of training with binary mask, is there anything need to be noticed/different with the polygon-training.
   _Many thanks for your great framework and your enthusiastic supports!_

[ppwwyyxx]

Evaluation of a generic (not COCO format) dataset with RLE is not yet supported. There is a TODO in https://github.com/facebookresearch/detectron2/blob/5e2a1ecccd228227c5a605c0a98d58e1b2db3640/detectron2/data/datasets/coco.py#L335-L337 (cc @botcs )

[ppwwyyxx]

The above issue about evaluation is hopefully fixed in https://github.com/facebookresearch/detectron2/commit/6901cc7e35ffcef6a4515efe26410a457612ac36

[siyuanfeng-tri]

The above issue about evaluation is hopefully fixed in 6901cc7

I think I need to do a bit more to get this actually working.

1. I had to cast this into int explicitly, otherwise json dump complains about TypeError: Object of type 'uint32' is not JSON serializable
   https://github.com/facebookresearch/detectron2/blob/master/detectron2/data/datasets/coco.py#L339
2. I had to decode segmentation["counts"] into an ascii string otherwise json dump complains about not being able to serialize bytes. TypeError: Object of type 'bytes' is not JSON serializable

[ppwwyyxx]

There are a few different RLE formats cocoapi recognizes and the 'counts' in coco's original json is actually a list of int.
Should figure out how to convert the bytes to that format.

[siyuanfeng-tri]

Thanks for the reply! I am not that familiar with different flavors of coco TBH. The only thing I really cared about here is to preserve the RLE annotation (which comes from pycocotools.mask.encode()) through json conversion and back through detectron2's wrappers, since I have a custom dataset, where everything is an image.

So I looked briefly around https://github.com/facebookresearch/detectron2/blob/master/detectron2/data/datasets/coco.py#L160 and looked up where imgToAnns comes from in https://github.com/cocodataset/cocoapi/blob/master/PythonAPI/pycocotools/coco.py
I think my workaround is probably fine for what I care about? As long as json is serializing and deserializing that bytes array the same way I should be fine. I did something like segmentation["counts"] = segmentation["counts"].decode(''ascii") before calling json dump.

[kharyal]

Now the default dataloader can work with RLE formats inside your dataset.
All you need is:

1. use RLE format (documented at https://detectron2.readthedocs.io/tutorials/datasets.html#standard-dataset-dicts) in your dataset
2. set INPUT.MASK_FORMAT='bitmask'.

Still not working for me. I am not getting any error but the network is not learning anything. It just runs for 500 and doesn't segment anything. The Epoch error is also dropping to 0.