Yolact: KeyError while running the model on Custom Dataset

[sainatarajan]

Hi! Thanks for this repo. I am trying to run this model on a custom dataset and I face this error. Here is the stack trace:

Traceback (most recent call last):
  File "train.py", line 386, in <module>
    train()
  File "train.py", line 215, in train
    for datum in data_loader:
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 582, in __next__
    return self._process_next_batch(batch)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 606, in _process_next_batch
    raise Exception("KeyError:" + batch.exc_msg)
Exception: KeyError:Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/user/yolact/data/coco.py", line 87, in __getitem__
    im, gt, masks, h, w, num_crowds = self.pull_item(index)
  File "/home/user/yolact/data/coco.py", line 126, in pull_item
    file_name = self.coco.loadImgs(img_id)[0]['file_name']
  File "/usr/local/lib/python3.6/dist-packages/pycocotools/coco.py", line 229, in loadImgs
    return [self.imgs[id] for id in ids]
  File "/usr/local/lib/python3.6/dist-packages/pycocotools/coco.py", line 229, in <listcomp>
    return [self.imgs[id] for id in ids]
KeyError: 7

The value changes every time. I have shown an example where the KeyError threw was 7. I followed the answer from issue #40 , but still, the error persists. My class id's start from 1 and end in 10 (inclusive). Here is my config.py:

my_dataset= dataset_base.copy({
    'name': 'my dataset',
    'train_images': '/home/user/dataset/train/',
    'train_info': '/home/user/dataset/train/via_region_data.json',

    'valid_images':'/home/user/dataset/val/',
    'valid_info': '/home/user/dataset/val/via_region_data.json',

    'has_gt': True,
    'class_names': ('BWK12', 'LWK1', 'LWK2', 'LWK3', 'LWK4', 'LWK5', 'SWK1/2', 'Cage', 'Schraube', 'Stab'),
    'label_map': {1:1, 2:2, 3:3, 4:4, 5:5, 6:6, 7:7, 8:8, 9:9, 10:10}
})

Here is the configuration:

yolact_base_config = coco_base_config.copy({
    'name': 'yolact_base',

    # Dataset stuff
    'dataset': my_dataset,
    'num_classes': len(spine_dataset.class_names)+1,
    .........
)}

[dbolya]

Notice that the key error isn't for classes:
return [self.imgs[id] for id in ids]
It's for images. Check your dataset json and make sure the image with ID 7 exists.

[sainatarajan]

@dbolya Thanks for your reply. I checked my JSON file and it contains the id value 7. As I had said before, it does not occur for only key 7. Every time I run, it throws a different key number. Here is a snapshot:

File "/usr/local/lib/python3.6/dist-packages/pycocotools/coco.py", line 229, in <listcomp>
    return [self.imgs[id] for id in ids]
KeyError: '2'

[dbolya]

So the particular portion of code in pycocotools that creates self.imgs is:

        if 'images' in self.dataset:
            for img in self.dataset['images']:
                imgs[img['id']] = img

So a couple things could have happened:
1.) The dataset didn't get initialized for some reason (to check, it says the loading dataset took x time at startup right?)
2.) Your annotation file doesn't have an 'images' field. Seems weird, but check if it's misspelled I guess.
3.) In your_dataset_json['images'], a couple of these objects don't have an 'id' field (namely 2, 7, etc.), so check if that exists.

If there's no error in any of those 3 cases, then I'm not sure how this could have happened.

One thing to note, is that my COCODataset class gets image IDs from the annotations (to ignore images with no annotations), while pycocotools get image IDs from the your_dataset_json['images'] field, so that indicates to me that you've filled out the image ids properly in your_dataset_json['annotations'] but not in your_dataset_json['images']. Note that each element of your_dataset_json['images'] must look like:

{
    "id": int,
    "width": int,
    "height": int,
    "file_name": str
}

[sainatarajan]

Thanks for the fast reply.

1. Yes, what you have said is correct. The dataset is loaded successfully.
2. I have the 'images' filed in my JSON file and it's not misspelled.
3. The JSON seems to be fine. It happens for all the id's I guess like whichever CPU core runs the code first, the corresponding id fails.
   This is how my train.json looks like:

{
"info": {},
"images": [{}, {}, ...., {}],
"annotations": [{}, {}, ...., {}],
"licenses": [{}],
"categories": [{}, {}, ...., {}]
}

The images field from the train JSON:

[{
            "id": 0,
            "width": 1345,
            "height": 2825,
            "file_name": "9afaad4734ec2ff240f2e6abfd4590f7.jpg",
            "license": 1,
            "flickr_url": "9afaad4734ec2ff240f2e6abfd4590f7.jpg",
            "coco_url": "9afaad4734ec2ff240f2e6abfd4590f7.jpg",
            "date_captured": ""
}, .....]

The annotations field from the train JSON:

[{
            "id": 0,
            "image_id": "0",
            "segmentation": [
                484,
                784,
                698,
                882,
            ],
            "area": 115898,
            "bbox": [
                424,
                780,
                347,
                334
            ],
            "iscrowd": 0,
            "category_id": 2
}, ....]

I have pasted only 4 points in the segmentation field above.
Is there any error in the above format? One thing I noticed is that the img_id that is passed to self.coco.loadImgs(img_id) is an integer, but we can see that the call for an array argument is executed. Here is the loadImgs function from coco.py from pycocotools.

def loadImgs(self, ids=[]):
        """
        Load anns with the specified ids.
        :param ids (int array)       : integer ids specifying img
        :return: imgs (object array) : loaded img objects
        """
        if _isArrayLike(ids):
            return [self.imgs[id] for id in ids]
        elif type(ids) == int:
            return [self.imgs[ids]]

When passed an integer id, this function executes the first return statement instead of the second. We can clearly see this from the stacktrace.

[dbolya]

_OH_, your last observation is actually critical.

Notice something wrong with your annotations?
Specifically on this line:

"image_id": "0"

That's a string not an integer!

It's executing the array branch because it's operating on each character of the string, and it can't find the strings '2' and '7', because of course your IDs are integers! I'm not sure why your first stack trace has KeyError: 7, but your second has KeyError: '2', which indicates that you passed in a string instead of an integer.

So, the solution would be to make sure all the image ids and annotation image_ids are integers (since your first error message had KeyError: 7, it might be that image id 7 has a string for its id field, while all annotations for image id 2 have a string for their image_id fields).

[sainatarajan]

@dbolya Thanks for pointing it out. The train.json file was actually exported from VGG Annotation Tool in COCO JSON format and I just appended the categories field to it. My bad that I didn't notice it. I ran the code after editing, it got past the above error and stopped at a new one. Here is the stacktrace:

Target:  [{'id': 80, 'image_id': 10, 'segmentation': [612, 345, 900, 454, 1018, 459, 926, 704, 695, 616, 529, 590, 581, 542, 612, 463], 'area': 175551, 'bbox': [529, 345, 489, 359], 'iscrowd': 0, 'category_id': 8}]

Traceback (most recent call last):
  File "train.py", line 386, in <module>
    train()
  File "train.py", line 215, in train
    for datum in data_loader:
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 582, in __next__
    return self._process_next_batch(batch)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
TypeError: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/user/yolact/data/coco.py", line 87, in __getitem__
    im, gt, masks, h, w, num_crowds = self.pull_item(index)
  File "/home/user/yolact/data/coco.py", line 138, in pull_item
    masks = [self.coco.annToMask(obj).reshape(-1) for obj in target]
  File "/home/user/yolact/data/coco.py", line 138, in <listcomp>
    masks = [self.coco.annToMask(obj).reshape(-1) for obj in target]
  File "/usr/local/lib/python3.6/dist-packages/pycocotools/coco.py", line 431, in annToMask
    rle = self.annToRLE(ann)
  File "/usr/local/lib/python3.6/dist-packages/pycocotools/coco.py", line 416, in annToRLE
    rles = maskUtils.frPyObjects(segm, h, w)
  File "pycocotools/_mask.pyx", line 292, in pycocotools._mask.frPyObjects
TypeError: object of type 'int' has no len()

Tried to figure out the error, but couldn't.

[dbolya]

When using points, the segmentation should be a list of lists. One list for every disjoint polygon you are segmenting. It's trying to find the length of ann['segmentation'][0] and finding that it's an integer, when it should in fact be a list.

If none of your segmentations are disjoint, you can just make every segmentation value be a list of one list.

[sainatarajan]

@dbolya This is my current JSON structure. I think somewhere it is wrong.

Suppose I have an image_id as 0 and there are 2 segmentations from the image, each segmentation is defined as a separate list in this JSON file as you can see. This format was obtained by exporting to COCO format from VGG annotation tool. If possible, can you give an example of the JSON format structure which the model is expecting?

[dbolya]

Yes, that is fine, except your segmentations need to be lists of lists. A segmentation can have multiple polygons if, let's say the middle portion is obscured. You can't have multiple disjoint polygons with one list of points, so COCO allows you to have multiple lists of points in order to have an arbitrary number of non-intersecting polygons in your segmentation. I mean your segmentation should look like (if it was 1 triangle):

"segmentation": [[ 5, 10,  3, 4,  20, 29 ]]

You can see the COCO data format specs here:
http://cocodataset.org/#format-data

[sainatarajan]

@dbolya Thanks for the help. I am able to execute the code successfully without any problems. But, when I execute eval.py, it does not create any segmentation masks. The outputs generated are exactly the same as inputs. I ran the code only for 10 epochs just to test whether any segmentation masks are created or not. Is it due to training for less no. of epochs?

[dbolya]

You mean the boxes are good but it doesn't generate any masks? Or something else?

Depending on the dataset size, 10 epochs could be nothing. Typically with minibatches this small, iterations matter a lot more than epochs. You can try fine tuning instead if you don't have a lot of data (#36).

[sainatarajan]

Boxes? Yes, the points of the mask for the training set are correct. But, unfortunately it doesn't generate any kind of segmentation. The inputs (test set) are just the same as outputs (generated ones), as a copy and paste.
I just ran for 10 epochs to see whether any segmentations are drawn on the test set. Doesn't matter whether it's correct or wrong.

[dbolya]

Ohhh you mean there weren't any detections. Yeah, you probably just didn't train long enough. Since it sounds like you don't have a lot of data, I recommend find tuning. You're goiung to requre at least a couple 10k iterations before you start seing anything, with full training taking multiple 100k iterations.

If after a large number of iterations (>50k) you dont see anything, then there might be more issues with your setup.

[sainatarajan]

Thank you for the clarification. I will test the dataset on large number of iterations and check for segmentations. This will take a lot of time for me. I will close this issue. Thanks for the help.

[CodeXiaoLingYun]

@dbolya Thanks for pointing it out. The train.json file was actually exported from VGG Annotation Tool in COCO JSON format and I just appended the categories field to it. My bad that I didn't notice it. I ran the code after editing, it got past the above error and stopped at a new one. Here is the stacktrace:

Target:  [{'id': 80, 'image_id': 10, 'segmentation': [612, 345, 900, 454, 1018, 459, 926, 704, 695, 616, 529, 590, 581, 542, 612, 463], 'area': 175551, 'bbox': [529, 345, 489, 359], 'iscrowd': 0, 'category_id': 8}]

Traceback (most recent call last):
  File "train.py", line 386, in <module>
    train()
  File "train.py", line 215, in train
    for datum in data_loader:
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 582, in __next__
    return self._process_next_batch(batch)
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
    raise batch.exc_type(batch.exc_msg)
TypeError: Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/usr/local/lib/python3.6/dist-packages/torch/utils/data/_utils/worker.py", line 99, in <listcomp>
    samples = collate_fn([dataset[i] for i in batch_indices])
  File "/home/user/yolact/data/coco.py", line 87, in __getitem__
    im, gt, masks, h, w, num_crowds = self.pull_item(index)
  File "/home/user/yolact/data/coco.py", line 138, in pull_item
    masks = [self.coco.annToMask(obj).reshape(-1) for obj in target]
  File "/home/user/yolact/data/coco.py", line 138, in <listcomp>
    masks = [self.coco.annToMask(obj).reshape(-1) for obj in target]
  File "/usr/local/lib/python3.6/dist-packages/pycocotools/coco.py", line 431, in annToMask
    rle = self.annToRLE(ann)
  File "/usr/local/lib/python3.6/dist-packages/pycocotools/coco.py", line 416, in annToRLE
    rles = maskUtils.frPyObjects(segm, h, w)
  File "pycocotools/_mask.pyx", line 292, in pycocotools._mask.frPyObjects
TypeError: object of type 'int' has no len()

Tried to figure out the error, but couldn't.

i also meet the first question. And i change the type to int,it is OK.
But i alse meet the second question as you.

File "pycocotools/_mask.pyx", line 292, in pycocotools._mask.frPyObjects
TypeError: object of type 'int' has no len()