Yolact: While training a custom data in intermediate i'm getting error

[CzJaewan]

While training with the dataset I created, I ran into the following problem:

(yolact) nscl@nscl-lab:~/catkin_ws/src/yolact_instance$ python train.py --config=yolact_base_config --batch=16
loading annotations into memory...
Done (t=0.06s)
creating index...
index created!
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
Initializing weights...
Dataset COCODetection
Number of datapoints: 1516
Root Location: ./data/nobrand/images/train
Transforms (if any):
Target Transforms (if any):
Begin training!

[ 0] 0 || B: 7.083 | C: 26.982 | M: 5.827 | S: 12.983 | T: 52.875 || ETA: 0:00:00 || timer: 12.687
Traceback (most recent call last):
File "train.py", line 408, in
train()
File "train.py", line 226, in train
for datum in data_loader:
File "/home/nscl/anaconda2/envs/yolact/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 568, in __next__
return self._process_next_batch(batch)
File "/home/nscl/anaconda2/envs/yolact/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
ValueError: Traceback (most recent call last):
File "/home/nscl/anaconda2/envs/yolact/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/nscl/anaconda2/envs/yolact/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/nscl/catkin_ws/src/yolact_instance/data/coco.py", line 100, in __getitem__
im, gt, masks, h, w, num_crowds = self.pull_item(index)
File "/home/nscl/catkin_ws/src/yolact_instance/data/coco.py", line 225, in pull_item
masks = np.vstack(masks)
File "<__array_function__ internals>", line 6, in vstack
File "/home/nscl/anaconda2/envs/yolact/lib/python3.6/site-packages/numpy/core/shape_base.py", line 282, in vstack
return _nx.concatenate(arrs, 0)
File "<__array_function__ internals>", line 6, in concatenate
ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 3145728 and the array at index 5 has size 921600

---

I think it's a mask size problem. Shouldn't the masks be different sizes?

[dbolya]

Have you messed with the data augmentation at all? The data augmentation resizes all masks to be the same size, since YOLACT works on fixed size input.

[ljjyxz123]

While training with the dataset I created, I ran into the following problem:

(yolact) nscl@nscl-lab:~/catkin_ws/src/yolact_instance$ python train.py --config=yolact_base_config --batch=16
loading annotations into memory...
Done (t=0.06s)
creating index...
index created!
loading annotations into memory...
Done (t=0.01s)
creating index...
index created!
Initializing weights...
Dataset COCODetection
Number of datapoints: 1516
Root Location: ./data/nobrand/images/train
Transforms (if any):
Target Transforms (if any):
Begin training!

[ 0] 0 || B: 7.083 | C: 26.982 | M: 5.827 | S: 12.983 | T: 52.875 || ETA: 0:00:00 || timer: 12.687
Traceback (most recent call last):
File "train.py", line 408, in
train()
File "train.py", line 226, in train
for datum in data_loader:
File "/home/nscl/anaconda2/envs/yolact/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 568, in next
return self._process_next_batch(batch)
File "/home/nscl/anaconda2/envs/yolact/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 608, in _process_next_batch
raise batch.exc_type(batch.exc_msg)
ValueError: Traceback (most recent call last):
File "/home/nscl/anaconda2/envs/yolact/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in _worker_loop
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/nscl/anaconda2/envs/yolact/lib/python3.6/site-packages/torch/utils/data/_utils/worker.py", line 99, in
samples = collate_fn([dataset[i] for i in batch_indices])
File "/home/nscl/catkin_ws/src/yolact_instance/data/coco.py", line 100, in getitem
im, gt, masks, h, w, num_crowds = self.pull_item(index)
File "/home/nscl/catkin_ws/src/yolact_instance/data/coco.py", line 225, in pull_item
masks = np.vstack(masks)
File "<array_function internals>", line 6, in vstack
File "/home/nscl/anaconda2/envs/yolact/lib/python3.6/site-packages/numpy/core/shape_base.py", line 282, in vstack
return _nx.concatenate(arrs, 0)
File "<array_function internals>", line 6, in concatenate
ValueError: all the input array dimensions for the concatenation axis must match exactly, but along dimension 1, the array at index 0 has size 3145728 and the array at index 5 has size 921600

I think it's a mask size problem. Shouldn't the masks be different sizes?

@CzJaewan
If you are using data augmentation method, then you can resize all of the input images and input masks to the same size, such as 128 * 128.
If you didn't use data augmentation method, then you need to add resize function to the dataset before training.

In my case, I used SSD_Augmentation method, and used the `resize' function to ensure that all of the input images and masks have the same sizes.

[DataXujing]

I have the same problem, did you solve it? @CzJaewan

[jasonkena]

I experienced the same problem, and found a solution at https://github.com/dbolya/yolact/issues/103. It is caused by annotations on an image being from multiple images.

You can easily fix it by replacing
target = self.coco.loadAnns(ann_ids) in coco.py with
target = [x for x in self.coco.loadAnns(ann_ids) if x["image_id"]==img_id], this will remove the mismatched annotations.

[dbolya]

@jasonkena Perfect, thanks for the fix. I'll push it shortly.