Yolact: Some questions

[Auth0rM0rgan]

Hi @dbolya, I have some questions about the training and validation phase. I would appreciate if you can help me!

1) If I want to eval the pretrained model on a full-HD video or HD webcam, is it necessary to train the model with bigger max_size? if yes, suppose my video is 1080p what should be the max_sizefor training?

2) Is it possible to turn off the boundary box in eval mode and get only masks without bb and detection accuracy?

3) any suggestion on how can I modify the code to get one color per instance instead of having different colors for one instance in an image?.

Thanks!

[abhigoku10]

@Auth0rM0rgan these are answers
Q1. since the model during training is resised to 500 or 700 even we give HD, UHD video the results will be smilar but it would be good to train the model with varying pixel of the objects in the same resolution
Q2. yup you can do it inthe eval code
Q3 There is a option obtain the same color for same objects just make the color =False you wil get it

[Auth0rM0rgan]

Thanks @abhigoku10

[dbolya]

Adding to @abhigoku10's answers:

1. Yeah no changes necessary, just run like normal with your 1080p footage and the code will automatically resize to the correct size.
2. I think #286 is exactly what you want.
3. Yup, the option is here: https://github.com/dbolya/yolact/blob/117921a0b2a82cf7b2fefca3d0055acaf8e934c1/eval.py#L135 Just change it to class_color=True.

For your latest question, the best way to detect shadows would just be to mask out the shadow manually too and add that as a separate class. You can create like a -shadow class for each of your classes if it's not important that you match instances to shadows. Otherwise, you might have to go with a hybrid approach where you train one model with the object and shadow in the same mask, and another model with them as separate classes and then subtract one from the other.

[Auth0rM0rgan]

@dbolya Thanks for the information!

About detecting shadow, Is there any dataset that contains both object and shadow in the same mask (I search for it but couldn't find anything)?
Also, what are the possible way to have a more precise mask for objects? in my project, I need to improve the mask to be more accurate. Do you have any suggestions?

Thanks a lot!

[dbolya]

@Auth0rM0rgan I don't know of any. Though you might be able to use some traditional CV methods to find dark blobs close to detected masks? Idk how that would work but if you don't want to make a dataset of your own, that's probably your best bet.

As for making masks more precise, YOLACT++'s base model gives the highest performance of all published real-time methods, but if you're okay with reducing speed you have options:

1. You can add an extra upsample layer for higher quality masks. To do that, change the [(None, -2, {}), (256, 3, {'padding': 1})] on this line:
   https://github.com/dbolya/yolact/blob/db81124874817895db69f2dc443f5c24e0e3f491/data/config.py#L691 to [(None, -2, {}), (256, 3, {'padding': 1})] * 2 where the * 2 at the end dictates how much higher resolution you want (*1 for 138x138, *2 for 276x276, *3 for 552x552, etc.) though be wary of how much GPU memory you have.
2. Increase the input image size (currently 550 for the base version). Check how yolact_im700_config is defined and you can define a similar one for YOLACT++.

[Auth0rM0rgan]

@dbolya Thanks for the information! I'm going to try it. I have access to a GPU cluster with 8 Titan X.
Do you think it's a good idea to train the Yolact or Yolact++ with "Open Images Dataset V5" instead of COCO since there are more images in Open Images Dataset?

[dbolya]

@Auth0rM0rgan The more images the better! Looks like you have the GPUs for it too ;^ )

[Auth0rM0rgan]

Hey @dbolya, do you have any suggestions for converting openImageDatasetV5 instance segmentation annotations files( the annotation files are CSV) to COCO JSON format? (I'm trying to do that but seems a little complicated)

Thanks!

[dbolya]

@Auth0rM0rgan I can't tell you exactly how to do it without looking at the annotations themselves, but I wrote a script for converting pascal to COCO a while ago that might help:
https://github.com/dbolya/yolact/blob/master/scripts/convert_sbd.py

The input format is not the same, but this will show you what information needs to be included.

[Auth0rM0rgan]

@dbolya Thanks a lot. I'm trying to modify your script for open image dataset segmentation. In open image dataset, annotations store in CSV file and the format is like below:

Instance segmentation masks
MaskPath,ImageID,LabelName,BoxID,BoxXMin,BoxXMax,BoxYMin,BoxYMax,PredictedIoU,Clicks

25adb319ebc72921_m02mqfb_8423aba8.png,25adb319ebc72921,/m/02mqfb,8423aba8,0.000000,0.998438,0.089062,0.770312,0.62821,0.15808 0.26206 1;0.90333 0.41076 0;0.17578 0.66566 1;0.00761 0.23197 1;0.07918 0.26058 0;0.31792 0.47737 1;0.12858 0.59262 0;0.73229 0.34016 1;0.01865 0.20001 1;0.52214 0.31037 0;0.83596 0.28105 1;0.23418 0.60177 0

MaskPath: name of the corresponding mask image.
ImageID: the image this mask lives in.
LabelName: the MID of the object class this mask belongs to.
BoxID: an identifier for the box within the image.
BoxXMin, BoxXMax, BoxYMin, BoxYMax: coordinates of the box linked to the mask, in normalized image coordinates.
PredictedIoU: if present, indicates a predicted IoU value with respect to ground-truth.
Clicks: if present, indicates the human annotator clicks, which provided guidance during the annotation process

Thanks!

[dbolya]

Yeah, the process should be pretty similar as what's in the script. The main difference is that looks like the masks are all independent, so you can just load the mask with pillow or something and convert it to numpy. Then the box is pretty straightforward, though it looks like those are relative coordinates and COCO expects absolute coordinates so you'll have to multiply them by the image width / height.

[Auth0rM0rgan]

@dbolya, Thanks a lot for the information! Could you please provide more details (I'm stuck in converting format :| )?

[dbolya]

@Auth0rM0rgan Which types of data specifically are you stuck in converting?