Yolact: direct drivable area segmentation with yolact

[pandamax]

Thanks for your wonderful job！I have implemented direct drivable area segmentation with yolact_darknet53_config.And the test video below shows that it really works.But for the speed,is there any solution that use more lightweight model to be the backbone,for example MobileNets?
https://youtu.be/ZzLeWhUnpmI

[dbolya]

Some people in #27 are trying out mobilenet, but I think you already saw that (@Wilbur529 shares his code there). Other than that, no I don't have plans to implement backbones other than the 4 we already have. However, adding your own backbone should be pretty easy. Check out the code in backbone.py and you should be able to follow along. If you do decide to try to implement it yourself, feel free to ask questions if you're stuck.

Though, mobilenet (because it's so lightweight) might actually struggle a lot with YOLACT, because we leave localization to the network (which is a difficult task).

Actually--if your application is drivable area segmentation, wouldn't it be better to use a semantic segmentation method? YOLACT makes compromises to be able to do instance segmentation, so if you don't need instance segmentation you're much better off just using semantic segmentation. (Though correct me if I'm wrong. I'm not familiar with drivable area segmentation)

[pandamax]

Yep!Thanks for your prompt reply.As you said, it's better to use semantic segmentation methed to implement coarse drivable area segmentation especially taking speed into account.

But other classes will be added in the forthcoming experiment,for example alternative drivable area and car.Semantic segmetation method is hard to find the every lane‘s boundary when number of lane and driving car’s perspective is uncertain.So I think Instance segmentation method should be adopted.

Above all,thanks again for your contribution.

[dbolya]

That makes sense. By the way, if you want it, there's actually a semantic segmentation branch in YOLACT used only for training. See #73 for where you can expose that branch.

[pandamax]

Thank you for the quick response!
I will try the experiment based on your instance segmentation method when alternative drivable area is added.Then test if the model works.of course,I will try the method that you suggest.
Thanks again~

[pandamax]

Here is the test image, green area is the direct drivable area;and the other is the alternative drivable area.it seems fine,of course it depends on the labeled datasets

[abhigoku10]

@pandamax thanks for sharing the idea even i wanted to have a look into it , so as the author suggested did u try the semantic segmentation method for driveable area
@dbolya the link you have pointed in your comment can be used for semantic segmentation right ?
i wanted to do lane marking detection also

[dbolya]

@abhigoku10 Yeah we already have a semantic segmentation module built in. You just need to expose it like in #73.

[abhigoku10]

@pandamax did u perform the instance segmentation or semantic segmentation for your dirveable are a application , if so what is the code changes made to obtain semantic segmentation

[abhigoku10]

@pandamax how many images have you trained you model with to get the output??
@dbolya to get drive-able and non drive-able area should we train the architecture with label

[dbolya]

@abhigoku10 Yeah, train with two instance annotations per image: one for the drivable area, and the other for the non-drivable error. Then, ignore the instance segmentation output of the model and just use the semantic segmentation.

Note: to make this actually useful (over using a semantic segmenter from the get-go), you can also include car and people, etc. instances in your annotation and just discard the semantic segmentation classes for instance segmentation and visa versa for semantic segmentation.

[pandamax]

@pandamax how many images have you trained you model with to get the output??
@dbolya to get drive-able and non drive-able area should we train the architecture with label

In my experience, I got the datasets from BDD100K, you can use the drivable-area data and convert the data format to cocostyle, here is a script may help you. And train dataset is 70K, val datasets is 10K.

meanwhle, I opt the instance segmentation method.

[bond9779]

hey @pandamax script which you have suggested to convert to cocostyle is not available

[abhigoku10]

@pandamax @dbolya i took up ur suggestion and created the derivable area as an instant class for the KITTI road surface dataset which training the model i am facing few issues , please find the below details
1.the overall dataset is 200 images
2.batch size =4 , num_workers=0
i getting nan values after certain iterations which is around 30,40or 60 . i have checked the input and its correct , any thought on this

[dbolya]

@abhigoku10 You mean the loss goes to Nan and never recovers? If so, disable batch norm. Your batch size is too low (set 'freeze_bn': True in the config).

[abhigoku10]

@dbolya thanks for the suggestions but when i do the following the mask loss is increasing incrementally which in turns making the mask value output at the validation map is zero

[dbolya]

@abhigoku10 Uhh I guess try fine tuning? #36

[abhigoku10]

@dbolya i am training the resent 50 model from scratch , i donot have any other classes besides driveable area and background

[dbolya]

@abhigoku10 Still use fine-tuning. It's okay to change the number of classes from the model you're fine tuning from. It's just that 200 images is really really low and woefully inadequate to train masks prediction network.

[abhigoku10]

@dbolya any method on training for these low images , any params we can tune so to see the model generated

[dbolya]

@abhigoku10 Wait you're training from scratch from scratch? i.e. not even imagenet pretraining? There are almost as many parameters as pixels in your training data...

You can try changing learning rate and stuff like that, but I doubt you'll get anywhere.

[abhigoku10]

@dbolya i am using resent50 imagenet pretrained model for training with kitti 200 images annotated with driveable area as a separate class

[dbolya]

@abhigoku10 Right, but you realize that even Mask R-CNN does abysmally on KITTI? There are so few images that the best methods are traditional computer vision hand-crafted algorithms. Modern machine learning techniques just don't work as well with so few images.

You can try reducing the learning rate, but it's probably just going to overfit.

[abhigoku10]

@dbolya @pandamax i trained the model using BDD100k data set i am getting the exact boxes but the mask is zero , even during training the mask value during validation is zero , but i am getting proper boxes of the road from the model currently the training set consists of 90k images

[dbolya]

@abhigoku10 Huh, that shouldn't be happening. Maybe you have a bug somewhere in your config, or if the dataset was not originally in COCO format, a bug in conversion?

[pandamax]

@abhigoku10 here is script to check if your data format is cocostyle and visualize your dataset. That may do some help.

[abhigoku10]

@pandamax @dbolya after a through evaluation i had set this (set 'freeze_bn': True in the config). which i made it false now its working on BDD100k dataset , now i shall train on the KITTI dataset and update the results

[abhigoku10]

@dbolya @pandamax i have got better results after training using the dataset . I have few queries
1.the mask value for the road surface detection during validation mAP is low its in the range of 5-10

1. i tried this for lane detection and i get the same the mask value of the range 0-1
2. Any metrics to validate the output obtained of driveable area and alternate driveable area

[aungriah]

@pandamax did you actually manage to add road objects to your direct drivable area segmentor?