Caffe: Training on new dataset with small objects

[miquelmarti]

I am trying to train SSD on an aerial view dataset in which the objects are quite small. It has 6 different classes and is quite large (I extracted around 100000 images for training) however most of the images are really similar as they come from frames of a few different videos.

After setting everything up and checking with ssd.ipynb that the dataset is correctly formatted etc. I started to train after changing num_classes and disabling expand in the data augmentation and the loss goes to nan within 60 iterations.

Might that be caused by the target objects being to small? The smallest prior bboxes seem to be about the same size as the objects.

Has anyone faced something similar? Any suggestions or possible causes?

Thanks!

[miquelmarti]

Diminished the learning rate and it worked, might have been caused by using a smaller batch size due to memory limitations.

Closing.

[rremani]

@miquelmarti Hi, it's 60+ gb data to download, you know any source from where I can get sufficient data to test.

Thanks

[miquelmarti]

It is indeed a pain to handle this dataset but it's the biggest by far, which is good for training.

I also considered these:
http://vision.cse.psu.edu/data/vividEval/datasets/datasets.html
https://ivul.kaust.edu.sa/Pages/Dataset-UAV123.aspx

Or using a simulator, like this GTAV-based one (https://github.com/ai-tor/DeepGTAV), to build your own dataset without having to annotate it.

[rremani]

Thanks for the datasets.

[rremani]

Hi @miquelmarti ,
I could download the data and tried training on it with taking only one class 'Pedestrian', but was getting 0 loss from the start, it's very strange. I also followed your steps like disabling expand. Could you suggest me things I should look at while training Stanford dataset?

Thanks

[miquelmarti]

You could start by checking that the LMDB files you created are correct with the following notebook: https://github.com/weiliu89/caffe/blob/ssd/examples/ssd.ipynb
... changing the plot so it shows the ground truth bounding boxes.

Loss is zero when there is no ground truth bbox so that could be the cause...

[rremani]

I stored the annotations in a text file in this order label_id, xmin, ymin, xmax, ymax without normalizing them, is this the correct way.

[miquelmarti]

I don't know, I transformed the annotation to the XML format used for Pascal VOC and then created the LMDB database using the create_data.sh script in data/VOC0712.

Check the scripts in my fork https://github.com/miquelmarti/caffe/tree/ssd/data/SDD

[rremani]

Thanks @miquelmarti but I was just wondering why it isn't working with text annotations. I went through the function (https://github.com/weiliu89/caffe/blob/ssd/src/caffe/util/io.cpp#L462) which parse the text. One thing I noticed in that function that it wasn't taking the height and width from the text annotated file compared to other functions for parsing like in xml and json.

It should be something with the order I was inserting the annotations.
Some observations into this problem would be very helpful @weiliu89 @miquelmarti

[miquelmarti]

To be honest I only saw after converting the annotation to XML that the txt option was possible so I never checked it because the other approach worked just fine. Tell me if you can make it work because I have another dataset to convert using the same format (VATIC txt files) than Stanford Dataset, would be helpful.

[rremani]

Ya, I am looking into it, if I get something will share it.

[rremani]

Hi,
I tried using text but despite mentioning height and width while creating lmdb it is not working. With your xml code I was able to train. I took 1e-5 as my learning rate, how much was yours and mAP you got?

[miquelmarti]

For VGGnet with size 500x500 I used 1e-4 for 80000 iter and 1e-5, 1e-6 for 20000 each as it was done for VOC on some other cases. Standard parameters for SSD from ssd_pascal.py. Total batch size 32 (2x1, iter_size 16).

Got this on the validation set I created with my script:

It's worth mentioning though that all videos appear in the training set in the way I did the split, even if frames in the training, val and test sets are all different, they are quite similar so this result should be taken with a grain of salt as it is probably heavily overfitting to the dataset characteristics.

I will run the model on another dataset and see how it performs, at least qualitatively. Will update.

What did you get @rremani ?

[rremani]

Hi @miquelmarti, thanks for the detailed description.

I took 800 train and 200 validation images with size 300*300, batch_size = 8 and accu_batch_size = 8. My results showed that map_val was highest for 10000 iterations (0.46 mAP) and it dropped gradually till 70000. I just took one class i.e. 'Pedestrians', I feel it should have got much better accuracy with just one class.

I totally agree with your point that train, val and test have images from one type (bookstore, nexus etc) with more or less same type of background which makes it more prone to over fitting and difficult to generalise it. But I think your splitting code took samples from different folders, if I am right, which should generalise a bit since each folder has a different theme?

BTW, which code you used for visualizing detections.

[rremani]

@miquelmarti can you also show me your loss plot.

Thanks

[miquelmarti]

I think you are right but the whole dataset looks somehow similar, image quality seems quite poor, especially color is quite off. I don't know why just taking Pedestrians didn't work but my guess from your network overfitting (val mAP going down after 10000 iter) is that it is because a very small number of images. I would try creating an LMDB including something like 10000 datapoints for train and also a bigger validation set.

For visualization I used ssd_detect.py with some minor modifications. Loss plot looks like this:

If you want to continue the conversation maybe write me a mail so we don't spam peopl with this closed issue any longer.

[rremani]

ya, that would be better, ill mail you. Taking iter_size mentioned by you is showing quite better results.

[roipony]

@miquelmarti
Can you please explain more about https://github.com/ai-tor/DeepGTAV especially how to create labeled database (class and bb).

Thanks

[miquelmarti]

Dear @roipony, for questions related to that repo write there, I'm sure @ai-tor will be happy to help.

Best

[fraukej]

Hi @miquelmarti,
I know I'm asking this over a year later, and you're probably in a totally different place now, but I'm currently working on SSD, trying to let it detect objects in my own dataset and seeing that you were able to do that, I thought you might be able to help me. I'm facing some issues considering my data and my lmdb files, I figured. So I tried to check that my annotated data is correct using the provided ssd.ipynb, as @weiliu89 suggested.

The problem is that I don't know how this notebook can validate my data. It seems to me that it can only be used after I've trained my network. But I wanted to make sure my data is correct before training.

I took a look at the ssd.ipynb in your fork and it seems to me that you are using the solver.prototxt of your own dataset (SDD), but you use the pretrained model (VGG_ILSVRC_16_layers_fc_reduced.caffemodel).

# We create a solver that fine-tunes from a previously trained network.
solver = caffe.get_solver(os.path.join(caffe_root, 'models/VGGNet/SDD/SSD_500x500/solver.prototxt'))
solver.net.copy_from(caffe_root + '/models/VGGNet/VGG_ILSVRC_16_layers_fc_reduced.caffemodel')

You are also using the labelmap of your dataset:

# load PASCAL VOC labels
labelmap_file = 'data/SDD/labelmap_SDD.prototxt'

I've tried these steps using my dataset, but when I'm executing the notebook, the images outputted in code block 4 are black without any bounding boxes. Does that mean that my annotated data is wrong? What else could I be doing wrong? Do you have any ideas/tips for me?

In #885 I provided a more detailed description of the problem.

I'd be very thankful for your help :)