Detr: Generalizing to New Dataset

[dvd42]

❓ How to do something using DETR

Thanks for providing code for this new and cool approach to object detection!!!

Describe what you want to do, including:

I am trying to train detr on the OpenLogo Dataset. The only thing "weird" about this dataset is that most images have 1 or 2 objects in them, however there are a considerable amount that spam between 20 all the way up to 116.

The problem I am facing is basically that I want to train the model from scratch (imagenet weights) but I cannot even get close to the detectron2 baseline that I have with the same model (36 AP).

I tried an experiment using only 5 queries during 150 epochs and got decent results (24 AP), however when training for 300 epochs and 100 queries I get stuck at around 15 AP.

Should I just try different values for the eof_coef parameter?
Is it possible to train this model from scratch?
Does this approach generalizes to other datasets?
How much should I tune the default hyper-parameters to get close near the detectron2 baseline for the same dataset?

[alcinos]

Hi @dvd42
Thank you for your interest in DETR.
It's a bit hard to understand what is going wrong based on these two training curves alone. I'd suggest you dig further in the failure modes of the predictions. In particular, is it generally unsure how many objects are there? Or maybe the bounding boxes are there but not quite precise enough (thus being counted as misses)?

I don't know much about the dataset you're using, but I would expect that logos are typically small, which is going to cause DETR some problems. One possible way to improve the situation is to switch to DC5 if you can afford the compute.

Another suggestion would be to try to finetune a model trained on coco. See #9 for some tips on that.

Is it possible to train this model from scratch?

Do you mean without imagenet pre-trained backbone? Most likely not using the current code, especially given the small size of your dataset.

Best of luck

[fmassa]

Hi,

I would like to understand a bit better the setup you are using.
I suppose your are using the dataset from https://qmul-openlogo.github.io/, in the Closed-set logo detection setting?

Also, what is the configuration that you have for the detectron2 baseline? Most probably you can't use the same types of data augmentation as we do by default, as horizontal flips don't apply anymore to logos.

If you could share the config file that you use for Detectron2 to get to 36 mAP, plus the breakdown of the mAP for different sizes / classes, it would be very helpful

[dvd42]

Hi. thanks for the quick replies

I suppose your are using the dataset from https://qmul-openlogo.github.io/, in the Closed-set logo detection setting?

Yes that is correct

Also, what is the configuration that you have for the detectron2 baseline?

The configuration can be found here. It is basically the default conf with a couple of changes I do apply the same data augmentation, which it is definetely an oversight due to the horizontal flip problem.

I should've said that I do not expect to reproduce the 36 mAP of detectron2 out of the box due to the fact that detr struggles
with small objects, which tend to be present in logo dataset as @alcinos pointed out. My main concern is to get a reasonable
AP while keeping the number of queries high.

is it generally unsure how many objects are there? Or maybe the bounding boxes are there but not quite precise enough (thus being counted as misses)?

This does not seems to be the case, however there are A LOT of very low confidence detections I can threshold this for visualization but I am not aware of how this affects the mAP metric. The only other problem I see is that small logos are missed but this is to be expected and it is not my main concern at the moment.

The losses for both of the experiments (5 queries 150 epochs and 100 queries 300 epochs) seem both very similar. The only major difference I notice is the cardinality error (I am not quite sure what this means).

Most probably you can't use the same types of data augmentation as we do by default, as horizontal flips don't apply anymore to logos.

Thank you very much for pointing this out. I will remove horizontal flips from the data augmentation ops.

Do you mean without imagenet pre-trained backbone? Most likely not using the current code, especially given the small size of your dataset.

Sorry, I meant without the COCO weights.

[fmassa]

I'll try to give this dataset (or another similar one) a try during the weekend and see what I get from finetuning it.

I wouldn't expect training from scratch to be the best option here, at least not without a lot of training iterations, which would be too time-consuming.

[dvd42]

Thank you so much @fmassa. You can find the openlogo dataset converted to coco format here I adapted it to detr might save you some time.

[dvd42]

Update: I finetuned the model trained on COCO with this dataset and the results are as expected now. Not quite as good as Faster-RCNN but that is to be expected because of small objects. Since I am getting the expected results now, I am closing the issue.

[fmassa]

Thanks for keeping us updated @dvd42 !

I didn't yet have the time to try finetuning, I'll keep you updated whenever I do it.