Detr: Recommendations for training Detr on custom dataset?

[lessw2020]

[Zumbalamambo]

+1

[PancakeAwesome]

agree

[alcinos]

Hello,
Thanks for your interest in DETR.
It depends on the size of your dataset. If you have enough data (say at least 10K), training from scratch should work just fine. You'll need to prepare the data in the coco format and then follow instructions from the Readme. Note that if your dataset has a substantially different average number of objects per image than coco, you might need to adjust the number of object queries (--num_queries) It should be strictly higher than the max number of objects you may have to detect, and it's good to have some slack (in coco we use 100, the max number of objects in a coco image is ~70)

Fine-tuning should work in theory, but at the moment it's not tested/supported. If you want to give it a go anyways, you just need to --resume from one of the checkpoint we provide. Feel free to report back any results you obtain :)

Best of luck

[raviv]

Hi,

When fine-tuning from model zoo, using my own dataset, how should I modify the number of classes?
Loading the model fails (as expected) on:

RuntimeError: Error(s) in loading state_dict for DETR:
    size mismatch for class_embed.weight: copying a param with shape torch.Size([92, 256]) from checkpoint, the shape in current model is torch.Size([51, 256]).
    size mismatch for class_embed.bias: copying a param with shape torch.Size([92]) from checkpoint, the shape in current model is torch.Size([51]).

As I have 50 labels, and the checkpointed model has 91.

Thanks!

[alcinos]

If you just want to replace the classification head, you need to erase it before loading the state dict. One approach would be:

model = torch.hub.load('facebookresearch/detr', 'detr_resnet50', pretrained=False, num_classes=50)
checkpoint = torch.hub.load_state_dict_from_url(
            url='https://dl.fbaipublicfiles.com/detr/detr-r50-e632da11.pth',
            map_location='cpu',
            check_hash=True)
del checkpoint["model"]["class_embed.weight"]
del checkpoint["model"]["class_embed.bias"]
model.load_state_dict(checkpoint["model"], strict=False)

Best of luck.

[cbasavaraj]

It would be easier (or at least more standard practice) to first load the pre-trained model, and then replace the classification head.

[lessw2020]

related question but how should we downgrade the query number for smaller classes ( in terms of continuing from the approach above)?
For example I only have 5 classes to detect and each image will have exactly 5 classes per image, so I was planning to run with queries = 12 instead of the default 100 (or should it be 5 if we know that's the max our images will ever have...)

I'm looking at model.query_embed with (100,256) and assume that is the right place to adjust but unclear. If we adjust via model.query_embed.num_embeddings=my_new_query_count, is that enough?
(update - I'm working on this and the DETR model stores a self.num_queries as well, but this is only referenced later for segmentation.
But to be correct should update both model.num_queries and the model.query_embed.num_embeddings would need to be adjusted together...)

[lessw2020]

Also wouldn't we want to re-init the weights in class_embed to normal or uniform after wiping the checkpoint weights to kick off the new training?

[alcinos]

If you're fine-tuning, I don't recommend changing the number of queries on the fly, it is extremely unlikely to work out of the box. In this case you're probably better off retraining from scratch (you can change the --num_queries arg from our training script).

As for the initialization of class_embed, the solution I posted above makes sure it is initialized as it should.

Best of luck

[lessw2020]

Hi @alcinos - excellent, thanks tremendously for the advice here, esp on a Sat night.
I will try both fine tuning for now (with smaller dataset and will not touch num_queries) and from scratch as we'll have a larger dataset soon, and update here to share results.
Thanks again!

[raviv]

My dataset has images of various sizes.
Do I need to resize them to a specific size?

[lessw2020]

My dataset has images of various sizes.
Do I need to resize them to a specific size?

I can't answer definitively but if you look at the code in datasets/coco.py, you can see how they handled their image resizing for coco training. Basically they do random rescaling per the scales list, with the largest size dimension maxed at 1333:
`
scales = [480, 512, 544, 576, 608, 640, 672, 704, 736, 768, 800]

if image_set == 'train':
    return T.Compose([
        T.RandomHorizontalFlip(),
        T.RandomSelect(
            T.RandomResize(scales, max_size=1333),
            T.Compose([
                T.RandomResize([400, 500, 600]),
                T.RandomSizeCrop(384, 600),
                T.RandomResize(scales, max_size=1333),

`

The colab example used a max size of 800, with half precision weights.

Thus if your images are all larger than 1333 in one dimension, then they'll all be resized below that with padding anyway.

Hopefully others can add more info here but hope this provides some starter info for you.

[alcinos]

My dataset has images of various sizes.
Do I need to resize them to a specific size?

As was noted by @lessw2020, the images will be randomly resized in an appropriate range by our data-augmentation. The images will then be padded, so having different sizes is not an issue.

Thanks for wonderful work,
What is your recommendation to use DETR for single object detection(e.g., scene text detection) datasets?

I'm not sure about the specifics of your dataset, but in general I'd say all the general advice provided in this thread apply to the case where there is only one object class.

[raviv]

@alcinos, @lessw2020 It seems that these resizes are for data augmentation when training.
As I'm using my own dataloader and augmentations, my question is does the architecture (or implementation) expects images to have some maximum size?
Thanks.

[fmassa]

@raviv no, the architecture doesn't expect a maximum size, but note that the Transformer Encoder is quadratic wrt the number of pixels in the feature map, so if your image is very large (like larger than 2000 pixels), you might face memory issues.

[raviv]

This is how my losses look like so far.
Would love to get other's input on their attempt to train on DETR on custom datasets.

[m-klasen]

Hi, currently working with my custom dataset. Relatively small with ~2k Train, 400 valid images (32 video sequence clips) and only 4 classes with a maximum of 6 instances per image.
For my first training attempt i set num_queries=20 and discared all transformer weights etc.
I trained 400 epochs with apex fp16 at lr 1e-4 with a lr_drop to 1e-5 at 200.

Evaluation at ep400 gives me a mAP of 0.45 which i can benchmark against a known good MaskRCNN from my colleague who achieves 0.63 mAP.
My questions now are, which are the primary reason for the weaker performance?

1. More training? Better LR adjustments with a decay for example (hard to do with the first attempt when you are going in blind)?
2. reduce num_queries further?
3. class/bg loss coef adjustment?
4. ...?

[fmassa]

@mlk1337 thanks for sharing the results!

I think you are at a good starting point. I would say that from the logs you might want to change the eos_coef a bit and try different values. I think the number of num_queries is ok, but the eos_coef probably needs to be adapted.

I don't know if using apex with fp16 affects something or not as I haven't tried, but maybe @szagoruyko can comment on this?

@raviv your training logs are very weird, it seems that the model stopped working at some point early in training. Are you using gradient clipping (it's on by default)

[raviv]

@fmassa I'm running with the default args.
To keep things simple, I'm using 1 class and disabled all augmentations.
The behavior was similar when training multiple classes and with aug enabled.
To speed things up I'm using a subset of my dataset with 8K train and 2K test

[alcinos]

@mlk1337 with such a small dataset, I'd recommend trying to fine-tune the class head, while starting from a pre-trained encoder/decoder. You'll have to keep the 100 queries if you do that, but unless you're after very marginal speed improvement it shouldn't hurt.

[tanulsingh]

Hey , I wanted to fine tune DETR myself on custom datasets , But I am new to all , I have been using torchvision models all the time to fine tune on my dataset . I would be glad if someone shares a demo code for fine-tuning @alcinos

[lessw2020]

@raviv - happy to share my training results but can you post your plot code for the graphs and I'll use that? Right now I just have text output as the detr plot_utils wasn't working (wasn't sure if I should debug that or just move it to tensorboard, looking at that now).
@mlk1337 - same question, can you share your plot code for the logs?

[m-klasen]

@tanulsingh I wrote quick gist on how you can modify DETR to finetune on your own coco-formatted dataset Link. Hope this helps.

[m-klasen]

@lessw2020
https://github.com/facebookresearch/detr/blob/5617b89475faa21d4010c81ee2533e34a06014b5/util/plot_utils.py#L20

changed to
pd.DataFrame(pd.np.stack(df.test_coco_eval_bbox.dropna().values)[:, 1]).ewm(com=ewm_col).mean()
worked for me (for bounding boxes)

[raviv]

@lessw2020 I'm using https://github.com/allegroai/trains/ to track training

[lessw2020]

Thanks very much @raviv and @mlk1337 - here's my first two training runs, I used num queries = 12 (6 classes) and trained from scratch.
I modified eos_coef from .1 to .01 to compare. As you can see, training loss looks great but validation not doing so well.
(one caveat though is I can't hflip b/c this is for medical and flipped = no no, so I will be adding in more augmentations which for EffDet made a big difference and may alone be the validation issue here.)
I'm trying with higher queries now just as a fast check and will go 10x higher on eos_coef, and then will also compare the fine tuning only option (with default 100 queries) and then kick in augmentations.
Anyway at least for train loss, it's learning rapidly and easily:

[raviv]

@lessw2020 What does the dotted line represent?

[lessw2020]

Here's fine tuning vs training from scratch - everything looks much better relatively. (not sure why test class error never changes though...need to review loss criterion?)

@raviv - dotted line represents test (validation) scores, sold is training scores.

[lessw2020]

related question - has anyone written visualization code for viewing sample output images with bboxes during training (i.e. with and/or without gt boxes in same image)?
edit - actually can leverage this code here for part of the visuals:
https://github.com/plotly/dash-detr/blob/master/model.py
https://github.com/plotly/dash-detr/blob/master/app.py

[lessw2020]

lastly, here's fine tuning with detr101-dc5 - for <30 epochs curves look great.
Still unclear what is happening with test_class_error and test_loss_ce (dotted lines = test):

[raviv]

@lessw2020 Re: visualizing sample output, most of the code is in the project's Colab notebook
You would just have to adapt it to show GT bboxes as well.

[ghost]

Hi,

If I want to train on Openimages v6 dataset with 600 classes in 30 GB sets, is it recommended to train all the layers or just the classification head?

And, does the classification head consist of class_embed and bbox_embed or just the class_embed

Finally, if I set num_queries = 700, and 500 epochs, would that be alright?

[m-klasen]

@kratosld

Hi,

If I want to train on Openimages v6 dataset with 600 classes in 30 GB sets, is it recommended to train all the layers or just the classification head?

And, does the classification head consist of class_embed and bbox_embed or just the class_embed

Finally, if I set num_queries = 700, and 500 epochs, would that be alright?

Just remove class_embed.weight & class_embed.bias and keep the rest.
Unless you literally have 700 items in each image, do not change num_queries and keep it at 100, this will give you 100 proposed boxes for each item. Changing num_queries will result in retraining the whole transformer, which is costly.

[ghost]

@kratosld

Hi,
If I want to train on Openimages v6 dataset with 600 classes in 30 GB sets, is it recommended to train all the layers or just the classification head?
And, does the classification head consist of class_embed and bbox_embed or just the class_embed
Finally, if I set num_queries = 700, and 500 epochs, would that be alright?

Just remove class_embed.weight & class_embed.bias and keep the rest.
Unless you literally have 700 items in each image, do not change num_queries and keep it at 100, this will give you 100 proposed boxes for each item. Changing num_queries will result in retraining the whole transformer, which is costly.

Alright, thank you!

[lessw2020]

Just a quick update that I am getting really outstanding results on my object detection inference with DETR (via fine tuning the res101 model).
I still have oddities in no mAP score, and class loss test curve is stuck at 100% error, but in running the actual detections on test images today it just smoked EfficientDet by comparison (D0 and D1). I'm sure this is b/c DETR can understand relationships, which is a big leap for this diagnostic work where all the items are inter-related and was the key reason I was super fired up to switch to DETR as soon as I read about the transformer architecture.
Anyway just wanted to post a big thanks to @fmassa and @alcinos esp. both for the help in getting training going (and also inventing DETR), and @raviv and @mlk1337 for additional feedback here.
This is for malaria and covid work fyi, so it has real life impact.
Thanks again!
(note I'm not signing off here, still have lots more datasets to train and fix mAP etc. but did want to provide an update and thanks!)

[MHI4]

Hello All,
I am quite a beginner in python. My experience is only in MATLAB based training. I was wondering whether anyone enthusiastically prepare a Google Colab notebook for us to train on our Custom Dataset. It might help us to learn the sequential training and validation steps.
I appreciate your contribution, @lessw2020 @mlk1337 @raviv @fmassa @alcinos.
Thank you all in Advance.

[lessw2020]

Hi @MHI4 - I can make a colab this weekend if no one beats me to it.

1 - Do you have a custom dataset I can use to test on and private email for testing it so we don't clog up this thread during dev/testing? My work datasets are private so I can't use those, or we can also pick a general smaller detection dataset for example.

2 - Did you already see @mlk1337 gist link to gist as that gives you the key steps needed to fine tune, though maybe a bit more knowledge to setup vs colab with pluggable params for a dataset.

[MHI4]

Hello @lessw2020,
Thank you in advance for your Colab notebook.

I have started with a small open dataset from here. I have re-annotated object (1 object) in MATLAB and converted it to XML files. Using voc2coco I've generated the json annotation files. All are uploaded here.

You can use this. It would be wonderful. May be you can add the command for voc2coco based XML to json conversion in the same Colab script for all users.

Thank You Again

[lgvaz]

@mlk1337

@lessw2020
https://github.com/facebookresearch/detr/blob/5617b89475faa21d4010c81ee2533e34a06014b5/util/plot_utils.py#L20

changed to
pd.DataFrame(pd.np.stack(df.test_coco_eval_bbox.dropna().values)[:, 1]).ewm(com=ewm_col).mean()
worked for me (for bounding boxes)

Do we have to do this when training with only bboxes (without masks)?

[AlexAndrei98]

Hi @MHI4 - I can make a colab this weekend if no one beats me to it.

1 - Do you have a custom dataset I can use to test on and private email for testing it so we don't clog up this thread during dev/testing? My work datasets are private so I can't use those, or we can also pick a general smaller detection dataset for example.

2 - Did you already see @mlk1337 gist link to gist as that gives you the key steps needed to fine tune, though maybe a bit more knowledge to setup vs colab with pluggable params for a dataset.

I think a notebook to run on a custom dataset would be very helpful 🙌🏼🙌🏼

[lgvaz]

For those having problems training on custom datasets,

I'm writing a library that unifies a data API for object detection, I just finished a tutorial on how to use it with Detr here.

The project provides a very flexible API for custom datasets while still using Detr original source code for training, be sure to take a look =)

---

@MHI4 , @AlexAndrei98 I'm tagging you because you're interested in a tutorial (the source code of the link I shared is a notebook btw, so you can run that).

[lessw2020]

I started on a colab notebook today to walk through fine tuning, though didn't get as far as I thought b/c have some design decisions to make re: easiest way to wrapper custom datasets and should we do the training with all the args right in the notebook or run it with the shell command as current.
(I wrote my own class to handle, but might be easier way and I have trained both by setting all args in a notebook and with shell command. Personally I like having the args listed and all available so I think I'll proceed with that...)
Here's a link to the start though it just ramps up to the real issues atm.
https://github.com/lessw2020/training-detr/blob/master/training_detr_colab.ipynb

[lessw2020]

I've made more progress on the colab for custom training - it's at the point of building the model/post-processor/criterion but I have a bit of a sticking point b/c num_classes in detr.py::build(args) is determined from dataset name.
I simply modified detr.py for my own training but that's not a good solution b/c it will break over time with new updates. I will open an issue and probably do a PR tonight if that's of interest, to close the loop on this so that simply passing in an args.num_classes is supported directly (i.e. defaults to 20 per the current code, but if not coco or coco_panoptic, it will adjust num_classes)?
I think that's the cleanest solution w/o disrupting anything and avoid the need to manually edit detr.py.

[lgvaz]

args.num_classes is supported directly

@lessw2020 In the example I shared I implemented exactly that, in a backwards compatible way:

def build(args):
    if args.num_classes is not None:
        num_classes = args.num_classes
    else:
        num_classes = 20 if args.dataset_file != 'coco' else 91
        if args.dataset_file == "coco_panoptic":
            num_classes = 250

If it's of interest I can do a PR, just let me know =)

[AlexAndrei98]

How does it compare in terms of time to a FasterRCNN in terms of training?

I have 2k images with roughly 10 classes each. I currently have a model that took me two day to train and performs rather well roughly 10000 iters batch size 4 using Colab Pro. Any tips to better tune some parameters? Thanks you

[lessw2020]

1 - I made a new PR to hopefully cleanly and robustly handle supporting args.num_classes with full backwards compat: https://github.com/facebookresearch/detr/pull/89

2 - @lgvaz - thanks for the code snippet! I initially had similar (a None check for args.num_classes and default None), but that will throw an exception if num_classes not present at all in args.
I have several modified main.py and others likely as well where args.num_classes would not be present at all, so I wrappered the check in a try/except block and also defaulted it to 20 in both cases to be back compat with the previous code). I also went with a simple if/then blocks to check for coco and coco_panoptic respectively to keep it uber-readable.

3 - @lgvaz or others - do you happen to have a lightweight wrapper class for supporting datasets in coco format ala handling class_id mapping I could use for the colab? I have my own coco class I made, but it needs reworking imo though it handles the class id mapping issue which blew up my initial detr training.
For the colab I'm trying to keep it very light and minimal any new/external requirements, and integrated with detr as cleanly as possible... so I don't want to integrate a larger project like mantisshrimp for it. But from a quick look tonight at the mantisshrimp project I definitely like some of the abstraction work you've done there with the parser and datasets (reminds me of fastai), so if it could be split off as just a custom class wrapper that would be ideal.

[lessw2020]

@AlexAndrei98 - I was able to train my custom dataset via fine-tuning in half a day on a v100.

For a 2K dataset though I would try fine-tuning first and I don't envision you would need 2 days though obviously what gpu is going to impact that.
I had good results with the default params supplied (bs = 2, etc) so I would also start with those, but you could do a shorter cycle up front and review before committing to the default 300 epochs.
As an initial test, I trained for 60 epochs, and did the lr drop at 50 as a first test. That was plenty to review the model with test data and get an idea, so you could try that and check to determine how much total training time you might need.

[m-klasen]

@lessw2020 Hi, did you have any success fine tuning on a different backbone in your experiments? Afaik the resnet used is quite standard except for a short stem. However, despite training quite extensively I never came close to the AP achieved with the provided resnet. Any thoughts on how to address this issue?

[tazu786]

Hi, while formatting my custom dataset into coco format, I came across this (datasets/transforms line 253 on):
>
if "boxes" in target:
boxes = target["boxes"]
boxes = box_xyxy_to_cxcywh(boxes)
boxes = boxes / torch.tensor([w, h, w, h], dtype=torch.float32)
target["boxes"] = boxes
Why should I apply xyxy_to_cxcywh in the normalization of the bbox target if the coco format for the bbox is already xywh (with xy top left corner)?

[fmassa]

@tazu786 in our implementation of COCO datasets, we convert the boxes to x1y1x2y2 format, see
https://github.com/facebookresearch/detr/blob/1fcfc65f5cfef0836d349d618aa4afe30aeb838e/datasets/coco.py#L67

[AlexAndrei98-LI]

After training with the following parameters I noticed that the model is not quite succesful at learning has anyone had success into fine tuning for their dataset?

args.num_classes = 11
args.epochs = 3000
args.batch_size = 2
args.lr = 0.05
args.train_only_head = True

[thomashossler]

yes, I was successful using this

If you just want to replace the classification head, you need to erase it before loading the state dict. One approach would be:

[m-klasen]

After training with the following parameters I noticed that the model is not quite succesful at learning has anyone had success into fine tuning for their dataset?

args.num_classes = 11
args.epochs = 3000
args.batch_size = 2
args.lr = 0.05
args.train_only_head = True

Your learning rate is really high for this architecture. Try something lower like 1e-4.

[lessw2020]

Hi @AlexAndrei98,
I have been very successful with fine tuning. My recommendation is stick with the provided defaults first and only modify after you get a feel for how those do on your dataset.
In your example above, as @mlk1337 correctly points out, your learning rate is too high. Thus you are just oscillating without learning.
1e-4 is the default lr for detr transformer and head and 1e-5 for the backbone. I would recommend starting with those same learning rates. Training for transformers tends to be more of a long steady process so I'd change your lr and run with that to start.
Also try with 300 epochs, not the 3000 you have and check your results at that point.
Hope that helps!

[vickraj]

Hey all,

I've been training DETR on a custom datasets that has a few pretty dense scenes (350-450ish objects) - Trained from scratch with 500 num-queries, default everything else (had to change the learning rate a bit to make it converge, but it appears to have converged at this point.

Only problem is that there are SEVERAL duplicate boxes on the images with <200 objects, sometimes with the same class sometimes with different classes... NMS can clearly help here, but I was hoping for a solution that doesn't use NMS since to my understanding DETR was not supposed to require this. The converged class error is also pretty high, which i imagine is contributing here. Any suggestions on other parameters to change, or should I just do nms and be done with it?

[alcinos]

Hi @vickraj ,

500 queries is quite high, it's fairly possible that the default hyper-parameters are not optimal for this setting. You can at least try to play with the eos-coef to see if it helps (maybe try 0.2 and 0.05 instead of the default 0.1)

As for duplicate boxes, are the duplicates high confidence?
In general, it really depends what is your end-goal and how you use the predicted boxes down the road. If you only care about AP, and your duplicate boxes are somewhat low confidence, then NMS is unlikely to improve the AP score. As a matter of fact, when evaluating AP for DETR, we provide to the evaluator ALL the queries, even those for which the highest scoring class is "no-object" (in this case we use the second best scoring class and its associated confidence).

Now if you don't really care about AP but care about the quality of the boxes, then it is likely that you can threshold the boxes based on the detection confidence to solve your issue of lesser-quality boxes (eg duplicates), similarly to what we do in the example collab. Note that the said threshold might be class-dependent, especially if the classes are un-balanced. In coco for eg, "person" is the majority class by far, and using a threshold like 0.9 or even 0.95 is likely to retain all the salient ppl in the image without duplicates. For rare classes like "hair-drier", you'll want to use a much lower threshold.

Hope this helps, and best of luck in your trainings with DETR.

[vickraj]

Hi @alcinos, thanks for the tips. These are all pretty high confidence scores for the duplicate boxes unfortunately, which is why I was thinking NMS would help a lot. I removed the "no-object" ones for visualziation purposes, and there were still quite a few of duplicate boxes.

The eos-coef isn't something I've played around with - will definitely try it out. Thanks for the suggestion.

The class-dependent thresholding is interesting, and certainly worth a try. It's possible that I could find some good thresholds to ensure one (or close to one) box per class for this case.

[Mihir-Mavalankar]

Hi,
I am trying to use the DETR model to fine tune it for a dataset with only one class. As expected the class_error for the model goes to zero immediately but the loss_bbox and loss_giou error don't seem to be going down.
loss_bbox error remains in the range of 1500-3000 and loss_giou error remains at about 1.5.
I have trained multiple times on a GPU for about 5 epochs but these losses keep randomly fluctuating and don't go down.

I checked the bboxes I am supplying they are correct and in the x_center,y_center, h, w format.
The learning rate is 1-e4 and I am using torch SGD optimizer. 100 queries as default and all other hyper-parameters are as default.

What might I be doing wrong that leads to the loss not going down? Any suggestions would be greatly appreciated!
@alcinos

[alcinos]

Hi @Mihir-Mavalankar

SGD is not a good idea here, for two reasons: 1) the network was pre-trained with a different optimizer and it's never recommended to switch 2) SGD is known to work poorly with transformer anyways. You should stick to the default (AdamW)

A loss of loss_box of 1500-3000 seems really high to me. Could you check that the targets you provide to the loss are normalized in [0, 1]? Normally the code does it for you here: https://github.com/facebookresearch/detr/blob/1fcfc65f5cfef0836d349d618aa4afe30aeb838e/datasets/transforms.py#L255-L257 but you may be skipping this part for some reason?

[martinj3456]

Hi @lessw2020,
I am looking forward to your colab script to train DETR on my custom dataset. When might the script be ready for us?

Thank you in advance for this awesome endeavor.

[lessw2020]

Hi @martinj3456 - thanks for the feedback! I was planning to finish it this weekend but I was asked to do an org wide demo of DETR on Monday so I'm working on that first.
I should be able to have a working setup posted later this week.
It looks like num_args support won't be added to the codebase so I'll just show how to modify the current code as needed for that and use my current Coco class to show how to setup a custom class, and that should enable anyone to start training their custom dataset. (will then work on improving the process and add image visualizations etc).

[Dicko87]

Great stuff @lessw2020 😃 waiting in anticipation haha... patiently for course!

[tanulsingh]

Hello everyone , @alcinos , I want to use Detr as nn.module type things and build a custom model with , just like we do with hugging face transformers , we can add different things like now fc on top of BERT,roberta,etc . I want to do something like:

class CustomDETR(nn.Module): def __init__(self): super(CustomDETR,self).__init__() self.model = torch.hub.load('facebookresearch/detr', 'detr_resnet50', pretrained=True)

Now here I want to be able to change the queries and everything and when I overrride the forward function I want to be able to use DETR model

The forward function of DETR only gives the labels and not the losses how to get the losses and then backpropagate and fine tune the network ? I really tried , but don't know what to do

[fmassa]

@tanulsingh I believe I've answered your question in #109

[alcinos]

Hi everyone, we released a Detectron2 wrapper for DETR. This can be used as an alternative option to train on custom datasets (though it requires more dependencies).
You can refer to Detectron2's tutorial and colab to get started.

Best of luck.

[Dicko87]

That’s great, is it possible to use this to fine tune the DETR using our custom dataset, rather than training from scratch? I only have about 2K images

[tanulsingh]

@fmassa after clonning now I am trying to import detr.models it gives an error while it was working perfectly before the latest commit , it says that util not found

ModuleNotFoundError Traceback (most recent call last)
in
21
22 #DETR
---> 23 import detr.models
24 #from detr.models import HungarianMatcher
25 #from detr.models.detr import SetCriterion

/kaggle/working/detr/models/__init__.py in

  1 # Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved

----> 2 from .detr import build
3
4
5 def build_model(args):

/kaggle/working/detr/models/detr.py in

  7 from torch import nn
  8

----> 9 from util import box_ops
10 from util.misc import (NestedTensor, nested_tensor_from_tensor_list,
11 accuracy, get_world_size, interpolate,

ModuleNotFoundError: No module named 'util'

[fmassa]

@tanulsingh can you please open a new issue with the error you are having, so that we keep this issue only for the original topic of custom DETR training and tips?

[Dicko87]

Hi there @lessw2020 , I have been following your lovely notebook and wondering how to execute training now that I have set up my custom dataset. I can see from @mlk1337 that to do this on the command line I can do python main.py --dataset_file your_dataset --coco_path data --epochs 50 --lr=1e-4 --batch_size=2 --num_workers=4 --output_dir="outputs" --resume="detr-r50_no-class-head.pth"
However I am doing this through the notebook and not sure how to go about it.... or/and how do I go about running what was done in the notebook, on the command line so that the command just given will work. Thank you very much 😃
I ran the script but got some missing Keys ... here's my output.

[Dicko87]

I manages to fix the above by reading https://gist.github.com/mlk1337/651297e28199b4bb7907fc413c49f58f#gistcomment-3331643
The model now seems to be training away :) looking forward to seeing the results. :)
just started it and it looks like this...

[tanulsingh]

@fmassa I managed to solve the issue and I was able to write complete pipeline , that can be used to fine-tune train etc , A DETR model on any dataset with some changes , I have tried to make as genric and reproducible I can
https://www.kaggle.com/tanulsingh077/end-to-end-object-detection-with-transformers-detr
I plan to upload it on github too , I would really appreciate your suggestions/views on this , it will be highly motivating for me to keep using DETR @alcinos

[lessw2020]

Hi @Dicko87 - great, glad to see you got your training going! I'll integrate the training aspect here shortly into the colab and of course include the strict=False requirement.
The one question I was debating was where to run the training -
a) in the notebook directly (i.e. a live training loop with code exposed) vs
b) running the modified main.py as a shell process, so that's why I hadn't added that yet.
I would like to look at using tensorboard for visuals basically.
I'll just setup with main.py first since that's faster though now that I think about it.

Note if anyone has a good public dataset for sample training that would be great if you can post a link (saw someone had a mask detector for example). The dataset that was posted earlier for stairs has some issues with how the train/test is setup (they had pre-augmented so you can get nearly identical images in train and test..) so hoping to use a different example to train on and demo with in the colab.

[tanulsingh]

Hi @lessw2020 Please see this https://www.kaggle.com/tanulsingh077/end-to-end-object-detection-with-transformers-detr
I don't know whether it helps or not but I would love to have some views on it

[Dicko87]

Hi @lessw2020 , thanks very much for the reply :) Brilliant, I'll take another look at your notebook once you've added to it, will keep checking.
Just something I've noticed while training is going on, on every epoch the class_error goes between 0.00, 25.00, 33.00, 50.00
these numbers seem a little ... emmm... high, just wondering if this is normal (I was expecting something like 0.6, 0.3 etc etc) Running for 50 epochs on a training set of 839 images and validation set of 160 images. Currently on epoch 14.

Oooo, one more thing ... was just thinking, what happens if for some reason the training got interrupted and it disconnected after epoch 14 say and I want to resume from where it left off, back from epoch 14 rather than starting from the beginning .... does anyone know how to do this ... if it's even possible !
Thanks :)

[Dicko87]

Hey @tanulsingh, great work.
I was wondering if you know how to create a test set and calculate the mAP of it, that would be a great thing to see.

[lessw2020]

Hi @tanulsingh - nice job! I like how you integrated Albumentations as I will be doing the same for my own work.

@Dicko87 - I had the same thing with the large class_error fluctuations, so it's pretty normal. Using the average meter that Tanul used in his kernel might smooth that out. Alternatively I hope to setup with tensorboard for the training visuals soon and can run an average there.

I did not get mAP working yet on my datasets so definitely seeing more info on that from anyone that has it working would be great.

[tanulsingh]

@Dicko87 Thanks ,yeah that is the next step , will soon do that , I have a lot of things planned , I will report the map here as well as in kaggle dicussion forums .
I have also prepared a separate kernel for calculating MAP for any dataset , will be updating that too soon @lessw2020

[Dicko87]

@tanulsingh Exciting stuff, I can’t wait to see the results! :)

[lessw2020]

oh forgot to answer @Dicko87 - yes you can pickup again if training is interrupted. Look for the latest weights in the output directory and then use the --resume option and point it to those weights to pick up again.

@tanulsingh - kernel for calculating mAP for any dataset sounds outstanding! Definitely look forward to that.

I also got the heatmaps going on my work datasets from the checkin that was done yesterday (thanks again @fmassa and @alcinos!) and was able to then interpolate them onto the test images directly to give a nicer visual result and will show how to do that soon.
I'm hoping to mimic the way they did it in the paper today as they apparently used seaborn vs I was using magma and jet cmaps which look good but not as nice as the ones in the paper.

[Dicko87]

@lessw2020 hiya, where do I find the latest checkin haha, wondering if I’ve been looking in the wrong place. I used was looking at the training ipynb in your repo ... detr

[lessw2020]

Hi @Dicko87 - it's actually not in the codebase proper (might be good to put it in a notebooks folder?) but they checked in and added a link on the readme to a new colab that shows how to run predictions and do heatmaps yesterday is what I am referring to. You can find it in the notebooks section on the github homepage for detr or here's the direct link (jumps you into colab):
https://colab.research.google.com/github/facebookresearch/detr/blob/colab/notebooks/detr_attention.ipynb

[Dicko87]

Ah right I see, well thank you very much for the link - much appreciated.
I was thinking .. is there a way to 'Save the Best Epoch' for example say we got an mAP of 0.5 at epoch 10 but at epoch 20 we got mAP 0.2 - would be best to use the model on epoch 10.
Very intrigued to see the heatmaps, will gave a gander :)

Also, has anybody tried it with Resnet101 ? just wondering if the results are much better, might try that sometime this week.

Just though of something else haha... augmentations, does DETR automatically apply data augmentations to my dataset to help it generalize ? All I did was keep my data in the same format as coco so it reads it as coco, not sure if the default network did some nice augmentations on the fly.

[eslambakr]

Hello All,

I am using DETR on custom data, which contains 2k images for training. I have followed your suggestion to fine-tune to avoid getting zeros, and I succeeded in achieving comparable accuracy.

But when I tried to train from scratch using the default configuration in main.py, I got zeros for the first 100 epochs until now, so should I wait for more epochs? I think it is so weird
So what do u think should I do to be able to get a good accuracy from scratch?

[lessw2020]

@Dicko87 - I tested with both 50 and 101 on my dataset. Got a bit better with 101 but the difference wasn't huge. This is likely going to vary a lot based on dataset though so I think you ultimately want to test both.
Note that the dc5 variants will require about 2x more memory and I hit CUDA out of memory on both...so I just tested regular 50 and 101.

[Dicko87]

@lessw2020 Cool, thanks for the info :) I am just curious ...
In my training set I have 1678 images and in my validation set I have 320 images. Now during training, besides the number of epochs it says Epoch:[n] [n/830] during training and Epoch:[n] [n/160] during Test. Only half of my data is shown, but I have just realised I have the batch size set to 2 - are all my images being used, I assume so, just wondering why there aren't two batches of 839 and two batches of 160

[Dicko87]

Woooohooo @lessw2020 my training has now completed ... after 5 hours, just wondering how I go about running my model in evaluation mode to run some predictions on a test image ?

[lessw2020]

Hi @Dicko87 - congrats, that's great to hear!
For predictions, I can post some code but the basic steps are to load the model with your final weights, be 1,000% sure you put it in eval mode, and then you can just setup a detect and show_results function.
The detect needs to take the image, resize,normalize, and totensor and then run it through the model. You can then screen out the confidence levels of the results to just use the confident predictions.
Then it's just a matter of resizing/rescaling the bboxes and plotting them onto the original image to 'see' the results.
Hang on let me post the link to their colab that has some good starting code

[lessw2020]

Here's a jump to their colab with their prediction process:
https://colab.research.google.com/github/facebookresearch/detr/blob/colab/notebooks/detr_demo.ipynb#scrollTo=-CVJRl28-_wS

Note that there's some gotchas if you want to use the gpu for predictions (much faster but you have to move things back and forth) and I'll add that to a prediction colab soon.

[Dicko87]

Awww @lessw2020 that will be great, thank you.
I was playing about and trying to do torch.load(‘latest.pth’) and then trying to load state dictionary but got an error saying attribute error: ‘dict ’ object has no attribute ‘load state dict’

[Dicko87]

Here's a jump to their colab with their prediction process:
https://colab.research.google.com/github/facebookresearch/detr/blob/colab/notebooks/detr_demo.ipynb#scrollTo=-CVJRl28-_wS

Note that there's some gotchas if you want to use the gpu for predictions (much faster but you have to move things back and forth) and I'll add that to a prediction colab soon.

Amazing, I’ll take a look! 😃😃😃😃

[Dicko87]

Well I managed to load an image and tried to run predictions and the bounding boxes that were predicted arent around any object at all, the class names aren't visible either. Was wondering if its a problem with how my data is being read into the model in the first place, is there a way to check and visualize the input data to amke sure the bounding boxes in the ground truth data are being read in properly? Or I've messed something up with trying to make predictions :s

I even ran a prediction on one of the training images and ... yeah it did rubbish haha, so something isn't right.

[lessw2020]

Hi @Dicko87 - might be easiest to start with the training image that didn't work and look at the underlying annotation vs the predictions and start working backwards. Issues could be scaling is off (i.e. if image was resized with padding but not accounted for).
For training data review, you can do a call your dataset via .__getitem__ and grab an image and annotation, and plot those to make sure your training data is looking good.
I'm currently testing out Detectron2 since that support was added yesterday and that has a nice visualization tool there as well, though a lot more overhead.
Also you asked about augmentation - the default detr.py coco.py and also the custom_dataset.py I posted already have standard augmentations of hflip and rescaling etc and that is likely enough to start with.
But if you ran with no augmentation then that could also produce poor results though it sounds like right now there's more of a fundamental issue related to either annotation interpretations/scaling or similar esp if your class names aren't showing.
Might be worth posting your work as a colab if it's not work confidential and then we can try to help resolve (we = me and/or others here).

[lessw2020]

hi @eslambakr - glad to hear the fine tuning went well!
For 'scratch' training, it was proposed that 10k images might be the size needed so not sure if 2k will work that well.
That said, it likely can be done.
Transformers take a long time to train so probably need to think about getting to 200 or 300 epochs to make a better determination.
Hope that helps!

[Dicko87]

Hi there, I have started again to try and resolve some of the issues I am having with regards to predicted bounding boxes being wayyyyy off. I know something isn't right, before I didn't create a custom data class, I kept everything the same and created the following directories: datasets/coco and in coco I have three folders, annotations, train2017 and val2017. In the annotations folder I have my two json files, for training I have named it instances_train2017.json and for validation I have renamed it instances_val2017. In the train2017 folder I have my training images and in the val2017 folder I have my validation images. I have been trying to find a way to view the data that was going into the model so I can see if the ground truth bounding boxes are in the right location. I know I need to use getitems and have been trying but can't seem to figure it out just yet. I have also tried to go about using the custom_dataset.py file by @lessw2020 this time round but I got the following error:

This was resolved by changing the following line in the __init__.py file in the datasets folder:

Any ideas on how to insepct my input data would be great, I will keep plugging away at it in the mean time, thank you :)

https://github.com/lessw2020/training-detr/blob/master/custom_dataset.py

[Dicko87]

Just an update, I managed to have a peep at the training data .... I think and the ground truth bounding boxes look to be in the correct place.... not sure whats going on :s I do have another side question haha, I keep hearing abobut setting the number of queries ... what is it? and how do you set it? because I've just left it as default and I have 2 classes.

[Dicko87]

Hi @lessw2020 is there a way to private message you on here?

[lessw2020]

Hi @Dicko87 - would be great if github had a PM system (sorely needed imo) but none that I am aware of.
Are you on the pytorch forums or fastai deep learning forums? Both have a PM setup, and my id is same there on both as here...so that would work - https://discuss.pytorch.org or https://forums.fast.ai

[Dicko87]

@lessw2020 cool, I’ll head over to pytorch and pm you there :D ... hehe I’ve found you on pytorch.. just trying to find the pm bit haha .... found it !

[Dicko87]

Oh @lessw2020 it says I can’t send you a message!

[lessw2020]

@Dicko87 - oh no, that must be some anti-spam issue (i.e. a new account can't send PM's till X days later). Try forums.fast.ai and hopefully it's less travelled so may not have that.

[Dicko87]

@Dicko87 - oh no, that must be some anti-spam issue (i.e. a new account can't send PM's till X days later). Try forums.fast.ai and hopefully it's less travelled so may not have that.

@lessw2020 I think I’ve managed it on fastai, I hope it is private haha hope I have done it right !

[gfiameni]

Anyone working to train with the Visual Genome dataset (https://visualgenome.org/)?

[zlyin]

Hi there, after reading through the post, I came up with a dumb but important question. What's the format of gt_boxes we feed to the model? Is it Normalized coco format, [x_min, y_min, w, h] or Normalized yolo format, [x_center, y_center, w, h]? I'm puzzled because:

someone stated that:

I checked the bboxes I am supplying they are correct and in the x_center,y_center, h, w format.

while in the Kaggle kernel published, I notice that

DETR takes in data in coco format

@lessw2020 Less, since you've achieved good success with fine-tuning, could you help me verify that? Thank you very much!

[lessw2020]

hi @zlyin - I think it's easiest to answer if I trace the full path of the bounding box from coco annotations to final output :)
1 - Coco annotation format = coco format:
[x,y,width,height]
example:
"bbox": [253, 703, 226, 120]

before you pass in to detr, you have to convert it to [xmin, ymin, xmax, ymax]:
DETR code converts it here with this one line:
boxes[:, 2:] += boxes[:, :2])
which is to add the width to x1, the height to y1 to produce the new xmax, ymax

example becomes:
"bbox": [253, 703, 479, 823]

src: detr/coco.py line 64sh

When DETR then outputs the predictions, it's in cx,cy, width, height format..so to plot you can:
1 - change from cxcywh to the above xmin,ymin,xmax,ymax
minmaxboxes = box_cxcywh_to_xyxy(out_bbox)
2 - rescale to original image size

Hope that helps clarify!

[zlyin]

@lessw2020 Hi Less, great thanks for your help! I

I'm following the Kaggle kernel created & mentioned above by @tanulsingh, and I found the wrapped class outputs pred_bboxes in normalized coco format (probably because I feed the network normalized coco format as well). I don't know why they are not formulated in the Yolo format as you explained. I'll check it later.

And I adopted your experiment setting yesterday and finally got a run successfully. The settings I used are as follows. Again thank you for your contribution & share in the thread!

- lr = 1e-5
- eos_coeff = 0.01 (originally used 0.5)
- num_queries = 100 as the same as the pretrained resnet50 weights 
- No augmentations
- epochs = 60, reduce lr once at epoch=50

(@alcinos @fmassa, Earlier this week I created another post to report #148 that train_mAP & val_mAP decrease with losses. Based on your advice and Less's guidance, I can have an update for that thread here.)

The learning curves of my experiment looks like this:

• My training_bbox_loss oscillates drastically while valid_bbox_loss decreases steadily. This leads to the training_total_loss oscillating as well.
• Another issue is about the mAP. The good thing is that the mAP values are in the normal range. But the val_mAP indicates no enhancement.

Here are my 2 cents & probably all of you can provide advice.

• The cls_loss plot indicates the training needs to run more epochs so that overfitting can appear.
• Something needs to be done to take care of the pred_boxes. Currently, I set num_queries=100, but in my task, the number of wheat spikes (object) in an image varies from [0, 116]. Based on the above discussion, I think will stick to the number for fine-tuning.
• I can plot both gt_boxes in red & pred_boxes in blue from a validation step at the last training epoch in the following image. The interesting fact is that the model predicts num_queries=100 boxes each time, and the confidence_score of boxes are all above 0.9. Though the model is pretty confident about its prediction, the number of objects in the images if quite lower than that of pred_boxes. Should I reduce num_queries or apply something like NMS?
  
• From my observation, it seems that many redundant pred_boxes are proposed randomly when fine-tune starts. As the training process goes on, the pred_boxes seem to reduce and shift to approximate the gt_boxes. Does my observation reflect what the architecture does? For example, the following image shows what boxes look like at the beginning of fine-tuning.
  

Thank you very much for your help! I appreciate your kindness!

[zlyin]

I got some updates.

After reading the paper, I found the detr model consumes & output boxes in [xc, yc, w, h] / image_size format. So I changed that in my dataloader.
I also reset all the parameters to default values as same as that in the main.py. So my learning curve become like this:

• A good signal is that my mAP curves indicate that the model is really "learning" from the dataset.
• But the bbox_loss is still oscillating drastically. Still have no clue...

Help & advice are needed, @lessw2020 @fmassa @alcinos. Great thanks!!

[lessw2020]

Hi @zlyin - great to see the progress!

1 - Can I ask what system you are using to produce your metrics graphs? They look really good.

2 - You mentioned you were now passing in cx,cy format but did you then comment out/bypass the prepare_targets function as that is where xyxy is converted to cxcy? (detr.py)

Basically the flow in the repo is coco->xyxy->cxcy is auto done as part of target prep->loss compare with cxcy in detr proper.
So you could use cxcy directly but you would have to avoid that prepare_target call and wondering how you bypassed that for my own info w/o modding detr directly?

3 - Re: ideas for your current training - I'm planning to modify the current detr loss a bit and test out distance loss and complete loss which in other detectors have shown better results than giou. wondering if that might be a good next step in your case here?

[zlyin]

Hi @lessw2020, thanks!

1 - Can I ask what system you are using to produce your metrics graphs? They look really good.

• Sure thing. I just used the tensorboard carried by PyTorch. You can simply import it by from torch.utils.tensorboard import SummaryWriter. The rest usage should be same as the original tensorboard package.

You mentioned you were now passing in cx,cy format but did you then comment out/bypass the prepare_targets function as that is where xyxy is converted to cxcy? (detr.py)

• No, I never did that. Actually I can't find the prepare_target() function in the file mydrive/DETR/detr/models/detr.py. Instead of finding box_xyxy_to_cxcywh in that file, I can find 2 appearance of box_cxcywh_to_xyxy(src_boxes) and box_cxcywh_to_xyxy(target_boxes) in the method of loss_boxes under SetCriterion() class.
• But I did find the prepare_targets() function in the file of DETR/detr/d2/detr/detr.py. However, It seems that my model does not call any functions from that folder at all. The way I followed the Kaggle pipeline to initiate a model is like the following. I don't think it's calling some function from d2/detr/detr.py. Did you initiate a model like this?

 class DETRModel(nn.Module):
    """
    args:
    - num_classes = coco has 91 classes + 1 bg class; 
    - num_queries = default 100, which means output 100 bboxes of each image
    """
    def __init__(self, num_classes=1, num_queries=100):
        super(DETRModel, self).__init__()
        #self.num_classes = num_classes
        self.num_classes = num_classes + 1
        self.num_queries = num_queries

        # load pretrained MODEL
        self.model = torch.hub.load('facebookresearch/detr', 'detr_resnet50', pretrained=True)
        self.in_features = self.model.class_embed.in_features

        # switch a new head
        self.model.class_embed = nn.Linear(in_features=self.in_features, out_features=self.num_classes)
        self.model.num_queries = self.num_queries
        assert self.model.class_embed.out_features == num_classes + 1, "num_classes mismatch" 
        pass

    def forward(self,images):
        return self.model(images)

3 - Re: ideas for your current training - I'm planning to modify the current detr loss a bit and test out distance loss and complete loss which in other detectors have shown better results than giou. wondering if that might be a good next step in your case here?

• From my experiment, my giou loss is steadily decreasing. So I think my first priority here is still to suppress the oscillation of bbox_loss error. I need to figure out this issue at first. But I'll keep an eye on your idea once I get it run.

Thank you!

[Chris-hughes10]

Hi @lessw2020 and @zlyin,

I have also been looking at training on a custom dataset, just a couple of observations on the discussion.

In terms of the bounding boxes, I have been inputting them in the format cxcywh (normalized) straight into the pretrained torchvision model. From what I can see, prepare_targets is only used with detectron2, so if you are replacing the coco dataset with a custom one, this is bypassed.

Also @zlyin, just a note on on your model definition:

```
class DETRModel(nn.Module):
"""
args:
- num_classes = coco has 91 classes + 1 bg class;
- num_queries = default 100, which means output 100 bboxes of each image
"""
def __init__(self, num_classes=1, num_queries=100):
super(DETRModel, self).__init__()
#self.num_classes = num_classes
self.num_classes = num_classes + 1
self.num_queries = num_queries

    # load pretrained MODEL
    self.model = torch.hub.load('facebookresearch/detr', 'detr_resnet50', pretrained=True)
    self.in_features = self.model.class_embed.in_features

    # switch a new head
    self.model.class_embed = nn.Linear(in_features=self.in_features, out_features=self.num_classes)
    self.model.num_queries = self.num_queries
    ## Need this -> self.model.query_embed = nn.Embedding(self.num_queries, 256)
    assert self.model.class_embed.out_features == num_classes + 1, "num_classes mismatch" 
    pass

def forward(self,images):
    return self.model(images)

```

although you are passing in num_queries as an input argument, I don't think that is enough to change the output; for this I believe you would have to replace the Embedding layer query_embed.

3 - Re: ideas for your current training - I'm planning to modify the current detr loss a bit and test out distance loss and >complete loss which in other detectors have shown better results than giou. wondering if that might be a good next step in
your case here?

This sounds really interesting! I would definitely be interested in hearing the results.

[zlyin]

@Chris-hughes10 Hi Chris, thanks for your reply.

I have been inputting them in the format cxcywh (normalized) straight into the pretrained torchvision model. From what I can see, prepare_targets is only used with detectron2, so if you are replacing the coco dataset with a custom one, this is bypassed.

Yes, I think I'm doing the same thing here, and therefore the input box format shouldn't be a problem.

## Need this -> self.model.query_embed = nn.Embedding(self.num_queries, 256)
I add this line to the model and found the no difference, the loss_bbox still oscillates drastically... Pretty annoying issue. If you look into it, the "loss_bbox" bounced up & down alternatively, do you have some experience about such circumstances? I'm new to object detection, so I have no idea at all...

Any input are welcomed. Thanks!

[lessw2020]

for those interested (cc @Chris-hughes10 and @zlyin) , I trained with both giou (detr default) and then complete iou loss on new, small dataset for equal training time today.
Complete iou was better in every aspect, and planning to standardize on it for our DETR use.
Definitely recommend you try on your datasets and see if it helps your results.

[Chris-hughes10]

Hi @lessw2020, thanks for the update, I will definitely try it out! Do you have a gist or anything you can share with your implementation?

[lessw2020]

@zlyin - I forgot to say thanks a bunch for letting me know about tensorboard. Last time I looked at it all their charts were an ugly orange, but I'll hook it up this week as it looks much improved.

Re: complete iou and impl - let me check at our team meeting today if we can release that code (think so but have to confirm).
I think it would be a great PR frankly :) b/c I don't see any downside to using it and I had good results, and we could easily make it a switchable choice for training.
To clarify the differences - ciou includes aspect ratio, and it includes/ builds on diou which includes center distance.

So you get those additional training signals (aspect and center distance), plus original giou for context.

[lessw2020]

ok can post our ciou and diou code shortly. Let me hook it up as a switchable param so it's easy to control. will just post a new box_utils.py and modified detr.py for now.

[zlyin]

@lessw2020 No worries! Actually they add more but limited colors. You can try to have a look.

I have to give up DETR my to competition since it does not work. But I do look forward to learn from your complete IoU scripts. Thank you!

[Chris-hughes10]

@lessw2020 Excellent! By the way, if you want to play around with tensorboard with relatively little effort, have you checked out PyTorch Lightning (https://pytorch-lightning.readthedocs.io/en/latest/)? It enabled me to simplify the code pretty drastically.
https://gist.github.com/Chris-hughes10/29589ff65a281931043c886b2c5d1ed1

[lessw2020]

@Chris-hughes10 - thanks for sharing the lightning impl. I have to say that lightning layout looks super elegant and clean. I may put it on a todo list, to run a test with it next week after our next internal demo. (my only comment is you aren't breaking out the lr for backbone vs transformer?)

We're training more datasets and getting a lot better at DETR training. CIou helps a bunch, and we've added in some augmentations that are really improving the training curves (beyond what is in the base DETR transforms) ala mAP of 99% in some cases of 1 class datasets.

I'll try and post the ciou code tomorrow - I think we'll just post the code and show to modify detr.py as under time pressure for next demo. Josh wrote it from the paper, so either he or I will post it out.
On other news I tested out AdaHessian 2nd order optimizer with a classifier last weekend - that is like training with an absolute rocket...per the paper it does well for both CNN and transformers so I think it might be perfect for DETR. The giant catch - it doubles your memory requirements and no way I can fit DETR x2 on a regular gpu (need 22GB roughly).

[lessw2020]

just wanted to share one of our beautiful DETR custom training curves. this is single class, 60 epochs of fine tuning. 1.4K images.

[tazu786]

@lessw2020 they look amazing! Looking forward to testing it with my dataset as well! You mentioned you applied extra data augmentations. What did you use?

[lessw2020]

hi all,
Here's the DIou and CIou loss functions.
https://github.com/PlasmaDuck/detr-ciou
It's a one line change to implment, and @plasmaduck has instructions right in the code file.
Just backup your box_ops.py and replace with the one in his repo.
( cc @Chris-hughes10, @zlyin )

We've seen a nice improvement in training results and plan to perm use CIou for all DETR work going forward.
Enjoy!

[alcinos]

Thank you @lessw2020 for sharing your results, it is indeed a very nice mAP you're getting there :)

As for the CIou/DIou, interesting that you got some improvements on your dataset, on Coco it didn't seem to help when we tried it out.
As far as I can tell, you're only using it in the loss, correct? A possible thing to try would be to use it in the matcher as well (instead of the GIOU that is used currently). We have good theoretical reasons to believe it's better for the matching cost and the loss to be as close as possible (even if they are currently slightly different in DETR for stability reasons). However in DL world theory and practice sometimes have different opinions, so I can't guarantee it'll help in your case :)

[lessw2020]

Hi @alcinos - thanks for the feedback and insight :)
Currently we just use CIoU in the loss as you suspected.
We did see a nice jump in mAP from that, but that makes perfect sense to keep the loss in the box_loss and matcher the same. Will add that in and test out on our datasets next week (have a demo early next week so locking down things on changes for now).
Thanks very much for the advice here, always appreciated!
I will post an update once we retrain with DIoU in both loss and matcher, and look forward to trying it out - thanks again for the insight and advice.

[lessw2020]

Hi @tazu786 - we added colorjitter and also a synthetic shadow maker.
For our specific use (malaria and covid-19 rapid diagnostics) shadows are a big problem for detection, so we made our own synthetic shadow generator.
We also have rotation transform in the works but it's still got a few bugs left but should be training with that as well next week.
I think we thus capture the biggest detractors from good detection - scale (DETR already has that nicely setup with the random scaling), lighting (shadows, colorjitter) and orientation (rotation).
RandomErasing is one other that we plan to try as I had good luck with that in the past on a classifier project so will try it out with DETR soon.
Hope that info helps!

[zlyin]

@lessw2020 Thanks for your new loss! I'm back to the fight with DETR for my competition again...Let me explore and update if I can get it run normally this time. Will update for sure.

[zlyin]

@lessw2020 Hi Less, I made some progress to fix the oscillating loss_bbox, it was caused by the box format messed up by my augmentations...I fixed it and the loss begins to converge well.

• As for earlier discussion between you and @alcinos about CIoU loss, I ran 2 experiments earlier today and found the loss curves of GIoU looks better than that of CIoU. I believe that's related to the discrepancy of GIoU loss of matcher and CIoU of criterion. Hope it helps.
• Also, what you posted above is a beautiful training process! Awesome work! I got my latest learning curves. This experiment still uses default GIoU for both matcher & criterion. I employed some learning rate decay scheduler and a relatively larger initial learning rate. Will definitely try out your CIoU later.
  

Currently, I'm suffering from the problem that mAP of my model can't reach 0.45, which can be shown in the following image. During my training & validation process, I calculate mAP@range(0.5, 0.76, 0.05), and curves at various iou threshold values look exactly the same. I have expected curves at higher thresholds present lower scores, e.g. [email protected] should be lower than [email protected]. This leads me to dig out the pred_boxes in the model output and find some wired issues.

• I set num_queries=100 & fine-tune from a ResNet50 weights. With training going on, the pred_probs of detections increase to above 0.80 quickly. This results in many overlapped boxes & False Positive predictions (Backgroud class). A typical example is like this:

So I'm wondering how you @lessw2020 post-process the pred_boxes to remove those duplicated & overlapped False Positive boxes? Also, which retained weights are you using?

@alcinos, any insights about such circumstances? Maybe I I need to train more epochs or my num_queries=100 is way too high? But my understanding is that num_queries constraints the max_num_of_predictions per image.

I think I'm just one bug away form a good model. Please help!
Thank you very much!

[AJ-RR]

Hi,
How exactly can I enable/disable data augmentation?
Thank you in advance!

[pritamrao746]

@zlyin I have a doubt regarding this global wheat detection kaggle problem with respect to DETR ! Can you provide me your email or any contact . I also have the same doubts the loss is oscillating very much . Thanks in advance !

[eslambakr]

Hello All,
I am running DETR on my custom dataset and using pre-trained weights however, I got this weird behavior.
Should I wait for more epochs to make the model able to converge?
I think there is no use of waiting more time, as the validation loss is increasing, so is there any suggestion?

I tried another dataset, which contains similar classes like COCO dataset and it works fine, but this one is totally different than COCO, I think that's why the model fails to converge.
Should I unfreeze the batch norm in this case?

---

@zlyin I am facing the same problem, please tell us if u could solve it
Thanks in advance

[lessw2020]

Hi @zlyin - great to see your progress with DETR! Have you added in things like color jitter to help with the training? I saw a nice improvement from that and also recommend you try batch size 4 (if you have gpu mem for it) as that seemed to improve our results as well.
Re: nms - we actually have not had to do any post processing on our results in most cases b/c DETR already understands the context. For our use we only have 1 item of each class per image so it's a bit different than a free-form variable instances per image.

I am going to try to update ciou today to integrate with the matcher and will share that out once done...but it's not as easy as I thought to do that b/c of the mismatch in len between target and predictions in the matcher.

[lessw2020]

@alcinos - is it possible to get your impl of ciou regarding integration with the matcher? I'm finding it tricky b/c of the len mismatch between target and predictions in the matcher (currently the basic loss always has equal length target ala [N,4] and [N,4] but in matcher we have [N,4] and [M,4].
I think I see what to do and will try today, but as backup would be great if that code is available for external use.
We still see consistent gains across multiple datasets with CIou and definitely want to integrate with the matcher for the potential improvement.

[zlyin]

Hi,
How exactly can I enable/disable data augmentation?
Thank you in advance!

Hi there, I write a dataset class & apply augmentations before fetching an image and send to a model.

[zlyin]

@zlyin I have a doubt regarding this global wheat detection kaggle problem with respect to DETR ! Can you provide me your email or any contact . I also have the same doubts the loss is oscillating very much . Thanks in advance !

Hi @pritamrao746, yes, you can reach my via [email protected]

[zlyin]

I tried another dataset, which contains similar classes like COCO dataset and it works fine, but this one is totally different than COCO, I think that's why the model fails to converge.
Should I unfreeze the batch norm in this case?

Hi @eslambakr, I would suggest just kill it. The model doesn't learn at all, indicated by your learning curves. I was stumbled by many issues and I don't really remember if I came across exactly the same issue before. But I would suggest checking the bounding box format at first. Just as you said, the dataset of this learning curve is quite different from COCO. Also, try to not introduce any data augmentations since those applications may mess up your bbox as well...

[zlyin]

Have you added in things like color jitter to help with the training?

Hi @lessw2020, haha thanks! Without you & @fmassa @alcinos, I can't solve the problem actually. Though some progress, I still have a lot of issues to fix, especially the prediction. Hope to learn with you guys continuously.

Have you added in things like color jitter to help with the training?

• I added something like the follow. Are these "color jitter" you're referring to?

A.HueSaturationValue(hue_shift_limit=0.2, sat_shift_limit= 0.2, val_shift_limit=0.2, p=0.9),
A.RandomBrightnessContrast(brightness_limit=0.2, contrast_limit=0.2, p=0.9)

I saw a nice improvement from that and also recommend you try batch size 4

• I found DETR model is pretty lightweight than EfficientDet, which limited me using batch_size=1 for 1024 image, and I've already used batch_size=4 + gradient accumulation. In that way, my batch_size becomes 16 in fact. Maybe it's helpful, but I haven't done an ablation experiment to verify it.

Re: nms - we actually have not had to do any post processing on our results in most cases b/c DETR already understands the context. For our use we only have 1 item of each class per image

OK, now I can understand the difference between your dataset & mine. What's the dimension of your object in the image? Maybe very large? In my case objects are relatively small & distributed densely. But I don't understand why there are so many False Positives. @alcinos @fmassa Any insights?

I am going to try to update ciou today to integrate with the matcher and will share that out once done...but it's not as easy as I thought to do that b/c of the mismatch in len between target and predictions in the matcher.

Thanks for your work! Actually I found the mismatch issue while applying CIoU to the matcher as well and tried to fix it. But my implementation leads to oscillation & divergence.

• To save you some time & provided some initial ideas, I drop my naive modifications here, hope it can help. box_ops_for_Less.txt
• Also I found an issue in you DIoU function, return center_distance2 / corner_distance2 + giou. I think it should be iou - return center_distance2 / corner_distance2. What do you think?

Thanks!

[AJ-RR]

Hi,
How exactly can I enable/disable data augmentation?
Thank you in advance!

Hi there, I write a dataset class & apply augmentations before fetching an image and send to a model.

Hi,
How exactly can I enable/disable data augmentation?
Thank you in advance!

Hi,
Then, augmentation is not included (during training) in the repository code?

[lessw2020]

@zlyin - thanks for the initial drop re: matcher ciou - that's a big help! (we are both doing nearly the same thing, so that means we are likely correct). I'll test and also check re: the possible bug you mentioned and update you in a couple hours.

re: color jitter - I am using the torchvision transform color jitter specifically. I don't like using the Albumentations stuff b/c you have to convert to numpy, swap the channel dimension etc. so we've been either using torchvision and also writing a lot of our own augmentations.

Anyway in our custom detr class:
T.ColorJitter(brightness=.4, contrast=.3, saturation=.3, hue=.1),
where T is:
import datasets.transforms as T

Now to use these you have to make a cover function - you can do that two ways but the quick way is just make it in datasets.transforms.py:
`class ColorJitter(object):
def __init__(self, args, *kwargs):
self.jitter = T.ColorJitter(args, *kwargs)

def __call__(self, img, target):
    return self.jitter(img), target`

*why does github do so poorly at showing code?? anyway you get the idea - you have to deal with the fact that you only want to pass in the img while keeping the target info along for the ride.
A better way is what Plasmaduck did and make a TargetlessTransform class and then you don't need to make a new class each time...but maybe that's something I can show this weekend.

[lessw2020]

Hi @AJ-RR - there is basic horizontal flip and a really nice scaling handled in the repository code.
You can copy the coco.py in the /detr/datasets directory and modify it to make your own class.
If you look at make_coco_transforms in there you'll see the default transforms:

if image_set == 'train': return T.Compose([ T.RandomHorizontalFlip(), T.RandomSelect( T.RandomResize(scales, max_size=1333), T.Compose([ T.RandomResize([400, 500, 600]), T.RandomSizeCrop(384, 600), T.RandomResize(scales, max_size=1333), ]) ), normalize, ])
Hope that helps!

[zlyin]

Hi @lessw2020, glad to see it help a little bit. Definitely look forward to your answers but I have to go to sleep...played with the model until 3:30 am last night...

• I think my augmentation mentioned above does the same job as you ColorJitter class. Both of them are adding randomness to HSV space. I found albumentaion is convenient to use since it takes care of box automatically with the affine changes to the image & objects. Such as flip, rotation etc. But I haven't got time to explore the detailed parameters.
• As for the CIoU & DIoU loss functions. If assuming we're doing something correct, I used 100 images to test 2 setting-ups. Exp2 seems to look better. But I need to run the test during the evening. You can find my comments in the script. box_ops_for_Less.txt

Looking forward to your updates & inputs.
Thank you!

[lessw2020]

@zlyin - thanks for the update. Yes I looked closer and your albumentations for brightness/contrast and hue/saturation is effectively identical to my posted colorjitter.
Thanks for pointing out the issue on distance box and testing. I will test shortly and confirm...unfortunately my server ran out of hdd space and having to rebuild new one now but should be back and running shortly.

[Dicko87]

Hey guys, I seem to have a good mAP on my dataset but when I try to plot it’s predictions on a training image, the bounding boxes are obviously in the wrong places which suggests there is a scaling problem somewhere that I am not finding. My json bounding boxes are already in coco format, has anybody faced a similar issue and managed to resolve this? Thanks 😊

[lessw2020]

@zylin - first, thanks very much for finding the bug in our distance iou. I've reviewed and you are correct regarding iou should be used and not giou. I've added a clamp as well after reviewing the inventors paper as well.
Still checking complete_iou and will post updates after that, but did want to say thanks a ton for finding this issue!
(note in first runs with the improved 'matcher' complete loss, iou oscillations were much improved, new high on map). Now to test with matcher integrated.

[lessw2020]

@zylin - looks great and working nicely inside of the matcher. I'm attaching latest version - main add was putting a torch clamp in and bit of tidying, and thanks a ton for doing this.
box_ops_final_zylin.txt

We got the best bbox_loss to date using this on the dataset I ran it on today, so it's clearly a nice improvement!

[lessw2020]

@zylin - I forgot to answer but to your question, our datasets are definitely different in that we only have large and medium items to detect...very rarely any 'small' detections b/c the photos are effectively zoomed images. At least from the paper comments, DETR outperforms on larger items and not as good on smaller.

[zlyin]

Hi, @lessw2020 thanks for your feedback. Glad to contribute. Will try it ou my dataset. I'm currently out of computation resource...If it works, can we have @alcinos add our script to this repository? It's my first time to contribute something to a DL model, haha, very happy to see it if possible.

[woctezuma]

In case someone is interested, I have created a tiny notebook where I fine-tuned DETR on the super tiny balloon dataset. I have not yet figured out how to create all the nice plots which are posted in this thread, but I have for sure used all kinds of information pointed out here (and I have tried my best to acknowledge it).

[lessw2020]

hi @woctezuma - thanks for posting out your notebook!
Re: "how to create the nice plots" - I posted a small notebook for you here:
https://github.com/lessw2020/Thunder-Detr
Look at the View_your_training_results.jpynb in the home dir.

That uses the built in detr plot utils.

[AlanZTY]

I am finetuning on single class dataset. Do you think the weight of cross central loss needs to increase?

[guysoft]

@ lessw2020 any chance you can share the snippet of the rotation code assuming its coco?

[pranav-ust]

I am also facing with overlapping boxes problems. Does anyone know how to troubleshoot that? cc @zlyin

[zlyin]

I am also facing with overlapping boxes problems. Does anyone know how to troubleshoot that? cc @zlyin

Hi @pranav-ust, you need to apply some NMS-like tricks to kill the duplicated boxes. My issue actually comes from many "small objects". If t that's the case for your data as well, I suggest try other models actually...

[pranav-ust]

Hi @zlyin, what do you mean by other models? Do you mean architectures other than DETR?

My overall loss was converging well, so I thought that overlapping issue would disappear soon, sigh.

[zlyin]

Hi @pranav-ust, I mean if you target data are relatively small objects, DETR can't perform well, based on its theoretical limitations. Even if you solved the overlapping boxes via NMS-like tricks, it's not guaranteed to perform well on new test data... My loss converged pretty well, but I my predictions on new test data gave me pretty lower scores.

[1chimaruGin]

I confused here!

I wonder why 91?
Should we need to add no object class for every ten classes?

[woctezuma]

I wonder why 91?
Should we need to add no object class for every ten classes?

Basically, the 2017 version of COCO has 80 classes, but they are not consecutively numbered, due to historical reasons: they are a subset of the paper version of COCO which had 91 classes.

So, when working with the 2017 version of COCO, the highest class ID is 90. DETR expects the parameter num_classes to be the highest class ID plus one (cf. https://github.com/facebookresearch/detr/issues/108#issuecomment-650269223), hence 91. This new ID will be reserved for the "no_object" class.

For illustration purpose, the first class has ID n°1 and refers to a person.

The last class has ID n°90 and refers to a toothbrush.

Reference: https://tech.amikelive.com/node-718/what-object-categories-labels-are-in-coco-dataset/

[woctezuma]

On a side-note:

1. if you have N classes,
2. if you started indexing your classes at 0 instead of 1, i.e. the first class has ID n°0,
3. if you ensured that you had no "hole" in your indexing, so that the last class has ID n°N-1,

then you could feed DETR with num_classes equal to N, and the no_object class would be assigned ID n°N.

I have not tested it yet, so I don't know yet if classes have to be indexed starting at 1, as it was the case with the COCO dataset. I suspect it would not be an issue to start indexing classes at 0 with DETR.

[1chimaruGin]

@woctezuma

Thank you for the clarification

"categories": [ { "id": 0, "name": "bg", "supercategory": "none" }, { "id": 1, "name": "soccer-ball", "supercategory": "ball" }, { "id": 2, "name": "basket-ball", "supercategory": "ball" } ]

For this annotations, I first set the num_classes = 2.
It leads to worst result.

Then, I set num_classes = 3
It give better training result.

[woctezuma]

For your example, you should set num_classes to 3 anyway, because your highest class ID is n°2. DETR will assign its own no_object class to the ID equal to what you feed it as num_classes. You cannot force DETR to assign this class to the ID n°0, as you tried to do in your example.

I think the parameter name num_classes is misleading, and you should always stick to the definition in https://github.com/facebookresearch/detr/issues/108#issuecomment-650269223 if in doubt.

[lessw2020]

Just to add to @woctezuma comment, you can absolutely start your class id at 0 and arguably should for custom training with detr. For our internal framework we have a json util that does exactly that (reindex to 0, along with many other automated checks) before we start training with detr.
Also num classes is misleading in that it needs to be equal to your max class id which could be higher if you have gaps in your indexing. But automated reindexing avoids that concern as well.

[EhsanAlahi]

Hi @lessw2020, I am trying to fine-tune detr on my custom data but the gist link you have provided above for fine-tuning is not opening (page not found). I have my dataset prepared and updated my num-classes issue by following your jupyter notebook. Just guide me with final command to start my training with pretrained weights.

[EhsanAlahi]

Hi @MHI4 - I can make a colab this weekend if no one beats me to it.

1 - Do you have a custom dataset I can use to test on and private email for testing it so we don't clog up this thread during dev/testing? My work datasets are private so I can't use those, or we can also pick a general smaller detection dataset for example.

2 - Did you already see @mlk1337 gist link to gist as that gives you the key steps needed to fine tune, though maybe a bit more knowledge to setup vs colab with pluggable params for a dataset.

I am asking about the link you mentioned here.

[woctezuma]

It looks like @mlk1337 disappeared from Github.

I knew Internet documentation could disappear without notice, but damn, that is unfortunate.

[woctezuma]

Pinging @m-klasen, would you have a copy of the gist lying around?

[m-klasen]

https://gist.github.com/m-klasen/651297e28199b4bb7907fc413c49f58f this should be working

[woctezuma]

Thanks. Pinging @EhsanAlahi, the Gist is above.

[EhsanAlahi]

Thanks. Pinging @EhsanAlahi, the Gist is above.

Thanks @woctezuma , @m-klasen

[lessw2020]

Given all the confusion over num_classes here, and also my experience when I first worked with DETR of blowing up due to the fact that our labeling service was exporting category_id's as randomized large ints ...last night I wrote and put two utilities out for those interested:
1 - coco_compress = takes an input coco json file, compresses all category_ids to be contiguous, and re-bases to zero based index. Saves out to new file by default.
2 - show_catids = takes input json file, shows all category_ids and provides "num_classes" recommendation.

Here's the usage:

and located at: https://github.com/lessw2020/Thunder-Detr/tree/master/td_utils

[woctezuma]

I am trying to find out which parameters should one try to tweak first, when trying to improve the fine-tuning results.

1) The first thing which comes to mind is the number of epochs.

2) I see people mentioned the eos_coef above. Would you say it is the first parameter to try to tweak a bit (rather than, say, the learning rate)?

3) There are other coefficients for the matching cost and the loss. These would be next?

    # Loss
    parser.add_argument('--no_aux_loss', dest='aux_loss', action='store_false',
                        help="Disables auxiliary decoding losses (loss at each layer)")
    # * Matcher
    parser.add_argument('--set_cost_class', default=1, type=float,
                        help="Class coefficient in the matching cost")
    parser.add_argument('--set_cost_bbox', default=5, type=float,
                        help="L1 box coefficient in the matching cost")
    parser.add_argument('--set_cost_giou', default=2, type=float,
                        help="giou box coefficient in the matching cost")
    # * Loss coefficients
    parser.add_argument('--mask_loss_coef', default=1, type=float)
    parser.add_argument('--dice_loss_coef', default=1, type=float)
    parser.add_argument('--bbox_loss_coef', default=5, type=float)
    parser.add_argument('--giou_loss_coef', default=2, type=float)
    parser.add_argument('--eos_coef', default=0.1, type=float,
                        help="Relative classification weight of the no-object class")

[lessw2020]

Hi @woctezuma - I would definitely recommend extending your epochs first, and also look at where you have lr drops.
If your curves are going flat for a while, then dropping the lr is the way to go imo and then continue training.
(i.e. dont' just blindly add X epochs and hope it will get better, take a look at the curves...you want to have some oscillation with plateau indicating its hovering around a minima, and then that's usually the right time to drop the lr. )

I tested with the eos_coef and didn't see much change and just reverted to default (I think I did +/- 10x each way on that to test). That may not hold for your dataset though, so it's worth testing.

I think the other biggest changes you might test:
a - batch size - I get consistently better results with bs 4 instead of default 2
b - move to complete_box_loss and not just use giou. (interestingly DETR team indicated they did not see much improvement on COCO vs we saw a very healthy improvement on real world datasets).

[woctezuma]

Thank you, @lessw2020!

I had to find the acronym (CIoU) to find information about complete_box_loss. So, for others, it is about #38.

CIoU loss [takes into account] 3 geometric factors:

1. overlap area,
2. normalized central point distance,
3. aspect ratio.

[Dicko87]

Hi Guys, does anybody know how to decouple the mAP scores for multiple classes. At present I have two classes and a single mAP score. I would like separate mAP score for each class.
Also if anybody knows how to plot sensitivity curves I would be very interested.
Thanks

[EhsanAlahi]

Hi there, can anyone please check #224. I created the new issue but no one is responding there. Please.

[DanielFennhagen]

Hi, thanks for the excellent work!

I'm currently working with a custom dataset. It is a small dataset (~100 images), and to compensate there are some generated images to expand the dataset (5k images).
The dataset only contains two classes.
I tried fine tuning with your provided weights and modfied the head with num_classes = 2.

Training with 5K generated images

Continued training with 100 real images

Class error during the second stage of training

The mAP has outperformed our previous models using R-CNN with various backbones. However the class error is surprisingly high.

Any ideas of what to try to decrease the class error.

I have not tried modifying the num_queries at all, should I begin there?

[edit: image formatting]

[woctezuma]

Be careful that class_error is actually a number between 0 and 100. class_error is defined here. accuracy is defined here.

[hitesh11]

Does the current DETR codebase work correctly if an element in the input batch has no (zero) ground truth bounding boxes?

As I understand and experimented with the DETR code for 2D bounding box detection, it seems to work fine with empty ground truth boxes, and I have been able to train successfully getting decent performance. I just wanted to confirm with the authors and other users in case I am missing something. cc: @fmassa @alcinos

Details:
I am trying to run DETR with multiple camera images, and in some cases a camera (say left camera) input images in the batch do not have any bouding boxes in the ground truth. It is not straightforward to skip this image from a particular camera in the batch, as images from all cameras are passed together in a batch. Hence I planned to update the loss functions to handle this case. Gladly, I found that all relevant loss functions seem to already work in this case.

In the case of a batch having an element with empty ground truths (target):

1. Starting with the Hungarian matcher function, it returns empty ([]) indices for the batch element with empty ground truths.
   Ref: https://github.com/facebookresearch/detr/blob/master/models/matcher.py#L12

1. Classification loss {loss_labels() } considers all target labels as no_class and computes the desired classification loss in this case. Ref: https://github.com/facebookresearch/detr/blob/master/models/detr.py#L116

2. The batch element with no ground truth bboxes does not contribute to the bounding box loss (loss_boxes) as the target_boxes do not contain any boxes for the batch element, as empty box is removed in torch.cat operation.
   Ref: https://github.com/facebookresearch/detr/blob/master/models/detr.py#L151

3. losses["class_error"] is incorrectly computed as 100 in this case irrespective of predictions. But this loss is only used for logging purposes and not for gradient computation, so it does not affect training.
   Ref: https://github.com/facebookresearch/detr/blob/master/models/detr.py#L126

Please let me know if I am missing something, or if there can be scenarios which would fail in this case, and any other recommendations to handle this case. Please let me know if you need any further information.

Thanks again to the authors for the great contribution, and the community for all the help.

Regards,
Hitesh

[alcinos]

@hitesh11 as you pointed out, the code supports empty annotations for some image just fine, and in fact there are images like that in the Coco dataset.

However, from a modeling perspective it is not innocuous at all to provide empty annotations. Indeed, you are explicitly training DETR to predict the empty set for the given image. If, however, there are objects in this image (that happen to be non-annotated), this will be quite damaging to DETR's performance, probably more than if you'd do the same for a classification-based detector like Fast R-CNN or Yolo, because they optimize for a different objective.

In summary, yes DETR can handle image with no annotations, but I strongly advise against giving no annotations if the image contains objects amongst the categories your are training for (ie objects that should be annotated if the annotations was perfect).

[hitesh11]

@alcinos Thanks a lot for your prompt and detailed response.

Also thanks for your advice on the modeling perspective. I believe the other option is just to skip the image with no (zero) ground truth annotations? I know there are some annotation errors in our dataset, but it is difficult to automatically determine if there are missing annotations for a particular image. I guess the erroneous annotations would harm in the general case too, where there are some annotations (and not necessarily zero annotations). Any advice on how to handle that apart from correcting the annotations? Just as FYI, I saw that DETR seems to be robust to some of these, as it predicted the right object, while GT annotation was missing.

Another reason for asking the question was a strange observation -
Reduced DETR performance on reducing the number of GPUs used for training :

Keeping the batch_size=32 fixed, I train on following two configurations:

(a). 4 nodes containing 8 GPUs each (32 total GPUs) -> 1 image / GPU : Achieves ~90% AP on relevant class.
(b). 1 node containing 4 GPUs each (4 total GPUs) -> 8 images / GPU : Achieves ~80% AP on relevant class.

I am currently debugging the cause for the above discrepancy as I would like to get same performance with less GPUs.

I realized that world_size() / total GPUs are different in both cases [(a)=32 and (b)=4], which would change the normalized num_boxes and hence change the loss_boxes (loss_giou and loss_bbox). I am using the default hyper-parameters from the repository and both training and validating on the same configuration. Also, all images are of same size in my dataset, so padding does not seem to be the issue.

`
# Compute the average number of target boxes accross all nodes, for normalization purposes

    num_boxes = sum(len(t["labels"]) for t in targets)

    num_boxes = torch.as_tensor([num_boxes], dtype=torch.float, device=next(iter(outputs.values())).device)

    if is_dist_avail_and_initialized():

        torch.distributed.all_reduce(num_boxes)

    num_boxes = torch.clamp(num_boxes / get_world_size(), min=1).item()

.............

  losses["loss_bbox"] = loss_bbox.sum() / num_boxes

  losses["loss_giou"] = loss_giou.sum() / num_boxes

`
Ref: https://github.com/facebookresearch/detr/blob/master/models/detr.py#L227

To verify this is the issue, I am currently experimenting with changing other parameters proportionately (bbox_loss_coef, giou_loss_coef) to get similar losses with reduced world size. If this is the issue, then it should be just a matter of hyper-parameter tuning.

Do you think having different world size is the potential issue for the discrepancy, and can you share what changes you made/ hyper-parameters used to run with smaller/larger number of GPUs/nodes? I was concerned if I am missing something fundamental which could be the possible cause for the issue. Would appreciate your guidance on this.

Thanks again,
Hitesh

[pratikkorat26]

Very impressed with the all new innovative architecture in Detr!
Can you clarify recommendations for training on a custom dataset?
Should we build a model similar to demo and train, or better to use and fine tune a full coco pretrained model and adjust the linear layer to desired class count?
Thanks in advance for any input.

https://github.com/pratikkorat26/DETR_FineTuning

Hey i have build the repo so easily it can be trained on the custom dataset
Check it out and i also provided small toy dataset of orange and apple

By following the step you can easily train it on custom dataset

Try it out and let me know how'd it go
Thank you...😊

[SwHaraday]

Dear Sirs,

Thanks to your discussion, checkpoint file for my custom dataset was obtained.
To get more accuracy, I would like to add data augmentation during training.
From experience, I wish I could add "RGB to GREYSCALE" and "ROTATE 90" randomly.

Would you please advice how to add above 2 augmentations to datasets\transformers.py.

Thanks in advance.

[ayberksener]

@lessw2020 I'm using https://github.com/allegroai/trains/ to track training

Hi, can you provide the snippet of your code for visualization. Thank you.

[Dicko87]

Hi there has anybody managed to get color jitter working? if so could you please tell me how you went about adding this to coco.py and transforms.py please, thank you.

[lessw2020]

Hi @Dicko87!
Yes, have been using colorjitter for a long time with DETR and highly recommend it.
There's actually two ways to do it - I'll show the way that is more true to the DETR codebase here and show the more elegant way later.
Summary is you need to link it into the ./datasets/transforms.py as so...(not sure why github just stinks at displaying code so I'm posting code 'images' here):

this links in the torchvision colorjitter reference via a passthrough object...then you just use it in your custom dataset class (coco.py or a derivative of it) as so:

Hope that helps!
ps - here's the code above so it's easy to cut and paste...edit - finally figured out that is you use three ~ before and after, github will do the right thing for display...using the gui codeblock button doesn't do the right thing:
~~~python # in detr/datasets/transforms.py
class ColorJitter(object):
def __init__(self, args, *kwargs):
self.jitter = T.ColorJitter(args, *kwargs)

def __call__(self, img, target):
    return self.jitter(img), target

~~~

and second one:
~python# in custom dataset class:
T.ColorJitter(brightness=0.2, contrast=0.3, saturation=0.3, hue=0.1)
~

[Dicko87]

Hi there Less, thank you for the reply. Well after playing about with it last night, I finally got the model to run using this approach. I am not sure if it is working or not, what do you think? If there are problems then I'll try your method above.

So I used this in coco.py and transforms.py respectively.

[Dicko87]

Sorry another tiny question, I am trying to get consistent results and before I run my model I enter PYTHONSEED= 42. But I ran the model again and the results weren’t identical. Is there a way to get consistent results, something I am missing? Thanks folks

[lessw2020]

Hi @Dicko87 - the way you have it setup will work but your params are effectively hardcoded in the call...i.e. if you change it in coco.py to to brightness = .2...it's still going to be .4 in actuality.
So to adjust it in your impl means having to go change it by hand in transforms vs the way I set it up it will flow through from coco.py settings which I find more intuitive (that's also the way DETR does it).
Anyway, yours will work just bear in mind when you want to change it, you have to go modify the call params directly.

[lessw2020]

Re: seed - you have more rng than just python. Thus, you'll want to seed cuda as well:
~python # single gpu
torch.cuda.seed(seed)
# or if multi-gpu
torch.cuda.seed_all(seed)
# and regardless, try to use deterministic convolutions:
torch.backends.cudnn.deterministic = True
~
You may also need to seed numpy, though most likely the missing seeding is the torch stuff above.
Hope that helps

[Dicko87]

Thanks Less that’s great, I am using anaconda, so do I just type that stuff into the terminal? Like in anaconda before training the model would I type those commands in. Thanks .... is it manual_seed ? ... also which .py files do I need to put this in hehe... sorry

[lessw2020]

Hi @Dicko87 -
1 - actually yes manual_seed is better b/c it does both cpu and gpu, so I'm updating it below.
2 - I would just add a function like this and call in your train.py or similar file that's directing your training. This way you aren't having to manually put it into the cmd line. i.e. just put it at the top before you get into any model creation etc.:
~~~python

seed everything

def seed_all(seed):
# python seeding
random.seed(seed)

# torch seeding and deterministic setting.
# single gpu assumed...
torch.manual_seed(seed)
# 1.7...beta: torch.set_deterministic(True)
torch.backends.cudnn.deterministic = True 
torch.backends.cudnn.benchmark = True

# np seeding
np.random.seed(seed)

print(f"--> np,python,torch,cuda (single gpu) all seeded with {seed}")

~~~

[Dicko87]

Thats great, thanks very much Less, I'll try it tomorrow and see if I get the same results.

[ayberksener]

@lessw2020 I want to use DETR with different backbone, can you provide any starter code for this. Thank you

[Dicko87]

@lessw2020 hmm I tried the seed and got this error.

I then entered CUBLAS_WORKSPACE_CONFIG=:4096:8 into the terminal and got this message.

But then I got the same error message at the top again.

I managed to get around this by following:

But end up with this error:

So today I set my num_workers to 0 instead of 4 and made the seed function as follows but I still can't get the same results!

I added these to the model.py and detr.py files

[lessw2020]

Hi @Dicko87 - are you running with Cuda 11?
Two quick comments that may help:
1 - Your code above you have torch.backends.cudnn.benchmark = False...but that could introduce randomness. It should be set to True per the code I posted earlier. Basically it means cuda will test a couple operations for speed ,then finalize on it and always use that. Otherwise, it may dynamically adjust operations on the fly which will add variability. (the flip side is if your inputs vary a lot...then it may need to be False).
2 - The seed_all function should just be invoked once, ideally in your train.py or similar, but basically right at the start of setting up for training and before you start creating anything. I wouldn't put it in model or detr and definitely not 2x.
Also when you say you get different results..what does that mean? i.e. what metric are you using to quantify the difference...just wondering as that may help debug it more.

[lessw2020]

@lessw2020 I want to use DETR with different backbone, can you provide any starter code for this. Thank you

Hi @ayberksener - here's a link with some initial code, and if you are using mobilenet there's a full implementation in a link further down in that thread:
https://github.com/facebookresearch/detr/issues/154

[Dicko87]

Hi @lessw2020 hmmm I am just using the DETR folders... emmm can’t see a train.py just detr.py and main.py
Sorry, the results I am on about is the mAP, it’s different every time I run the model.