Detr: Embedding size is too small.

[yelantf]

Thanks for sharing this amazing work! I was trying to train a dilated model with your code, but simply running
python main.py --coco_path /xxx/ --position_embedding learned --dilation
will raise a CUDA error. It seems that the positional embedding size (50) is too small for dilation model. The upper bound of the feature map size for dilation model should be 1333/16=83.3125 instead of 50.

[fmassa]

Hi,

Thanks for opening the issue!

If you are using the learned position embedding, then yes, the current formulation will not fit for dilation.

You would need to change https://github.com/facebookresearch/detr/blob/be9d447ea3208e91069510643f75dadb7e9d163d/models/position_encoding.py#L57-L58 to be twice as large, as you noted.

But as discussed in https://github.com/facebookresearch/detr/issues/37, the learned positional embeddings in its current formulation might not be ideal, because some embeddings might never be trained, which could lead to unexpected results.

We could make the PositionEmbeddingLearned to be more generic, by accepting a max_num_embeddings, and perform the max_size // (32 // dilation) to get the number of embeddings, but given that our main model uses sine positional embedding, I'm not sure it's worth the added complexity.

Thoughts?

[yelantf]

Thanks for your clarification! I also agree that sine embedding is more robust than the current design of learned embedding. I think it is good to keep the code that way for now.

BTW, in the README.md, it is said that "DC5 models show a significant drop in AP if evaluated with more than 1 image per GPU", which I just noticed. What is the reason for this significant performance drop? Would you mind explaining it with more details? Many thanks in advance!

[fmassa]

@yelantingfeng it's just because we have trained the DC5 models with batch size of 1.

This means that the transformer model has not seen during training padded images (nor padded masks), and thus it has no way of learning to only predict boxes inside the mask in a robust manner.

I believe that if we add random padding (both at the image and at the mask) during training, it would make training with batch size 1 robust to be evaluated with batch size larger than 1, but we haven't tried it.

I believe I have answered your questions, and as such I'm closing this issue, but let us know if you have further questions.