Detr: Strage behavior potententially bug.

[MaratZakirov]

I am trying to adopt this repository to OCR task and facing same dilemma

While training you have 3 different sizes of image encoded in dataset

1. Actual tensor size
2. filed 'size' - which means what?
3. field 'orig_size' - which I believe means original size of an image in the dataset

So if you try to print boxes of a dataset of an element for batch size bigger than 1 (I check it with 5)
You will get behaviour there for the same picture due to random batch sampling will have different box coordinates

Look below. This function will print you boxes on image correctly but only in case of batch_size=1 or if all pictures in your dataset are the same size or in case if you use for W, H scaling from target["size"] which is wrong.

# img = (3, H, W) tensor from batch with the samme H and W
# target - labels for this particular image
def showImageFromBatch(img, target):
    from PIL import Image, ImageDraw, ImageFont

    draw = ImageDraw.Draw(img)
    boxes = target['boxes']
    cl = target['labels']

    if 1:#boxes.max() <= 1:
        boxes = box_cxcywh_to_xyxy(boxes)

        print('Image:', (img.height, img.width), target['size'], target['orig_size'])


        H, W = target['size'] <<< Works well only with that
        W, H = img.width, img.height <<< But must works with this!!!

        boxes[:, 0::2] *= W
        boxes[:, 1::2] *= H

    for i in range(len(boxes)):
        x1, y1, x2, y2 = boxes[i]
        draw.rectangle((x1, y1, x2, y2), outline=(0, 255, 0) if cl[i] >= 0 else (0, 0, 0), width=3)
        draw.text((x1, y1), str(cl[i].item()), (0, 255, 0) if cl[i] >= 0 else (0, 0, 0),
                  font=ImageFont.truetype("DejaVuSansMono.ttf", 20))
    img.show()

Please clarify this situation. Thank you in advance.

[alcinos]

Hi @MaratZakirov
Let me try to clarify the situation with an example. Imagine your input consists of two images, img1 of size (200, 300) and img2 of size (300, 200)

• "orig_size" corresponds to the size of each image in the batch, before any pre-processing. In this situation, the tensor will contain [[200, 300], [300, 200]]. See for example how the postprocessor uses it to compute the boxes respective to the original size: https://github.com/facebookresearch/detr/blob/be9d447ea3208e91069510643f75dadb7e9d163d/models/detr.py#L279-L282

• "size" corresponds to the size of each image in the batch, after pre-processing but before batching. In our validation pre-processing for example, all images are resized so that their small side is 800. The tensor will thus contain [[800, 1200], [1200, 800]]. This field is mostly used in panoptic segmentation, if you're only interested in detection you can ignore it.

• The actual tensor size will be obtained after batching, which implies padding. In our example, the final size will be something like [2, 3, 1200, 1200]

DETR's predictions, are, by design, always made with respect to the original image size, disregarding potential padding. That's why one can't use the size of the batched tensor as H,W, and must rely instead on the "orig_size" field.

Hope this helps. I think I have answered your question, and as such I'm closing this, but feel free to reach out if anything remains unclear.

[MaratZakirov]

!!!! @alcinos code fragment you provide work ONLY in evaluation stage in training problem which I have been described remains.

Although it seems that we understood each other let me describe it more distinctly

Suppose we have batch_size=2 and 3 images in our dataset

1. with size (500, 500)
2. with size (1000, 500)
3. with size (500, 1000)

Lets consider that somehow we switched off preprocessing/augmentations (resizes, crops etc) so "orig_size" will be always equal to "size". But due to random batching we could obtain following batches:

Batch one
with image 1 orig_size=500,500 size=500, 500 as a part of tensor with size = 2, 3, 1000, 500
and image 2 orig_size=1000,500 size=1000, 500 as a part of tensor with size = 2, 3, 1000, 500

Batch two
with image 1 orig_size=500,500 size=500, 500 as a part of tensor with size = 2, 3, 500, 1000
and image 3 orig_size=500, 1000 size=500, 1000 as a part of tensor with size = 2, 3, 500, 1000

In these two cases boxes coordinates for image 1 will be ABSOLUTELY the same. So at training stage DETR will learn same Image 1 coordinates for DIFFERENT Image 1 tensors. This is do not seems to be normal. Is not it?

PS

Maybe DETR implicitly (as super duper strong CNN) clean off padding zeros in tensor and somehow account this fact and at the end refer to the same tensor features in both cases? ...
.

[MaratZakirov]

I have made some investigation and put print just before criterion call where all modification to target already have been done and it goes to HungrianMatcher and other cool stuff as it is

    for samples, targets in metric_logger.log_every(data_loader, print_freq, header):
        samples = samples.to(device)
        targets = [{k: v.to(device) for k, v in t.items()} for t in targets]

        <<<<<< I show image here see images bellow

        outputs = model(samples)
        loss_dict = criterion(outputs, targets)

This is what I get for batch size equal to 1

This is what I get for batch size equal to 2

I print bounding rectangles according Input tensor size, but please pay attention that despite tensor size are completely different in there two cases boxes coordinates absolutely the same at training stage!!!!

[fmassa]

Hi,

I believe the issue arises due to a bug in your visualization code.

Before pasting the boxes in the image, you need to crop the image so that it is of target['size'], in order to take into account the padding that was added by batching multiple images together.

Here is what I would add to your implementation:

H, W = target['size']
img = img[:, :H, :W]
boxes[:, 0::2] *= W
boxes[:, 1::2] *= H

Note that our predictions are made in 0-1, and we take extra care in our positional encoding to account for the regions which are only inside the image, see https://github.com/facebookresearch/detr/blob/be9d447ea3208e91069510643f75dadb7e9d163d/models/position_encoding.py#L32-L34
which is basically a batched mesh-grid, but that takes the original image size into account to avoid the issues you mentioned.

[MaratZakirov]

Hi @fmassa my question was not about visualization, visualization I made is needed just for illustration. And if you read my post carefully you would see following.

        H, W = target['size'] <<< Works well only with that
        W, H = img.width, img.height <<< But must works with this!!!

So I already know what you are trying tell me, but my question is about training, not visialization which is just an illustration, and you may forget about green rectangles look at white numbers at the left top of an image which I printed.

The point I tried to make clear as possible is that DETR is forced to learn _SAME_ boxes coordinates for _DIFFERENT_ tensor sizes (due to batch padding) which is as I believe is completely wrong. Look at two images above. They have different WxH but coupled with SAME coordinates which I printed (using white color).

So in one case (one batch) we asking DETR to output one set of coordinates (which are {0:1}) for phone for example.
And in batch two we asking DETR to output same set of coordinates as for previous case but for padded differently same image.

This scheme could be trained successfully only in case if DETR is smart enough to account paddings implicitly to pay attention to the same visual features in both batches (or you must provide 'size' filed explicitly as an NN input and hope for the best)

[MaratZakirov]

So for example two batches of size 2 both has same image at position of 0
So formally we have:
B1.shape = (2, 3, H1, W1)
B2.shape = (2, 3, H2, W2)
H_c = min(H1, H2)
W_c = min(W1, W2)
B1[0, :, 0:H_c, 0:W_c] == B2[0, :, 0:H_c, 0:W_c]
B1.targets[0][boxes] == B2.targets[0].boxes

Suppose we have important 'phone' feature at address _[0, :, i, j]_
So DETR according to it is asked to produce same coordinates _(a, b, h, w)_

[fmassa]

Hi again,

So, I think that the main thing that we need to take into account here is that a Transformer encoder is permutation-equivariant. This means that we can shuffle all the pixels in feature map, and the output of the encoder will be shuffled accordingly. In the same vein, the Transformer decoder is permutation-invariant wrt the feature maps that we feed, which means that the order in which we feed the input pixels doesn't matter.

With that in mind, the only way the transformer can predict relative coordinates is by feeding the positional encoding. I've described in my previous post that the positional encoding takes care for the objects inside the image. But I didn't describe what / how the mask is calculated, which I'm doing now.

If you look at https://github.com/facebookresearch/detr/blob/be9d447ea3208e91069510643f75dadb7e9d163d/util/misc.py#L283-L300, which is used in the collate_fn that we pass to the DataLoader, you'll see that images of different sizes are padded so that they have the same size. But we also keep another tensor (named mask) around, which hold as information which pixels belong to the image and which ones are just padded and should not be taken into account.

This mask is used at a few places:

• the positional encoding, in order to properly compute the coordinates inside the image without taking padding into account
• the transformer, which does not take the padded regions into account

So while the feature maps from the CNN will indeed be different for different image sizes, the transformer will only look at the regions which does not correspond to padding. And for predicting the 0-1 relative coordinates of the boxes, it will also only look at the features inside the image.

This can be a bit hard to explain, but please let us know if this isn't clear and we will try to explain it better.

[MaratZakirov]

@fmassa Thank you that is exactly I was asking for! So you provide information about real image via mask

[linzzzzzz]

Hi @MaratZakirov
Let me try to clarify the situation with an example. Imagine your input consists of two images, img1 of size (200, 300) and img2 of size (300, 200)

* "orig_size" corresponds to the size of each image in the batch, _before_ any pre-processing. In this situation, the tensor will contain [[200, 300], [300, 200]]. See for example how the postprocessor uses it to compute the boxes respective to the original size: https://github.com/facebookresearch/detr/blob/be9d447ea3208e91069510643f75dadb7e9d163d/models/detr.py#L279-L282

* "size" corresponds to the size of each image in the batch, _after_ pre-processing but _before_ batching. In our validation pre-processing for example, all images are resized so that their small side is 800. The tensor will thus contain [[800, 1200], [1200, 800]]. This field is mostly used in panoptic segmentation, if you're only interested in detection you can ignore it.

* The actual tensor size will be obtained after batching, which implies padding. In our example, the final size will be something like [2, 3, 1200, 1200]

DETR's predictions, are, by design, always made with respect to the original image size, disregarding potential padding. That's why one can't use the size of the batched tensor as H,W, and must rely instead on the "orig_size" field.

Hope this helps. I think I have answered your question, and as such I'm closing this, but feel free to reach out if anything remains unclear.

Hi @alcinos, I have a quick question on this last comment "DETR's predictions, are, by design, always made with respect to the original image size, disregarding potential padding".

I saw in transforms.py you are adjusting boxes and area of target when doing resizing. However, why would you need to adjust target if DETR's predictions always made with respect to the original image size?

https://github.com/facebookresearch/detr/blob/10a2c759454930813aeac7af5e779f835dcb75f5/datasets/transforms.py#L114-L123

[fmassa]

@MaratZakirov about the boxes:
This is needed because we perform the box normalization (to be between 0-1) as the last step of our transformation. Doing it this way is easier because handling crops and some other transformations becomes simpler. Because of that, and in order to keep the transformations correct, we need to scale the boxes as well, so if one day we decide to predict everything in absolute coordinates we just need to change one line, see https://github.com/facebookresearch/detr/blob/10a2c759454930813aeac7af5e779f835dcb75f5/datasets/transforms.py#L256

For the area, this is not actually used anymore in the codebase. We would use it to be able to separate between small-medium-large objects. In fact, we might not need to scale it even when it was needed, but for consistency, it was preferable to perform the data transformation over all fiels.

Let us know if you have further questions!

[jiangwei221]

Hi DETR team, I have the same question as @MaratZakirov mentioned before.
I understand that Transformer encoder is permutation-equivariant, and the Transformer decoder is permutation-invariant.
But actually the black border will change the values of feature maps through convolutions, if I understand correctly.

I did following runs, and printed the values after this line:
https://github.com/facebookresearch/detr/blob/d45a12c72fd9877f896aad566eee8478e49cac95/models/detr.py#L68

python main.py --batch_size 1 --no_aux_loss --eval --resume https://dl.fbaipublicfiles.com/detr/detr-r50-e632da11.pth --coco_path "d:/dataset/coco"

In [5]: outputs_coord[-1][0,0,:]
Out[5]: tensor([0.0791, 0.5397, 0.0183, 0.0526], device='cuda:0')

python main.py --batch_size 2 --no_aux_loss --eval --resume https://dl.fbaipublicfiles.com/detr/detr-r50-e632da11.pth --coco_path "d:/dataset/coco"

In [1]: outputs_coord[-1][0,0,:]
Out[1]: tensor([0.0787, 0.5397, 0.0188, 0.0531], device='cuda:0')

The only difference between these two runs is the batch size, and you can see the predicted values are not exactly the same.

Here is a paper I'd like to mention: https://arxiv.org/abs/2001.08248, But I'm not sure it's is related.

[fmassa]

@jiangwei221 correct, the padding will affect the convolution output, and thus potentially change the output of the model.

I would like to note thought that the same happens with other methods as well, such as Faster RCNN.

[jiangwei221]

I see, thanks for the clarification!