Detr: Unable to understand positional encoding and masks.

[saahiluppal]

1. Can someone please explain me how you calculated the positional encoding?
   I know what positional encoding is, but models.positional_encoding.py is but overwhelming. I want to know what are considered as positional encoding while working with images. Are these calculated for feature maps or somewhat else?

2. How do you calculate masks when using images in transformers?
   I know what masks are, but how do we calculate these when dealing with images?

I found no answers to these questions anywhere so posting it here.

[fmassa]

Hi,

Question 1

I want to know what are considered as positional encoding while working with images.

Positional encoding takes a xy coordinate in [0, 1] and convert the xy into a vector of 256 elements. The encoding for x and y are the same, so for the sake of simplicity let's only look at the x part.
In https://github.com/facebookresearch/detr/blob/0fb754c4e592b4da87e6ec0010c767265809666e/models/position_encoding.py#L30-L31
we create an image tensor which is similar in spirit to meshgrid, but that supports images with different sizes (read masks) in each batch. This way, we have a grid of xy, which we normalize afterwards so that they are between 0 and 1 (in this case, we scale by 2 * pi as well, but that's a detail) https://github.com/facebookresearch/detr/blob/0fb754c4e592b4da87e6ec0010c767265809666e/models/position_encoding.py#L32-L35
Then, in https://github.com/facebookresearch/detr/blob/0fb754c4e592b4da87e6ec0010c767265809666e/models/position_encoding.py#L40-L43 we apply standard sine embedding in a vectorized fashion for x and y separately, and concatenate them afterwards for x and y, yielding the spatial positional embedding.
The positional embeddings only depend on the feature map shapes and the masks (as there could be padding between different images), and not on the content of the feature maps.

Question 2

How do you calculate masks when using images in transformers?

Those are calculated in https://github.com/facebookresearch/detr/blob/0fb754c4e592b4da87e6ec0010c767265809666e/util/misc.py#L299
Basically, everything that corresponds to zero padding the image so that they have the same size are filled with True for the mask.

I believe I have answered your questions, and as such I'm closing the issue, but let us know if you have further questions.

[saahiluppal]

Perfect explanation!