Detr: Generate attention decoder heatmap images for model feedback as shown in DETR paper?

[lessw2020]

❓ How to do something using DETR

Describe what you want to do, including:

1. what inputs you will provide, if any:
2. what outputs you are expecting:

Will code be added/released to generate the attention decoder heatmaps like in the paper? (i.e. the zebra and elephant images).
I've found heatmaps to be very useful for helping with training and understanding model performance, so hoping the code used in the paper will be added so we can generate these for our DETR models?

NOTE:

1. Only general answers are provided.
   If you want to ask about "why X did not work", please use the
   Unexpected behaviors issue template.

2. About how to implement new models / new dataloader / new training logic, etc., check documentation first.

3. We do not answer general machine learning / computer vision questions that are not specific to DETR, such as how a model works, how to improve your training/make it converge, or what algorithm/methods can be used to achieve X.

[szagoruyko]

we have a colab in preparation, should be ready within the next few days, stay tuned

[lessw2020]

@szagoruyko - colab sounds fantastic! Thanks very much and look forward to it.

[thomashossler]

any update on this? looking forward to try the colab!

[thomashossler]

for anyone interested, you can replicate the figure by registering a forward hook on the last multihead attention layer of the decoder.

out = {}
def get_output(name):
    def hook(model, input, output):
        out[name] = output[1].detach()
    return hook

model.transformer.decoder.layers[5].multihead_attn.register_forward_hook(get_output('last_decoder'))

after a forward pass :

act_decoder = out['last_decoder'].squeeze()

will get you the heatmap (need to resize and select the map corresponding to an object though).

[lessw2020]

@thomashossler - thanks for the informative post.
I've hooked and gotten the activations but I'm unclear re: the last step of "select the map corresponding to an object" and how to display.
Would you be able to post a full working example to just finish off the process?

[thomashossler]

@lessw2020 so you need to first reshape the attention map. If you are using the elephant image without any resizing, your attention map should have a shape of torch.Size([1, 100, 300]). When reshaping, you get a map of shape torch.Size([100, 1, 15, 20]). The first dimensions is the number of proposals.

Now when you forward pass the image, you should look at the labels predictions. For the elephant image, only 2 of the 100 proposals have a predicted label of 22 (elephant class id). You can use the indices of these two proposals to extract the attention maps of the objects. You just need to resize them to the image size and you should be done.

[lessw2020]

@thomashossler - thanks very much.
You mean look at the output tensor and use the predictions from that to index in reverse back to the attention maps?

[thomashossler]

@lessw2020 I tried cleaning my code a bit:

# forward pass
d = transforms(image)
model.eval()
pred = model(d.unsqueeze(0))

# post processing
postprocessors = {'bbox': PostProcess()}
_, h, w = d.shape
preds = postprocessors['bbox'](pred, torch.tensor([h, w]).unsqueeze(0))[0]

# find the high score proposals
obj_ids = torch.where(preds['scores'] > 0.9)[0]

Then you can resize the reshaped maps (where maps is your torch.Size([100, 1, 15, 20]) tensor ) and display them

upsampled_maps = F.interpolate(maps, (h, w), mode='bilinear', align_corners=False)[obj_ids, 0]

[fmassa]

Hi,

FYI I've sent a PR in https://github.com/facebookresearch/detr/pull/112 which adds a Colab notebook for visualizing both self-attention in the encoder as well as inter-attention in the decoder.

[fmassa]

The PR has been merged, so you you can access the colab with
https://colab.research.google.com/github/facebookresearch/detr/blob/colab/notebooks/detr_attention.ipynb

Let us know if you have further questions!

[lessw2020]

Thanks very much for this colab @fmassa!
(And thank you @thomashossler for posting the additional code!)

Question for @fmassa - What cmap or matplotlib settings did you guys use to make the glow coloration in the paper examples (attached)?
I used jet and magma cmap with an alpha blend for putting my heatmaps onto my images (from the attention decoder map, now that I understand how to hook those thanks to all the info posted here), but I like the look of your color coded glow better :)

[fmassa]

@lessw2020 I believe we used seaborn for that (but @szagoruyko can correct me if I'm wrong)

import seaborn as sns
colors = sns.color_palette(n_colors=n_objects)