Captum: On custom architecture

[abhigoku10]

@orionr @zpao @asmeurer @asuhan @kostmo great work by the team , this is what i was looking for , i have few queries .

1. can captum used be used for architectures like object detection and semantic segmentation
2. would i be able to see the intermediate learnings during training

[orionr]

Hi @abhigoku10 - good questions!

We haven't really used the library for detection / segmentation or other non-classification tasks yet, but definitely something we're curious about and want to explore more. @bilalsal (or @NarineK and @vivekmig) might have some thoughts here, but you could maybe attribute back on a per-pixel basis of the output (heat mask, segmentation, etc) to start. Regions (with thresholds) is probably want you'd want to try next.

For intermediate results, you definitely should be able to run attribution at anytime during the training loop. We generally use it for trained models, but I don't think there's anything stopping you from using it periodically every couple of epochs.

Let us know if you try it and have any issues!

[abhigoku10]

@orionr any updated on this issue??

[orionr]

@vivekmig is working on some examples for segmentation and detection. Hopefully he can give details soon!

[austinmw]

@orionr Hi, are there any updates on segmentation and detection examples?

[vivekmig]

Hi @austinmw , yes, we now have a tutorial on segmentation available here!