Captum: evaluation metric

[berleon]

Hi, captum includes many different attribution method. But only one evaluation metric: the infidelity metric [1]. I think it is quite important show results on different evaluations -- each evaluation measures something differently. Having different evaluations would allow the user to make a better decision which attribution method to use.

Are there any plans to add for example the sanity checks [2]? For our paper [3], we developed a bounding box task. It measures if the ImageNet bounding boxes are scored highest by an attribution method. I have code for both. If you are interested, let me know. I would be happy to contribute.

[1] https://arxiv.org/pdf/1901.09392.pdf
[2] https://arxiv.org/abs/1810.03292
[3] https://openreview.net/forum?id=S1xWh1rYwB

[NarineK]

Hi @berleon , Thank you for the links! Sanity Checks are definitely on our list.
RESTRICTING THE FLOW: INFORMATION BOTTLENECKS FOR ATTRIBUTION looks like a recent paper. It looks interesting. I haven't seen it before I'll read it. Thank you for the reference :D

Btw. I've put out the PR for the sensitivity metric and planning to look into more options.

[berleon]

Hi @NarineK, thanks for you interest in our work. If you registered for the virtual ICLR, you can also watch our video (https://iclr.cc/virtual/poster_S1xWh1rYwB.html). Once ICLR is over, I should have time to adapt our sanity checks code to Captum. I would then just create a PR orientation myself on the sensitivity metric?

[NarineK]

Thank you very much for the link @berleon ! Great presentation! Sanity checks are definitely very important. Some of the simple sanity checks / metrics can be added by orienting on infidelity or sensitivity API design. The ones that are more complex and require, for example, model retraining might require more architectural code changes. Currently I've been working on designing the infrastructure for it as well.
Which sanity checks are you thinking to add first ?

[berleon]

Thanks for your positive feedback! I would only add the parameter randomization sanity check. It is an important check and easy to perform for any attribution & model. The data randomization task is also interesting, but - as you mentioned - is more complex and requires knowledge of how to train the model. I would not consider the data randomization task to be essential for captum.