Captum: Returning only the gradients/"multipliers"

[AvantiShri]

Hi all,

Just wanted to put this particular use-case on your radar. Sometimes we find that it is useful to get access to just the gradients ("multipliers"), before they are multiplied by the difference-from-reference to get the final attribution. Specifically, we use the multipliers to estimate how the network _might have_ responded had it seen slightly different inputs. We refer to these estimates as "hypothetical contribution scores". If you are curious how these hypothetical contributions look, here's a notebook (on a fork of the DeepSHAP repository) where I compute hypothetical contributions in the context of genomic data: https://github.com/AvantiShri/shap/blob/0b0350ba3a42af275f6e99ca2e3c5877d7d94f8a/notebooks/deep_explainer/PyTorch%20Deep%20Explainer%20DeepSEA%20example.ipynb

You've all done an awesome job with this repository, and I will definitely point it to the pytorch users in my lab once the release is formally announced. I totally understand if the ability to return just the multipliers is not something that you are likely to incorporate in the main release; I'm sure we can easily fork the repository and add that feature in for our lab's purposes.

Thanks again!
Av

[NarineK]

Hi Avanti, Thank you very much for the feedback! Happy to hear that you liked it.
I think that it shouldn't be an issue to return the multipliers.

Thank you very much for jupyter notebook. I'll check it out :)

[orionr]

Not going to close this, since we can consider it as a feature request that we aren't working on yet. Thanks, @AvantiShri!

[AvantiShri]

Thanks @orionr! One thing to note about how I implemented support for this feature in my fork of the shap repository (which is linked to in the notebook above) is that I allow the user to optionally provide a custom function that takes as input the (multipliers, original input, references) and returns the final scores - I think the equivalent in Captum would be allowing a user to provide a custom function that varies the logic in this line:
https://github.com/pytorch/captum/blob/2fea0d764a19e621e2e8da86aa6424f9cd74def2/captum/attr/_core/deep_lift.py#L207

The main reason to provide some flexibility here (in terms of how the user might like to combine the multipliers with the inputs/references to give the final contribution scores) is that for one-hot encoded sequences, sometimes getting the contributions is not as simple as doing multiplier*difference-from-baseline. Specifically, imagine you have a one-hot encoded sequence where only one character at each position can be a 1 (as is often done in character encodings in NLP, or sequence encodings in genomics). Say there are four characters, and the encoding at some position in the input sequence is [1,0,0,0] while the reference at that position is [0,1,0,0]. DeepLIFT will assign a contribution to the first character having a difference-from-reference of +1 and the second character having a difference-from-reference of -1. However, because of the one-hot encoding constraint that only one character at each position can have a 1, the difference-from-reference of -1 in the second character is in some sense "due to" the difference-from-reference of +1 in the first character. Thus, for ease of visualization, we add the contribution of the second character (the one that is absent from the input) to the contribution of the character that is actually present in the input, and then only visualize the total contribution on the character that is actually present in the input. If we did not do this, then at visualization time we would we get scores on more than one character at each position, which is extremely confusing given the one-hot encoding constraint!

Just something I thought I would highlight since @NarineK referenced this feature and I wasn't sure if you had decided to start working on it yet. Please let me know if you would like me to clarify more, or if anything I have said does not make sense!

Thank you again for the hard work!

Avanti

[NarineK]

Hi @AvantiShri, sure, that makes sense to me. We can take an additional custom function as an argument and in case it is not provided we can use our default logic. This shouldn't be a big change. I'm currently adding layer/neuron based support and will add this functionality as next.

[AvantiShri]

Awesome, thank you!