Captum: The convergence delta of DeepLIFT is not equal to zero

[chenbai1002]

According to the paper of DeepLIFT, the convergence delta should be zero when the model is pure DNNs without any structures (such as LSTM etc.) and the activation function is ReLU. However, when I try to explain a regression DNN model using DeepLIFT, the convergence delta is always greater than zero. So in which conditions could the delta greater than zero ?

[NarineK]

Hi @chenbai1002, what is the DNN architecture that you're using ? delta is close to zero, it doesn't necessarily have to be exactly zero. What delta value do you observe ?

[chenbai1002]

Hi @chenbai1002, what is the DNN architecture that you're using ? delta is close to zero, it doesn't necessarily have to be exactly zero. What delta value do you observe ?

Thank you for your reply!
My DNN architecture is very simple:

The average of delta of 5000 samples is 2.0304 while the outputs of model is around 0.1. I tried Integrated Gradients and the average delta of the same 5000 samples is 7.6120e-05. So I think there is something wrong when I use DeepLIFT.
The code to interpret model using DeepLIFT and IG are highly similar:

When I use DeepLIFT, I set the interpret_method = DeepLift(net), when I user IG, I set interpret_method = IntegratedGradients(net).

I don't know why the results of DeepLIFT is abnormal, wish for some suggestions!

[NarineK]

Hi @chenbai1002,

thank you for the example. Are you using random model initialization weights or did you train your model ? Would you drop the code snippet so that I can copy it in the notebook and execute ? Or if you have a collab notebook and could share it that would be great!

Thank you,
Narine

[NarineK]

@chenbai1002 , let us know if you were able to get it working ?

[chenbai1002]

@NarineK , sorry for replying late, the delta of DeepLIFT still has a problem.
The result of Integrated Gradients is reasonable, so I think the model to interpret is well trained.
It is not convenient for me to share the model and data, do you have some methods to debug?
Thank you for you reply!

[vivekmig]

Hi @chenbai1002 , I took a quick look at your code, I think the problem might be usage of F.relu, since we are not able to hook functional non-linearities, only those that are using the corresponding modules. Can you try switching to nn.ReLU and see if that resolves the issue?

[chenbai1002]

Hi @chenbai1002 , I took a quick look at your code, I think the problem might be usage of F.relu, since we are not able to hook functional non-linearities, only those that are using the corresponding modules. Can you try switching to nn.ReLU and see if that resolves the issue?

@vivekmig Thank you very much for your advice! I will try it soon.

[chenbai1002]

@vivekmig @NarineK I have tried to use nn.ReLU instead and the average delta of 5000 samples reduces to -0.0163 from 2.0304, while the sum of all features' attributes is 0.0397. Your advice works!
But the delta of Integrated Gradients is -1.6187e-05 while the sum of all features' attributes is 0.0561. Compared to Integrated Gradients, the delta of deepLIFT is still very large. Is that reasonable? I guess this gap might caused by the integrated effect of IG (deepLIFT does not integrate multipliers), what is your opinion?

[vivekmig]

Hi @chenbai1002 , the remaining error could just be an effect of floating point error increasing gradually through the layers, or there could possibly be a bug somewhere. To investigate further, it would be best to try LayerDeepLift for each layer, starting from the end moving forward, and look at the convergence delta. The attributions and delta of DeepLift should match LayerDeepLift with the first layer (and attribute_to_layer_input=True). A sudden increase in the delta at a particular layer might suggest that there's something wrong with the propagation of multipliers at that layer.

[chenbai1002]

@vivekmig Good point! However, when I updated captum from 0.1.0 to 0.2.0 and changed python environment, the convergence delta decrease to -4.6566e-09. It seems that there are some bugs in the previous environment.
Thank you for you help!

[vivekmig]

Ok, great, I'm glad the issue was resolved with 0.2.0!