Shap: Time Series LSTM with SHAP

[oshapeman]

Confused about the shap_values not equal to the model output and how the shap deepExplainer deal with the multi input of the lstm ?

A minimal for debugging:
https://github.com/oshapeman/lstm-shap/blob/master/shap_lstm_issue.ipynb

Thanks a lot !

[slundberg]

When I ran your code I got no errors:

max: 1.260343674980291e-07
min: 1.1843082070583932e-10
mean: 1.4680117375043267e-08

But I am on my laptop without a GPU, so I had to change CuDNNLSTM to LSTM. It could be there is some issue with CuDNNLSTM.

To help find these problems in the future I have now added a check (turned on by default) that verifies the sum of the SHAP values equals the model's output in DeepExplainer.

[oshapeman]

Thanks sir!
I also tested LSTM with shap, it worked as you showed. I think that it is a issue about the CuDNNLSTM when using shap.
Finally, thanks for your adding the check to the DeepExplainer, it helps a lot!