Shap: Sequence data with recurrent neural nets?

[denfromufa]

Any examples of using SHAP for explaining the importance of features in RNN models such as LSTM?
I understand that SHAP is model agnostic approach, but maybe there are pitfalls with sequence data, e.g. in applying time series predictions with multiple input features?

[denfromufa]

Just found a paper from 2018 comparing LIME, SHAP with a new method applied to LSTM sequence predictions:

https://arxiv.org/abs/1802.07814

[slundberg]

That's a good paper. I would caution that the results of Kernel SHAP can depend on the number of samples used (I think they used Kernel SHAP's previous default of 1k samples in that paper).

The challenge/pitfalls with sequence models is 1) how many features you have, if you get long sequences then you can get more sampling variability. 2) treating the features as independent (i.e. perturbing a single word independently) can create break some patterns in the sequences.

Both of these issues are something we are working on addressing in an upcoming DeepExplainer. Once we get that out putting together an example of a sequence based model seems like a good idea. Hope that helps!

[denfromufa]

@slundberg the code for that paper has been added to github, really looking forward to trying both SHAP and L2X:

https://github.com/Jianbo-Lab/L2X

[denfromufa]

Here is the immediate error that I'm getting with sequence data: batch dimensions.

X_train.shape = (7755, 20, 49)

# use Kernel SHAP to explain test set predictions
explainer = shap.KernelExplainer(model.predict_proba, X_train[:5000], link="logit")
shap_values = explainer.shap_values(X_train[5000:], nsamples=100)

# plot the SHAP values for the Setosa output of the first instance
shap.force_plot(shap_values[0][0,:], X_train[5000:].iloc[0,:], link="logit")

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-104-a764dfb10a89> in <module>()
      1 # use Kernel SHAP to explain test set predictions
      2 explainer = shap.KernelExplainer(model.predict_proba, X_train[:5000], link="logit")
----> 3 shap_values = explainer.shap_values(X_train[5000:], nsamples=100)
      4 
      5 # plot the SHAP values for the Setosa output of the first instance

C:\Python\Anaconda3\lib\site-packages\shap\explainers\kernel.py in shap_values(self, X, **kwargs)
    148 
    149         assert str(type(X)).endswith("'numpy.ndarray'>"), "Unknown instance type: " + str(type(X))
--> 150         assert len(X.shape) == 1 or len(X.shape) == 2, "Instance must have 1 or 2 dimensions!"
    151 
    152         # single instance

AssertionError: Instance must have 1 or 2 dimensions!

[denfromufa]

I was able to use this LIME example on my dataset, but due to number of variables, sequence length and noise in data, probability distribution issues, it appears the results are not interpretable. What is hard with sequence predictions is dimensionality that blows up very fast: n_features * n_sequences.

https://github.com/marcotcr/lime/pull/57