Shap: Summary plot for Keras NN

[hmodea]

Hello,

I'm trying to visualize a summary plot for a NN model implemented using keras but I'm getting this error:

AttributeError Traceback (most recent call last)
in ()
----> 1 shap.summary_plot(shap_values2, test_data.iloc[:50,:])

~/anaconda3/lib/python3.6/site-packages/shap/plots.py in summary_plot(shap_values, features, feature_names, max_display, plot_type, color, axis_color, title, alpha, show, sort, color_bar, auto_size_plot)
262 """
263
--> 264 assert len(shap_values.shape) != 1, "Summary plots need a matrix of shap_values, not a vector."
265
266 # convert from a DataFrame or other types

AttributeError: 'list' object has no attribute 'shape'

is this visualization only available for XGBoost model or is it model agnostic?

[slundberg]

This works for the model agnostic approach, but it requires a matrix with many explained predictions. It looks like you are just passing a list. You might be passing the list output from a multi-task Keras model. In that case you need to create a summary plot for each of the outputs separately. (i.e. shap_values2[0]) That help?

[hmodea]

Thanks for your response. It's actually a single regression task I pass the shap_values for 50 predictions as follows:
explainer2=shap.KernelExplainer(model_opt.predict,train_data.iloc[:100,:])
shap_values2 = explainer2.shap_values(test_data.iloc[:50,:],nsamples=500)
shap.summary_plot(shap_values2, test_data.iloc[:50,:])

[slundberg]

Could you paste the output of model_opt.predict? Is it an array of 50
numbers, or a list of arrays? Even a single task can have two outputs if
you consider both "yes" and "no"

On Tue, Apr 3, 2018 at 3:29 PM Haytham Modea [e-mail]
wrote:

Thanks for your response. It's actually a single regression tasks I pass
the shap_values for 50 predictions as follows:
explainer2=shap.KernelExplainer(model_opt.predict,train_data.iloc[:100,:])
shap_values2 = explainer2.shap_values(test_data.iloc[:50,:],nsamples=500)
shap.summary_plot(shap_values2, test_data.iloc[:50,:])

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/slundberg/shap/issues/52#issuecomment-378291699, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADkTxWMiTKiGkHOsWJ14gnNMLU9Frkazks5tk5VtgaJpZM4TEMyO
.

[hmodea]

Here it is....it seems like it's an array of 50 numbers, sorry I'm a bit new to python
array([[23.81435 ],
[24.920784],
[18.394382],
[32.466373],
[26.936054],
[20.75615 ],
[27.826164],
[28.82502 ],
[29.35595 ],
[19.585823],
[31.1749 ],
[23.818367],
[20.986301],
[21.990189],
[26.480679],
[27.435806],
[25.687576],
[25.030403],
[30.75037 ],
[28.854046],
[26.443665],
[25.836979],
[30.658043],
[30.002487],
[27.904613],
[24.56041 ],
[26.050224],
[27.367702],
[27.79565 ],
[27.298489],
[29.100311],
[23.541214],
[27.832752],
[26.319263],
[28.547066],
[24.584953],
[23.648563],
[30.290405],
[26.649635],
[29.261406],
[29.21804 ],
[25.310356],
[27.680351],
[28.777126],
[30.405788],
[26.985176],
[26.834045],
[25.840912],
[29.40435 ],
[26.746555]], dtype=float32)

[slundberg]

Ah...I should probably check for that, but you can just flatten the matrix
to a vector and it should work. Since it is a 2D matrix SHAP thinks it is a
multi-task output...with only one task. Or you could just use
shap_values2[0]

On Tue, Apr 3, 2018 at 3:47 PM Haytham Modea [e-mail]
wrote:

Here it is
array([[23.81435 ],
[24.920784],
[18.394382],
[32.466373],
[26.936054],
[20.75615 ],
[27.826164],
[28.82502 ],
[29.35595 ],
[19.585823],
[31.1749 ],
[23.818367],
[20.986301],
[21.990189],
[26.480679],
[27.435806],
[25.687576],
[25.030403],
[30.75037 ],
[28.854046],
[26.443665],
[25.836979],
[30.658043],
[30.002487],
[27.904613],
[24.56041 ],
[26.050224],
[27.367702],
[27.79565 ],
[27.298489],
[29.100311],
[23.541214],
[27.832752],
[26.319263],
[28.547066],
[24.584953],
[23.648563],
[30.290405],
[26.649635],
[29.261406],
[29.21804 ],
[25.310356],
[27.680351],
[28.777126],
[30.405788],
[26.985176],
[26.834045],
[25.840912],
[29.40435 ],
[26.746555]], dtype=float32)

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/slundberg/shap/issues/52#issuecomment-378297826, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADkTxRp9BmU3IRMJCjoHTHM9Fnj7sEvXks5tk5mAgaJpZM4TEMyO
.

[hmodea]

OK thanks I'll try that ! could you please explain what does the index in shap_value represent ? I was under the impression that shap_values[0] for example correspond to the shap_value of the first feature but clearly that is not the case? also would flattening and taking shap_value[0] yield same results?

[slundberg]

when doing multi-task prediction the result of explainer.shap_values is a
list of matrices, each matrix has the SHAP values for each prediction on
each row. When you don't do multi-task the output is just a single matrix,
but with multi-task you get a list of matrices. Since you have a multi-task
prediction with only one task you should get a list with only one matrix in
it.

On Tue, Apr 3, 2018 at 3:56 PM Haytham Modea [e-mail]
wrote:

OK thanks I'll try that ! could you please explain what does the index in
shap_value represent ? I was under the impression that shap_values[0] for
example correspond to the shap_value of the first feature but clearly that
is not the case? also would flattening and taking shap_value[0] yield same
results?

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/slundberg/shap/issues/52#issuecomment-378301080, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADkTxReUWeUsnsgj5nGFVqSzKY7U5l-gks5tk5urgaJpZM4TEMyO
.

[hmodea]

Thanks for your responsiveness. I have two final question though,
1- In case of a single task: shouldn't the no. of shap values be equivalent to no.of features * no. of predictions.
2- I see that there is an explanation step which you typically perform on training data then use that explainer to compute shape values for test data. It seems to me analogous to training the model then using it for prediction to test data. However, I understand that your model does explanations on a per prediction basis so there is no generalization here and that Shap values are measure of feature contribution per prediction, so could you please clarify what the explainer step actually does?

[slundberg]

1. It will be a matrix of # of predictions X # of features + 1. The +1 is
   for a bias term which represents the expected output of the model.
2. The explainer step just builds an object with a given background
   distribution used for computing expectations. There is no training, just
   the creation of a python object so you could call shap_values many times.
   The idea was is allow some processing of the background dataset in the
   future, but right now it just builds an object and all the real work
   happens when you call the shap_values() method.

On Tue, Apr 3, 2018 at 4:42 PM Haytham Modea [e-mail]
wrote:

Thanks for your responsiveness. I have two final question though,
1- In case of a single task: shouldn't the no. of shap values be
equivalent to no.of features * no. of predictions.
2- I see that there is an explanation step which you typically perform on
training data then use that explainer to compute shape values for test
data. It seems to me analogous to training the model then using it for
prediction to test data. However, I understand that your model does
explanations on a per prediction basis so there is no generalization here
and that Shap values are measure of feature contribution per prediction, so
could you please clarify what the explainer step actually does?

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/slundberg/shap/issues/52#issuecomment-378433718, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADkTxYRSZQfb9VpOMXMNs-Vx2_LFbyB8ks5tlAjogaJpZM4TEMyO
.

[hmodea]

So currently it doesn't really matter if I can the explainer on one sample and compute the shap values for the full set or call it on 100 samples and still compute shap values for the full set? As I noticed a big difference in terms of processing time

[slundberg]

The samples you pass when creating the explainer are used to integrate out
held-out features while computing the SHAP values. So if you pass 100
background samples it will need 100 times more function evaluations than if
you just pass 1 sample (but it should better approximate a missing sample).
I would either just pass a single reference value for the background, or
use weighted means as in:
https://github.com/slundberg/shap/blob/master/notebooks/Diabetes%20regression.ipynb

On Tue, Apr 3, 2018 at 9:19 PM Haytham Modea [e-mail]
wrote:

So currently it doesn't really matter if I can the explainer on one sample
and compute the shap values for the full set or call it on 100 samples and
still compute shap values for the full set? As I noticed a big difference
in terms of processing time

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/slundberg/shap/issues/52#issuecomment-378475669, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADkTxQrdTZbk_jf7UZ7VtWDi0D0ya8bnks5tlEnWgaJpZM4TEMyO
.

[hmodea]

Thanks for your response, all clear now ! and thanks for the interesting package!

[slundberg]

Great!

[xavierbourretsicotte]

I had the same problem on a regression task and solved by flattening shap_values - cheers