Shap: Can The kernel Explainer be used for a neural network model

[niloofardadras]

Hi,
I have a neural network model developed with tensorflow estimator API, I have tried to calculate shap values from my model with Deep explainer and Gradient explainers but all attempts have failed. I eventually used kernel explainer and got results from it after i encoded my categorical data and decoded inside my function.

My question is, does using kernel explainer provide reliable results?

Also i have realized that strangely using my explainer on same data creates different shap values each time i run the explainer. is this normal or is there anything missing for calculating stable result?

[jtduran]

Having similar problems with TF 2.2 Sequential networks and Shap 0.35.0, but havent gotten KernelExplainer to work yet. I'm guessing it's not quite compatible with TF 2.1+ ?

[alexcoca]

Hi @niloofardadras ,

The calculations of the Shapley values using KernelShap needs to treat, in turn, each possible combination of inputs as "missing" (aka you estimate a contribution by seeing what happens to the function value if you remove that feature from the input). In general, there are 2**M such subsets, so it is not tractable to compute the exact values. Instead, the algorithm excludes one feature at a time, first, since this process gives the most information about a feature's contribution. Then the subset size increases to 2, 3 and so on. The aglorithm tries to fill up nsamples (which defaults to something 2 * X.shape[1] + 2048 with subsets of increasing sizes, but, if, at some point, it doesn't have enough remaining samples to include all subsets of a certain size it samples subsets of various sizes at random (see line 368 in kernel.py). This is a random step, so, if you have not set the seed you might see some differences because of this sampling effect.

Another reason for the observed randomness across different runs could be the state of randomness in your model. To compute the shap values, the KernelExplainer solves a regression problem where the targets are obtained by calling your model with inputs that are obtained by combining what you're trying to explain with the background data (which is used to replace the "missing values" mentioned above). If your model makes different predictions across runs, then the explanation will also change since you regress on a different target. I suggest looking at the ey property of the explainer across different runs to see if this is the case. Additionally, you can check how self.maskMatrix changes - this is a binary mask that indicates which subsets have been used in the computation across runs. If the first n subsets are the same but the rest differ, then this might be indicating that different subsets have been subsampled as described above.

It is difficult to say how "reliable" the results will be. As mentioned above, the computational complexity of this algorithm is exponential so the estimates are noisy. How good they are depends on the alloted computational budget. So you could try fixing the seed and then increasing the nsamples to understand how much the explanations change and select a sensible number of samples. The larger the dimension of your problem (aka number of features), the more samples you will need to obtain accurate results. KernelShap has been shown to converge (empirically) for cases where the exact values can be computed (e.g., TreeModels).

I hope this helps.

[niloofardadras]

@alexcoca thank you so much for your amazing reply. appreciate it so much. :)
so if i understood correctly the kernel explainer may be used for neural network also and reliability depends on factors such as output and number of samples.

now the question is how do i know how many number of samples is enough for the best results. after calculations with kernel explainer I realized that it takes pretty long time to calculate results for 1 observation if i use 1000 samples for example (around 2 mins!). I have around 50 features which needs calculation for shap values. would you have any suggestions for me?

also another thing i am not sure of is how the explainer handles encoded categorical features.

[alexcoca]

@niloofardadras , yes, you understood correctly. But using a model agnostic method is going to come with a significant computational cost. It is better to use the specialised models as per your original attempt, but as it stands it seems that the implementation might not have caught up with advances in the underlying libraries.

I guess my suggestion would be to see what is the minimum number of samples you need in order to get an explanation that is sensible. So start with, say, 200 samples. Obtain explanations for a few instances, then increase to something like 250, see how much the values change and continue until you think the change is small enough. It's a good idea to relate the variation in the shap value to some baseline (e.g., 10% of the nth most important feature) so as to put the error in some context. A way to reduce runtime might be to reduce the size of the background dataset - this will impact the estimation of the shap values since there will be a smaller number of samples from which each target for the regression is calculated.

Is the dimensionality 50 before or after encoding? The KernelExplainer doesn't do anything about the encoded categorical variables. So if you put encoded data in the explainer, you will get a shap value for each encoded dimension. A way to deal with this is to sum up the shap values for the encodings of a given features to obtain 1 single value per categorical feature. Another slightly more advanced method is to perform a grouping of the categorical variables so that an length m encoding is treated as a single shap value during the estimation process. Here's an example how to do this. Assume you have a dataset that has the 3 features, 2 continous, one categorical with 4 encoded dimensions. Let's give them names: 'height', 'width', 'hair_colour'[brown/blond/brunette/white]

from shap.common import DenseData

group_names = ['height', 'width', 'hair_colour']
# assume first two columns are the continuous variables and the next the enc. categorical var
groups = [[0], [1],  [2, 3, 4, 5]]
background_data = DenseData(my_data, group_names, groups)

Pass this object to the KernelExplainer as opposed to the numpy array containing your data. This may give you small runtime savings.

[niloofardadras]

@alexcoca Thank you so much for your great help.
makes sense to check shap values for certain explanations. will look into it.

does the dense data createonehot encoded input for the function?
the dimentionality is before any encoding. so here is the problem i might have with densedata. my model is trained in a way that it takes categorical data as string without encoding and passes the input to the tensorflow model function to process. as a results the input to the prediction function I have created can not be one hot encoded.

I have tried to do label encoding before feeding into the shap explainer and label decoding inside my model function. but i am not sure if that is correct. the results seem not that great.
is there any better approach for me to take without changing model training pipeline?

[niloofardadras]

@alexcoca hi again, hope you are doing well.

regarding shap.common.dense data I am having an error. hope you can help me with it.
I am using onehot encoded data passed to DenseData.

File "C:\Anaconda3\lib\site-packages\shap\common.py", line 133, in __init__
assert valid, "# of names must match data matrix!"
AssertionError: # of names must match data matrix!