Shap: TFDeepExplainer broken with TF2.1.0

[FRUEHNI1]

Hi,

With the latest HEAD revision of shap from git and TF 2.1.0 I get the following errors.

1.) custom_record_gradient() missing 1 required positional argument: 'name'
I assume this is due to the way the custom gradient record is defined between shap and tensorflow.
If it's not used, this would be a simple fix by either removing the argument, or setting it to None per default?

See also https://github.com/tensorflow/tensorflow/commit/6d8eafa78834e197a04086a928d9bb3e1339a9aa

    TypeError: custom_record_gradient() missing 1 required positional argument: 'name'

If I fix this, I get:

2.) AttributeError: 'TFDeepExplainer' object has no attribute 'between_tensors'

 out[i] = t.name in self.between_tensors
    AttributeError: 'TFDeepExplainer' object has no attribute 'between_tensors'

This is due to the fact that in deep_tf#L236 init_between_tensors is called, but it is expected to be there in deep_tf#L232

[forough71]

I have the exact same problem.

[MathiesW]

I am getting the same error with SHAP 0.35 and TF2.1.0

[talipini]

I am having the same issue with nightly and 2.2.0rc4. I am using SHAP 0.35. Is there any version TF that works with SHAP DeepExplainer?

[MathiesW]

I figured that shap deep explainer works with TensorFlow 1.15. I set up a virtual Python environment to calculate the shapley values and export them as numpy arrays. It's not comfortable but works well enough as a workaround.

[talipini]

Great, Thanks MathiesW. I will try that.
FYI, 2.1.0rc0 seems to work and is not failing with this error. but I am still trying to get everything together and test it.

[MathiesW]

Glad I could help. I just tried TensorFlow 2.1.0-rc0 but it resulted in the following error message:
RecursionError: maximum recursion depth exceeded in comparison

[threadgoldc]

I figured that shap deep explainer works with TensorFlow 1.15. I set up a virtual Python environment to calculate the shapley values and export them as numpy arrays. It's not comfortable but works well enough as a workaround.

thanks. Worked for me :)

[qemtek]

The same problem exists in tensorflow 2.1.1

[jonathanglima]

Same problem in TF 2.2

[darien-schettler]

The same problem exists when using the nightly build as of July 21, 2020 -- 2.4.0-dev20200721

[GlorianY]

The same problem exists in TF 2.3.0

[mattnorel]

I know it's not a solution but setting tf.compat.v1.disable_v2_behavior() right after importing tensorflow seems to work with shap's DeepExplainer. It might be a way for someone who don't want to install previous version.

[talipini]

I got a strange error when I disable v2_behavior. shap_values throws a value error with the 2x batchsize of the actual batch size in the model. My sample data is exactly the batch size of 1024. I guess SHAP is doubling the batchsize?

ValueError: Cannot feed value of shape (2048, 124, 257) for Tensor '0_lstm_layer_input:0', which has shape '(1024, 124, 257)'