Shap: DeepExplainer - TypeError: Fetch argument None has invalid type <type 'NoneType'>

[ZeonTrevor]

I get this error using the DeepExplainer while trying to get the shap values for the Keras model as defined here. The DeepExplainer instantiates correctly for the above model giving the right expected value for the given set of background samples. I have tried DeepExplainer before using another model and it works fine, so I feel it could possibly be due to the use of a custom layer DynamicMaxPooling that requires TF gather_nd op. I use SHAP (0.25.2), TF(1.11), Keras(2.2.2).

```---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
in ()
----> 1 shap_values = deep_explainer.shap_values([background_query[:1], background_doc[:1], background_dpool_index[:1]])

/home/fernando/neuir/local/lib/python2.7/site-packages/shap/explainers/deep/__init__.pyc in shap_values(self, X, ranked_outputs, output_rank_order)
115 were chosen as "top".
116 """
--> 117 return self.explainer.shap_values(X, ranked_outputs, output_rank_order)

/home/fernando/neuir/local/lib/python2.7/site-packages/shap/explainers/deep/deep_tf.pyc in shap_values(self, X, ranked_outputs, output_rank_order)
248 # run attribution computation graph
249 feature_ind = model_output_ranks[j,i]
--> 250 sample_phis = self.run(self.phi_symbolic(feature_ind), self.model_inputs, joint_input)
251
252 # assign the attributions to the right part of the output arrays

/home/fernando/neuir/local/lib/python2.7/site-packages/shap/explainers/deep/deep_tf.pyc in run(self, out, model_inputs, X)
268 for t in self.learning_phase_flags:
269 feed_dict[t] = False
--> 270 return self.session.run(out, feed_dict)
271
272 def custom_grad(self, op, *grads):

/home/fernando/neuir/local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in run(self, fetches, feed_dict, options, run_metadata)
927 try:
928 result = self._run(None, fetches, feed_dict, options_ptr,
--> 929 run_metadata_ptr)
930 if run_metadata:
931 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/home/fernando/neuir/local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in _run(self, handle, fetches, feed_dict, options, run_metadata)
1135 # Create a fetch handler to take care of the structure of fetches.
1136 fetch_handler = _FetchHandler(
-> 1137 self._graph, fetches, feed_dict_tensor, feed_handles=feed_handles)
1138
1139 # Run request and get response.

/home/fernando/neuir/local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in __init__(self, graph, fetches, feeds, feed_handles)
469 """
470 with graph.as_default():
--> 471 self._fetch_mapper = _FetchMapper.for_fetch(fetches)
472 self._fetches = []
473 self._targets = []

/home/fernando/neuir/local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in for_fetch(fetch)
259 elif isinstance(fetch, (list, tuple)):
260 # NOTE(touts): This is also the code path for namedtuples.
--> 261 return _ListFetchMapper(fetch)
262 elif isinstance(fetch, collections.Mapping):
263 return _DictFetchMapper(fetch)

/home/fernando/neuir/local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in __init__(self, fetches)
368 """
369 self._fetch_type = type(fetches)
--> 370 self._mappers = [_FetchMapper.for_fetch(fetch) for fetch in fetches]
371 self._unique_fetches, self._value_indices = _uniquify_fetches(self._mappers)
372

/home/fernando/neuir/local/lib/python2.7/site-packages/tensorflow/python/client/session.pyc in for_fetch(fetch)
256 if fetch is None:
257 raise TypeError('Fetch argument %r has invalid type %r' % (fetch,
--> 258 type(fetch)))
259 elif isinstance(fetch, (list, tuple)):
260 # NOTE(touts): This is also the code path for namedtuples.

TypeError: Fetch argument None has invalid type ```

[slundberg]

Hmmm...I don't know after looking briefly. Could you check and see if the line self.phi_symbolics[i] = tf.gradients(out, self.model_inputs) in deep_tf.py is saving a None value at any point? (Just print self.phi_symbolics[i] right after that line)

[ZeonTrevor]

Yes self.phi_symbolics[i] has a None value for the 3rd Input Tensor, which is the input required for the custom layer (DynamicMaxPooling) that I had mentioned above. Below is the output I get while I print self.phi_symbolics[i]:

[<tf.Tensor 'gradients/embedding_1/embedding_lookup_grad/Select:0' shape=(?, 4) dtype=float32>, <tf.Tensor 'gradients/embedding_1_1/embedding_lookup_grad/Select:0' shape=(?, 500) dtype=float32>, None]

[slundberg]

Thanks. The way DeepExplainer works is by overriding the gradient back propagation steps. I assume that gather_nd will end up being a "GatherV2" operation. The overload of that operation's gradient has some specific code in place to handle embedding layers so credit can get passed back to the incoming indexes. To debug this I would need to setup a running example and start printing out what is happening in the gather function of deep_tf.py. My guess is that something in there is going wrong with the dynamic max pooling. If you could post a minimal self contained example I'll take a look when I get a chance. Or feel free to poke around in there yourself.

[akshayuppal3]

I got the same error with deep explainer

[VincentBonnivard]

Hello,
I'm facing a similar issue using a Keras implementation of RetinaNet object detection,
with shap 0.35, keras 2.3.1, tensorflow 1.13.1.
The implementation is from https://github.com/fizyr/keras-retinanet

You can find below a minimal working example. To make it work, you need :

• To clone the https://github.com/fizyr/keras-retinanet repo
• To install the package, with pip install .
• To download the resnet50_coco_best_v2.1.0.h5 model from https://github.com/fizyr/keras-retinanet/releases. I'm using version from tag 0.5.1
• To copy the downloaded model in the "snapshots" folder of the keras-retinanet repo.

Then, here is the code I've used (in a Jupyter Notebook)

import keras
from keras_retinanet import models
from keras_retinanet.utils.image import read_image_bgr, preprocess_image, resize_image
from keras_retinanet.utils.visualization import draw_box, draw_caption
from keras_retinanet.utils.colors import label_color
from keras_retinanet.utils.gpu import setup_gpu

import matplotlib.pyplot as plt
import cv2
import os
import numpy as np
import time

gpu = 0
setup_gpu(gpu)

# adjust this to point to your downloaded/trained model
# models can be downloaded here: https://github.com/fizyr/keras-retinanet/releases
model_path = os.path.join('..', 'snapshots', 'resnet50_coco_best_v2.1.0.h5')

# load retinanet model
model = models.load_model(model_path, backbone_name='resnet50')

# if the model is not converted to an inference model, use the line below
# see: https://github.com/fizyr/keras-retinanet#converting-a-training-model-to-inference-model
#model = models.convert_model(model)

# load label to names mapping for visualization purposes
labels_to_names = {0: 'person', 1: 'bicycle', 2: 'car', 3: 'motorcycle', 4: 'airplane', 5: 'bus', 6: 'train', 7: 'truck', 8: 'boat', 9: 'traffic light', 10: 'fire hydrant', 11: 'stop sign', 12: 'parking meter', 13: 'bench', 14: 'bird', 15: 'cat', 16: 'dog', 17: 'horse', 18: 'sheep', 19: 'cow', 20: 'elephant', 21: 'bear', 22: 'zebra', 23: 'giraffe', 24: 'backpack', 25: 'umbrella', 26: 'handbag', 27: 'tie', 28: 'suitcase', 29: 'frisbee', 30: 'skis', 31: 'snowboard', 32: 'sports ball', 33: 'kite', 34: 'baseball bat', 35: 'baseball glove', 36: 'skateboard', 37: 'surfboard', 38: 'tennis racket', 39: 'bottle', 40: 'wine glass', 41: 'cup', 42: 'fork', 43: 'knife', 44: 'spoon', 45: 'bowl', 46: 'banana', 47: 'apple', 48: 'sandwich', 49: 'orange', 50: 'broccoli', 51: 'carrot', 52: 'hot dog', 53: 'pizza', 54: 'donut', 55: 'cake', 56: 'chair', 57: 'couch', 58: 'potted plant', 59: 'bed', 60: 'dining table', 61: 'toilet', 62: 'tv', 63: 'laptop', 64: 'mouse', 65: 'remote', 66: 'keyboard', 67: 'cell phone', 68: 'microwave', 69: 'oven', 70: 'toaster', 71: 'sink', 72: 'refrigerator', 73: 'book', 74: 'clock', 75: 'vase', 76: 'scissors', 77: 'teddy bear', 78: 'hair drier', 79: 'toothbrush'}

# Try SHAP
import shap, os

# For this minimal example, just use the image available in the "examples" folder of retinanet repo
list_background_files = ['000000008021.jpg']*100

# preprocess image for network
background = []
for file in list_background_files:
    image = read_image_bgr(os.path.join('./', file))
    image = preprocess_image(image)
    image_resize = cv2.resize(image, (600,600), interpolation=cv2.INTER_AREA)
    image_resize = np.expand_dims(image_resize, axis=0)
    background.append(image_resize)

# Define model for DeepExplainer
model_in = model.input
model_in.set_shape((None,)+(600, 600, 3))
# Here, we select only the "class" outputs from the model, not the bounding boxes or the scores
model_out = model.outputs[2]
model_tuple = (model_in, model_out)

# Create explainer
e = shap.DeepExplainer(model_tuple, background)

# Compute SHAP values, here on the image from the background set for simplicity purpose
shap_values = e.shap_values(background[0])

I end up with the error:
"TypeError: Fetch argument None has invalid type "

The weird behavior is that I'm able to get the shap values if I try to explain the model.outputs[1]
instead of model.outputs[2], which correspond to the scores associated to each predicted class (while model.outputs[2] is the integer associated to each class).

Looking a bit into SHAP, it appears to be caused by the same problem as mentionned ealier in the issue, i.e. a "None" gradient computation: when printing the tf.gradients(out, self.model_inputs) within SHAP code, I get a "[None]"

In case it can help, I can list the layers of the model using model.summary(), but it's quite a long list :)

Thanks in advance!