Shap: ResourceExhaustedError with DeepExplainer after Keras model load

[austinmw]

I have my x_train and x_test data loaded, and gpu memory utilization is 0%.
They are both np.arrays of size (10000,224,224,3).

I then load my Keras model:

from tensorflow.keras.models import load_model
model = load_model(./model.h5)
optimizer = tf.keras.optimizers.SGD(lr=0.0001, momentum=0.99, nesterov=True)
model.compile(loss='categorical_crossentropy', optimizer=optimizer)

And GPU memory utilization jumps to around 96%.

I then try to use DeepExplainer just like the MNIST example:

import shap
import numpy as np

# select a set of background examples to take an expectation over
background = x_train[np.random.choice(x_train.shape[0], 100, replace=False)]

# explain predictions of the model on three images
e = shap.DeepExplainer(model, background)

shap_values = e.shap_values(x_test[1:5])

However I end up getting:

ResourceExhaustedError: OOM when allocating tensor with shape[200,64,224,224] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc

Am I doing something incorrectly?

[slundberg]

It could be that your GPU does not have enough space for all 100 background samples. DeepExplainer runs all 100 through as a batch. You can try just 10 and see if that fixes it. (you can average 10 runs of 10 to get the effect of 100). Alternately the GradientExplainer accepts a batch_size argument that controls how many samples to run at once.

[austinmw]

Thanks, I think that was the only issue. Taking an average of a smaller number of backgrounds seems to work.

[slundberg]

Great!

[vedantbhatia]

It could be that your GPU does not have enough space for all 100 background samples. DeepExplainer runs all 100 through as a batch. You can try just 10 and see if that fixes it. (you can average 10 runs of 10 to get the effect of 100). Alternately the GradientExplainer accepts a batch_size argument that controls how many samples to run at once.

Hi,

Could you please explain what you mean by taking the average of 10 runs of 10?

[slundberg]

@vedantbhatia it just means that you can run DeepExplainer 10 times with 10 different sets of background samples and average the values you get (across the 10 runs) to get the same effect as running it once, but with 10x the number of samples.

[vedantbhatia]

@vedantbhatia it just means that you can run DeepExplainer 10 times with 10 different sets of background samples and average the values you get (across the 10 runs) to get the same effect as running it once, but with 10x the number of samples.

Thank you very much!