Shap: Problem in LIGHTGBM and CatBoost using TreeExplainer with model_output = "probibility"

[SaadAhmed96]

I have created the following LighGBM model but I cant get it to work with the TreeExplainer when I use model_output = "probability".

Exception Traceback (most recent call last)
in ()
1
----> 2 explainer_3 = shap.TreeExplainer(model_3, model_output = "probability", data = X_train, feature_dependence = "independent")
3 shap_values_3 = explainer_3.shap_values(X)

D:\Anaconda\lib\site-packages\shap\explainers\tree.py in __init__(self, model, data, model_output, feature_dependence)
109 raise Exception(msg)
110 elif self.model.objective is None:
--> 111 raise Exception("Model does have a known objective! When model_output is not \"margin\" then we need to know the model's objective")
112
113 # A bug in XGBoost fixed in v0.81 makes XGBClassifier fail to give margin outputs

Exception: Model does have a known objective! When model_output is not "margin" then we need to know the model's objective

Same happens when I try to do it with CatBoost despite the fact I have specified objective = "MultiClass" there as well. Any particlular reason why is this happening or how can I get it to work on my system.

[slundberg]

This is because explaining the probability requires dealing with a non-linear transform of the output coming from the trees. In order to do that we need to know what that non-linear transform is, and then specifically build out support in the C++ code. Right now only binary classification (logistic) transforms are supported. It would be nice to also support multiclass, but I am not sure when I'll have the bandwidth to tackle that. Leaving a todo note here to track that this is feature it would be nice to have.

[SaadAhmed96]

Thanks for the detailed response. I will get back to you if I encounter any other problem.

[jayantsahewal]

In my case, I am getting the same issue even with a binary CatBoost classifier. Based on the code base, it looks like if model output is not margin then it will never work because objective for CatBoost classifiers will always be None. Is it possible to make it work for at least binary CatBoost classifiers?

if model_output != "margin":
    if self.model.objective is None and self.model.tree_output is None:
        raise Exception("Model does have a known objective or output type! When model_output is " \
"not \"margin\" then we need to know the model's objective or link function.")


elif str(type(model)).endswith("catboost.core.CatBoostClassifier'>"):
    assert_import("catboost")
    self.model_type = "catboost"
    self.original_model = model