Shap: name '_cext' is not defined

[gschivley]

I just installed shap 0.16.1 in a conda environment using pip. Trying to use it on a sklearn RandomForestRegressor model gave me the following error:

shap_values = shap.TreeExplainer(rf_reg).shap_values(X_train)

```
NameError Traceback (most recent call last)
in ()
----> 1 shap_values = shap.TreeExplainer(rf_reg)#.shap_values(X_train)

~/anaconda/envs/marginal_emiss/lib/python3.6/site-packages/shap/explainers/tree.py in __init__(self, model, **kwargs)
41
42 if str(type(model)).endswith("sklearn.ensemble.forest.RandomForestRegressor'>"):
---> 43 self.trees = [Tree(e.tree_) for e in model.estimators_]
44 elif str(type(model)).endswith("sklearn.tree.tree.DecisionTreeRegressor'>"):
45 self.trees = [Tree(model.tree_)]

~/anaconda/envs/marginal_emiss/lib/python3.6/site-packages/shap/explainers/tree.py in (.0)
41
42 if str(type(model)).endswith("sklearn.ensemble.forest.RandomForestRegressor'>"):
---> 43 self.trees = [Tree(e.tree_) for e in model.estimators_]
44 elif str(type(model)).endswith("sklearn.tree.tree.DecisionTreeRegressor'>"):
45 self.trees = [Tree(model.tree_)]

~/anaconda/envs/marginal_emiss/lib/python3.6/site-packages/shap/explainers/tree.py in __init__(self, tree, normalize)
187
188 # we compute the expectations to make sure they follow the SHAP logic
--> 189 self.max_depth = _cext.compute_expectations(
190 self.children_left, self.children_right, self.node_sample_weight,
191 self.values

NameError: name '_cext' is not defined

On further inspection, looks like there's an ImportError. The code in Tree.py seems to suppress ImportErrors. When I comment out the `except ImportError:` portion I do get the printed error.

```Python
try:
    from .. import _cext
except ImportError:
    pass
except:
    print("the C extension is installed...but failed to load!")
    pass

First attempt at installation gave some errors because cython wasn't installed. I uninstalled shap, installed cython from conda, and reinstalled shap with pip.

[gschivley]

Uninstalling again and reinstalling with pip from the repository (pip install git+https://github.com/slundberg/shap.git) solved the problem.

I don't think it affects my issue but this morning I realized that gcc wasn't installed on this machine. So I installed it and tried an additional round of uninstall/install from PyPI. Ran into the same problem accessing _cext.

[slundberg]

It must have failed to build the extension during the original pip install. I didn't want errors for people without a compiler if they just used XGBoost or LightGBM since they don't need the C-extension, hence the ImportError suppression. I added what should be a better error when you actually try and use it.

[marlanc]

We too have had this issue. We installed cython and made sure we had gcc installed and then reinstalled. No luck. @gschivley did you finally solve the problem with an install from git directly? I'm asking because the last sentence of your most recent comment suggested you hadn't yet solved the problem.

[slundberg]

Did you notice any build errors during the pip install process? Those might indicate why the extension failed to build.

[marlanc]

No, the admin who installed the package said he saw no errors. Are there any changes in the latest version in this repository vs. the version that is installed by pip that might affect this. I ask because gschivley mentioned that installing from the repo solved the problem (although I'm unclear whether it solved the extension build problem).

[slundberg]

Unless you are using a non-standard pypi mirror that is not up-to-date, then there should be no new changes in git that would address this. That being said, if you just do a git clone https://github.com/slundberg/shap and then run python setup.py build inside the shap directory you should see any potential build problems.

[gschivley]

@marlanc Installing from the repository did solve the problem for me. Tricky to determine the actual issue, but here's an overview of what happened.

1. Installed from PyPI via pip, got error.
2. Installed cython from conda, uninstalled/reinstalled shap from PyPI. Still got error.
3. Googled a bunch, determined that gcc might not be installed on my machine (mac). Installed xcode-select via command line to get a gcc compiler (LLVM version 9.1.0).
4. Uninstalled shap with pip. Tried installing from PyPI again. Still got error.
5. Uninstalled shap with pip. Used pip to install from the github repo. No error!

This was not scientific, and my memory could be off about the order of events.

Further testing just now:

• I installed shap via pip/PyPI (shap 0.18.0) in a separate conda environment and had no problems.
• Uninstalled 0.18.0 and installed 0.16.1 in this new conda environment, got error! I don't know enough about pip and conda environments to know if some cached files from my original flawed builds are being used.
• Uninstalled 0.16.1 and installed 0.17.1, no problems.

Hope this is of some use!

[slundberg]

Thanks for the details! You could be right that pip wasn't rerunning the build process after the first failure (when you didn't have a compiler). At some point getting automatic wheels built would be good, but that's more than I can bite off right now.

[marlanc]

Yes, thanks much @gschivley! The additional information is very helpful! I am working with our system admin so haven't had a chance to work with him further on this issue. We'll try what solved it for you.

[fredtony]

I had a similar problem, but I was not able to resolve it using the suggestions above. In the end, I cloned the repo, modified setup.py so that on the last line, "__main__" calls:
try_run_setup(with_binary=False, test_xgboost=True, test_lightgbm=True)

I then ran "pip install -e ." from the command line. This was able to finish the installation without error. I saw that try_run_setup is supposed to do this automatically if the build raises an exception, but it seems to be failing altogether instead of raising an exception.

I hope this helps anybody else who runs into this problem.

[slundberg]

Thanks @fredtony! If I get time I'll have to look into why it didn't throw an exception.

[bichbich]

I have been trying to install shap and everytime i get the same error i tried even what @fredtony did and i didn't get any results

Traceback (most recent call last):
File "C:Users\wiemAnaconda3\envsdatascience\Scripts\pip-script.py", line 6, in
from pip import main
File "C:Users\wiemAnaconda3\envsdatascience\lib\site-packages\pip__init__.py", line 28, in
from pip.vcs import git, mercurial, subversion, bazaar # noqa
File "C:Users\wiemAnaconda3\envsdatascience\lib\site-packages\pip\vcs\mercurial.py", line 9, in
from pip.download import path_to_url
File "C:Users\wiemAnaconda3\envsdatascience\lib\site-packages\pipdownload.py", line 34, in
from pip.utils.logging import indent_log
File "C:Users\wiemAnaconda3\envsdatascience\lib\site-packages\pip\utils\logging.py", line 66, in
class ColorizedStreamHandler(logging.StreamHandler):
File "C:Users\wiemAnaconda3\envsdatascience\lib\site-packages\pip\utils\logging.py", line 72, in ColorizedStreamHandler
(logging.ERROR, _color_wrap(colorama.Fore.RED)),
AttributeError: module 'pip._vendor.colorama' has no attribute 'Fore'

[slundberg]

Looks like a problem with pip. Can you install anything else?

On Mon, Jul 2, 2018 at 7:14 AM bichbich [e-mail] wrote:

I have been trying to install shap and everytime i get the same error i
tried even what @fredtony https://github.com/fredtony did and i didn't
get any results

Traceback (most recent call last):
File "C:Users\wiemAnaconda3\envsdatascience\Scripts\pip-script.py",
line 6, in
from pip import main
File "C:Users\wiemAnaconda3\envsdatascience\lib\site-packages\pip_
init_.py", line 28, in
from pip.vcs import git, mercurial, subversion, bazaar # noqa
File
"C:Users\wiemAnaconda3\envsdatascience\lib\site-packages\pip\vcs\mercurial.py",
line 9, in
from pip.download import path_to_url
File
"C:Users\wiemAnaconda3\envsdatascience\lib\site-packages\pipdownload.py",
line 34, in
from pip.utils.logging import indent_log
File
"C:Users\wiemAnaconda3\envsdatascience\lib\site-packages\pip\utils\logging.py",
line 66, in
class ColorizedStreamHandler(logging.StreamHandler):
File
"C:Users\wiemAnaconda3\envsdatascience\lib\site-packages\pip\utils\logging.py",
line 72, in ColorizedStreamHandler
(logging.ERROR, _color_wrap(colorama.Fore.RED)),
AttributeError: module 'pip._vendor.colorama' has no attribute 'Fore'

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/slundberg/shap/issues/96#issuecomment-401819248, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADkTxdGaXgFf0VljexLzDOGzzdevfYFsks5uCiq2gaJpZM4UH8WT
.

[bichbich]

Thank you for your observation, actually no i can't. In fact after
activating my environment i get this (set "KERAS_BACKEND=theano" ) and i
think keras is making me in this trouble.

On Mon, Jul 2, 2018 at 5:14 PM, Scott Lundberg [e-mail]
wrote:

Looks like a problem with pip. Can you install anything else?

On Mon, Jul 2, 2018 at 7:14 AM bichbich [e-mail] wrote:

I have been trying to install shap and everytime i get the same error i
tried even what @fredtony https://github.com/fredtony did and i didn't
get any results

Traceback (most recent call last):
File "C:Users\wiemAnaconda3\envsdatascience\Scripts\pip-script.py",
line 6, in
from pip import main
File "C:Users\wiemAnaconda3\envsdatascience\lib\site-packages\pip_
init_.py", line 28, in
from pip.vcs import git, mercurial, subversion, bazaar # noqa
File
"C:Users\wiemAnaconda3\envsdatascience\lib\site-packages\
pip\vcs\mercurial.py",
line 9, in
from pip.download import path_to_url
File
"C:Users\wiemAnaconda3\envsdatascience\lib\site-packages\
pipdownload.py",
line 34, in
from pip.utils.logging import indent_log
File
"C:Users\wiemAnaconda3\envsdatascience\lib\site-packages\
pip\utils\logging.py",
line 66, in
class ColorizedStreamHandler(logging.StreamHandler):
File
"C:Users\wiemAnaconda3\envsdatascience\lib\site-packages\
pip\utils\logging.py",
line 72, in ColorizedStreamHandler
(logging.ERROR, _color_wrap(colorama.Fore.RED)),
AttributeError: module 'pip._vendor.colorama' has no attribute 'Fore'

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/slundberg/shap/issues/96#issuecomment-401819248,
or mute
the thread
ADkTxdGaXgFf0VljexLzDOGzzdevfYFsks5uCiq2gaJpZM4UH8WT>
.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/slundberg/shap/issues/96#issuecomment-401856695, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AYVGd9TLNgbBkvvyVySUYSZDHpdgn4LKks5uCkbbgaJpZM4UH8WT
.

[slundberg]

As of v0.20.1 there are now x64 Window's wheels on PyPI for Python 3.6 and 3.7 :) Let me know if they don't work.

[bichbich]

Thanks i will let you know

Le mar. 17 juil. 2018 à 5:18 PM, Scott Lundberg [e-mail]
a écrit :

As of v0.20.1 there are now x64 Window's wheels pn PyPI for Python 3.6 and
3.7 :) Let me know if they don't work.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/slundberg/shap/issues/96#issuecomment-405641011, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AYVGd_m-x4gxtzTzOQ-8rIPCNjFs0tg6ks5uHg5LgaJpZM4UH8WT
.

[Anggi-Permana-Harianja]

pip install shap==0.22.2 is the stable version of shap library

[luisaporrasm]

I have tried to install shap so many times, but I always get this error: cannot import name '_cext' from 'shap'. @slundberg Do you know what should I do?

[Syuukakou]

@luisaporrasm I get the error: C extension was not built during install! ....... cannot import name '_cext' from 'shap'
for this error, I uninstalled shap firstly via pip, and then

• installed the Cygwin, gcc via pip
• installed Microsoft vs 2019, and build tools for vs2019
• use command "pip3 install git+https://github.com/Syuukakou/shap.git" to install shap
  No Errors!!!
  So, I think this error occurred because of the c compiler environment not complete. (Maybe)

[qingfeng-ui]

@marlanc Installing from the repository did solve the problem for me. Tricky to determine the actual issue, but here's an overview of what happened.

1. Installed from PyPI via pip, got error.
2. Installed cython from conda, uninstalled/reinstalled shap from PyPI. Still got error.
3. Googled a bunch, determined that gcc might not be installed on my machine (mac). Installed xcode-select via command line to get a gcc compiler (LLVM version 9.1.0).
4. Uninstalled shap with pip. Tried installing from PyPI again. Still got error.
5. Uninstalled shap with pip. Used pip to install from the github repo. No error!

This was not scientific, and my memory could be off about the order of events.

Further testing just now:

• I installed shap via pip/PyPI (shap 0.18.0) in a separate conda environment and had no problems.
• Uninstalled 0.18.0 and installed 0.16.1 in this new conda environment, got error! I don't know enough about pip and conda environments to know if some cached files from my original flawed builds are being used.
• Uninstalled 0.16.1 and installed 0.17.1, no problems.

Hope this is of some use!

you are right, the version 0.18.0 is ok.
however, it has new issues.

AttributeError: 'TreeExplainer' object has no attribute 'expected_value'
I thought that the v0.20.0 is different with 0.18.0.