Shap: Unable to install SHAP

[akhilsurya]

Command to install the library:
sudo pip3 install shap

Output of pip3 --version:
pip 9.0.3 from /usr/lib/python3.7/site-packages (python 3.7)

Environment:
Amazon Linux 2 on the master node of an Amazon EMR Cluster

Error and logs have been attached here - pip3_install_logs.txt. I was able to able to understand to the extent that, it is tied to the numba requirement. But I was not to figure out more details than that. Let me know if additional details are required.

Any help here is greatly appreciated.

[ManuelZ]

The following line makes me thing that you are missing the header files for Python (python3-dev in Ubuntu; I don't know about Amazon Linux 2).

  shap/_cext.cc:3:10: fatal error: Python.h: No such file or directory

I would start there.

Also your pip seems pretty old. Current version is 20.2, yours is 9.

[akhilsurya]

Seems like there are couple of issues going on including what @ManuelZ guessed.

1. Python.h is missing because of missing equivalent package of python3-dev. I had to use sudo yum install python3-devel -y to install
2. llvmlite is required because of shap's dependency on numba. Recently there have been changes due to which a minimum of pip-19 is required for installing llvmlite. See: https://github.com/numba/llvmlite/issues/531 Sadly Amazon Linux 2 comes with the really old pip-9. Solution is to either upgrade pip to a >19 version or use older versions of numba and llvmlite. Specifically sudo pip3 install llvmlite==0.31.0 && sudo pip3 install numba==0.48.0 && sudo pip3 install shap worked for me.

I'm not entirely clear how the issue could have been mitigated or the failure reason made clearer. I'll leave the issue open as I don't have much experience with dependency management in this context.

Feel free to close if there is not much scope for improvement of experience for example by shap's version also incorporating critical dependency version upgrades.