Termux-packages: How to install Scikitlearn Sklearn

[dekubi]

@its-pointless @community
How do you install scikitlearn without getting errors when importing certain modules in termux

[ginacordova]

The below instructions are for installing scikitlearn on Android using the Termux app. I have Jupyter Notebook and Python 3 installed on my smartphone:

1. Install curl. Then run the below in Termux:

$ curl -L https://its-pointless.github.io/setup-pointless-repo.sh | sh

_This script essentially installs gnupg on your device and downloads and adds a public key to your apt keychain ensuring that all subsequent downloads are from the same source._

1. Then Run:
   $ pkg install scipy

_You need to have scipy installed to install scikit-learn. You also have to have about a dozen other things installed like numpy etc in order for scipy/scikit-learn to work. I would install python 3, jupyter, notebook, ipython, pyzmq, pandas, numpy, matplotlib first. As well as the latest version of pip before starting. I can send you a list of the packages with versions that I have installed on my Android if you need it._

1. Then Run:
   $pip3 install scikit-learn
   _This installs scikit-learn_

References I found helpful for this install:
http://www.leouieda.com/blog/scipy-on-android.html
http://bretahajek.com/2016/12/jupyter-notebook-opencv-android-machine-learning/
https://wiki.termux.com/wiki/Installing_Scipy_The_Easy_Way

[dekubi]

@ginacordova
"I can send you a list of the packages with versions that I have installed on my Android if you need it." I will be very happy if you do that

[dekubi]

@ginacordova
After following the steps, i still get this error when i try to import some modules.

"UserWarning: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.. joblib will operate in serial mode
warnings.warn('%s. joblib will operate in serial mode' % (e,)) "

[david-macleod]

@ginacordova I would also be interested if you could outline the exact steps you took to get the scikit-learn installation to work

I have followed the links and tried the following

$ apt install clang python python-dev fftw libzmq libzmq-dev freetype freetype-dev libpng libpng-dev pkg-config
$ curl -L https://its-pointless.github.io/setup-pointless-repo.sh | sh
$ pkg install scipy
$ pip install numpy matplotlib pandas

$ pip install scikit-learn

But the last step produces the error recorded here #https://github.com/termux/termux-app/issues/443

[ryanyuerong]

$ termux-chroot

Had the same issue for days and found a way out.
Pip install sklearn under chroot works.

[david-macleod]

@ryan15858 I have just tried this and can confirm it also worked for me, thanks!

[sogaiu]

After the various prerequisites, the following allows the installation portion to complete without errors:

LDFLAGS=" -lm -lcompiler_rt" pip install -U scikit-learn

Not sure if the LDFLAGS portion is necessary, but that's what I've been doing for pip-related things as mentioned in various places.

I do get the message mentioned above after importing, i.e.:

"UserWarning: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.. joblib will operate in serial mode
warnings.warn('%s. joblib will operate in serial mode' % (e,)) "

[jrgriffiniii]

Many thanks @its-pointless and @ryanyuerong , for those attempting this more recently, I found that the following works with a fresh install of termux:

pkg install python wget proot clang python-dev fftw libzmq libzmq-dev freetype freetype-dev libpng libpng-dev pkg-config 
wget https://its-pointless.github.io/setup-pointless-repo.sh
bash setup-pointless-repo.sh
pkg update
pkg install numpy scipy
termux-chroot
pip install --update pip
pip install scikit-learn==0.19.2

0.20.x releases fail due to https://github.com/scikit-learn/scikit-learn/issues/11378 for me :(

[JulianWangS]

@jrgriffiniii
I follow your steps, but it did not work.
When I import sklearn, report issue 3770 again.

[xeffyr]

When I import sklearn, report issue 3770 again.

It is not possible to fix UserWarning: This platform lacks a functioning sem_open implementation, therefore, the required synchronization primitives needed will not function, see issue 3770.. joblib will operate in serial mode warnings.warn('%s. joblib will operate in serial mode' % (e,)).

Termux doesn't implement sem_open() as this is libc/kernel thing.

[JulianWangS]

@xeffyr
So, it is not possible to use sklearn lib in termux?

[jrgriffiniii]

@JulianWangS the warning never prevented me from importing any modules, classes or the like from sklearn. At worst it can just be ignored in my environment. @xeffyr is this also the case for you?

[xeffyr]

So, it is not possible to use sklearn lib in termux?
At worst it can just be ignored in my environment. @xeffyr is this also the case for you?

It is possible to use sklearn, just ignore warning. Only multiprocessing will not work, everything else should be fine.

warning != error

[thebestnom]

Everytime I'm trying to do pip install scikit-learn it does

creating build/temp.linux-aarch64-3.8/sklearn/svm/src/liblinear
                                                 compile options: '-Isklearn/svm/src/liblinear -Isklearn/utils -I/data/data/com.termux/files/usr/lib/python3.8/site-packages/numpy-1.18.0-py3.8-linux-aarch64.egg/numpy/core/include -I/data/data/com.termux/files/usr/lib/python3.8/site-packages/numpy-1.18.0-py3.8-linux-aarch64.egg/numpy/core/include -I/data/data/com.termux/files/home/hafifa-env/include -I/data/data/com.termux/files/usr/include/python3.8 -c'
                                         extra options: '-fopenmp'                               Traceback (most recent call last):
                        File "/data/data/com.termux/files/usr/lib/python3.8/multiprocessing/synchronize.py", line 28, in <module>
         from _multiprocessing import SemLock, sem_unlink    ImportError: cannot import name 'SemLock' from '_multiprocessing' 
(/data/data/com.termux/files/usr/lib/python3.8/lib-dynload/_multiprocessing.cpython-38.so)

As an error and it stops installing (sorry for the bad console output)

Android 10
OnePlus 7t

[thebestnom]

Finally manged to install it!
To anyone whose getting this problem run
NPY_NUM_BUILD_JOBS=1 pip install scikit-learn

[sogaiu]

@thebestnom Haven't verified yet, but thanks for sharing :+1:

[thebestnom]

@sogaiu After testing it for a little bit importing the lastest version raises no sem lock, haven't managed to get rid of it... Also haven't tried older version (which had the same problem while installing)

Thinking about patching joblib to add try in the import multiprocess

[sogaiu]

@thebestnom i've tried:

NPY_NUM_BUILD_JOBS=1 pip install scikit-learn

a few times, but haven't successfully reached the end (may be due to timeouts).

Will make a few more attempts.

[thebestnom]

@sogaiu I would suggest adding -vv at the end for verbose output, and see why it's stopping, also make sure numpy, scipy and cython are already installed

[sogaiu]

@thebestnom Thanks for the tip. I'll do that next time.

What I remember ATM is something about mk_lrt not being available.

It's been a long time since I tried this so there's a good chance I didn't fulfill the prerequisites :)

[sogaiu]

Yes, inadequate prerequisites. Installation appears to have been successful. Thanks @thebestnom

For future strugglers: the -dev items mentioned in:

https://github.com/termux/termux-packages/issues/1618#issuecomment-445509037

didn't exist when I tried, so I left them out.

I also ended up doing pip install cython along the way before the final NPY_NUM_BUILD_JOBS=1 pip install scikit-learn worked.

[thebestnom]

@sogaiu have you managed to import it?

[jrgriffiniii]

Thank you very much @sogaiu, this worked for me with a reinstall.

@thebestnom I could only get it to import with version 0.19.2:

pip install scikit-learn==0.19.2

I receive a warning, but it does actually function for me. It is not https://github.com/scikit-learn/scikit-learn/issues/11378 as I suspected before (I was incorrect).

[thebestnom]

@jrgriffiniii yeah, guessed as much, will try to make a fix for newer versions

[jrgriffiniii]

I have no idea what I am doing, but the timing for https://github.com/scikit-learn/scikit-learn/pull/11741 looks like it confirms:

Thinking about patching joblib to add try in the import multiprocess

Please let me know if I can assist in any way, I have not really worked with any of these projects.

[thebestnom]

@jrgriffiniii downgrading joblib to version 0.11 seems to work on the latest scikit learn, looks like it is an already open issue on joblib
https://github.com/joblib/joblib/issues/825

[jrgriffiniii]

Thank you again @thebestnom, I can confirm that the following worked in my environment:

(Do not forget to termux-chroot)

pip uninstall joblib
# Do not specify a patch version, they don't use SemVer
pip install joblib==0.11
NPY_NUM_BUILD_JOBS=1 pip install scikit-learn

I would recommend this to others, as I now have no warnings logged when I import from the package.

[jrgriffiniii]

If anyone is available to review/correct https://github.com/joblib/joblib/pull/999 please, I would be most grateful. You can also test this locally with:

pip install git+https://github.com/jrgriffiniii/joblib.git@issues-825-jrgriffiniii-no-sem-support

[thebestnom]

@jrgriffiniii tried installing it, importing works! Currently I have no code testing it against, soim trying to run scikit-learn pytest on it!
Thank you very much!!

[sogaiu]

@thebestnom No, I hadn't successfully imported, but with @jrgriffiniii's joblib instructions, import succeeded :+1:

[sogaiu]

@jrgriffiniii https://github.com/termux/termux-packages/issues/1618#issuecomment-578530963 seems to work here so far in local testing -- at least importing works :)

[thebestnom]

@jrgriffiniii did some testing by running notebooks from https://github.com/ogrisel/notebooks, and everything looks to work fine (after updating the old code), thanks again!

[jrgriffiniii]

I need to still write a test for https://github.com/joblib/joblib/pull/999, and then request a release. Sorry for falling behind on this, work issues arose and kept me from returning to this.

[dekubi]

Guys check this out https://gist.github.com/thetdg/53ed30126078a16b49a2fff9094f3524#file-install_python_termux_android-txt and @jrgriffiniii your fix works

[cuze]

@jrgriffiniii Thanks, I success installed sklearn.

[gengishkhan]

@jrgriffiniii Thanks. I confirm it works!