Termux-packages: Python 3.5 -> 3.6 upgrade breaks everything.

[tigran123]

I have done "apt upgrade" today and discovered that python has moved to the ultra-latest 3.6 (I hoped that 3.5 would still be "current" for at least 10 years, just like "2.x" is still current in some circles and will probably remain for generations to come).

Anyway, I am happy to discover that $PREFIX/lib/python3.5 is still there (with site-packages containing all the precious packages like numpy and scipy) but the python3.5 binary is gone, so how do I install it in order to use the modules compiled for 3.5?

I will of course try to re-compile the modules under 3.6 but knowing how extremely difficult (nearly impossible) it was to compile for 3.5 I have little hope of success under 3.6...

[tigran123]

Just as I suspected, python 3.6.0 is useless, look (this is after doing pip install numpy):

$ python3 -c "import numpy"
Traceback (most recent call last):
  File "/data/data/com.termux/files/usr/lib/python3.6/site-packages/numpy/core/__init__.py", line 16, in <module>
    from . import multiarray
ImportError: dlopen failed: cannot locate symbol "__unorddf2" referenced by "/data/data/com.termux/files/usr/lib/python3.6/site-packages/numpy/core/multiarray.cpython-36m-aarch64-linux-android.so"...

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/data/data/com.termux/files/usr/lib/python3.6/site-packages/numpy/__init__.py", line 142, in <module>
    from . import add_newdocs
  File "/data/data/com.termux/files/usr/lib/python3.6/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/data/data/com.termux/files/usr/lib/python3.6/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/data/data/com.termux/files/usr/lib/python3.6/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/data/data/com.termux/files/usr/lib/python3.6/site-packages/numpy/core/__init__.py", line 24, in <module>
    raise ImportError(msg)
ImportError: 
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.

So I need a way to bring back python 3.5 binary. Or, better, remember how I compiled numpy before --- something to do with "OPT=-ffast-math" or something like that? Anyone remembers what was the magic line to prefix "pip install numpy" in order to succeed in Termux environment?

[tomaklutfu]

Check out https://github.com/termux/termux-packages/issues/136

[tigran123]

No, the #136 doesn't help here, but compiling numpy with "OPT=-ffast-math" solved the numpy problem. But scipy still doesn't compile, so we need the magic which @its-pointless did in the issue #605 when he compiled and put in

https://github.com/its-pointless/gcc_termux/tree/master/required_armlibs

Oh wait, look in the above URL! He did his magic for Python 3.6 already yesterday! Hoorah! I'll try his packages and see if they work :)

UPDATE: No, those are for armv7l architecture only, not aarch64. So, I'll wait until he does his magic for aarch64 as well.

[its-pointless]

yeah i will around to it today shouldn't be too long.

[tigran123]

Thank you, I didn't realise they went into a different directory, so I was still waiting :) I'll test the aarch64 versions (and armv7l as well) versions now.

[tigran123]

@its-pointless Unfortunately, it doesn't work -- openblas refuses to install:

$ md5sum openblas_0.19_aarch64.deb 
6e9903e560dd4ac4433cedea0b04797c  openblas_0.19_aarch64.deb
$ dpkg -i openblas_0.19_aarch64.deb 
dpkg: error processing archive openblas_0.19_aarch64.deb (--install):
 parsing file '/data/data/com.termux/files/usr/var/lib/dpkg/tmp.ci/control' near line 8 package 'openblas':
 end of file before value of field 'Conflicts' (missing final newline)
Errors were encountered while processing:
 openblas_0.19_aarch64.deb

Should I force the installation? Or is this a corrupted deb file --- please check the MD5 checksum above.

[its-pointless]

Looks like i messed up with packaging because i was in a hurry. installing it probably won't hurt if it doesn't extract dpkg -x openblas_0.19_aarch64.deb ./ and copy over the file... i should of tested it. Anyway i will upload a proper one shortly.

[tigran123]

Ah, I see, it's just libopenblas.so inside --- ok, I copied it to $PREFIX/lib. Let's see what happens next... (I need to install matplotlib first, before I can test them)

[tigran123]

Thank you, I confirm that on aarch64 everything is working absolutely fine. Thank you very much. I'll test armv7l architecture later.

[tigran123]

Works fine on armv7l as well --- THANK YOU VERY MUCH! :)

[tigran123]

Oh, I spoke too soon concerning armv7l --- on it I get a crash from matplotlib:

ImportError: dlopen failed: cannot locate symbol "cos" referenced by "_qhull.cpython-36m-arm-linux-androideabi.so"...

So, I'll try to remember if there was some magic prefix for installing matplotlib on armv7l (like OPT=-ffast-math or something) and re-install it. Luckily I always edit HISTSIZE and HISTFILESIZE to contain at least a few billions last commands :)

LDFLAGS=" -lm -lcompiler_rt" pip install matplotlib

Yes, the above invocation fixes everything!

[yaman1987]

tigran123 please can you mention the step to solve this problem because i am a beginner in python
and i am using python 3.5.0b1 and installed numpy successfully but when i try to import it i got this error :+1: < Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try git clean -xdf (removes all
files not under version control). Otherwise reinstall numpy>>

[tigran123]

@yaman1987 Yes, that error is what you get if you try to do "pip install numpy" manually. You shouldn't do that. Instead, you should install both numpy and scipy packages prepared by @its-pointless in his repository here:

https://github.com/its-pointless/gcc_termux/tree/master/required_aarch64libs

or here:

https://github.com/its-pointless/gcc_termux/tree/master/required_armlibs

(one is for aarch64 architecture and the other is for armv7l architecture).

After you have done this you should install matplotlib using the special parameters that I pointed out above, i.e.:

LDFLAGS=" -lm -lcompiler_rt" pip install matplotlib

After all these steps you will have a working numpy+scipy+matplotlib. But the other important packages (mpmath, sympy, pandas etc) you can install via "pip install" with no extra magic involved --- they just work.

There is another important package PyFFTW which @its-pointless compiled, but when I tried it --- it didn't work (i.e. was 10x slower than scipy.fftpack when it should be about 1.5x faster), so I didn't pursue this further.

[sem-geologist]

how to install from egg?

[tigran123]

@sem-geologist Use easy_install file.egg

[yaman1987]

the problem was solved
thanks so much bro

On Sun, Jan 29, 2017 at 5:04 AM, Tigran Aivazian [e-mail]
wrote:

@yaman1987 https://github.com/yaman1987 Yes, that error is what you get
if you try to do "pip install numpy" manually. You shouldn't do that.
Instead, you should install both numpy and scipy packages prepared by
@its-pointless https://github.com/its-pointless in his repository here:

https://github.com/its-pointless/gcc_termux/tree/
master/required_aarch64libs

or here:

https://github.com/its-pointless/gcc_termux/tree/master/required_armlibs

(one is for aarch64 architecture and the other is for armv7l architecture).

After you have done this you should install matplotlib using the special
parameters that I pointed out above, i.e.:

LDFLAGS=" -lm -lcompiler_rt" pip install matplotlib

After all these steps you will have a working numpy+scipy+matplotlib. But
the other important packages (mpmath, sympy, pandas etc) you can install
via "pip install" with no extra magic involved --- they just work.

There is another important package PyFFTW which @its-pointless
https://github.com/its-pointless compiled, but when I tried it --- it
didn't work (i.e. was 10x slower than scipy.fftpack when it should be about
1.5x faster), so I didn't pursue this further.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/termux/termux-packages/issues/687#issuecomment-275904418,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AWMXlvMX4QQcbmpr490Gnmuc4cUfELAkks5rXGQhgaJpZM4Lpnu0
.

[sem-geologist]

Well I get error with numpy import:
```
$ python
Python 3.6.0 (default, Jan 19 2017, 03:36:19)
[GCC 4.2.1 Compatible Android Clang 3.8.256229 ] on linux
Type "help", "copyright", "credits" or "license" for more information.

import numpy
Traceback (most recent call last):
File "/data/data/com.termux/files/usr/lib/python3.6/site-packages/numpy-1.12.0-py3.6-linux-aarch64.egg/numpy/core/__init__.py", line 16, in
from . import multiarray
ImportError: dlopen failed: library name "/data/data/com.termux/files/usr/lib/python3.6/site-packages/numpy-1.12.0-py3.6-linux-aarch64.egg/numpy/core/multiarray.cpython-36m-aarch64-linux-android.so" too long

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "/data/data/com.termux/files/usr/lib/python3.6/site-packages/numpy-1.12.0-py3.6-linux-aarch64.egg/numpy/__init__.py", line 142, in
from . import add_newdocs
File "/data/data/com.termux/files/usr/lib/python3.6/site-packages/numpy-1.12.0-py3.6-linux-aarch64.egg/numpy/add_newdocs.py", line 13, in
from numpy.lib import add_newdoc
File "/data/data/com.termux/files/usr/lib/python3.6/site-packages/numpy-1.12.0-py3.6-linux-aarch64.egg/numpy/lib/__init__.py", line 8, in
from .type_check import *
File "/data/data/com.termux/files/usr/lib/python3.6/site-packages/numpy-1.12.0-py3.6-linux-aarch64.egg/numpy/lib/type_check.py", line 11, in
import numpy.core.numeric as _nx
File "/data/data/com.termux/files/usr/lib/python3.6/site-packages/numpy-1.12.0-py3.6-linux-aarch64.egg/numpy/core/__init__.py", line 24, in
raise ImportError(msg)
ImportError:
Importing the multiarray numpy extension module failed. Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try git clean -xdf (removes all
files not under version control). Otherwise reinstall numpy

[sem-geologist]

It says the link is too long.. wtf... it is android not windows/dos wtf....?

[dekubi]

@its-pointless @tigran123 It seems cool if you can build a custom #scikit-learn to integrate with the #numpy and #scipy to complete the #Data+Machine Learning cycle.
Looking up to you guys, Thanks

[ahartikainen]

Hey, the problem with too long comes from that the maximum path on termux / python can be only 250-255 (or something like this). I had the same kind of problem with tmp folder. The solution was to move it manually lower in the folder tree.

This + your .egg folder are too long.

/data/data/com.termux/files/usr/lib/

[fornwall]

@sem-geologist @ahartikainen The problem with dlopen and paths longer than 128 chars was a system limitation in Android 5.0, and was fixed for Android 6.0 and later (reference: https://github.com/android/platform_bionic/commit/aae859cc3ca127d890e853cbf12b731e05624a22).

So, the workaround to reduce path length should only be needed on Android 5.0.

[ahartikainen]

@fornwall thanks for the info. I did not know that limit was that low.