Spksrc: Python 2.7.14 fails to load for ARM

[ymartin59]

@Safihre I would like to track this specific issue here to get it fixed

Related issues: #3201

<<
It's not clear to me if it's in cross/cryptography or in cross/openssl.
The compilation of cross/cryptography is supposed to pick-up the same flags as Python (for example -pthread), but for some reason it doesn't.

>

References:

• https://github.com/pyca/cryptography/issues/3344

My understanding is that cryptography wheel build from requirements.txt dynamically links with DSM original openssl library (.so) instead of package specific version - due to lacking rpath option in binary.

My proposal is to either be able to pass arguments to pip when building wheels, or to get back cross/cryptography you have removed when upgrading.

[Safihre]

I already re-added the cross/cryptography in all the test builds that I created for people #3201. It doesn't work.
But at least then you can see the compilation steps (it doesn't show when using the requirements.txt) which shows that it's not picking up the same flags as the compilation of Python.
The error is specially in the _openssl of the Cryptography files, not in Openssl itself it seems to say in the errors. The _openssl.c file gets dynamically created when setup.py is run.
I tried forcing -pthreads everywhere using the flags, both in Python and in the cross compile of Python packages but the setup.py just wasn't picking them up.
It should be picked up from the _sysconfigdata.py that's generated during Python compilation. But even when I get those flags in there during Python compilation, it ignores them during the module building.

[Safihre]

Or maybe indeed it needs the 'rpath`, didn't think of that! I also don't see that being picked up during module compilation.
Not an expert really in these compilation things :/

[ymartin59]

My understanding:

• there is no trouble at compilation, custom "openssl" is used and expects symbol "pthread_atwork"
• at runtime, libraries fails to find symbol probably because linking with DSM openssl

To confirm with a test case, you may try to run Python with LD_LIBRARY_PATH to /var/packages/python/target/lib. If failure wanish, then adding RPATH to cryptography binary is the right way to fix

[ymartin59]

@Safihre Have you got any "simple" test case to run as basic "python" command line to reproduce trouble?

[Safihre]

The trouble is that it only happens on some platforms, for example not on my 88f681 or on the x64 when I tested. The whole reason I got the 88f681 system was to test this bug, but doesn't seem to apply 😞

This simple import should trigger it:

    from cryptography.hazmat.bindings._openssl import ffi, lib

[ymartin59]

Probably it depends on how DSM openssl version has been built...
What are impacted architectures? monaco, armada...?

[Safihre]

That's the only one that were reported so far. Didn't you have Evansport? It works fine on that one?

[ymartin59]

Seems to work fine:

yma@DiskStation:~$ /usr/local/python/bin/python -c "from cryptography.hazmat.bindings._openssl import ffi, lib"
yma@DiskStation:~$ 

[kantjer]

This is the output on my DS216PLAY

root@DS216PLAY:~# /usr/local/python/bin/python -c "from cryptography.hazmat.bindings._openssl import ffi, lib"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: /usr/local/python/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so: undefined symbol: pthread_atfork
root@DS216PLAY:~#

[ymartin59]

@kantjer Thanks for reporting. I am reviewing requirements for this method availability according to libc and kernel versions... probably some architecture runs quite old versions.

[Safihre]

@ymartin59 can @kantjer maybe try the LD_LIBRARY_PATH method? Is it just as simple as prepending it before the executable?

[ymartin59]

Some status about investigations done on x64: inside wheel wheelhouse/cryptography-2.1.4-cp27-none-any.whl (a simple zip file)
After unzip in temporary location

ldd ./cryptography/hazmat/bindings/_openssl.so
    linux-vdso.so.1 (0x00007ffd641c1000)
    libssl.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.0 (0x00007f12af848000)
    libcrypto.so.1.0.0 => /usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0 (0x00007f12af44c000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f12af0a0000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f12aee9c000)
    /lib64/ld-linux-x86-64.so.2 (0x000055a3db364000)

What is wrong here is lacking of libpthread dependency, whereas symbols are expected:

strings ./cryptography/hazmat/bindings/_openssl.so | grep pthread
pthread_mutex_init
pthread_mutex_unlock
pthread_mutex_lock
pthread_atfork

I guess it may work because of libpthread linking from Python itself

ldd ./install/var/packages/python/target/lib/libpython2.7.so
    linux-vdso.so.1 (0x00007ffc37b31000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fcf235e2000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fcf233de000)
    libutil.so.1 => /lib/x86_64-linux-gnu/libutil.so.1 (0x00007fcf231da000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fcf22ed9000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fcf22b2e000)
    /lib64/ld-linux-x86-64.so.2 (0x000055fc1f35b000)

From my point of view, build chain of cryptography's native parts is lacking -lpthread linkage option... leading to our trouble on some architecture where loader may have more strict behavior than on Intel.
It probably makes sense to report (again?) upstream to pyca maintainers.

[Safihre]

What if you do the same analysis on the armada version?

[ymartin59]

@Safihre Problem is that ldd, readelf or objdump are not available on stock DSM... it would require a full build chain with compilation tools on DSM (maybe from entware-ng-3x...)

[Safihre]

So let's assume then that it should work by picking up from core Python but doesn't, what if we try your linking trick?

[bjorn1981]

I've subscribed to this issue. When the time comes, I can test a package for apollolake DS118.

[Safihre]

@ymartin59 What do we do? 😢
Python 2.7.15 is coming and I want to get SAB 2.3.2 (and 2.3.3, which is coming soon) to our users.
How do we debug the wrong-links? Can users do something?

[echel0n]

It be nice if we can get a run down of any packages that get updated so we can adjust accordingly our requirements.txt files please

[ymartin59]

@Safihre @hgy59 For information, I am testing LDFLAGS="-lpthreads" in mk/spksrc.wheels.mk for "pip wheels" command with "monaco" arch to confirm generated so files now loads currently lacking libpthreads. I may be wrong but from my point of view, this is a trouble with ARM toolchains... When gcc "-pthreads" option is used, it is expected that linker also include reference to libpthreads. Notice libraries generated in wheels have libpthread dependency for x86 and ppc architectures.

[Safihre]

Aah, yes makes sense.

[Safihre]

Maybe also push 2.7.15 and new OpenSSL :)
https://www.python.org/downloads/release/python-2715/

[ymartin59]

@kantjer Here is a test package for monaco/DS216play, I hope include proper fix for cryptography / openssl / pthread to work: http://dl.free.fr/h66zz8ueM
May you please install it manually and test again Deluge to confirm it fixes this issue?

[kantjer]

@ymartin59 thanks I will try the test package when I'm back home.

[kantjer]

@ymartin59 unfortunately Deluge will still not start

https://hastebin.com/qepuxaqoxu.sql

[ymartin59]

@kantjer I ask for your help and it requires some expertise. Please first install "Diagnostic Tools" packages with command "synogear" as documented in https://github.com/SynoCommunity/spksrc/wiki/FAQ-synogear

When done, still in root shell after synogear install, please report output of command ldd /var/packages/python/target/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so

Please also report md5sum /var/packages/python/target/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so so that I can confirm it is expected version.

[kantjer]

@ymartin59

ldd /var/packages/python/target/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so

libssl.so.1.0.0 => /var/packages/python/target/lib/libssl.so.1.0.0 (0x76e59000)
libcrypto.so.1.0.0 => /var/packages/python/target/lib/libcrypto.so.1.0.0 (0x76ccb000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x76c9f000)
libc.so.6 => /lib/libc.so.6 (0x76b62000)
libdl.so.2 => /lib/libdl.so.2 (0x76b4f000)
/lib/ld-linux.so.2 (0x76f6e000)

md5sum /var/packages/python/target/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so

77616f6c6150884fe4a71d0c1821b48d  /var/packages/python/target/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so

[ymartin59]

@kantjer Here is yet another trial: http://dl.free.fr/czffD5Pjz

Still after "sudo synogear install", still as root:

• strace /usr/local/python/bin/python -c "from cryptography.hazmat.bindings._openssl import ffi, lib" 2> strace-import-crypto-openssl.log
• ldd /usr/local/python/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so

If properly fixed, ldd should report linkage to libpthread and python command should return without any error message. If any error, please attach produced log file strace-import-crypto-openssl.log to a comment here.

Many thanks for your help

[kantjer]

@ymartin59 installed Python test build and now Deluge is installed and running fine.

ldd /usr/local/python/lib/python2.7/site-packages/cryptography/hazmat/bindings/_openssl.so

libpthread.so.0 => /lib/libpthread.so.0 (0x76ec5000)
libssl.so.1.0.0 => /var/packages/python/target/lib/libssl.so.1.0.0 (0x76e66000)
libcrypto.so.1.0.0 => /var/packages/python/target/lib/libcrypto.so.1.0.0 (0x76cd8000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x76cac000)
libc.so.6 => /lib/libc.so.6 (0x76b6f000)
/lib/ld-linux.so.2 (0x76fa3000)
libdl.so.2 => /lib/libdl.so.2 (0x76b5c000)

FIY, I'm running DSM 6.1.7-15284
Other Python dependant packages like "SickBeard Custom" and "NZBHydra" are also running fine.

[ymartin59]

Great. I try to publish it today

[Safihre]

Woohoo! 🤗

[ymartin59]

I fear this python update may break other online packages, not yet published for DSM 6.x... but it is expected that deploying beta/test versions should be OK.

[Safihre]

Why do you think it might break them? If they just use the python command?

[ymartin59]

We already had surprised, didn't we? For instance old application version may be incompatible with recent dependency wheels in Python package...

[kantjer]

I'm running "SickBeard Custom" and "NZBHydra" and now Deluge all from post #3138

[ymartin59]

... problems I expect concern running other applications from online repository (built for DSM 5.x and previous - but not available yet in #3138) on that new python package.

[Safihre]

I can make some time to convert packages to new style packages, if they get broken bij this update! :)