Nixpkgs: ctypes.util.find_library() is broken on Python 3.4, 3.5

[abbradar]

Issue description

ctypes.util.find_library() doesn't return anything on Python 3.4 and 3.5. This breaks python35.pkgs.cffi which is needed for TensorFlow on Python 3 (prebuilt TensorFlow doesn't support Python 3.6).

Steps to reproduce

 nixpkgs git:(master) ✗ nix-build -A python3 
/nix/store/g8prw6jy2yvrww8kfmsdyzja66dbm0sl-python3-3.6.2
 nixpkgs git:(master) ✗ result/bin/python
Python 3.6.2 (default, Jul 17 2017, 03:45:41) 
[GCC 6.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes.util
>>> ctypes.util.find_library("m")
'libm.so.6'
>>> 
 nixpkgs git:(master) ✗ nix-build -A python35
/nix/store/8lh4pxhhi4cx00pp1zxpz9pqyy44kjm6-python3-3.5.4
 nixpkgs git:(master) ✗ result/bin/python
Python 3.5.4 (default, Aug  7 2017, 08:13:11) 
[GCC 6.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ctypes.util
>>> ctypes.util.find_library("m")
>>> 

Technical details

• Nixpkgs version: (run nix-instantiate --eval '<nixpkgs>' -A lib.nixpkgsVersion) e960eb0bf579b1e5afc58a207539d46bf21b7416

cc @FRidh

[FRidh]

Yea, I noticed this. I think its caused by a7ddca6e3dce865a6a07cbdaea3963026b7bb3d3.

[Mic92]

What did this function returned before? I think we can hard-code this information as it seems to already set at build time (location of libc and so on)

[abbradar]

@Mic92 See result from python3 above (Python 3.6 works okay). Hardcoding this is impossible as find_library() should resolve libraries from LD_LIBRARY_PATH and other sources (if I understand correctly).

[roberth]

I have tried adding LD_LIBRARY_PATH to the cffi derivation, to no avail.

Also, reverting commit e960eb0bf579b1e5afc58a207539d46bf21b7416 does not change the test failures in cffi. My revert was on top of fbe46c7a2419.

[domenkozar]

@FRidh what's the plan? Should we revert the commit you linked?

[roberth]

@domenkozar Seems like I tested the wrong commit. Please ignore my previous comment.

[roberth]

@domenkozar @FRidh reverting that commit does fix cffi on 3.4 but it does not fix cffi on 3.5

[FRidh]

@roberth you would need to revert also a7ddca6e3dce865a6a07cbdaea3963026b7bb3d3, I suppose that it should be fine then.

[roberth]

@FRidh that's what I did today because I noticed my mistake. I have checked out release-17.09 (at commit 64a563f15b120c76d1379148e5dc6035d6ca9ee8) and then reverted a7ddca6. Doing so makes cffi build on python 3.4 but not on python 3.5. I am now building cffi on python 3.5 with d64e798c822343ab1bfe902c905f6fa460e61db7 reverted.

[roberth]

Ok, that solves the problem. Reverting both a7ddca6 and d64e798 fixes the problems on 3.4 and 3.5, but 3.6 does not need these reverts, so this needs some more manual work. Perhaps you can see what's different between these python versions, @FRidh ? I suppose it's best that you fix it, because you seem to be more familiar with these changes and the reasons behind.

[FRidh]

Yes, I'm going to have a look at this, but likely not until Wednesday.

[domenkozar]

FYI: this is blocking the darwin channel https://github.com/NixOS/nixpkgs/issues/29221#issuecomment-328515653

[domenkozar]

@FRidh sounds good :)

[FRidh]

Hardcoding this is impossible as find_library() should resolve libraries from LD_LIBRARY_PATH and other sources (if I understand correctly).

Either we hardcode paths (preferable) or we set LD_LIBRARY_PATH, in which case ctypes.utils.find_library should work.

FYI: this is blocking the darwin channel #29221 (comment)

That was unrelated (Python 3.6 cffi).

The no-ldconfig.patch seems to have caused this regression.

The reason is that prior to 3.6 LD_LIBRARY_PATH was not checked.
https://bugs.python.org/issue9998
It did work before, but maybe that was because it used gcc linker to find it? That code seems to have changed between 2.7 and 3.x.

In any case, we should be able to backport the portion in 3.6 that checks LD_LIBRARY_PATH.
https://github.com/python/cpython/blob/v3.6.2/Lib/ctypes/util.py#L278

[FRidh]

Note that on Darwin it does not check for DYLD_LIBRARY_PATH. We may in the future choose to support this.

[FRidh]

Fix in https://github.com/NixOS/nixpkgs/pull/29357.