Shapely needs a patch to work with conda on OS X

[shoyer]

I'm not 100% what the right solution is for shapely (which is why this issue is not a PR), but we needed to apply this patch to get Shapely to successfully build a conda package on OS X that uses the conda install of libgeos:
https://github.com/TheClimateCorporation/conda-recipes/blob/cd57916a02a4f97b71ca5937ee98e1499f131c73/shapely/osx-geos.patch

It would be nice to get a fix for this in Shapely. Here is a discussion with more context for this fix:
https://groups.google.com/a/continuum.io/d/msg/conda/kw2xC4wjI-Y/wDHMYeTZDeEJ

Note that in my version of the patch, I needed to change it look in sys.prefix _before_ the other locations, because I had another, non-conda version of geos installed (installed via brew) which shapely was linking to instead.

CC @ilanschnell @pelson @rsignell-usgs

[sgillies]

@shoyer I'd like to help you out, but it looks like your patch stomps on two other categories of users (py2app and QGIS). Can you find a way to make Conda builds work with our existing users or maybe suggest a better general approach?

[shoyer]

@sgillies I know, that patch only works for conda (obviously!).

For whatever reason, ctypes.util.find_library finds the wrong version of libgeos (instead of the version we want, which is in sys.prefix). Thus, it doesn't even suffice to add sys.prefix + '/lib' to alt_paths.

@asmeurer do you have suggestions for the "right way" to handle this so conda can work along with other packaging systems?

[pelson]

I've just had a quick look at this, something like the following would do the trick for the conda:

diff --git a/shapely/geos.py b/shapely/geos.py
index 20945bb..a7b9853 100644
--- a/shapely/geos.py
+++ b/shapely/geos.py
@@ -71,7 +71,13 @@ elif sys.platform == 'darwin':
             # macports
             '/opt/local/lib/libgeos_c.dylib',
         ]
-    _lgeos = load_dll('geos_c', fallbacks=alt_paths)
+    # By default, try to get the libgeos dylib in the sys prefix (as exists
+    # with conda), otherwise fall back to the load_dll mechanism.
+    dylib_path = os.path.join(sys.prefix, 'lib', 'libgeos_c.dylib')
+    if os.path.exists(dylib_path):
+        _lgeos = CDLL(dylib_path)
+    else:
+        _lgeos = load_dll('geos_c', fallbacks=alt_paths)
     free = load_dll('c').free
     free.argtypes = [c_void_p]
     free.restype = None

I'm not sure if that prevents some usecases such as allowing system variables to override load_dll (e.g. does LD_LIBRARY_PATH change the behaviour of load_dll?).

[sgillies]

@pelson yes, LD_LIBRARY_PATH does impact load_dll (because it calls ctypes find_library()). Which it should, and I've got deployed applications that count on it.

I think the key is to understand why find_library() doesn't work with conda's libgeos. If the answer is because it's incompatible with find_library() then we make conda (and sys.prefix) a special case in load_dll(), not the primary.

[asmeurer]

Is this issue with OS X, Linux, or both?

[shoyer]

Only on OS X, apparently. Our Linux builds work fine. No idea why that makes a difference. This might also be related to using the homebrew geos in particular.

On Fri, Sep 19, 2014 at 8:53 AM, Aaron Meurer [email protected]
wrote:

Is this issue with OS X, Linux, or both?

Reply to this email directly or view it on GitHub:
https://github.com/Toblerity/Shapely/issues/177#issuecomment-56196046

[pelson]

Is this issue with OS X, Linux, or both?

We've found this on OSX.

The python docs state:

On Linux, find_library() tries to run external programs (/sbin/ldconfig, gcc, and objdump) to find the library file. It returns the filename of the library file.

On OS X, find_library() tries several predefined naming schemes and paths to locate the library, and returns a full pathname if successful.

So it looks like find_library on OSX is not particularly clever, which I'd guess was the problem...

[asmeurer]

find_library on OS X is quite different than on Linux https://docs.python.org/2/library/ctypes.html#finding-shared-libraries.

[asmeurer]

It's all written in Python, so you can look at the source code to see what it is doing (https://hg.python.org/cpython/file/c0b0dda16009/Lib/ctypes/util.py and https://hg.python.org/cpython/file/c0b0dda16009/Lib/ctypes/macholib/dyld.py).

[asmeurer]

Maybe the Anaconda Python should add its lib to the DEFAULT_LIBRARY_FALLBACK at the top of that file. @ilanschnell thoughts?

[asmeurer]

Can someone verify if adding "%s/lib" % sys.prefix to the DEFAULT_LIBRARY_FALLBACK in lib/python2.7/ctypes/macholib/dyld.py (or lib/python3.4/ctypes/macholib/dyld.py) fixes this problem? Maybe we should open a bug at bugs.python.org about this. I didn't see any related bugs in my searches.

[pelson]

Can someone verify if adding "%s/lib" % sys.prefix to the DEFAULT_LIBRARY_FALLBACK

Confirmed.

Maybe we should open a bug at bugs.python.org about this

Agreed. I also think we should fix the file for any conda builds of Python. Want me to submit a PR against https://github.com/conda/conda-recipes?

[asmeurer]

I'm not clear what change you have in mind but feel free to open a pull request.

[pelson]

I'm not clear what change you have in mind but feel free to open a pull request.

I was proposing modifying the dyld.py file in the CPython recipe, but alas, there is no CPython recipe in https://github.com/conda/conda-recipes, as far as I can see.

[pelson]

:+1: - I agree. This is not a shapely issue, though it clearly manifests itself as a problem when using Shapely (specifically the ctypes find_library function). I think we need to raise a bug with python.org for this, and in the short/medium term we can try to find a fix for the conda python build (@shoyer's patch for OSX for example). I also have a build of geos and shapely at binstar.org/pelson which fixes the problem (built from github.com/SciTools/conda-recipes-scitools).

[pelson]

For what it is worth, between v1.5.7 and master (specifically 2212deb2105d380345c9b558f8f4c3d3be903496) this is now fixed.

For my own records, to reproduce with conda for the bad versions:

from shapely import speedups
speedups.enable()
from shapely.geometry import LineString
LineString([(0, 0), (10, 0)]).wkt

Liking the new libgeos.py module! :tada:

[rhattersley]

Maybe we should open a bug at bugs.python.org about this. I didn't see any related bugs in my searches.

Did this ever happen?

[ilanschnell]

When we build shapely for Anaconda, we always apply a patch which ensures that it is linking against <sys.prefix>/lib/libgeos_c.dylib. It would be nice, if shapely would always try to look at his location for the geos library.

[MichaelCPell]

I've tried

1. adding "%s/lib" % sys.prefix to the DEFAULT_LIBRARY_FALLBACK of my python "macholib/dyld.py"
2. installing shapely with conda install -c SciTools shapely

and am still on square one. I've also exhausted SO. Has anyone been able to install shapely with anaconda on Mac 10.10?

My particular error shows as ImportError: No module named geometry

[shoyer]

@MichaelCPell Try the ioos and conda-forge channels as well. @ocefpaf has does some impressive work getting conda builds of the Python geoscience stack working reliably.

[ocefpaf]

BTW go straight to conda-forge. (The IOOS channel is deprecated now.)

Here is the patch in case you want to roll your own conda solution.