Shapely: Error installing shapely via pip on MacOS Big Sur and Python 3.9

[pvcraven]

Expected behavior and actual behavior.

Would like to see Shapely install on MacOS BigSur and Python 3.9

Steps to reproduce the problem.

pip install shapely

Operating system

MacOS Big Sur

Shapely version and provenance

Shapely 1.7.1

Details

Output from install:

(venv) @Pauls-Mac-mini arcade % pip install shapely
Collecting shapely
  Downloading Shapely-1.7.1.tar.gz (383 kB)
     |████████████████████████████████| 383 kB 4.5 MB/s 
    ERROR: Command errored out with exit status 1:
     command: /Users/paulcraven/PycharmProjects/arcade/venv/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/np/f6_qtx353lnblsj211k_291c0000gn/T/pip-install-k3hv2ed2/shapely/setup.py'"'"'; __file__='"'"'/private/var/folders/np/f6_qtx353lnblsj211k_291c0000gn/T/pip-install-k3hv2ed2/shapely/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/np/f6_qtx353lnblsj211k_291c0000gn/T/pip-pip-egg-info-spht05a3
         cwd: /private/var/folders/np/f6_qtx353lnblsj211k_291c0000gn/T/pip-install-k3hv2ed2/shapely/
    Complete output (12 lines):
    Failed `CDLL(/Library/Frameworks/GEOS.framework/Versions/Current/GEOS)`
    Failed `CDLL(/opt/local/lib/libgeos_c.dylib)`
    Failed `CDLL(/usr/local/lib/libgeos_c.dylib)`
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/np/f6_qtx353lnblsj211k_291c0000gn/T/pip-install-k3hv2ed2/shapely/setup.py", line 85, in <module>
        from shapely._buildcfg import geos_version_string, geos_version, \
      File "/private/var/folders/np/f6_qtx353lnblsj211k_291c0000gn/T/pip-install-k3hv2ed2/shapely/shapely/_buildcfg.py", line 190, in <module>
        lgeos = load_dll('geos_c', fallbacks=alt_paths)
      File "/private/var/folders/np/f6_qtx353lnblsj211k_291c0000gn/T/pip-install-k3hv2ed2/shapely/shapely/_buildcfg.py", line 162, in load_dll
        raise OSError(
    OSError: Could not find library geos_c or load any of its variants ['/Library/Frameworks/GEOS.framework/Versions/Current/GEOS', '/opt/local/lib/libgeos_c.dylib', '/usr/local/lib/libgeos_c.dylib']
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
(venv) paulcraven@Pauls-Mac-mini arcade % 

[sgillies]

This is a known issue. Dear everyone, please refrain from making "me too" comments. There is no need. Here is the heart of the matter: there is no binary wheel for any version of osx and python 3.9 on PyPI: https://pypi.org/project/Shapely/#files. We don't have a job for this yet in https://github.com/shapely/shapely-wheels/blob/master/.travis.yml#L86, but will overcome that soon.

And there's another issue, Python 3.9 is not ready for macOS 11.0. See https://github.com/pypa/packaging/pull/319#issuecomment-733290582. There's nothing the shapely project can do about that but wait and upgrade its build infrastructure after there are new releases.

[AlbertWigmore]

@sgillies Is there anything blocking the addition of a Python 3.9 build for Linux?

[sgillies]

@AlbertWigmore there were multiple blockers: I was tied up by something big at work and python 3.9 wheels required a new version of auditwheel, which required a shapely hotfix. Good news: Shapely-1.7.1-cp39-cp39-manylinux1_x86_64.whl is at https://pypi.org/project/Shapely/1.7.1/#files now. This also unblocks a release of rasterio 1.2b1 for Python 3.9.

On the macos front: our Travis CI infra can't build wheels for Python 3.9 and those will have to wait until we can switch to GitHub Actions or AppVeyor.

[Tom-Brouwer]

@AlbertWigmore there were multiple blockers: I was tied up by something big at work and python 3.9 wheels required a new version of auditwheel, which required a shapely hotfix. Good news: Shapely-1.7.1-cp39-cp39-manylinux1_x86_64.whl is at https://pypi.org/project/Shapely/1.7.1/#files now. This also unblocks a release of rasterio 1.2b1 for Python 3.9.

On the macos front: our Travis CI infra can't build wheels for Python 3.9 and those will have to wait until we can switch to GitHub Actions or AppVeyor.

@sgillies For me (Ubuntu/PopOS 20.04, Python 3.9) this wheel causes issues. I spend the last hour trying to find out why the Shapely 1.7.1 that I installed to a virtual environment last week was working, but the new virtual environment today with the same version was not working. I found it was because of the new wheel for shapely, which causes an OSError for me.

This seems to be because installing the wheel, causes a 'Shapely.libs' directory with .so files to be created in the 'site-packages' directory. When I delete the 'Shapely.libs' directory manually, it starts working again. Below a replication of the issue, by installing from source first, and then installing from the wheel.

To replicate, I first install it with the --no-binary, which means it builds from source (I have the 'libgeos-dev' ubuntu package installed). When I remove that and install from the wheel, the Shapely.libs directory is created, and an error occurs:

(.env) brouwer@thinkpad-tom:~/$ pip install shapely --no-cache-dir --no-binary :all:
Collecting shapely
  Downloading Shapely-1.7.1.tar.gz (383 kB)
     |████████████████████████████████| 383 kB 7.6 MB/s 
Skipping wheel build for shapely, due to binaries being disabled for it.
Installing collected packages: shapely
    Running setup.py install for shapely ... done
Successfully installed shapely-1.7.1

# Testing the import with this version, gives no errors:

(.env) brouwer@thinkpad-tom:~/$ python
Python 3.9.1 (default, Dec  8 2020, 02:26:20) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from shapely import geometry
>>> exit()

(.env) brouwer@thinkpad-tom:~/$ pip uninstall shapely
Found existing installation: Shapely 1.7.1
Uninstalling Shapely-1.7.1:
  Would remove:
    /home/brouwer/venv/lib/python3.9/site-packages/Shapely-1.7.1-py3.9.egg-info
    /home/brouwer/venv/lib/python3.9/site-packages/shapely/*
Proceed (y/n)? y
  Successfully uninstalled Shapely-1.7.1

(.env) brouwer@thinkpad-tom:~/OpenDaL/repositories/ingestion-and-transformation$ pip install shapely --no-cache-dir
Collecting shapely
  Downloading Shapely-1.7.1-cp39-cp39-manylinux1_x86_64.whl (1.0 MB)
     |████████████████████████████████| 1.0 MB 6.4 MB/s 
Installing collected packages: shapely
Successfully installed shapely-1.7.1

# Testing the import with this version, gives an OSError:

(.env) brouwer@thinkpad-tom:~/$ python
Python 3.9.1 (default, Dec  8 2020, 02:26:20) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from shapely import geometry
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/brouwer/venv/lib/python3.9/site-packages/shapely/geometry/__init__.py", line 4, in <module>
    from .base import CAP_STYLE, JOIN_STYLE
  File "/home/brouwer/venv/lib/python3.9/site-packages/shapely/geometry/base.py", line 19, in <module>
    from shapely.coords import CoordinateSequence
  File "/home/brouwer/venv/lib/python3.9/site-packages/shapely/coords.py", line 8, in <module>
    from shapely.geos import lgeos
  File "/home/brouwer/venv/lib/python3.9/site-packages/shapely/geos.py", line 81, in <module>
    CDLL(geos_whl_so[0])
  File "/usr/lib/python3.9/ctypes/__init__.py", line 375, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: libgeos--no-undefined-b94097bf.so: cannot open shared object file: No such file or directory
>>> 

Can you investigate this?

[aaliddell]

The Linux issue above is separate from the original issue in this thread. The OSX one appears during install due to no wheel, whilst the Linux one appears during load due to an issue with the Linux wheel. The below is specific to the Linux issue, which I think should be split to a separate issue:

At least for Linux, the error appears somewhat random as it depends on the return order of glob, which has no guarantees of determinism:

    geos_whl_so = glob.glob(
        os.path.abspath(os.path.join(os.path.dirname(__file__), ".libs/libgeos*.so*"))
    ) or glob.glob(
        os.path.abspath(
            os.path.join(
                os.path.dirname(__file__), "..", "Shapely.libs", "libgeos*.so*"
            )
        )
    )

    if len(geos_whl_so) > 0:
        CDLL(geos_whl_so[0])
        _lgeos = CDLL(geos_whl_so[-1])
        LOG.debug("Found GEOS DLL: %r, using it.", _lgeos)

If glob returns the filenames in one order, it will try to load the wrong .so file first and hence fail due to trying to load the second .so file automatically. If glob returns in the opposite order, the .so files will be loaded in the correct way and there will be no error.

I can't determine where the above code (copied from geos.py) has come from, as the tagged 1.7.1 version and master has a different set of code for loading the shared libs.

[sgillies]

@aaliddell you're right. See https://github.com/Toblerity/Shapely/pull/1047#issuecomment-745482654 about the fix.