Pysal: Fedora 33 with LLVM 11 and Python 3.9 breaks giddy and splot installation

[rsbivand]

The pysal metapackage fails in part to install on Fedora 33. The cause is that llvmlite, used by numba, quantecon, and then by giddy and splot, has not been extended to LLVM 11. Other Fedora components use LLVM 11, so multiple installs are not advisable. I have not been able to reconstruct the dependency graphs, so do not know whether the use of llvmlite is directly in giddy and splot, or via quantecon and numba, or numba seperately. It would be advisable to make at least some of the dependencies conditional. Note that I'm only using pip, and will never consider conda or similar mechanisms, so suggesting such a workaround isn't helpful.

[rsbivand]

The problem is in giddy, which splot then uses. The ergodic part of giddy uses quantecon but instead of checking that import is possible and working around its absence/failure, it makes import unconditional. I'm pretty certain that nothing in giddy needs the JIT compiler offered by numba and llvmlite, and used (also unconditionally) by quantecon. Can giddy make quantecon import conditional, and still install even if quantecon is missing (with omitted functionality)?

[weikang9009]

@rsbivand thank you for pointing this out!

I think it is reasonable to make quantecon import conditional especially if pysal is planning to support Python 3.9 very soon. Since I was absent from the recent dev meetings and do not about our plan, I would like to solicit comments/suggestions from other devs/maintainers @sjsrey @ljwolf @jGaboardi @knaaptime @darribas.

[ljwolf]

Sure, I think that's reasonable! In esda & libpysal, we make anything that relies on numba (and thus llvmlite) an optional import. It's tough, because that definitely complicates the code writing & maintenance a little bit, but I think it's generally a good idea to try to make these dependencies function-level, rather than module-level.

It looks like it's only used in steady_state() and fmpt()?

[ljwolf]

No, I'm wrong: it's used in the underlying Markov object, too, which is basically fundamental to the package.

@rsbivand have you filed this with quantecon?

Even if we can make a reasonable work around (which I think would entail disabling or re-implementing the Markov classes if quantecon is not available, as well as ergodic.py stuff), the issue is fundamentally with quantecon, numba, and llvmlite?

[rsbivand]

No, I have not - the problem arose on installing pip install pysal on a factory-fresh Fedora 33, which comes with LLVM 11 and Python 3.9. The problem is with llvmlite/numba, not even quantecon. Could you ask quantecon to condition on available upstream components? I certainly have insufficient insight to do so, and no willingness to shift to a conda or similar which would move me further from my base platform. CRAN is much better at getting package maintainers to pre-load fixed to system changes before they happen.

[jGaboardi]

I was absent from the recent dev meetings and do not about our plan

@weikang9009 There is no official plan, just something @martinfleis and I started trying.

[martinfleis]

I would say that this should be resolved in upstream packages. PySAL does not work (yet) with Python 3.9 because its dependencies do not support it. Although we can make some of them optional, it is not the ultimate solution. Not even conda-forge migration to Python 3.9 has finished yet (https://conda-forge.org/status/#python39) and PyPI wheels may take even longer.

[rsbivand]

OK, so I was lucky to find that geopandas and almost all of pysal work (for what I need), and the only breakage was because the JIT compiler numba/llvmlite combination needs LLVM < 11. I shouldn't have expected package upstream (reverse) dependencies to be prospectively preparing for new versions (here llvmlite for LLVM 11, probably irrespective of Python3 version). And I think fiona installed against my source-built local GDAL, since:

>>> import fiona
>>> fiona.get_gdal_release_name()
'3.2.0'

So this is a case of someone used to the R package cathedral being confused by a Python bazaar, maybe. Our current scrambling is around Big Sur (any architecture), and getting a Fortran compiler to work on the Apple Silicon architecture.