Ray: On Mac OS X, importing ray before numba dies with llvm error: "Expected all immutable passes to be initialized"

[olivierlefloch]

System information

• OS Platform and Distribution: Mac OS X 10.14.5
• Ray installed from (source or binary): ray-0.7.2-cp37-cp37m-macosx_10_6_intel.whl
• Ray version: 0.7.2
• Python version: 3.7.2
• Exact command to reproduce:

import ray
import numba

Describe the problem

Versions of ray>=0.7.1 cannot be imported before numba or an issue with llvmlite will be triggered (see also https://github.com/numba/numba/issues/4256 for additional minimal reproduction steps). Importing numba first seems to work around the issue.

The error reporting is an assertion failure:

Assertion failed: (PassInf && "Expected all immutable passes to be initialized"), function addImmutablePass, file /Users/buildbot/miniconda3/conda-bld/llvmdev_1556270736866/work/lib/IR/LegacyPassManager.cpp, line 849.

that leads to the python interpreter aborting. The issue does not seem to occur on Linux.

Version 0.7.0 seems unaffected.

[robertnishihara]

Thanks for reporting this, I was able to reproduce this with just pyarrow by doing

pip install -U pyarrow
python
> import pyarrow
> assert pyarrow.__version__ == '0.14.1'
> import numba

Note that Ray actually vendors its own version of pyarrow, which can be found under ray/pyarrow_files/pyarrow.

This is probably related to how pyarrow bundles llvm. We'll probably need to fix this in pyarrow or build the pyarrow wheels differently.

[aclyde11]

I can confirm that the work arounds proposed in above references solve the import issue but will simply delay the error with numba code....

[MQQ]

To follow up on this, I just tried this on the newest pyarrow==0.15.1 and it seems this is fixed on the pyarrow side. Is there a plan to include the updated pyarrow in ray in the near future?

Python 3.6.8 (default, Jan 22 2019, 15:04:02) 
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow
>>> pyarrow.__version__
'0.15.1'
>>> import numba
>>> 

[robertnishihara]

@pcmoritz, we started building Arrow wheels for Arrow 15.1 in https://github.com/ray-project/arrow-build/pull/12, right? But did we ever start using those? It should be a simple change to incorporate the latest Arrow code I think.

[robertnishihara]

@MQQ so yes, we are planning to do it. Not sure if it will be in the upcoming release or the following one.

[piojanu]

Hi, @robertnishihara! What is the status of this issue? When can we expect the next release with updated pyarrow?

[robertnishihara]

Fixed in ray==0.8.2.