Onnxruntime: "ImportError: cannot import name 'get_all_providers'" for macOS in TravisCI after 1.5.1 update

[joshuacwnewton]

Describe the bug

We use onnxruntime in a Python project called SpinalCordToolbox, and test the project using pytest and TravisCI. Under onnxruntime==1.4.0, our test suite for macOS 10.14 runs without error. But, after the update to onnxruntime==1.5.1, the following error interrupts the test suite:

ImportError while importing test module '/Users/travis/build/neuropoly/spinalcordtoolbox/unit_testing/test_deepseg.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
python/envs/venv_sct/lib/python3.6/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
unit_testing/test_deepseg.py:14: in <module>
    import spinalcordtoolbox.deepseg.core
spinalcordtoolbox/deepseg/core.py:12: in <module>
    import ivadomed.utils
python/envs/venv_sct/lib/python3.6/site-packages/ivadomed/utils.py:8: in <module>
    import onnxruntime
python/envs/venv_sct/lib/python3.6/site-packages/onnxruntime/__init__.py:13: in <module>
    from onnxruntime.capi._pybind_state import get_all_providers, get_available_providers, get_device, set_seed, \
E   ImportError: cannot import name 'get_all_providers'

By a stroke of luck, we had two identical Travis jobs run withn 30m, one before and one after the PyPI update (see https://github.com/neuropoly/spinalcordtoolbox/issues/2930#issuecomment-701495447), so I think the version bump is the cause.

Potentially related to #2572, #3943, #4196, #4806.

Urgency

We have a quick workaround (pinning 1.4.0), so it's not terribly urgent, but that isn't ideal, I think.

System information

• OS Platform and Distribution: Mac OS X 10.14.6 (Build Version:18G3020)
  
  
  
  • Specifically, https://docs.travis-ci.com/user/reference/osx/ with osx_image: xcode11.2
  
  
• ONNX Runtime installed from (source or binary): binary
• ONNX Runtime version: onnxruntime-1.5.1-cp36-cp36m-macosx_10_12_x86_64.whl
• Python version: 3.6.12
• Visual Studio version (if applicable): N/A
• GCC/Compiler version (if compiling from source): N/A
• CUDA/cuDNN version: Unsure
• GPU model and memory: Unsure

To Reproduce

The error occurred in a CI job, so it may be a little tricky to reproduce in a different environment. But, it amounts to:

• Install onnxruntime in a conda environment.
• Attempt to run pytest test suite that includes a test that imports onnxruntime.

Relevant CI links:

• Failing CI job: https://travis-ci.org/github/neuropoly/spinalcordtoolbox/jobs/731359167#L937
• CI script: https://github.com/neuropoly/spinalcordtoolbox/blob/master/.ci.sh

Expected behavior

Test imports onnxruntime without failure (like it does in 1.4.0).

Thanks much!

[pranavsharma]

What error did you get? Are you sure you've installed all the dependencies? On my machine it complained of missing libomp. After installing libomp, I didn't see any issue.

(py36) Pranavs-MacBook-Pro:~ pranav$ otool -L /Users/pranav/anaconda3/envs/py36/lib/python3.6/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.so 
/Users/pranav/anaconda3/envs/py36/lib/python3.6/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.so:
    /usr/local/opt/libomp/lib/libomp.dylib (compatibility version 5.0.0, current version 5.0.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 400.9.4)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.200.5)

(py36) Pranavs-MacBook-Pro:~ pranav$ python
Python 3.6.12 |Anaconda, Inc.| (default, Sep  8 2020, 17:50:39) 
[GCC Clang 10.0.0 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import onnxruntime as rt
>>> rt.get_all_providers()
['TensorrtExecutionProvider', 'CUDAExecutionProvider', 'MIGraphXExecutionProvider', 'NGRAPHExecutionProvider', 'OpenVINOExecutionProvider', 'DnnlExecutionProvider', 'NupharExecutionProvider', 'VitisAIExecutionProvider', 'ArmNNExecutionProvider', 'ACLExecutionProvider', 'DmlExecutionProvider', 'CPUExecutionProvider']
>>> 

[joshuacwnewton]

What error did you get? Are you sure you've installed all the dependencies? On my machine it complained of missing libomp. After installing libomp, I didn't see any issue.

Good call. When looking at our TravisCI log, a Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib warning was indeed raised:

python/envs/venv_sct/lib/python3.6/site-packages/onnxruntime/capi/_pybind_state.py:14
  /Users/travis/build/neuropoly/spinalcordtoolbox/python/envs/venv_sct/lib/python3.6/site-packages/onnxruntime/capi/_pybind_state.py:14: UserWarning: Cannot load onnxruntime.capi. Error: 'dlopen(/Users/travis/build/neuropoly/spinalcordtoolbox/python/envs/venv_sct/lib/python3.6/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.so, 2): Library not loaded: /usr/local/opt/libomp/lib/libomp.dylib
    Referenced from: /Users/travis/build/neuropoly/spinalcordtoolbox/python/envs/venv_sct/lib/python3.6/site-packages/onnxruntime/capi/onnxruntime_pybind11_state.so
    Reason: image not found'.
    warnings.warn("Cannot load onnxruntime.capi. Error: '{0}'.".format(str(e)))

I'll try your fix and report back. :)

[joshuacwnewton]

Thanks so much. It looks like editing our .travis.yml to make sure libomp is installed does the trick.

For anyone reading in the future, see: https://docs.travis-ci.com/user/installing-dependencies/#installing-packages-on-macos

addons:
  homebrew:
    packages:
    - libomp

Or brew install libomp, outside of Travis.