Cudf: [BUG] AttributeError: module 'pyarrow.lib' has no attribute 'IpcWriteOptions'

[pykeras]

Describe the bug

Rapids 0.16 installation on ubuntu 20.04
installation command :
conda create -n rapids-0.16 -c rapidsai -c nvidia -c conda-forge \ -c defaults rapids=0.16 python=3.8 cudatoolkit=10.1
conda (latest Minicodna) - version 4.9.2

>>> import cudf

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/kingeng/miniconda3/envs/pykeras/lib/python3.8/site-packages/cudf/__init__.py", line 11, in <module>
    from cudf import core, datasets, testing
  File "/home/kingeng/miniconda3/envs/pykeras/lib/python3.8/site-packages/cudf/core/__init__.py", line 2, in <module>
    from cudf.core import buffer, column
  File "/home/kingeng/miniconda3/envs/pykeras/lib/python3.8/site-packages/cudf/core/column/__init__.py", line 3, in <module>
    from cudf.core.column.categorical import CategoricalColumn
  File "/home/kingeng/miniconda3/envs/pykeras/lib/python3.8/site-packages/cudf/core/column/categorical.py", line 8, in <module>
    from cudf import _lib as libcudf
  File "/home/kingeng/miniconda3/envs/pykeras/lib/python3.8/site-packages/cudf/_lib/__init__.py", line 4, in <module>
    from . import (
  File "cudf/_lib/gpuarrow.pyx", line 1, in init cudf._lib.gpuarrow
AttributeError: module 'pyarrow.lib' has no attribute 'IpcWriteOptions'

__Pyarrow version : '0.17.1'__

Environment overview (please complete the following information)

• Environment location: [Bare-metal]
• Method of cuDF install: [conda]

conda (latest Minicodna) - version 4.9.2
Ubuntu 20.04
rapids 0.16
cuda 10.1

[kkraus14]

@pykeras what does conda list | grep arrow return in your environment? It looks like it's not pulling a cuda variant build while it should be.

[pykeras]

@kkraus14

Here the result

arrow-cpp                 1.0.1           py38h9444e92_16_cuda    conda-forge
arrow-cpp-proc            2.0.0                      cuda    conda-forge
pyarrow                   1.0.1           py38h3e2403a_16_cuda    conda-forge

[kkraus14]

Hmm that all looks fine. Are you sure you're checking everything in the pykeras environment? Typically we've seen this issue when someone has an outdated version of pyarrow.

[pykeras]

yes,
I'm not a fan of anaconda so all the packages installed for RAPIDS.
by the way, before this (0.14 release), I had a chance to install and work with RAPIDS

[kkraus14]

Can you run this and report the output:

import pyarrow
print(pyarrow.__version__)
print(pyarrow.__file__)

[pykeras]

Here the results:

>>> import pyarrow
>>> pyarrow.__version__
'0.17.1'
>>> pyarrow.__file__
'/home/kingeng/.local/lib/python3.8/site-packages/pyarrow/__init__.py'
>>> 

[kkraus14]

Looks like you have a pyarrow install outside of the conda environment that is being picked up before the pyarrow in your conda environment. Uninstalling that should likely resolve the issue.

[pykeras]

Thanks for your reply, for anyone who may have this issue
as I've understood Anaconda should stop using the global package when you activate virtualenv
but in my case, it doesn't so I have to type export PYTHONNOUSERSITE=True
before running python, that command solve my problem

[beckernick]

Glad you've resolved this @pykeras ! Thanks very much for sharing your solution in this issue for others.