Pandas: BUG: Issue with latest version (1.1.0) of pandas when imported

[rakekris]

• [Y ] I have checked that this issue has not already been reported.

• [ Y] I have confirmed this bug exists on the latest version of pandas.

• [ ] (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

# Your code here

Problem description

[this should explain why the current behaviour is a problem and why the expected output is a better solution]

We are using latest version of pandas in project .

We are getting following issue when we import pandas

import pandas as pd

File "/usr/local/lib64/python3.6/site-packages/pandas/__init__.py", line 22, in
from pandas.compat.numpy import (
File "/usr/local/lib64/python3.6/site-packages/pandas/compat/numpy/__init__.py", line 21, in
"this version of pandas is incompatible with numpy < 1.15.4n"
ImportError: this version of pandas is incompatible with numpy < 1.15.4
your numpy version is 1.14.5.
Please upgrade numpy to >= 1.15.4 to use this pandas version

Expected Output

Output of pd.show_versions()

[paste the output of pd.show_versions() here leaving a blank line after the details tag]

1.1.0

[TomAugspurger]

How'd you install pandas? We require NumPy>==1.15.4 and you seem to have an older version: https://github.com/pandas-dev/pandas/blob/bdcc5bffaadb7488474b65554c4e8e96a00aa4af/setup.py#L36

Your package manager shouldn't have installed pandas into an environment with NumPy 1.14.5.

[rakekris]

Below code snippet is the setup.py . we always use to pull latest py packages. We are not explicitly pulling numpy. I believe numpy is installed as part of pandas. By the way when i check the version of numpy it shows 1.19.1 which is again latest. We were not getting the mentioned issue while we used pandas 1.0.5. We started to get this issue starting from July 28

`from distutils.core import setup

from setuptools import find_packages

setup(
name='sharp',
use_scm_version={
'version_scheme': 'post-release'
},
setup_requires=['setuptools_scm'],
packages=find_packages(include=['sfo3.', 'sharp.', ], exclude=['tests.']),
package_data={
'': ['.yaml', '.json', '.html', '.ini', '.j2', '.js', '.css', '*.png'],
},
install_requires=['boto3', 'requests', 'jinja2', 'pandasql', 'pandas', 'pygrok', 'prison', "future"],
zip_safe=False,
)
`

[letianw91]

Below code snippet is the setup.py . we always use to pull latest py packages. We are not explicitly pulling numpy. I believe numpy is installed as part of pandas. By the way when i check the version of numpy it shows 1.19.1 which is again latest. We were not getting the mentioned issue while we used pandas 1.0.5. We started to get this issue starting from July 28

`from distutils.core import setup

from setuptools import find_packages

setup(
name='sharp',
use_scm_version={
'version_scheme': 'post-release'
},
setup_requires=['setuptools_scm'],
packages=find_packages(include=['sfo3._', 'sharp._', ], exclude=['tests._']), package_data={ '': ['_.yaml', '_.json', '_.html', '_.ini', '_.j2', '_.js', '_.css', '*.png'],
},
install_requires=['boto3', 'requests', 'jinja2', 'pandasql', 'pandas', 'pygrok', 'prison', "future"],
zip_safe=False,
)
`

I'm encountering the same issue. When I do pip-3.6 freeze | grep numpy, my numpy version is numpy==1.19.1.

But when I import pandas as pd from python3 REPL, I get

  File "<stdin>", line 1, in <module>
  File "/usr/local/lib64/python3.6/site-packages/pandas/__init__.py", line 22, in <module>
    from pandas.compat.numpy import (
  File "/usr/local/lib64/python3.6/site-packages/pandas/compat/numpy/__init__.py", line 21, in <module>
    "this version of pandas is incompatible with numpy < 1.15.4\n"
ImportError: this version of pandas is incompatible with numpy < 1.15.4
your numpy version is 1.14.5.
Please upgrade numpy to >= 1.15.4 to use this pandas version

It seems that even if I installed Pandas in Python3, it is picking up my Python2 numpy version.

pip freeze | grep numpy
-- numpy==1.14.5

After I downgrade Pandas to 1.0.5, problem disappears.

[TomAugspurger]

Did anyone here install pandas into a clean environment and get numpy<1.15.4? If so, that's an issue with pandas that needs to be fixed. Otherwise, it's an issue with how you / your package manager is installing pandas.

[rakekris]

@TomAugspurger We install the pandas on a transient EMR , hence it is a fresh install each time we spin up the EMR.

[TomAugspurger]

You're installing with pip? Can you post the full output of pip install pandas?

[letianw91]

@TomAugspurger We install the pandas on a transient EMR , hence it is a fresh install each time we spin up the EMR.

Looks like this might be an EMR specific issue. We are encountering this on EMR too (5.30.0)

[TomAugspurger]

Thanks for confirming. Right now my guess is that pandas is doing things correctly.

[rtenshi]

@letianw91
I had the same problem, so if you are having the problem at the EMR it seems to be really an EMR error. It's fixed on the 5.30.1 .
they changed the standard python version from the 2.7 to the 3.7.6, which solved the problem