Pandas: [BUG] Dataframe.rank() produces wrong results for float columns

[karthikeyann]

Code to reproduce the issue

import numpy as np
import pandas as pd

indx = np.array([5, 4, 3, 2, 1, 6, 7, 8, 9, 10])
col1 = np.array([5, 4, 3, 5, 8, 5, 2, 1, 6, 6])
col2 = np.array([5, 4, np.nan, 5, 8, 5, np.inf, np.nan, 6, -np.inf])
pdf = pd.DataFrame(index=indx)
pdf["col1"] = col1.astype('f8')
pdf["col2"] = col2.astype('f8')

ranked_df = pdf.rank()

ranked_series = pd.DataFrame(index=indx)
ranked_series["col1"] = pdf["col1"].rank()
ranked_series["col2"] = pdf["col2"].rank()

print(ranked_df.equals(ranked_series))
#False

Problem description

Dataframe rank produces individual column ranks. But when float column is present, individual ranks are not equal to Dataframe rank.

Expected Output

True

Output of pd.show_versions()

pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.8.1.final.0
python-bits : 64
OS : Linux
OS-release : 4.15.0-72-generic
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.0.1
numpy : 1.18.1
pytz : 2019.3
dateutil : 2.8.1
pip : 20.0.2
setuptools : 46.0.0.post20200309
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 7.12.0
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : None

[dsaxton]

From what I can tell the difference has to do with the handling of np.inf? If so this is a slightly smaller example:

In [1]: import numpy as np                                                                                                                                                                                                                

In [2]: import pandas as pd                                                                                                                                                                                                               

In [3]: values = [-np.inf, 0, np.inf]                                                                                                                                                                                                     

In [4]: print(pd.Series(values).rank())                                                                                                                                                                                                   
0    1.0
1    2.0
2    3.0
dtype: float64

In [5]: print(pd.DataFrame({"a": values}).rank())                                                                                                                                                                                         
     a
0  1.0
1  2.0
2  NaN

Does seem odd that np.inf doesn't rank as the largest value in the DataFrame case.

[letapxad]

take

[peterpanmj]

From what I can tell the difference has to do with the handling of np.inf? If so this is a slightly smaller example:

In [1]: import numpy as np                                                                                                                                                                                                                

In [2]: import pandas as pd                                                                                                                                                                                                               

In [3]: values = [-np.inf, 0, np.inf]                                                                                                                                                                                                     

In [4]: print(pd.Series(values).rank())                                                                                                                                                                                                   
0    1.0
1    2.0
2    3.0
dtype: float64

In [5]: print(pd.DataFrame({"a": values}).rank())                                                                                                                                                                                         
     a
0  1.0
1  2.0
2  NaN

Does seem odd that np.inf doesn't rank as the largest value in the DataFrame case.

A very good example. It is related to an old issue #6945. When missing values and inf values appear in one Series, the ranks will be wrong. I've worked on it before. It seems that DataFrame rank still cannot handle it, might be a regression.