Pandas: BUG: indexing with -1 doesn't work anymore in expanding function

[CloseChoice]

• [x] I have checked that this issue has not already been reported.

• [x] I have confirmed this bug exists on the latest version of pandas.

• [ ] (optional) I have confirmed this bug exists on the master branch of pandas.

---

Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Code Sample, a copy-pastable example

# Your code here
import pandas as pd
import numpy as np
# setup
df = pd.DataFrame([5,7,4,12,3,4,1], columns=['Value'])
# calculate countif
df['Count'] = df.Value.expanding(1).apply(lambda x: np.sum(np.where(x > x[-1], 1, 0))).astype('int')

raises Traceback (most recent call last): File "<stdin>", line 2, in <module> File "/home/tobias/anaconda3/envs/py38/lib/python3.8/site-packages/pandas/core/window/expanding.py", line 152, in apply return super().apply(func, raw=raw, args=args, kwargs=kwargs) File "/home/tobias/anaconda3/envs/py38/lib/python3.8/site-packages/pandas/core/window/rolling.py", line 1300, in apply return self._apply( File "/home/tobias/anaconda3/envs/py38/lib/python3.8/site-packages/pandas/core/window/rolling.py", line 507, in _apply result = calc(values) File "/home/tobias/anaconda3/envs/py38/lib/python3.8/site-packages/pandas/core/window/rolling.py", line 495, in calc return func(x, start, end, min_periods) File "/home/tobias/anaconda3/envs/py38/lib/python3.8/site-packages/pandas/core/window/rolling.py", line 1326, in apply_func return window_func(values, begin, end, min_periods) File "pandas/_libs/window/aggregations.pyx", line 1375, in pandas._libs.window.aggregations.roll_generic_fixed File "<stdin>", line 2, in <lambda> File "/home/tobias/anaconda3/envs/py38/lib/python3.8/site-packages/pandas/core/series.py", line 871, in __getitem__ result = self.index.get_value(self, key) File "/home/tobias/anaconda3/envs/py38/lib/python3.8/site-packages/pandas/core/indexes/base.py", line 4405, in get_value return self._engine.get_value(s, k, tz=getattr(series.dtype, "tz", None)) File "pandas/_libs/index.pyx", line 80, in pandas._libs.index.IndexEngine.get_value File "pandas/_libs/index.pyx", line 90, in pandas._libs.index.IndexEngine.get_value File "pandas/_libs/index.pyx", line 138, in pandas._libs.index.IndexEngine.get_loc File "pandas/_libs/hashtable_class_helper.pxi", line 998, in pandas._libs.hashtable.Int64HashTable.get_item File "pandas/_libs/hashtable_class_helper.pxi", line 1005, in pandas._libs.hashtable.Int64HashTable.get_item KeyError: -1

Problem description

[this should explain why the current behaviour is a problem and why the expected output is a better solution]

Expected Output

On 0.25.2 this resulted in

    Value   Count
0   5        0
1   7        0
2   4        2
3   12       0
4   3        4
5   4        3
6   1        6

Output of pd.show_versions()

INSTALLED VERSIONS

commit : None
python : 3.8.0.final.0
python-bits : 64
OS : Linux
OS-release : 4.10.0-35-generic
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : de_DE.UTF-8
LOCALE : de_DE.UTF-8

pandas : 1.0.5
numpy : 1.17.4
pytz : 2019.3
dateutil : 2.8.1
pip : 19.3.1
setuptools : 42.0.2.post20191203
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : None
IPython : 7.10.1
pandas_datareader: None
bs4 : None
bottleneck : None
fastparquet : None
gcsfs : None
lxml.etree : None
matplotlib : 3.1.2
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pytables : None
pytest : None
pyxlsb : None
s3fs : None
scipy : 1.3.3
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
xlsxwriter : None
numba : None

[Akshaysehgal2005]

Same error, pandas version 1.0.3 and 1.0.5, just to add, x[1], x[2] ... etc all give keyerror, except x[0]

[zky001]

Same error, pandas version 1.0.3 and 1.0.5, just to add, x[1], x[2] ... etc all give keyerror, except x[0]

You are absolutely right，I think we can change this function

[AlexKirko]

This works:

IN:
df['Count'] = df.Value.expanding(1).apply(lambda x: np.sum(np.where(x > x[-1], 1, 0)), raw=True).astype('int')

OUT:
Value   Count
0   5   0
1   7   0
2   4   2
3   12  0
4   3   4
5   4   3
6   1   6

This also works:

df['Count'] = df.Value.expanding(1).apply(lambda x: np.sum(np.where(x > x.iloc[-1], 1, 0))).astype('int')

@CloseChoice Looks like your code doesn't work, because x is a Series within the lambda function, not an ndarray. As far as I can tell, this is desired behavior.

From what I was able to dig up, before 1.0.0 we passed ndarray to the apply function by default, but then we changed the behavior in #20584 and began to pass the Series or DataFrame object by default starting with 1.0.0. This broke this kind of indexing, and now an explicit cast to ndarray is required (or raw=True, which is equivalent).