pandas.rolling.apply skip calling function if window contains any NaN

[jwchi]

Code Sample, a copy-pastable example if possible

import numpy as np
import pandas as pd

# sample data with NaN
df = pd.DataFrame([np.arange(8) + i * 10 for i in range(3)]).T
df[0][3] = np.nan
df[1][2] = np.nan
df[2][6] = np.nan
print(df)

# sample function to apply to window
def my_std(x): 
    print(x)
    return x.std()

# apply my_std to rolling window(5)
print(df.rolling(5).apply(my_std, raw=False)) 

Problem description

I wonder if this is an intended/unintended behavior. Apparently when a Rolling object runs the apply method, it skips calling the function completely if data in the window contains any np.nan.

df looks like this:

     0     1     2
0  0.0  10.0  20.0
1  1.0  11.0  21.0
2  2.0   NaN  22.0
3  NaN  13.0  23.0
4  4.0  14.0  24.0
5  5.0  15.0  25.0
6  6.0  16.0   NaN
7  7.0  17.0  27.0

Output of df.rolling(5).apply(my_std, raw=False):

    0         1         2
0 NaN       NaN       NaN
1 NaN       NaN       NaN
2 NaN       NaN       NaN
3 NaN       NaN       NaN
4 NaN       NaN  1.581139
5 NaN       NaN  1.581139
6 NaN       NaN       NaN
7 NaN  1.581139       NaN

I understand that in older versions, pandas calls numpy primitives to handle rolling windows, which leads to NaNs as numpy function propagates it. But here, the NaNs are not caused by my_std, because data in the first column are not even printed, i.e. my_std is not even called for the first column as there is no single non-NaN window.

Confusingly, the Expanding object would not do this. Expanding.apply actually calls the function with the expanding window despite if there is any NaN or not.

Expected Output

expected output of running: df.rolling(5).apply(my_std, raw=False)

          0         1         2
0       NaN       NaN       NaN
1       NaN       NaN       NaN
2       NaN       NaN       NaN
3       NaN       NaN       NaN
4       NaN       NaN  1.581139
5  1.825742  1.707825  1.581139
6  1.707825  1.290994  1.290994
7  1.290994  1.581139  1.707825

currently this is achieved by running: df.expanding(5).apply(lambda x: x[-5:].std(), raw=False)
it's confusing that one has to use expanding(5) to do the job of rolling(5) when NaN is present in data. Performance-wise is probably not ideal either...

Output of pd.show_versions()

INSTALLED VERSIONS

commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Windows
OS-release: 7
machine: AMD64
processor: Intel64 Family 6 Model 63 Stepping 0, GenuineIntel
byteorder: little
LC_ALL: None
LANG: None
LOCALE: None.None
pandas: 0.23.0
pytest: None
pip: 10.0.1
setuptools: 39.2.0
Cython: None
numpy: 1.14.3
scipy: 1.1.0
pyarrow: None
xarray: None
IPython: 6.4.0
sphinx: None
patsy: None
dateutil: 2.7.3
pytz: 2018.4
blosc: None
bottleneck: None
tables: None
numexpr: None
feather: None
matplotlib: 2.2.2
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: None
lxml: None
bs4: None
html5lib: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.10
s3fs: None
fastparquet: None
pandas_gbq: None
pandas_datareader: None

[WillAyd]

The difference you are seeing is a result of min_periods which defaults to the length of the window you've provided in the rolling function. In this case, because you don't have 5 non-NA records within your window the function automatically returns NA.

Note the difference in behavior when passing this explicitly:

>>>  df.rolling(5, min_periods=4).apply(my_std, raw=False)
          0         1         2
0       NaN       NaN       NaN
1       NaN       NaN       NaN
2       NaN       NaN       NaN
3       NaN       NaN  1.290994
4  1.707825  1.825742  1.581139
5  1.825742  1.707825  1.581139
6  1.707825  1.290994  1.290994
7  1.290994  1.581139  1.707825

See also the rolling documentation:
https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.rolling.html

I think this issue is worth leaving open if only to update the documentation of that keyword argument. There is an example that shows how it functions, but the description of that argument could use some clarification on what None actually means

[jwchi]

Thank you, WillAyd. You are right. It is the effect of min_periods indeed! Really appreciate your reply. Should I be leaving this open for a few more days to wait for the change in the documentation to happen? Or should I make change to the documentation myself? Sorry for the questions, this is the first time I posted a question in the forum.. Thanks

[WillAyd]

If you are open to making the change yourself contributions like that are welcome and encouraged. Just be sure to check out the pandas contributor guide:

https://pandas.pydata.org/pandas-docs/stable/contributing.html

[alphaCTzo7G]

@jwchi, are you planning on making the changes to the documentation or can I take this up?

[mattboggess]

Is this issue currently being worked on? If not, I'd like to take it up.

[jschendel]

@mattboggess : Go for it! Doesn't look like anyone is currently working on it.