Evalml: Unit test fails after upgrading to pandas 1.0.0

[angela97lin]

Steps to reproduce the bug:

1. Pull from master. This will reproduce the s3fs ImportError.
2. Locally upgrade s3fs via pip to 0.3.0 or higher. Now call load_fraud() to see the second numpy / pandas error.

Issue:
CircleCI unit test now fails because pandas 1.0.0 is installed, and pandas 1.0.0 now requires s3fs 0.3.0.

evalml/demos/fraud.py:17: in load_fraud
    n_rows=n_rows
evalml/preprocessing/utils.py:31: in load_data
    feature_matrix = pd.read_csv(path, index_col=index, nrows=n_rows, **kwargs)
test_python/lib/python3.7/site-packages/pandas/io/parsers.py:676: in parser_f
    return _read(filepath_or_buffer, kwds)
test_python/lib/python3.7/site-packages/pandas/io/parsers.py:431: in _read
    filepath_or_buffer, encoding, compression
test_python/lib/python3.7/site-packages/pandas/io/common.py:182: in get_filepath_or_buffer
    from pandas.io import s3
test_python/lib/python3.7/site-packages/pandas/io/s3.py:9: in <module>
    "s3fs", extra="The s3fs package is required to handle s3 files."
E                   ImportError: Pandas requires version '0.3.0' or newer of 's3fs' (version '0.2.2' currently installed).

However, upgrading s3fs to 0.3.0 causes this following issue with X.corwith(y) while building the docs instead:

~/evalml/test_python/lib/python3.7/site-packages/numpy/lib/function_base.py in corrcoef(x, y, rowvar, bias, ddof)
   2524         warnings.warn('bias and ddof have no effect and are deprecated',
   2525                       DeprecationWarning, stacklevel=3)
-> 2526     c = cov(x, y, rowvar)
   2527     try:
   2528         d = diag(c)

<__array_function__ internals> in cov(*args, **kwargs)

~/evalml/test_python/lib/python3.7/site-packages/numpy/lib/function_base.py in cov(m, y, rowvar, bias, ddof, fweights, aweights)
   2429             w *= aweights
   2430 
-> 2431     avg, w_sum = average(X, axis=1, weights=w, returned=True)
   2432     w_sum = w_sum[0]
   2433 

<__array_function__ internals> in average(*args, **kwargs)

~/evalml/test_python/lib/python3.7/site-packages/numpy/lib/function_base.py in average(a, axis, weights, returned)
    426 
    427     if returned:
--> 428         if scl.shape != avg.shape:
    429             scl = np.broadcast_to(scl, avg.shape).copy()
    430         return avg, scl

AttributeError: 'float' object has no attribute 'shape'

[angela97lin]

Just digging around but seems like the error goes away if y is a DataFrame instead... perhaps pandas is not handling Series data properly?

[jeremyliweishih]

pandas is calling numpys corrcoef() function so something internal to pandas must have changed with the new release.

[jeremyliweishih]

I checked their issues page but nothing popped up yet.

[dsherry]

I just rewound master to before 1.0.0 (#325). Now we can debug this issue at our own pace

[jeremyliweishih]

After debugging a little, this should be the problem: https://github.com/pandas-dev/pandas/issues/31466.

This is because if y is a pd.Series object, pandas will reroute pd.corrwith(other) to pd.apply(lambda x: other.corr(x)).

[jeremyliweishih]

Adding this to pandas.core.nanops.py.get_corr_func._pearson fixes the issue:

def _pearson(a, b):
        b = b.astype(np.float32)
        return np.corrcoef(a, b)[0, 1]

[dsherry]

Ooh, great detective work @jeremyliweishih !

So, our options:

1. Do nothing. Keep our pandas version capped below 1.0.0 and wait for pandas to put out a fix (@jeremyliweishih perhaps you could write to them with your findings?)
2. Find a way to apply the patch you found in our installs

@angela97lin you mentioned converting something to a dataframe seemed to fix things; is that something we should add to this list of options?

Option 1 seems like the winner to me. Thoughts?

[jeremyliweishih]

If we see supporting the newest version of pandas as a big win I believe a patch (either converting to dataframe or looping and calculating the corr) would be great! We can always circle back and remove the patch when they fix it. If not then I agree we can just wait.

[kmax12]

I think it should be a goal to stay current with pandas. who know if there are other errors we're not catching in our unit tests.

if there is an easy fix that comes at a performance cost, let's just do it and leave a note saying it can be optimized later.

[dsherry]

Cool, sounds good. Looks like @jeremyliweishih already has a PR, awesome!