Evalml: OneHotEncoder introduces NaNs when dataframe has custom index

[freddyaboulton]

Reproducer:

from evalml.pipelines.components import OneHotEncoder
import pandas as pd
import numpy as np

df = pd.DataFrame({"a": [1]*100, "b": ["foo"] * 50 + ["bar"] * 50}, index=range(100, 200))
ohe = OneHotEncoder()

ohe.fit(df)

ohe.transform(df)

Output:

I think the issue is the call to reindex (it introduces NaNs when there is no value in the new index). Maybe we should just pass in the index of X_t when X_cat is being created?

[dsherry]

Nice writeup! Super clear.

Yep, I see the call to reindex you're talking about.

I bet there are other places in which we don't support custom indexes, other components which would break. That said, we should certainly fix the one hot encoder to work with custom indexes. It would be a step in the right direction.

So the goal of this issue is to have the one-hot encoder support custom indexes, in particular by not inserting extra rows with nans.

@freddyaboulton would you be interested in taking this on?

[freddyaboulton]

@dsherry Yep! Would love to fix this.