Related to #1999
Repro
import evalml
import pandas as pd
import woodwork as ww
df = pd.read_csv('/Users/dylan.sherry/Downloads/bos_311_balanced.csv')
dt = ww.DataTable(df, logical_types={'reason': 'Categorical'})
dt = dt.drop('closed_dt') # ignore datetime feature because nans in it produce another bug
automl = evalml.automl.AutoMLSearch(X_train=dt, y_train=y, problem_type='multiclass')
automl.search()
Produces:
File "/Users/dylan.sherry/development/evalml/evalml/pipelines/components/transformers/preprocessing/text_featurizer.py", line 121, in transform
X_lsa = self._lsa.transform(X[self._text_columns]).to_dataframe()
...
File "/Users/dylan.sherry/development/evalml/evalml/pipelines/components/transformers/preprocessing/lsa.py", line 63, in transform
transformed = self._lsa_pipeline.transform(X[col])
...
File "/Users/dylan.sherry/.pyenv/versions/evalml/lib/python3.8/site-packages/sklearn/feature_extraction/text.py", line 219, in decode
raise ValueError("np.nan is an invalid document, expected byte or "
...
Fold 0: Exception during automl search: np.nan is an invalid document, expected byte or unicode string.
Had discussion with @jeremyliweishih @boopboopbeepboop @cmancuso
The short-term plan (tracked by this issue) is to add a data check error when natural language features contain nans, asking users to either drop those rows or fill in the missing values on their own through some other means.
Long-term we will discuss a) options for handling missing natural language features beyond simply dropping rows and b) ways our transformers and/or estimator can still benefit from the information contained in the rows with missing values for the natural language features in question.
Most helpful comment
Had discussion with @jeremyliweishih @boopboopbeepboop @cmancuso
The short-term plan (tracked by this issue) is to add a data check error when natural language features contain
nans, asking users to either drop those rows or fill in the missing values on their own through some other means.Long-term we will discuss a) options for handling missing natural language features beyond simply dropping rows and b) ways our transformers and/or estimator can still benefit from the information contained in the rows with missing values for the natural language features in question.