Evalml: Str target failure in automl: "ValueError: y contains previously unseen labels: [2.0]" (Looking Glass)

[dsherry]

This looking glass job failed with the error above. Failed on the first run of dataset Engine1_1.csv which is available here or in S3.

Can't get the first stack trace yet from looking glass but here's the relevant log entries:

2020-07-23T11:36:09.842-04:00 Running dataset Engine1_1.csv trial 1 / 20
...
2020-07-23T11:36:09.910-04:00 (1/6) Mode Baseline Multiclass Classificati... Elapsed:00:00
2020-07-23T11:36:09.910-04:00 Starting cross validation
2020-07-23T11:36:09.924-04:00 Fold 0: Encountered an error scoring the following objectives: AUC Micro, AUC Macro, AUC Weighted.
2020-07-23T11:36:09.924-04:00 Fold 0: The scores for these objectives will be replaced with nan.
2020-07-23T11:36:09.924-04:00 Fold 0: Please check /evalml_looking_glass/evalml_debug.log for the current hyperparameters and stack trace.
...
2020-07-23T11:36:15.781-04:00 Finished cross validation - mean Log Loss Multiclass: 0.216
2020-07-23T11:36:15.783-04:00 Search finished after 00:05
2020-07-23T11:36:15.788-04:00 Best pipeline: CatBoost Classifier w/ Simple Imputer
2020-07-23T11:36:15.788-04:00 Best pipeline Log Loss Multiclass: 0.146039
2020-07-23T11:36:16.740-04:00 Traceback (most recent call last):
2020-07-23T11:36:16.740-04:00 File "/evalml_looking_glass/evalml/evalml/pipelines/classification_pipeline.py", line 51, in _encode_targets
2020-07-23T11:36:16.740-04:00 return pd.Series(self._encoder.transform(y))
2020-07-23T11:36:16.740-04:00 File "//.pyenv/versions/3.8.3/lib/python3.8/site-packages/sklearn/preprocessing/_label.py", line 273, in transform
2020-07-23T11:36:16.740-04:00 _, y = _encode(y, uniques=self.classes_, encode=True)
2020-07-23T11:36:16.740-04:00 File "//.pyenv/versions/3.8.3/lib/python3.8/site-packages/sklearn/preprocessing/_label.py", line 117, in _encode
2020-07-23T11:36:16.740-04:00 return _encode_numpy(values, uniques, encode,
2020-07-23T11:36:16.740-04:00 File "//.pyenv/versions/3.8.3/lib/python3.8/site-packages/sklearn/preprocessing/_label.py", line 49, in _encode_numpy
2020-07-23T11:36:16.740-04:00 raise ValueError("y contains previously unseen labels: %s"
2020-07-23T11:36:16.740-04:00 ValueError: y contains previously unseen labels: [2.0]
2020-07-23T11:36:16.740-04:00 During handling of the above exception, another exception occurred:
2020-07-23T11:36:16.740-04:00 Traceback (most recent call last):
2020-07-23T11:36:16.740-04:00 File "looking_glass/container/perf_test_runner.py", line 139, in <module>
2020-07-23T11:36:16.740-04:00 perf_test_runner_cli()
2020-07-23T11:36:16.740-04:00 File "//.pyenv/versions/3.8.3/lib/python3.8/site-packages/click/core.py", line 829, in __call__
2020-07-23T11:36:16.740-04:00 return self.main(*args, **kwargs)
2020-07-23T11:36:16.740-04:00 File "//.pyenv/versions/3.8.3/lib/python3.8/site-packages/click/core.py", line 782, in main
2020-07-23T11:36:16.740-04:00 rv = self.invoke(ctx)
2020-07-23T11:36:16.740-04:00 File "//.pyenv/versions/3.8.3/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
2020-07-23T11:36:16.740-04:00 return ctx.invoke(self.callback, **ctx.params)
2020-07-23T11:36:16.740-04:00 File "//.pyenv/versions/3.8.3/lib/python3.8/site-packages/click/core.py", line 610, in invoke
2020-07-23T11:36:16.740-04:00 return callback(*args, **kwargs)
2020-07-23T11:36:16.740-04:00 File "looking_glass/container/perf_test_runner.py", line 131, in perf_test_runner_cli
2020-07-23T11:36:16.740-04:00 run(datasets_yaml_file,
2020-07-23T11:36:16.740-04:00 File "looking_glass/container/perf_test_runner.py", line 102, in run
2020-07-23T11:36:16.740-04:00 trial_measurements_dict = measure_evalml_automl(X, y,
2020-07-23T11:36:16.740-04:00 File "/evalml_looking_glass/looking_glass/container/evalml_runner.py", line 44, in measure_evalml_automl
2020-07-23T11:36:16.740-04:00 best_pipeline_validation_scores = new_pipeline.score(X_validation, y_validation, objectives=[automl.objective])
2020-07-23T11:36:16.740-04:00 File "/evalml_looking_glass/evalml/evalml/pipelines/classification_pipeline.py", line 123, in score
2020-07-23T11:36:16.740-04:00 y = self._encode_targets(y)
2020-07-23T11:36:16.740-04:00 File "/evalml_looking_glass/evalml/evalml/pipelines/classification_pipeline.py", line 53, in _encode_targets
2020-07-23T11:36:16.740-04:00 raise ValueError(str(e))
2020-07-23T11:36:16.740-04:00 ValueError: y contains previously unseen labels: [2.0]
2020-07-23T11:36:17.738-04:00 Finished running tests.

It appears this is related to #215 / #932 (@angela97lin FYI)

[dsherry]

Reproducer:

import pandas as pd
import numpy as np
import evalml
data = pd.read_csv("/Users/dylan.sherry/Downloads/Engine1_1.csv")
target = "class"
X = data.drop([target], axis=1)
y = data[target]
automl = evalml.automl.AutoMLSearch(problem_type='BINARY')
automl.search(X, y)

When I run that I see the following in stdout:

/Users/dylan.sherry/.pyenv/versions/evalml/lib/python3.8/site-packages/sklearn/model_selection/_split.py:665: UserWarning:
The least populated class in y has only 1 members, which is less than n_splits=3.

In the debug log, I don't see a stack trace. I do see Fold 1: Exception during automl search: y contains previously unseen labels: [2.0]

So that's the problem! y.value_counts() shows:

0.0    257
1.0    125
2.0      1

Next steps

• This is further proof we need a problem-type verification data check as discussed in #931 . This stack trace is hideous!
• I just updated #931 to track updating data checks to accept parameters, and then updating InvalidTargetDataCheck to validate errors like this based on the problem_type. This issue should be fixed by that.

[freddyaboulton]

Closing because #931 was closed with #1167