Evalml: Multiindex Lightgbm problem

[grayskripko]

I'm not sure whether the problem arises on evalml side or Lightgbm but I have a problem with multiindex X passing to AutoMLSearch.search()

Batch 1: (4/9) LightGBM Regressor w/ Imputer            Elapsed:00:01
    Starting cross validation
            Fold 0: Encountered an error.
            Fold 0: All scores will be replaced with nan.
            Fold 0: Please check ...\evalml_debug.log for the current hyperparameters and stack trace.
            Fold 0: Exception during automl search: logical_types contains columns that are not present in dataframe: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]
            Fold 1: Encountered an error...

Sorry, I can't give you a proper example. I still think this is better than nothing.

On my side, I fixed the problem with

X_train.pipe(lambda df: df.set_axis(['_'.join(col).strip() for col in df.columns.values], axis=1))

In addition, being still a novice in using the library, but I can't find a proper way to see logs of errors. I see the same information in evalml_debug.log without a clear traceback of the error. This is why I don't know the exact reason for the problem.

[angela97lin]

Hi @grayskripko, thank you for filing! RE a proper way to surface the traceback of the error, try:

from evalml.automl.callbacks import raise_error_callback
automl = AutoMLSearch(..., error_callback=raise_error_callback)
automl.search()

Having that traceback could be helpful for us to understand what is happening and better assist you 😄

[grayskripko]

Looks like a woodwork bug

Batch 1: (4/14) LightGBM Regressor w/ Imputer            Elapsed:00:01
    Starting cross validation
AutoMLSearch raised a fatal exception: logical_types contains columns that are not present in dataframe: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
  File "C:\Users\Gray\AppData\Roaming\Python\Python38\site-packages\evalml\automl\automl_search.py", line 677, in _compute_cv_scores
    cv_pipeline.fit(X_train, y_train)

  File "C:\Users\Gray\AppData\Roaming\Python\Python38\site-packages\evalml\utils\base_meta.py", line 18, in _set_fit
    return_value = method(self, X, y)

  File "C:\Users\Gray\AppData\Roaming\Python\Python38\site-packages\evalml\pipelines\regression_pipeline.py", line 32, in fit
    self._fit(X, y)

  File "C:\Users\Gray\AppData\Roaming\Python\Python38\site-packages\evalml\pipelines\pipeline_base.py", line 197, in _fit
    self._component_graph.fit(X, y)

  File "C:\Users\Gray\AppData\Roaming\Python\Python38\site-packages\evalml\pipelines\component_graph.py", line 89, in fit
    self._compute_features(self.compute_order, X, y, fit=True)

  File "C:\Users\Gray\AppData\Roaming\Python\Python38\site-packages\evalml\pipelines\component_graph.py", line 201, in _compute_features
    component_instance.fit(input_x, input_y)

  File "C:\Users\Gray\AppData\Roaming\Python\Python38\site-packages\evalml\utils\base_meta.py", line 18, in _set_fit
    return_value = method(self, X, y)

  File "C:\Users\Gray\AppData\Roaming\Python\Python38\site-packages\evalml\pipelines\components\estimators\regressors\lightgbm_regressor.py", line 90, in fit
    X_encoded = self._encode_categories(X, fit=True)

  File "C:\Users\Gray\AppData\Roaming\Python\Python38\site-packages\evalml\pipelines\components\estimators\regressors\lightgbm_regressor.py", line 75, in _encode_categories
    X_encoded = _rename_column_names_to_numeric(X_encoded)

  File "C:\Users\Gray\AppData\Roaming\Python\Python38\site-packages\evalml\utils\gen_utils.py", line 235, in _rename_column_names_to_numeric
    return ww.DataTable(X_renamed, logical_types=renamed_logical_types)

  File "C:\Users\Gray\AppData\Roaming\Python\Python38\site-packages\woodwork\datatable.py", line 81, in __init__
    _validate_params(dataframe, name, index, time_index, logical_types,

  File "C:\Users\Gray\AppData\Roaming\Python\Python38\site-packages\woodwork\datatable.py", line 1106, in _validate_params
    _check_logical_types(dataframe, logical_types)

  File "C:\Users\Gray\AppData\Roaming\Python\Python38\site-packages\woodwork\datatable.py", line 1162, in _check_logical_types
    raise LookupError('logical_types contains columns that are not present in '

[dsherry]

@grayskripko thank you for filing! We're looking into it. We'll follow up if we have more questions for you.

@gsheni

[gsheni]

@dsherry I am able to reproduce the error

import pandas as pd
import numpy as np
import woodwork as ww
from evalml.utils.gen_utils import _rename_column_names_to_numeric

df = pd.DataFrame([[1,2,3], [10,20,30], [100,200,300]])
df.columns = pd.MultiIndex.from_tuples((("a", "b"), ("a", "c"), ("d", "f")))
df = ww.DataTable(df)
df = _rename_column_names_to_numeric(df)

[angela97lin]

Thanks @gsheni for the simple repro! Yup, it looks like the issue is that _rename_column_names_to_numeric doesn't handle pd.MultiIndex very well. In general, I don't think we've done too much testing with passing in dataframes with MultiIndex 😬 . I can put up a fix for this particular case and try to run AutoMLSearch to see if anything else breaks.

To explain better, _rename_column_names_to_numeric was introduced to help handle weird string names that LightGBM and XGBoost don't play well with. It remaps column names to numeric values so that we don't have to worry about conflicting names that might result from stripping the unaccepted characters. However, this assumes that each column name can be mapped to an int. I can update the impl to use Woodwork's newer rename method (rather than having to convert to pandas and then back as before), but if a column name was originally a tuple, we have to give a tuple in return or else a ValueError will be thrown.

Edit: even after updating _rename_column_names_to_numeric to handle MultiIndex, we still run into issues because the LightGBM classifier/regressor cannot handle tuples as column names, resulting in the following stack trace:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
~/Desktop/evalml/evalml/pipelines/components/component_base.py in fit(self, X, y)
     99         try:
--> 100             self._component_obj.fit(X, y)
    101             return self

~/Desktop/venv/lib/python3.7/site-packages/lightgbm/sklearn.py in fit(self, X, y, sample_weight, init_score, eval_set, eval_names, eval_sample_weight, eval_class_weight, eval_init_score, eval_metric, early_stopping_rounds, verbose, feature_name, categorical_feature, callbacks, init_model)
    837                                         categorical_feature=categorical_feature,
--> 838                                         callbacks=callbacks, init_model=init_model)
    839         return self

~/Desktop/venv/lib/python3.7/site-packages/lightgbm/sklearn.py in fit(self, X, y, sample_weight, init_score, group, eval_set, eval_names, eval_sample_weight, eval_class_weight, eval_init_score, eval_group, eval_metric, early_stopping_rounds, verbose, feature_name, categorical_feature, callbacks, init_model)
    599                               verbose_eval=verbose, feature_name=feature_name,
--> 600                               callbacks=callbacks, init_model=init_model)
    601 

~/Desktop/venv/lib/python3.7/site-packages/lightgbm/engine.py in train(params, train_set, num_boost_round, valid_sets, valid_names, fobj, feval, init_model, feature_name, categorical_feature, early_stopping_rounds, evals_result, verbose_eval, learning_rates, keep_training_booster, callbacks)
    230     try:
--> 231         booster = Booster(params=params, train_set=train_set)
    232         if is_valid_contain_train:

~/Desktop/venv/lib/python3.7/site-packages/lightgbm/basic.py in __init__(self, params, train_set, model_file, model_str, silent)
   1982             # construct booster object
-> 1983             train_set.construct()
   1984             # copy the parameters from train_set

~/Desktop/venv/lib/python3.7/site-packages/lightgbm/basic.py in construct(self)
   1324                                 silent=self.silent, feature_name=self.feature_name,
-> 1325                                 categorical_feature=self.categorical_feature, params=self.params)
   1326             if self.free_raw_data:

~/Desktop/venv/lib/python3.7/site-packages/lightgbm/basic.py in _lazy_init(self, data, label, reference, weight, group, init_score, predictor, silent, feature_name, categorical_feature, params)
   1150         # set feature names
-> 1151         return self.set_feature_name(feature_name)
   1152 

~/Desktop/venv/lib/python3.7/site-packages/lightgbm/basic.py in set_feature_name(self, feature_name)
   1627                                  .format(len(feature_name), self.num_feature()))
-> 1628             c_feature_name = [c_str(name) for name in feature_name]
   1629             _safe_call(_LIB.LGBM_DatasetSetFeatureNames(

~/Desktop/venv/lib/python3.7/site-packages/lightgbm/basic.py in <listcomp>(.0)
   1627                                  .format(len(feature_name), self.num_feature()))
-> 1628             c_feature_name = [c_str(name) for name in feature_name]
   1629             _safe_call(_LIB.LGBM_DatasetSetFeatureNames(

~/Desktop/venv/lib/python3.7/site-packages/lightgbm/basic.py in c_str(string)
    130     """Convert a Python string to C string."""
--> 131     return ctypes.c_char_p(string.encode('utf-8'))
    132 

AttributeError: 'tuple' object has no attribute 'encode'

I believe the other estimators avoid this issue because the input is transformed to a numpy array. Since I don't think indices are taken into consideration (same DF with and without multi-index are treated the same), I will update _rename_column_names_to_numeric and to handle tuples, and LightGBM to handle tuples by mapping tuple names to a flattened string equivalent.

[angela97lin]

@grayskripko We recently released a new version of EvalML (0.18.2) which introduces a fix for MultiIndex issues. Please try it out and let us know if this fixes your original issue! 😁