Autogluon: Docs request: detailed docs for data preprocessing procedure

[kaheicanaan]

My question is similar to #545. Preprocessing procedure described in docs and paper Chapter 2.2 are oversimplified and I found difficulties to decide whether I should manually generate certain commonly used features (e.g. z-score, bin, etc.).

Reading source code does help me to understand how data preprocessing work has been done but it is time consuming. Is it possible to provide a detailed description about the data preparation process like this page: Data preparation that AutoML Tables does for you?

[Innixma]

Great suggestion! I think this is one of the most opaque and confusing aspects of AutoGluon as present.

I am in the process of refactoring various aspects of feature generation to be more standardized and flexible, as well as allowing multiple feature generators at once as opposed to just a single feature generator (important for multi-modal data and pipeline optimization). With these changes, I hope to also allow users to pass in pre-fit feature generators as input.

Alongside these improvements, I will be expanding documentation heavily, including adding an in-depth tutorial notebook on specifying and customizing feature generation. This is planned for 0.1 release.

If anyone has particular recommendations on what aspects of feature generation are most important to focus on in documentation, please feel free to discuss in this thread.

[diff7]

If anyone has particular recommendations on what aspects of feature generation are most important to focus on in documentation, please feel free to discuss in this thread.

I guess something like that:
(I am not sure if the information below it is correct by the way )

There are two types or feature reprocessing pipelines common for all models and specific for each model.
Common process mainly includes data cleaning:

1. We drop features with high variance var > max_th.
2. Remove columns with many missing values num_na > max_th.
3. Drop duplicates

Specific to each model:

1. Linear regression and NN.
   
   
   
   • Box cox for skewed data
   
   
   • Normalization
   
   
   • etc
   
   

1. Random forest and LightGBM.
   
   
   
   • binning ( I saw it somewhere not sure if it was about this model )
   
   
   
   

Handling categorical features:

• max cardinality for cat_features is ?
-

I guess I could wrap it up for you if I had some raw material. But I think it is not difficult to do once you have it ready ))