Autogluon: Export feature matrix

[braaannigan]

I'd like to be able to export the final feature matrix that autogluon creates. For example, in the tabular case I'd like to see if I can replicate the model performance using a synthetic dataset created from a tabular GAN model. Is this possible?

[Innixma]

This is technically possible, but it depends on what level of preprocessing we are talking about.

Preprocessing in Tabular works at different stages:

Stage 1: The generic preprocessing stage which does automated FE, type inference, ngram generation, etc. This is then passed to the training logic as the 'synthetic dataset' which models are trained with.

Stage 2: Stacker preprocessing stage. This effectively is the same as Stage 1, but with the OOF (Out of Fold) predictions of lower layer stacker models added as features to the dataset.

Stage 3: Then there is the model specific preprocessing. Each ML model does an additional preprocessing stage to get the data in its preferred form. For the NN, this is converting the data to an MXNet representation, for LightGBM, it is converting it to a lgb.Dataset(), for KNN, it is dropping categorical columns and imputing NaNs.

For stage 1 (Generic preprocessing), this is trivial to add for both training data and test data.

For stage 2 (Stacker preprocessing), this is trivial to add for test data. For training data it is complex because between stage 1 and 2, AutoGluon has the ability to augment the training set how it sees fit such as duplicating rows (to avoid edge-case crashes during bagging) and dropping rows (if it considers them 'invalid' and in future for subsampling huge datasets). If you are ok with the training data returned to you missing some amount of rows (or some rows appearing multiple times), this is an easy add (effectively it would return to you the input dataset to the next stacker layer if one were to be trained), otherwise if you require the exact original training set rows, it becomes significantly more complex to implement. It is still possible, although it would be a mash up of OOF prediction generated data (non-bagged inference) and inference-time generated data (bagged inference).

For stage 3 (Model specific preprocessing), this is trivial to add for test data. For training data it shares all the same complexities with stage 2. Furthermore, the output of this stage is highly specialized to the particular model, and likely not very useful to the user who wishes to make their own custom model from the data.

Which of stage 1, stage 2, and stage 3 are you interested in, and for which of training/test data (or both)? Is there anything that you are looking for that I missed?

[Innixma]

I gave this a shot and have created #431 which should enable the retrieval of feature matrices for stage 1 and 2 as described above. I will notify you when it has been merged. Let me know if this addresses your request!

[braaannigan]

Thanks!

[Innixma]

Hi @braaannigan,

I believe I have added the functionality you requested. Please use the latest mainline to try-out the new functionality.

As an example code:

predictor = task.load('my_predictor/')
test_data = task.Dataset('test.csv')
test_data_transformed = predictor.transform_features(test_data)
test_data_transformed_with_lightgbm_l1_features = predictor.transform_features(test_data, model='lightgbm_l1') # If you are using stacking and want the inputs to a particular stacker model
test_data_labels_transformed = predictor.transform_labels(test_data[LABEL])
test_data_labels_original = predictor.transform_labels(test_data_labels_transformed, inverse=True)
train_data_transformed, train_data_labels_transformed = predictor.load_data_internal(dataset='train')
val_data_transformed, val_data_labels_transformed = predictor.load_data_internal(dataset='val') # Only for non-bagged
train_data_transformed_with_out_of_fold_stack_features = predictor.transform_features(base_models=[{list of base model names for your stacker}]) # Only for bagged mode

your_new_stacker_model = your_new_stacker_model_func.fit(X_train=train_data_transformed_with_out_of_fold_stack_features, y_train=train_data_labels_transformed)
your_new_normal_model = your_new_normal_model_func.fit(X_train=train_data_transformed, y_train=train_data_labels_transformed, X_val=val_data_transformed, y_val=val_data_labels_transformed)

test_data_transformed_stacker_input = predictor.transform_features(dataset=test_data, base_models=[{list of base model names for your stacker}])

test_data_transformed_pred_proba = your_new_stacker_model_func.predict_proba(test_data_transformed_stacker_input)

test_data_pred_proba = predictor.transform_labels(test_data_transformed_pred_proba, inverse=True, proba=True)

your_model_test_score = predictor.evaluate_predictions(y_true=test_data_labels_original, y_pred=test_data_pred_proba) # Must have used one of the pred_proba metrics, such as logloss or AUC)

Let me know how it works for your use case and if anything is missing from this implementation.

[braaannigan]

Hi @Innixma that works great, thanks! Will let you know if the GAN approach is useful for my problem