Autogluon: Is there a way to view feature importance and pruned features of a trained predictor?

[ccll]

Since autogluon has a feature_prune parameter for the fit() method, there should have a way to get the importance of all features, and to get a list of pruned features.

[Innixma]

This is a feature we are currently working on. If a user tries to specify feature_prune in v0.0.5 it will display a warning and automatically set to False, as the current feature pruning code is not robust enough to be ready for users. Once feature_prune functionality is ready, obtaining feature importance for all features and the list of pruned features will be made available at the same time.

[ccll]

@Innixma Thanks for the info.
Can't wait to see the new release. :smile:

[Innixma]

Hi @ccll ,

The latest AutoGluon 0.0.6 now has built in feature importance support for Tabular. After getting your fitted predictor object, simply call predictor.feature_importance(X_test) or predictor.feature_importance().

Documentation: https://autogluon.mxnet.io/api/autogluon.task.html#autogluon.task.tabular_prediction.TabularPredictor.feature_importance

Note that feature pruning is still not implemented at the moment, and I plan to add this in a later version.

If you get a chance to try out feature importance, let me know what you think!

[ccll]

@Innixma
Yohoo! Thank you for the great work.
I'll try that later.

[diff7]

Guys, awesome library. Thank you!

How do you calculate feature importance, is it something similar to Shap value?
And do use the same strategy for dropping features during training as in feature_importance(X_test)?

Also, do you only prune features with negative values?

Thank you!

[Innixma]

@diff7 We use permutation-shuffle importance, a black-box feature importance method. It is very similar (and potentially identical) to the recently added sklearn permutation_importance: https://scikit-learn.org/stable/modules/permutation_importance.html

We currently do not do any feature pruning during training, but in future we do plan to do this, and when we do it will be using the permutation-importance method for dropping. Regarding which features to prune, small positive values can also be harmful, but it is difficult to identify without retraining the model. It is generally helpful to have a rising prune threshold with repeated model training, and then using a form of early_stopping to pruning iterations to find the best feature set.

[diff7]

Hi, thank you for your reply.

If you do not do feature pruning during training how
feature_prunebool, default = False in fit is used?

Or it can be used only after one session of training?

[Innixma]

feature_prune = True will log a warning indicating that feature pruning is not available, and will force it to false. We do this to maintain backwards compatibility.

[diff7]

Got it, thank you!