Xgboost: Determine feature importances

[dam-xx]

Hi; could you please indicate me the way to determine the importance of each feature? I'm using the Python interface and I could not find anything in the doc. Thanks!

[tqchen]

The algorithm automatically select features that can improve loss most. So
there is no need to specify feature importance...

On Friday, May 30, 2014, Damien Lefortier [email protected] wrote:

Hi; could you please indicate me the way to determine the importance of
each feature? I'm using the Python interface and I could not find anything
in the doc. Thanks!

—
Reply to this email directly or view it on GitHub
https://github.com/tqchen/xgboost/issues/11.

[dam-xx]

I understand what the algorithm does; my aim is different: looking at features importances, I could get some knowledge about which features work and which do not.

[tqchen]

In current python module, you can use text dump to dump out the tree, to
see what features are used.
There is undocumented extra option in console mode, you can dump with
xgboost console tool, with option dump_stats=1, this will dump additional
statistics ( gain for loss gain in that split and cover for sum weights in
the branch), that can help you to determine the importance of the feature.

Hope this answers your question

On Fri, May 30, 2014 at 9:24 AM, Damien Lefortier [email protected]
wrote:

I understand what the algorithm does; my aim is different: looking at
features importances, I could get some knowledge about which features work
and which do not.

—
Reply to this email directly or view it on GitHub
https://github.com/tqchen/xgboost/issues/11#issuecomment-44670216.

Sincerely,

Tianqi Chen
Computer Science & Engineering, University of Washington

[dam-xx]

Nice! Thanks, that works indeed although it requires some post processing :).

I think that a function such as dump_feature_strength(), which would return the weighted contribution of each feature to the model, would be nice. See e.g. Section 10.13 of the book "the elements of statistical learning" where such weights are defined.

Just an idea ;).

[tqchen]

Thanks for the suggestion. It is really hard to define a "strength score"
of a feature in GBM that all people agree, due to the complex model
contribution in tree ensemble.
So we allow user to dump out the model to make their own judgement.. But
you are right, this part could be improved.

On Fri, May 30, 2014 at 10:05 AM, Damien Lefortier <[email protected]

wrote:

Nice! Thanks, that works indeed although it requires some post processing
:).

I think that a function such as dump_feature_strength(), which would
return the weighted contribution of each feature to the model, would be
nice. See e.g. Section 10.13 of the book "the elements of statistical
learning" where such weighted are defined.

Just an idea ;).

—
Reply to this email directly or view it on GitHub
https://github.com/tqchen/xgboost/issues/11#issuecomment-44674560.

Sincerely,

Tianqi Chen
Computer Science & Engineering, University of Washington

[tqchen]

This is supported in the newest version.

https://github.com/tqchen/xgboost/blob/master/wrapper/xgboost.py#L400

[pommedeterresautee]

Hi, may be I am asking a lot, but would it be possible to have the same function for R implementation? If it represents work R people may take time to learn Python :-)

Kind regards
Michael

[tqchen]

Yes, @FeiYeYe and @hetong007 are working on this at issue #114

[StevenLOL]

Hi, seems same code in feature_importances can also work for XGBRegressor, thought it is not a function of XGBRegressor.

Can help to add this function to XGBRegressor ?

[gautham20]

We can get the feature importance for XGBRegressor, With the following code.

regr.booster().get_score(importance_type='weight')

Should be added as a function to XGBRegressor

[clarkzhao]

regr.booster().get_fscore()

works as well