Autogluon: How to only train some of the models in TabularPrediction.fit()
In my task the neural network component is adding a lot to the training time - which isn't a big deal - and a lot to the inference time - which is more important. The NN component isn't adding a lot to the already excellent performance of the axis-aligned models in this case.
I'd like to fit the model without the NN. It appears that models can be specified in the hyperparameter option as
hyperparameters={'custom':['GBM']}
but that only GBM is supported at the moment. Can this be extended to the other models?
braaannigan
All 12 comments
Further info: in my case it takes about 30 seconds to do predictions on 1000 rows of data with the level 0 cat model, but I gave up before making any predictions with the full model - even on a single row it ran for at least 10 minutes without returning anything
braaannigan
on 2 Apr 2020
Apologies we should clarify our TabularPrediction documentation.
The hyperparameters dict currently has the following possible keys:
'NN', 'GBM', 'CAT', 'RF', 'XT', 'KNN', 'custom'
corresponding to 7 different models that can be trained.
To omit one of these models during task.fit(), you simply drop that key from the
hyperparameters dict. For example, you can tell AutoGluon to only consider RF and GBM models (with their default hyperparameter settings) via:
task.fit(..., hyperparameters={'RF':{}, 'GBM':{}})
The 'custom' key offers different functionality than you described. You should not modify the GBM value for this key, but you can drop the key from hyperparameters if you'd like to omit the custom model. If you include 'custom' in hyperparameters, then AutoGluon will train additional custom models. For now, the only additional custom model is a _second_ GBM model with different (preset) hyperparameter settings than the default GBM model that gets trained when hyperparameters contains the GBM key.
jwmueller
on 2 Apr 2020
Note: if you mainly care about prediction time (inference latency), you also don't necessarily need to prevent AutoGluon from training all of its models. If you don't require a high-accuracy ensemble, you can instead just tell AutoGluon to use the fastest of these trained models at inference time, like this:
First, invoke your default predictor = task.fit() call without specifying the auto_stack, hyperparameter_tune or bagging-related arguments. Here there is no need to specify the hyperparameters dict, so AutoGluon will train all of its models (if the fit() time is ok with you; if you find the fit() time too long, then specify only a subset of the models in the hyperparameters dict as I outlined in the previous comment).
Second, call predictor.leaderboard(only_pareto_frontier=True) and find the model with the smallest value of 'pred_time_val' (this model is the fastest for predictions), suppose its name is: MODEL_NAME.
Note: make sure you've installed the latest version of autogluon from the master branch to access this functionality.
To use solely this faster model for your predictions, simply call:
predictor.predict(test_data, model = MODEL_NAME)
If you care about online inference with individual test data-points聽one-at-a-time, then you should first persist the models in memory like this (Note: this functionality is still under active development and not documented):
predictor._learner.persist_trainer()
pred1 = predictor.predict(test_datapoint1, model=MODEL_NAME)
pred2 = predictor.predict(test_datapoint2, model=MODEL_NAME)
...
To remove NN from training, set hyperparameters to:
hyperparameters = {
# 'NN': {'num_epochs': 500},
'GBM': {'num_boost_round': 10000},
'CAT': {'iterations': 10000},
'RF': {'n_estimators': 300},
'XT': {'n_estimators': 300},
'KNN': {},
'custom': ['GBM'],
}
Further info: in my case it takes about 30 seconds to do predictions on 1000 rows of data with the level 0 cat model, but I gave up before making any predictions with the full model - even on a single row it ran for at least 10 minutes without returning anything
I'm curious to know the scale of the data you are dealing with for a single row to take 10 minutes to infer with the ensemble. Does it contain text features? I don't think I've ever seen a model take that long on inference. Do you have Intel MKL installed? If so, I would recommend uninstalling it -> Intel MKL drastically slows down the NN train and inference time. I actually suspect it is KNN causing the extreme slowdown, and not the NN.
Can you provide the output of calling predictor.leaderboard()? This will confirm which models are causing the slowdown.
Innixma
on 2 Apr 2020
Great stuff everyone!
Thinking about it today I was wondering if my use case is a bit different from the standard tabular task. My dataset are news articles pulled from a news api so one of the columns is a potentially long text article in contrast to typical tabular examples where text entries are generally short. This gives rise to a high dimensionality in the text vectorizer (often about 30,000 dimensions). The fit_time I was getting for the NN was then about 3 hours - leaderboard shown below:
model score_val fit_time pred_time_val stack_level
26 weighted_ensemble_k0_l3 0.939854 0.797751 0.001496 3
22 CatboostClassifier_STACKER_l2 0.937605 245.529041 39.122814 2
25 weighted_ensemble_k0_l2 0.935919 0.801807 0.001595 2
14 CatboostClassifier_STACKER_l1 0.935919 308.329618 47.665054 1
24 weighted_ensemble_k0_l1 0.933671 1.045179 0.003294 1
6 CatboostClassifier_STACKER_l0 0.933671 447.286274 44.032478 0
16 RandomForestClassifierGini_STACKER_l2 0.931984 42.927477 33.485102 2
19 ExtraTreesClassifierEntr_STACKER_l2 0.931422 46.821788 33.330483 2
8 RandomForestClassifierGini_STACKER_l1 0.930860 52.614044 38.979910 1
17 RandomForestClassifierEntr_STACKER_l2 0.929174 42.595070 33.397289 2
9 RandomForestClassifierEntr_STACKER_l1 0.927487 51.144368 38.967965 1
18 ExtraTreesClassifierGini_STACKER_l2 0.927487 47.120970 33.517305 2
10 ExtraTreesClassifierGini_STACKER_l1 0.923553 57.305387 37.488566 1
11 ExtraTreesClassifierEntr_STACKER_l1 0.921866 56.856822 37.970667 1
7 NeuralNetClassifier_STACKER_l0 0.912310 12413.722548 2074.693989 0
15 NeuralNetClassifier_STACKER_l1 0.910624 10131.377533 1746.513610 1
23 NeuralNetClassifier_STACKER_l2 0.908375 8606.215128 1371.583073 2
3 ExtraTreesClassifierEntr_STACKER_l0 0.907251 57.921314 37.779187 0
2 ExtraTreesClassifierGini_STACKER_l0 0.905003 58.957216 38.242619 0
0 RandomForestClassifierGini_STACKER_l0 0.896571 52.312319 38.171415 0
1 RandomForestClassifierEntr_STACKER_l0 0.894323 51.920280 38.247511 0
13 KNeighborsClassifierDist_STACKER_l1 0.868465 99.857924 95.230456 1
21 KNeighborsClassifierDist_STACKER_l2 0.867903 76.748443 71.594591 2
12 KNeighborsClassifierUnif_STACKER_l1 0.864531 93.522354 84.638263 1
5 KNeighborsClassifierDist_STACKER_l0 0.862844 88.654842 82.695507 0
20 KNeighborsClassifierUnif_STACKER_l2 0.862282 76.861186 71.923305 2
4 KNeighborsClassifierUnif_STACKER_l0 0.858909 89.526703 81.870069 0
Obviously in this case the NN is helping much and I should just run without it.
braaannigan
on 2 Apr 2020
Great stuff everyone!
Thinking about it today I was wondering if my use case is a bit different from the standard tabular task. My dataset are news articles pulled from a news api so one of the columns is a potentially long text article in contrast to typical tabular examples where text entries are generally short. This gives rise to a high dimensionality in the text vectorizer (often about 30,000 dimensions). The fit_time I was getting for the NN was then about 3 hours - leaderboard shown below:
model score_val fit_time pred_time_val stack_level
26 weighted_ensemble_k0_l3 0.939854 0.797751 0.001496 3
22 CatboostClassifier_STACKER_l2 0.937605 245.529041 39.122814 2
25 weighted_ensemble_k0_l2 0.935919 0.801807 0.001595 2
14 CatboostClassifier_STACKER_l1 0.935919 308.329618 47.665054 1
24 weighted_ensemble_k0_l1 0.933671 1.045179 0.003294 1
6 CatboostClassifier_STACKER_l0 0.933671 447.286274 44.032478 0
16 RandomForestClassifierGini_STACKER_l2 0.931984 42.927477 33.485102 2
19 ExtraTreesClassifierEntr_STACKER_l2 0.931422 46.821788 33.330483 2
8 RandomForestClassifierGini_STACKER_l1 0.930860 52.614044 38.979910 1
17 RandomForestClassifierEntr_STACKER_l2 0.929174 42.595070 33.397289 2
9 RandomForestClassifierEntr_STACKER_l1 0.927487 51.144368 38.967965 1
18 ExtraTreesClassifierGini_STACKER_l2 0.927487 47.120970 33.517305 2
10 ExtraTreesClassifierGini_STACKER_l1 0.923553 57.305387 37.488566 1
11 ExtraTreesClassifierEntr_STACKER_l1 0.921866 56.856822 37.970667 1
7 NeuralNetClassifier_STACKER_l0 0.912310 12413.722548 2074.693989 0
15 NeuralNetClassifier_STACKER_l1 0.910624 10131.377533 1746.513610 1
23 NeuralNetClassifier_STACKER_l2 0.908375 8606.215128 1371.583073 2
3 ExtraTreesClassifierEntr_STACKER_l0 0.907251 57.921314 37.779187 0
2 ExtraTreesClassifierGini_STACKER_l0 0.905003 58.957216 38.242619 0
0 RandomForestClassifierGini_STACKER_l0 0.896571 52.312319 38.171415 0
1 RandomForestClassifierEntr_STACKER_l0 0.894323 51.920280 38.247511 0
13 KNeighborsClassifierDist_STACKER_l1 0.868465 99.857924 95.230456 1
21 KNeighborsClassifierDist_STACKER_l2 0.867903 76.748443 71.594591 2
12 KNeighborsClassifierUnif_STACKER_l1 0.864531 93.522354 84.638263 1
5 KNeighborsClassifierDist_STACKER_l0 0.862844 88.654842 82.695507 0
20 KNeighborsClassifierUnif_STACKER_l2 0.862282 76.861186 71.923305 2
4 KNeighborsClassifierUnif_STACKER_l0 0.858909 89.526703 81.870069 0
Obviously in this case the NN is helping much and I should just run without it.
braaannigan
on 2 Apr 2020
Great stuff everyone!
Thinking about it today I was wondering if my use case is a bit different from the standard tabular task. My dataset are news articles pulled from a news api so one of the columns is a potentially long text article in contrast to typical tabular examples where text entries are generally short. This gives rise to a high dimensionality in the text vectorizer (often about 30,000 dimensions). The fit_time I was getting for the NN was then about 3 hours - leaderboard shown below:
model score_val fit_time pred_time_val stack_level
26 weighted_ensemble_k0_l3 0.939854 0.797751 0.001496 3
22 CatboostClassifier_STACKER_l2 0.937605 245.529041 39.122814 2
25 weighted_ensemble_k0_l2 0.935919 0.801807 0.001595 2
14 CatboostClassifier_STACKER_l1 0.935919 308.329618 47.665054 1
24 weighted_ensemble_k0_l1 0.933671 1.045179 0.003294 1
6 CatboostClassifier_STACKER_l0 0.933671 447.286274 44.032478 0
16 RandomForestClassifierGini_STACKER_l2 0.931984 42.927477 33.485102 2
19 ExtraTreesClassifierEntr_STACKER_l2 0.931422 46.821788 33.330483 2
8 RandomForestClassifierGini_STACKER_l1 0.930860 52.614044 38.979910 1
17 RandomForestClassifierEntr_STACKER_l2 0.929174 42.595070 33.397289 2
9 RandomForestClassifierEntr_STACKER_l1 0.927487 51.144368 38.967965 1
18 ExtraTreesClassifierGini_STACKER_l2 0.927487 47.120970 33.517305 2
10 ExtraTreesClassifierGini_STACKER_l1 0.923553 57.305387 37.488566 1
11 ExtraTreesClassifierEntr_STACKER_l1 0.921866 56.856822 37.970667 1
7 NeuralNetClassifier_STACKER_l0 0.912310 12413.722548 2074.693989 0
15 NeuralNetClassifier_STACKER_l1 0.910624 10131.377533 1746.513610 1
23 NeuralNetClassifier_STACKER_l2 0.908375 8606.215128 1371.583073 2
3 ExtraTreesClassifierEntr_STACKER_l0 0.907251 57.921314 37.779187 0
2 ExtraTreesClassifierGini_STACKER_l0 0.905003 58.957216 38.242619 0
0 RandomForestClassifierGini_STACKER_l0 0.896571 52.312319 38.171415 0
1 RandomForestClassifierEntr_STACKER_l0 0.894323 51.920280 38.247511 0
13 KNeighborsClassifierDist_STACKER_l1 0.868465 99.857924 95.230456 1
21 KNeighborsClassifierDist_STACKER_l2 0.867903 76.748443 71.594591 2
12 KNeighborsClassifierUnif_STACKER_l1 0.864531 93.522354 84.638263 1
5 KNeighborsClassifierDist_STACKER_l0 0.862844 88.654842 82.695507 0
20 KNeighborsClassifierUnif_STACKER_l2 0.862282 76.861186 71.923305 2
4 KNeighborsClassifierUnif_STACKER_l0 0.858909 89.526703 81.870069 0
Obviously in this case the NN is helping much and I should just run without it.
braaannigan
on 3 Apr 2020
Thanks for the info! I have mainly focused on standard tabular problems and haven't had a chance to deep dive into text problem performance. Very interesting that NN is impacted so heavily by text features. I may have to look into this and potentially disable text features for NN to keep it in-line with the other models. In the near future we intend to start adding fine-tuned BERT to the stack ensemble when text features are detected, which should lead to significantly improved results even above what AutoGluon already obtains.
Innixma
on 3 Apr 2020
Great - AG already matches what I get with a fine-tuned roberta model (using the text column alone in the latter case), so think a bert element would be very powerful
braaannigan
on 3 Apr 2020
If you want a neat trick right now to get a better result, you can probably do a weighted ensemble with equal weight of your BERT model and AG's model to get a major boost in score.
Innixma
on 3 Apr 2020
@braaannigan
If you are interested in using the bleed-edge mainline of AutoGluon, we just merged in #422 which dramatically speeds up NN inference times, particularly when doing inference on small amounts of data, as you attempted in this PR. We haven't tested it with truly massive datasets, but I suspect this may have been one of the reasons inference took so long for you. I'd be very interested in seeing if re-running this dataset on the latest mainline AutoGluon speeds up NN inference, it is possible this could result in an over 10x speedup.
Innixma
on 20 Apr 2020
With the merging of #440 and #451, I think AutoGluon should be able to handle these text problems reasonably and without long inference times. #440 disables NLP features in the NN, as I was also experiencing unreasonable inference latency on other problems, which returns NN to being similar to the other models for inference.
Innixma
on 6 May 2020
Related issues
mli
路
6Comments
songfh
路
5Comments
jwmueller
路
6Comments
sxjscience
路
6Comments
Pentar0o
路
8Comments