Autogluon: TabularPredictor's predict is very slow!

[elife33]

I had trained a TabularPredictor model, when I called evaluate() at 50k+ samples, it took 3.991s. But when I called predict() at a sample for 100 times, it took 151.334s. Why predict() is so slow?!

[Innixma]

Could you provide a code sample? Predictor.evaluate() should take at least as long as Predictor.predict(), as evaluate calls predict internally.

Can you also clarify what you mean by 'called predict() at a sample for 100 times', are you saying that you are only passing in 1 row, and that the average runtime was 1.5s?

If so, you can speed-up these kinds of predictions with the following line of code:

predictor._learner.persist_trainer(low_memory=False)

Be warned: This is not memory safe, and may run OOM if the AutoGluon ensemble is large and consists of many models. Inference speed optimizations are an ongoing effort and will be improved in future.

[elife33]

Thanks Innixma! Yes, I mean only passing in 1 row, and that the average runtime was 1.5s.
After I add predictor._learner.persist_trainer(low_memory=False), the average runtime reduced to 0.6s.

The code is below:
fn='/Users/elife/work/ai/ml/dl/autogluon/dtlQT/NNQT-Bn/AutogluonModels/ag-20200321_133804'
predictor_0= ag.task.tabular_prediction.predictor.TabularPredictor.load(fn)
fn='/Users/elife/work/ai/ml/dl/autogluon/dtlQT/NNQT-Bn/AutogluonModels/ag-20200321_134232'
predictor_1= ag.task.tabular_prediction.predictor.TabularPredictor.load(fn)
predictor_1._learner.persist_trainer(low_memory=False)

t = pd.read_csv('NNQT-Bn2-2010-2018.csv')
print(t.values)
print(t.describe())
N=int(t.describe().values[0][0])
print("N="+ str(N))

print(t.values[0][1])

print(t.values[N-1][45])

t1 = time.time()
performance = predictor_1.evaluate(task.Dataset(df = t))
t2 = time.time()
print("Time elapsed: %.3fs" % (t2 - t1))

test = pd.read_csv('test2.csv') #only a fieldname row and a data row
print(test)

def test3():
iModel=1

t1 = time.time()
N=10
for i in range(0,N):
    for j in range(0,45):
        test.iloc[0][j]=t.values[i+1][j]
    test_data = task.Dataset(df = test) # convert to AutoGluon dataset

    if iModel==0:
        y_predproba = predictor_0.predict_proba(test_data)
        assert(false)
    else:
        #y_predproba = predictor_1.predict_proba(test_data)
        y_predproba = predictor_1.predict(test_data)
    #print(i)
t2 = time.time()
print("N="+ str(N))
print("Time elapsed: %.3fs" % (t2 - t1))

and the output is:
elife@MBP16 Release % python3 t2.py
/usr/local/lib/python3.7/site-packages/mxnet/optimizer/optimizer.py:167: UserWarning: WARNING: New optimizer gluonnlp.optimizer.lamb.LAMB is overriding existing optimizer mxnet.optimizer.optimizer.LAMB
Optimizer.opt_registry[name].__name__))
/usr/local/lib/python3.7/site-packages/catboost/core.py:5: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
import imp
/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/imp.py:342: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
return _load(spec)
[[ 1. 0.585841 0.198659 ... 0.979662 1.030981 1.046633]
[ 1. -0.725186 -1.044784 ... 0.942425 0.953653 0.97273 ]
[ 0. 1.254479 0.149221 ... 0.967385 0.987973 0.971796]
...
[ 0. -0.362446 0.64556 ... 1.014428 1.01223 1.00414 ]
[ 1. -0.38475 -0.410357 ... 0.990597 0.996829 0.995052]
[ 0. 0.475772 -0.659354 ... 1.021401 1.048634 1.093851]]
class f1 f2 f3 f4 f5 f6 ... f39 f40 f41 f42 f43 f44 f45
count 53668.000000 53668.000000 53668.000000 53668.000000 53668.000000 53668.000000 53668.000000 ... 53668.000000 53668.000000 53668.000000 53668.000000 53668.000000 53668.000000 53668.000000
mean 0.500000 0.004941 0.002503 0.006410 0.003824 0.006323 0.004974 ... 1.009177 1.007724 0.999153 0.997875 0.999856 1.011356 1.009695
std 0.500005 1.004191 0.998373 1.007743 1.005423 1.007640 1.009622 ... 0.033706 0.038586 0.021763 0.033517 0.042279 0.041729 0.047446
min 0.000000 -8.589355 -3.428146 -4.119226 -3.425174 -8.859541 -3.262384 ... 0.798870 0.791047 0.917562 0.856822 0.810352 0.786655 0.793390
25% 0.000000 -0.630761 -0.512844 -0.627229 -0.520720 -0.673530 -0.535668 ... 0.991389 0.988462 0.990176 0.983227 0.981367 0.988229 0.984447
50% 0.500000 -0.161565 -0.186717 -0.200205 -0.203711 -0.211572 -0.210423 ... 1.007931 1.008118 1.000648 1.001025 1.003213 1.010004 1.010300
75% 1.000000 0.489563 0.180818 0.473610 0.192943 0.573805 0.206839 ... 1.027579 1.029411 1.010011 1.016046 1.023033 1.033613 1.034644
max 1.000000 12.162383 26.084646 11.789606 16.351372 6.586908 22.992096 ... 1.118984 1.126895 1.064539 1.095898 1.142495 1.152852 1.160384

[8 rows x 46 columns]
N=53668
Predictive performance on given dataset: accuracy = 0.5343034955653275
Time elapsed: 3.134s
f1 f2 f3 f4 f5 f6 f7 f8 f9 f10 f11 ... f35 f36 f37 f38 f39 f40 f41 f42 f43 f44 f45
0 -1.208727 -0.741521 -0.820433 -0.541281 2.978588 2.269558 -0.487983 -0.923333 -1.499667 -0.526631 -0.82941 ... 0.0 0.991852 1.000588 0.936482 0.94535 0.918283 0.993123 0.993891 0.920942 0.923123 0.877193

[1 rows x 45 columns]
N=10
Time elapsed: 6.344s
elife@MBP16 Release %

[Innixma]

Thanks for the logs! If you want faster prediction times, there are a few ways to do it. Note that several of these recommendations are not official features yet and are subject to change at any time.

1. Call predictor.leaderboard(X_test) to see prediction times (pred_time_test_full) and scores (score_test) for each of the models. You can choose the model which is fast from there and specifically get predictions from it with predictor.predict_proba(X_test, model=)
2. Ensure auto_stack=False for ~10x - 20x speedup (Default is False). Note that this sacrifices model quality.
3. Disable KNN model
   
   
   
   • Pass parameter with KNN removed:
   
   

hyperparameters = {
               'NN': {'num_epochs': 500},
               'GBM': {'num_boost_round': 10000},
               'CAT': {'iterations': 10000},
               'RF': {'n_estimators': 300},
               'XT': {'n_estimators': 300},
               'custom': ['GBM'],
}

• If still too slow, train with NN, RF, and XT removed as well.
  
  
  
  1. (Experimental) Do the following to get the absolute fastest result, note that this might sacrifice model quality as it disables ensembling. If you do this, I would recommend training the predictor with auto_stack=True to improve the performance of the final single model without increasing inference time:
  
  

# best_compressed_model can be re-used, it is the best single-model solution, which will have fast inference
best_base_model = predictor._learner.load_trainer().best_single_model(stack_name='core', stack_level=0)
best_compressed_model = predictor._learner.refit_single_full(models=[best_base_model])[0]
predictor._learner.persist_trainer(low_memory=False)
preds = predictor.predict_proba(dataset=X_test, model=best_compressed_model)

1. If you want it even faster, update hyperparameter values to adjust epochs / n_estimators to smaller values, such as 'RF': {'n_estimators': 50} instead of the default 300. (GBM/CAT/NN should go to ~100/100/20 to speed things up). This will be a heavy hit to model quality.

Using method 1 and 4 (using fastest model instead of best_base_model) I am able to get prediction times from 1.72s on 1 row to 0.02s. Let me know how this goes, and what kind of speed you are hoping to achieve!

[elife33]

Thanks for the kind and helpful answer! In my case, predictor.leaderboard shows
model score_test score_val fit_time pred_time_test_full pred_time_test pred_time_val stack_level
0 CatboostClassifier 0.536633 0.579560 3.186842 0.011617 0.011617 0.011825 0
1 RandomForestClassifierGini 0.535645 0.650602 3.384887 0.334701 0.334701 0.115491 0
2 RandomForestClassifierEntr 0.535217 0.654757 5.234998 0.330769 0.330769 0.113999 0
3 weighted_ensemble_k0_l1 0.534303 0.678853 1.015663 27.253049 0.037662 0.001440 1
4 LightGBMClassifierCustom 0.533036 0.656834 5.729446 0.217263 0.217263 0.019709 0
5 LightGBMClassifier 0.533018 0.631491 2.104079 0.085246 0.085246 0.016396 0
6 ExtraTreesClassifierEntr 0.530912 0.663482 1.353004 0.528460 0.528460 0.116623 0
7 ExtraTreesClassifierGini 0.529627 0.664312 1.141636 0.527142 0.527142 0.115134 0
8 NeuralNetClassifier 0.524372 0.616951 67.403422 0.852643 0.852643 0.102713 0
9 KNeighborsClassifierDist 0.516956 0.601994 0.062110 12.785751 12.785751 0.630741 0
10 KNeighborsClassifierUnif 0.513099 0.539260 0.071212 11.541796 11.541796 0.642612 0

so I picked the CatboostClassifier which is the fastest one and run time of 10 predicts is 0.210s.

I also tried method 4, and the run time of 10 predicts is 1.232s. It seems using ExtraTreesClassifierGini since the log said "
Fitting model: ExtraTreesClassifierGini_FULL ...
1.24s = Training runtime
"

In this my project, the speed as CatboostClassifier above is needed.

[Innixma]

Great to hear that this worked for you! You can combine this with

predictor._learner.persist_trainer(low_memory=False)

To speed it up even more.

Regarding method 4: You can use it with CatBoost for a better model overall:

improved_catboost_model = predictor._learner.refit_single_full(models=['CatboostClassifier'])[0]
predictor._learner.persist_trainer(low_memory=False)
preds = predictor.predict_proba(dataset=X_test, model=improved_catboost_model)

This should give similar runtimes but better predictions. The model returned from here is trained on 100% of the original training data, so you won't get a validation score, but it should be a better model.

We will be adding more options to make this easier to use and more visible to users. Specifically we plan to have a way for a user to specify a desired speed and then AutoGluon automatically selects the best model to satisfy that constraint.

I am marking this as resolved for now. If you have any further questions, feel free to reopen!