Lightgbm: how to accelerate the predict?

[l616q]

[l616q]

Hello:
I use C++ API to call the function of c_api.h to load model and to do predict, online. But there are so many threads to call the predictor, so the predictions are very slow. Is there some method to accelerate the predict? thank you.

[dmitryikh]

• Do you use categorical variables along with numerical? Categorical variables use search in bitmask for decision rule and sometimes it could be inefficient
• Which API function do you call?
  
  
  
  • LGBM_BoosterPredictForFile seems not good choice because of I/O latency (especially in multithreading).
  
  
  • LGBM_BoosterPredictForCSR or LGBM_BoosterPredictForMat are good choice. Try to combine your feature vectors into large batches!
  
  

C API use OMP to parallelize predictions. But in you case, when you call API from different threads, turning off this parallelization could be a solution. Try to set environment variable:
OMP_NUM_THREADS=1

[l616q]

• Do you use categorical variables along with numerical? Categorical variables use search in bitmask for decision rule and sometimes it could be inefficient

• Which API function do you call?

  • LGBM_BoosterPredictForFile seems not good choice because of I/O latency (especially in multithreading).
  • LGBM_BoosterPredictForCSR or LGBM_BoosterPredictForMat are good choice. Try to combine your feature vectors into large batches!

C API use OMP to parallelize predictions. But in you case, when you call API from different threads, turning off this parallelization could be a solution. Try to set environment variable:
OMP_NUM_THREADS=1

Thank you for replay.
1、We only use numeric feature， no categorical variables
2、We use LGBM_BoosterPredictForMat .But due to the limit of out system, we can only pass one feature vector to the model for one pass of prediction.
3、We have tested just using one thread to call the predict function(LGBM_BoosterPredictForMat), the time cost is 50us. It is too slow.
so:
1、Is there some global variable or lock in the process of predict to maintain status to ensure the thread safe in lgbm? I feel it should be more quick than 50us for doing just one prediction through the tree model

[wagnew3]

https://github.com/dmlc/treelite has often been helpful for me.

[l616q]

https://github.com/dmlc/treelite has often been helpful for me.

Thanks a lot. I will try it!

[StrikerRUS]

Linking #856, #906.