Lightgbm: [python] Adding learning_rates to args of sklearn API

[mlisovyi]

The training API in the python package has a very handy feature of adjusting learning rate with iterations. However, this is not on the list of args of LGBMModel.fit(). Is there any particular reason not to do so? Otherwise, this feature would be very welcome.

From the sklearn.py code structure it seems to be as easy as adding an arg to the fit() method and passing it over to the train() call.

Thanks in advance,
Misha

[guolinke]

I am not very familiar with the sklearn interface. However, I think learning_rates isn't the standard features of sklearn. I don't know adding this is right or not, since it may break the compatibility with sklearn.
@StrikerRUS any ideas about this ?

[mlisovyi]

@guolinke you are right, learning_rates is not a standard feature of the sklearn interface. However, it seems that there are many non-standard args of the LGBMModel.fit() method. Since I'm a new user I might have overlooked a basic conceptual reason from the LightGBM philosophy.

[StrikerRUS]

@guolinke I have the same thoughts as you. Sklearn interface itself is very conservative one. On the one hand, I don't think that overloading sklearn interface by all possible features is good idea, because it will confuse new users, who expect only standard (for sklearn estimatior) fit(), .predict() behavior with minimum params. On the other hand, LightGBM's sklearn wrapper has already additional functionality, so maybe it's OK to add another one.

[mlisovyi]

Any decision on this issue?

I personally find the learning_rates very useful. But as a user I also strongly prefer to use the sklearn API, as it allows to make use of many nice sklearn functionalities. Thus, it would be wonderful to have such advanced (and useful) feature available in this interface. This arg can have the default disabled value and a beginner user doesn't have to be exposed to it

[MangoTheCat5]

@mlisovyi

Can't you do this by passing an argument as a reset_parameter callback in the sklearn api?

[mlisovyi]

@stevenknguyen This sounds interesting. Since I'm a new user, I have not mastered LightGBM callbacks yet. Do you have an example of what you have in mind?

[guolinke]

@mlisovyi refer to https://lightgbm.readthedocs.io/en/latest/Python-API.html#lightgbm.reset_parameter
you can use callbacks= [callback.reset_parameter(learning_rate=learning_rates)]

you can close this issue if you think that is okay.

[MangoTheCat5]

@mlisovyi

Something like this should work; this is using the default example where the learning rate is multiplied by .99 at every iteration.

import numpy as np
import lightgbm as lgb
model = lgb.LGBMRegressor()

model.fit(X, y,
callbacks = [lgb.reset_parameter(learning_rate=lambda iter: base_learning rate  * np.pow(.99, iter))]
)

[mlisovyi]

Thanks a lot! This looks easy and elegant. Let me try it out.
Just to be sure: base_learning_rate set to the learning_rate input arg value, isn't it?

[MangoTheCat5]

@mlisovyi

It's just a placeholder. You need to replace base_learning_rate with whatever learning rate is best for your problem.

[StrikerRUS]

Just to make it clear: under the hood learning_rates param works exactly via reset_parameter callback.

https://github.com/Microsoft/LightGBM/blob/04d37f9b38cb851c5201081c2b8289eb0ecbca02/python-package/lightgbm/engine.py#L167-L168

So I think we shouldn't duplicate the same functionality and issue may be closed.

[mlisovyi]

Thanks for help and advices! I think it would be useful to add a remark.instruction about learning rate shrinkage implementation in the sklearn interface

[StrikerRUS]

@mlisovyi You are welcome!

Yeah, I'll try to do it. If you know the appropriate place where to insert a note, please let me know or create a PR.