Lightgbm: how to use dynamic learning_rate in lightgbm?

[tonylibing]

Is there any methods to dymically decrease learning_rate parameter by training steps or some pre-designed function in lightGBM ?

[StrikerRUS]

It is possible via learning_rates argument of train() function or, in general case, via reset_parameter() callback.

Examples:
https://github.com/Microsoft/LightGBM/blob/a694712b7fb86cd532eea2c1781b58d4ba58436a/examples/python-guide/advanced_example.py#L117-L122
https://github.com/Microsoft/LightGBM/blob/a694712b7fb86cd532eea2c1781b58d4ba58436a/examples/python-guide/advanced_example.py#L127-L132

[tonylibing]

thanks, i had found the learning_rates and callbacks parameters. just a little cofusion, in the function i designed to decrease the learning_rate, how do i get the current iteration num or other informations like the last n iterations grid or error which help to calculate learning_rate?

[StrikerRUS]

You can find examples of different callbacks in the callback module.

And here is the environment which is available for callbacks:
https://github.com/Microsoft/LightGBM/blob/a694712b7fb86cd532eea2c1781b58d4ba58436a/python-package/lightgbm/callback.py#L31-L38

[julioasotodv]

However, keep in mind that shrinking the learning rate over increasing boosting rounds could lead to tremendously high overfitting; so I guess that you will rarely want to decrease the learning rate over time (as opposed to in deep learning, where the lower bias in the model across epochs helps mitigating the overfit caused by the shrinkage)

[louis925]

@julioasotodv Could you explain more about why reducing learning rate while increasing boosting rounds can lead to overfit? In my own experiments, I found that this approach improve the CV scores.

[julioasotodv]

@louis925 It sure does it in the CV scores. Just try in a separate test set ;)