Modernmt: Resume a training restarting with a different learning rate ?

[EtienneAb3d]

Is it possible to resume a training restarting with a different (higher) learning rate ?

In the log, it seems the initial training is launched with LR=2, isn't it ? How to understand the values on this curve ?

[davidecaroselli]

Hi @EtienneAb3d

@nicolabertoldi is working on this feature in the branch features/batch_tuing. You will be able to start a training process from a model already trained.

That LR plot is very common and perfectly ok. You can easily observe two steps:

1. During the warmup steps the leraning rate is increased from 0.0002 up to 0.001.
2. At this point, the training continues gradually reducing the learning rate (usually with a reversed square root algo).

[EtienneAb3d]

Training from an already trained model would solve the question.

But, here, I would only like to change the learning rate during a given training. As said in an old discussion, decreasing the learning rate is making a simulated annealing. This should bring a much better result, if it is warmed up several times, doing several successive convergences.

For the curve, what I do not understand, is this line in the LOG. Why 2.0, and not 0.0002 ?
INFO:tensorflow:Base learning rate: 2.000000

[davidecaroselli]

But, here, I would only like to change the learning rate during a given training.

I see, once the feature is ready you can "restart" the training from where you stop it with a different learning rate.

For the curve, what I do not understand, is this line in the LOG. Why 2.0, and not 0.0002 ?

Yeah.. this is because tensor2tensor uses a combination of "learning rate" and "scaling factor", not very clear indeed. I think the intention was to use the same learning rate with different Optimizers, having every one of them the right scaling factor.

[davidecaroselli]

Hi @EtienneAb3d

now with v4 you can restart a training from an already existing model with option --from-model. You can also change every possibile option of the underlying fairseq-train, including learning rate!

Thanks for your patience with this.