Ignite: ReduceLROnPlateau LRScheduler error

[sudarshan85]

Hi,

I tried to use the LRScheduler handler for ReduceLROnPlateau from torch.optim and it threw an a TypeError:

plateau_scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer=optimizer, mode='min', factor=0.5, patience=1)
ignite_plateau_scheduler = LRScheduler(plateau_scheduler)

TypeError                                 Traceback (most recent call last)
<ipython-input-13-56cb18e5af57> in <module>
      1 plateau_scheduler = optim.lr_scheduler.ReduceLROnPlateau(optimizer=optimizer, mode='min', factor=0.5, patience=1)
----> 2 ignite_plateau_scheduler = LRScheduler(plateau_scheduler)

/net/vaosl01/opt/NFS/su0/anaconda3/envs/nlpbook/lib/python3.7/site-packages/pytorch_ignite-0.2.0-py3.7.egg/ignite/contrib/handlers/param_scheduler.py in __init__(self, lr_scheduler, save_history, **kwds)
    424         if not isinstance(lr_scheduler, _LRScheduler):
    425             raise TypeError("Argument lr_scheduler should be a subclass of torch.optim.lr_scheduler._LRScheduler, "
--> 426                             "but given {}".format(type(lr_scheduler)))
    427 
    428         if len(lr_scheduler.optimizer.param_groups) > 1:

TypeError: Argument lr_scheduler should be a subclass of torch.optim.lr_scheduler._LRScheduler, but given <class 'torch.optim.lr_scheduler.ReduceLROnPlateau'>

The example from the documentation with StepLR works without problems. It seems that the objects belong to different base classes. This is indeed true as can be from the source here. StepLR inherits from _LRScheduler, while ReduceLROnPlateau does not have a base class (inherits from object). Is there a way to bypass that check for this scheduler?

Thanks.

[vfdev-5]

@sudarshan85 thanks for the feedback ! Yes, you are correct about ReduceLROnPlateau that is not derived from _LRScheduler:
https://github.com/pytorch/pytorch/blob/77a7285764334cc50eb573c3d2614ffefba4a29f/torch/optim/lr_scheduler.py#L268

class ReduceLROnPlateau(object):`

However, it is possible to use ReduceLROnPlateau as it suggested by pytorch and just call step method in a handler:

optimizer = torch.optim.SGD(model.parameters(), lr=0.1, momentum=0.9)
scheduler = ReduceLROnPlateau(optimizer, 'min')

@evaluator.on(Events.COMPLETED)
def reduct_step(engine):
    # engine is evaluator
    # engine.metrics is a dict with metrics, e.g. {"loss": val_loss_value, "acc": val_acc_value}
    scheduler.step(engine.metrics['loss'])

However, mixing a normal pytorch scheduler (e.g. StepLR) and ReduceLROnPlateau should be checked explicitly whether one does not overwrite another.

HTH

[sudarshan85]

@vfdev-5 Thank you for your prompt reply! Yes, I thought that I could something like that. But shouldn't the event be EPOCH_COMPLETED? COMPLETED refers to the entire training process completed right?

This might be a bit off-topic, but I'm confused about using the evaluator.on decorator vs using trainer.on decorator and using the evaluator inside the that function as is shown in the quickstart example here:

@trainer.on(Events.EPOCH_COMPLETED)
def log_training_results(trainer):
    evaluator.run(train_loader)
    metrics = evaluator.state.metrics
    print("Training Results - Epoch: {}  Avg accuracy: {:.2f} Avg loss: {:.2f}"
          .format(trainer.state.epoch, metrics['accuracy'], metrics['nll']))

[vfdev-5]

@sudarshan85 as the handler is attached to the evaluator which goes through validation dataset only once it is correct to set Events.COMPLETED.

To answer your off-topic question, it would be better represent the training loop and the events happening during its run.

The decorator just means at which event the engine should execute decorated function:

# trainer should execute `in_training_loop_on_epoch_completed` every time when epoch ends
# to compute validation metrics
@trainer.on(Events.EPOCH_COMPLETED)
def in_training_loop_on_epoch_completed(engine):
    evaluator.run(val_loader) # this starts another loop on validation dataset to compute metrics

# evaluator should execute `when_validation_loop_is_done` when run 
# over validation dataset is over.
@evaluator.on(Events.COMPLETED)
def when_validation_loop_is_done(engine):
    # do something with computed metrics etc
    # -> early stopping or reduce LR
    # or just log them 

@sudarshan85 if it is more clear for you, please feel free to contribute this knowledge to our documentation, such that other users would easily understand how to use ignite.
We appreciate very much community contributions :)

[sudarshan85]

The example in the documentation about early stopping shows a non-decorator version, where the score function is declared and passed onto the handler. How would we write the decorator version of it?

[vfdev-5]

Decorators are useful with user defined functions. In case of EarlyStoping
handler, as it is a callable class it is more simple to add it as described
in the docs

On Fri, Mar 22, 2019, 21:30 sudarshan <[email protected] wrote:

Thanks. The picture does add more clarity on whats going on. The example
in the documentation about early stopping shows a non-decorator version,
where the score function is declared and passed onto the handler. How would
we write the decorator version of it?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/pytorch/ignite/issues/462#issuecomment-475771376, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACWHH8Mcr7OoFqsLpt6FngaGD19yCIgbks5vZT1ogaJpZM4cDhRo
.

[sudarshan85]

Ok thanks. I just started using ignite today. Thats why I'm having some basic questions. Thanks for your help.