Pytorch-lightning: Change the way the configure_optimizers() returns

[danieltudosiu]

🚀 Feature

Force the method LightningModule.configure_optimizers() to return two lists.

Motivation

Right now you offer flexibility in return one of the following:
- Single optimizer
- List or Tuple - List of optimizers
- Two lists - The first list has multiple optimizers, the second a list of LR schedulers

But, that can be simplified to just returning two lists, one for optimizer and one for lr_schedulers since it does not make any sense to have that flexibility when you actually just convert it to two lists at the end of the day

Pitch

Forcing to return two lists, adds minimum overhead to researches and coders while being compliant to your codebase already.

The change aims at reducing variability while maintaining the same degree of flexibility. This will make memorizing the behavior of your framework easier and more straight forward.

It can be done as follows:
- Single optimizer - [optimizer], []
- List or Tuple - List of optimizers, [optimizer_1, optimizer_2], []
- Two lists - Multiple optimizers, single lr_scheduler - [optimizer_1, optimizer_2], [lr_scheduler]
- Two lists - Multiple optimizers, single lr_scheduler - [optimizer_1, optimizer_2], [lr_scheduler_1, lr_scheduler_2]

Alternatives

N/A

Additional context

Past Slack discussion: https://pytorch-lightning.slack.com/archives/CRBLFHY79/p1584064294116400

[yukw777]

I don't have a strong preference on this one. I think if enough people agree on this, we should do it.

[festeh]

I don't see clear benefits from this, probably because I got used to the current behavior. Drawbacks are:

• if one just wants to return an optimizer, additional cluttering with two lists will be required
• it's a backwards incompatible change, users will have to update their code

[Borda]

@PyTorchLightning/core-contributors ^^

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.