Pytorch-lightning: How optimizer frequencies works

[7rick03ligh7]

❓ Questions and Help

What is your question?

I don't understand how optimizer frequencies works
https://github.com/PyTorchLightning/pytorch-lightning/pull/1269

Code

When I tried to work with them as a list of optimizers, I got this error
It arises after training_step()

What's your environment?

• OS: [Win]
• Packaging [conda]
• Version [0.9.0]

P.S.

If you have a pytorch-lightning WGAN implementation or something with n_critics I would appreciate if you could share )

[rohitgr7]

it's fixed on master #3229

[7rick03ligh7]

@rohitgr7
Oh, thanks
But... How can I install from master in conda?

[awaelchli]

pip install --upgrade git+https://github.com/PyTorchLightning/pytorch-lightning.git in your conda env

[7rick03ligh7]

Ok, but I'm still interested in the sequence of function calls.

1) Am I right that the pseudocode with multiple optimizers looks like this?

2) If I use multiple optimizers with different outputs in training_step() how it will work in training_epoch_end()? I mean how I can aggregate outputs only from the first optimizer and something like that. In other words, how structured outputs from different optimizers in training_epoch_end()

3) How progress_bar will change with multiple optimizers and different outputs?
For example:
optimizer_idx == 0 will HAVE a 'acc' output, which will be monitored
optimizer_idx == 1 will NOT HAVE a 'acc' output

[awaelchli]

1. I think yes, this is roughly it
2. With multiple optimizers, you will get a list of lists in training_epoch_end. If using Results, you get List[List[Result]] and with dict you get List[List[dict]]. The top list has length equal to the number of optimizers. The inner list has length equal to the number of training steps.
3. Don't know, never tried it, but I would assume the progress bar dict gets merged.

[7rick03ligh7]

@awaelchli
Thanks for the reply

I tried it and got this:

All outputs from optimizers sent to a single list
Remind that I used frequencies (opt1 freq=1, opt2 freq=4)

Is it ok, or it's a bug?

[rohitgr7]

@7rick03ligh7
if you are using multiple optimizers with no frequencies defined you will get:

With multiple optimizers, you will get a list of lists in training_epoch_end. If using Results, you get List[List[Result]] and with dict you get List[List[dict]]. The top list has a length equal to the number of optimizers. The inner list has a length equal to the number of training steps.

but, when you have some frequencies defined, each training_step will use only 1 optimizer depending upon the current training_step, and outputs from the other step with different optimizer won't be transferred. So you will have a list with the size equal to the number of training_steps.

Your output is excepted so it's not a bug.

[awaelchli]

@rohitgr7 yes! subtle difference there, but very important! this is indeed working as intended.
@7rick03ligh7 for reference, what @rohitgr7 explained is also written here in the "Note" section:
https://pytorch-lightning.readthedocs.io/en/latest/api/pytorch_lightning.core.html#pytorch_lightning.core.LightningModule.configure_optimizers

[7rick03ligh7]

@awaelchli yeah, but in the documentation, it is not clear that with frequencies the outputs will be in a single list (at least for me)

[7rick03ligh7]

@awaelchli
@rohitgr7

Anyway, thanks for your replies)