Pytorch-lightning: LearningRateMonitor does not work with wandb

[stathius]

🐛 Bug

Wandb rejects the logging of the learning rate. I guess this is because Lightning keeps another step index for epoch-wide metrics like LR.

Epoch 1: 100%|██████████| 626/626 [00:10<00:00, 60.98it/s, wandb: WARNING Step must only increase in log calls.  Step 2 < 1125; dropping {'lr-Adam': 0.001}.

My code:

    lr_monitor_callback = pl.callbacks.LearningRateMonitor()
    early_stop_callback = pl.callbacks.EarlyStopping(monitor=hparams.monitor, min_delta=0.00, 
                patience=hparams.early_stopping_patience, verbose=True, mode='min')
    checkpoint_callback = pl.callbacks.ModelCheckpoint(filepath=checkpoint_dir,
                monitor=hparams.monitor, save_top_k=1, verbose=True, mode='min')
    best_validation_callback = BestValidationCallback(hparams.monitor, hparams.use_wandb)

    trainer = pl.Trainer.from_argparse_args(hparams, logger=logger,
                log_every_n_steps=1,
                checkpoint_callback=checkpoint_callback,
                callbacks=[early_stop_callback, 
                            lr_monitor_callback, 
                            best_validation_callback
                           ])

Environment

python 3.7
ubuntu 18.04
pytorch-lightning 1.0.7
wandb 1.0.11

[stathius]

https://github.com/wandb/client/issues/1534

[rohitgr7]

fixed by https://github.com/PyTorchLightning/pytorch-lightning/pull/4376 recently.
you can try master.

[awaelchli]

@stathius let us know if latest master version works for you. :)

[stathius]

@rohitgr7 @awaelchli Thanks a lot it works indeed. Actually solved another problem I had with testing a model with multiple dataloaders. Sticking with master for now.