Pytorch-lightning: early stopping callback on_epoch_end called twice

[aamster]

What is your question?

I am confused on the early stopping behavior. The on_epoch_end method seems to get called 2 times after a single validation epoch. It is called first from here https://github.com/PyTorchLightning/pytorch-lightning/blob/v0.7.5.x/pytorch_lightning/trainer/training_loop.py#L453 and next from here https://github.com/PyTorchLightning/pytorch-lightning/blob/v0.7.5.x/pytorch_lightning/trainer/training_loop.py#L363. This results in self.wait += 1 in EarlyStopping getting called I believe incorrectly. I.e what is happening is the current metric is checked, then it is checked immediately again and since it has not changed self.wait is incremented.

Code

I am passing this into the trainer

early_stop_callback = EarlyStopping(
        monitor='validation_mean_precision',
        min_delta=0.00,
        patience=3,
        verbose=True,
        mode='max'
    )

What's your environment?

• Packaging [conda]
• Version [0.7.6]

[github-actions[bot]]

Hi! thanks for your contribution!, great first issue!

[HansBambel]

That is an issue in 0.7.6.

There are multiple issues with this issue as well and it should be fixed in the next release.
As a workaround for 0.7.6 you can double the number of patience to get the expected behavior.

[LucFrachon]

Yes, I noticed the same issue. I tried modifying the EarlyStopping logic by replacing the on_epoch_end() hook with on_validation_end(), but it is never called inside the training loop.
I think there should be a validation_epoch_end() hook like in LightningModule.

[Borda]

@aamster @HansBambel @LucFrachon mind checking the latest v0.8.0rc2

[aamster]

@Borda early stopping seems to work correctly in 0.8.0rc2. But in my mind this is a major bug in 0.7.6

[Borda]

well as v0.8.0 will be released shortly, we will not prepare any further fixes in v0.7.x but if you want and send a fix PR we can arrange it for you in v0.7.6.x branch :rabbit: