Pytorch-lightning: Log validation metrics before training

[bartonp2]

❓ Questions and Help

Is there an easy way to run a full evaluation on the the validation set before starting training. I would like this as a kind of benchmark to see where I'm starting from and if the network learns anything at all.

While https://github.com/PyTorchLightning/pytorch-lightning/issues/1715 allows running the sanity check on the complete validation set, this does not log any metrics.

I tried the code as recommended:

class run_validation_on_start(Callback):
    def __init__(self):
        pass

    def on_train_start(self, trainer: Trainer, pl_module):
        return trainer.run_evaluation(test_mode=False)

_Originally posted by @dvirginz in https://github.com/PyTorchLightning/pytorch-lightning/issues/1715#issuecomment-642480058_

but this gives me the following error:

Traceback (most recent call last):██████████████████████████████████████████████████████████████████████████████████████████████████████████████████           | 12/13 [00:00<00:00, 10.56it/s]
  File "/scratch/bartonp/miniconda/envs/eco/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/scratch/bartonp/miniconda/envs/eco/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/scratch/bartonp/avamap/trainer/train.py", line 95, in <module>
    main(hparams)
  File "/scratch/bartonp/avamap/trainer/train.py", line 52, in main
    trainer.fit(model, train_loader, val_loader)
  File "/scratch/bartonp/miniconda/envs/eco/lib/python3.7/site-packages/pytorch_lightning/trainer/states.py", line 48, in wrapped_fn
    result = fn(self, *args, **kwargs)
  File "/scratch/bartonp/miniconda/envs/eco/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1073, in fit
    results = self.accelerator_backend.train(model)
  File "/scratch/bartonp/miniconda/envs/eco/lib/python3.7/site-packages/pytorch_lightning/accelerators/gpu_backend.py", line 51, in train
    results = self.trainer.run_pretrain_routine(model)
  File "/scratch/bartonp/miniconda/envs/eco/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 1239, in run_pretrain_routine
    self.train()
  File "/scratch/bartonp/miniconda/envs/eco/lib/python3.7/site-packages/pytorch_lightning/trainer/training_loop.py", line 363, in train
    self.on_train_start()
  File "/scratch/bartonp/miniconda/envs/eco/lib/python3.7/site-packages/pytorch_lightning/trainer/callback_hook.py", line 111, in on_train_start
    callback.on_train_start(self, self.get_model())
  File "/scratch/bartonp/avamap/trainer/train.py", line 17, in on_train_start
    return trainer.run_evaluation(test_mode=False)
  File "/scratch/bartonp/miniconda/envs/eco/lib/python3.7/site-packages/pytorch_lightning/trainer/evaluation_loop.py", line 603, in run_evaluation
    self.on_validation_end()
  File "/scratch/bartonp/miniconda/envs/eco/lib/python3.7/site-packages/pytorch_lightning/trainer/callback_hook.py", line 176, in on_validation_end
    callback.on_validation_end(self, self.get_model())
  File "/scratch/bartonp/miniconda/envs/eco/lib/python3.7/site-packages/pytorch_lightning/utilities/distributed.py", line 27, in wrapped_fn
    return fn(*args, **kwargs)
  File "/scratch/bartonp/miniconda/envs/eco/lib/python3.7/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 357, in on_validation_end
    filepath = self.format_checkpoint_name(epoch, ckpt_name_metrics)
  File "/scratch/bartonp/miniconda/envs/eco/lib/python3.7/site-packages/pytorch_lightning/callbacks/model_checkpoint.py", line 253, in format_checkpoint_name
    groups = re.findall(r'(\{.*?)[:\}]', self.filename)
  File "/scratch/bartonp/miniconda/envs/eco/lib/python3.7/re.py", line 223, in findall
    return _compile(pattern, flags).findall(string)
TypeError: expected string or bytes-like object

Is there no simple way to run and log the validation set before training?

[github-actions[bot]]

Hi! thanks for your contribution!, great first issue!

[oyj0594]

It's because model checkpointing occur with run_evaluation.
Just use ModelCheckpoint simultaneously, or you can bypass the issue with setting filename of checkpoint inside of your function.

[S-aiueo32]

I faced this issue too. It seems to be an important functionality in transfer learning or fine-tuning,
I roughly think it should be done by a simple flag of Trainer like:

trainer = Trainer(
    ...,
    eval_before_training=True
)

How do you think about it? @williamFalcon and .@othercontributors

[hantoine]

I agree that it would be nice to have a flag like eval_before_training to do this easily.
In the meantime, I managed to make it work for me by calling the method on_train_start of the checkpoint callback:

        class run_validation_on_start(pl.callbacks.Callback):
            def __init__(self):
                pass

            def on_train_start(self, trainer, pl_module):
                trainer.checkpoint_callback.on_train_start(trainer, pl_module)
                return trainer.run_evaluation(test_mode=False)

[stale[bot]]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!