Pytorch-lightning: Incompatible Trainer.test()

[isvoboda]

🐛 Bug

Trainer.test() stops on missing definitions of train & validation dataloaders methods.

To Reproduce

Steps to reproduce the behavior:

• Define LightningModule without train & validation dataloaders
• Train such model with data loaders provided to Trainer.fit(...) method
• Load model from checkpoint and test it with Trainer.test(...) method

test failes on :
~python
'No train_dataloader() method defined. Lightning Trainer expects as minimum a'
pytorch_lightning.utilities.exceptions.MisconfigurationException: No train_dataloader() method defined. Lightning Trainer expects as minimum a training_step(), training_dataloader() and configure_optimizers() to be defined.
~

Code sample

~python
test_data = data_loader(...)
estimator = MyEstimator.load_from_checkpoint(checkpoint_path=CKPT_PATH)
trainer = pl.Trainer(...)
trainer.test(model=estimator, test_dataloaders=test_data)
~

Expected behaviour

Run the test with provided test data loader, and do not stop on condition related to missing train & validation dataloaders which actually are not mandatory.

Environment

~~~

• Packages:
  - numpy: 1.18.4
  - pyTorch_debug: False
  - pyTorch_version: 1.5.0
  - pytorch-lightning: 0.7.5
  - tensorboard: 2.2.1
  - tqdm: 4.46.0
• System:
  - OS: Linux
  - architecture:
  - 64bit
  - ELF
  - processor:
  - python: 3.8.2
  - version: #1 SMP Wed, 29 Apr 2020 16:23:03 +0000
  ~~~

[github-actions[bot]]

Hi! thanks for your contribution!, great first issue!

[isvoboda]

This issue relates to #1195, fit and test should be decoupled. Is there any consensus or proposition of further steps?

[rohitgr7]

Can't we just add

if not self.testing:
    self.check_model_configuration(model)

here because during testing the testing configuraton is already checked here.

[iakremnev]

Also, using trainer.test(model) on loaded checkpoint doesn't work without explicitly passing test_dataloader parameter even if all *_dataloader methods are defined. Example.
This use case isn't documented in the 0.7.5 documentation but evidently was relevant at some point: https://pytorch-lightning.readthedocs.io/en/latest/test_set.html#test-pre-trained-model.
I suggest to either:

• fix Test set documentation
• (preferred) allow to call trainer.test(model) without specifying test_dataloader parameter. After all, what's the point of defining Model.test_dataloader method?

[rohitgr7]

@iakremnev can you fix the link? It's asking for request access.

[iakremnev]

@rohitgr7 oops, fixed it.

[rohitgr7]

Error coming from this line. Why is it checking whether test_dataloader() returns something or not. If we just check whether it is overridden or not wouldn't that be enough just like it is done in case of .fit() here.

[rohitgr7]

I would suggest

gave_test_loader = isinstance(model.test_dataloader, _PatchDataLoader) or self.is_overridden('test_dataloader', model)

here, and

if not self.testing:
    self.check_model_configuration(model)

here

[Borda]

it is the same as #1754