Pytorch-lightning: convert examples to doctests

[Borda]

🚀 Feature

Converting examples to doctests...

Motivation

The examples now are static so there is no guarantee that they are still valid...
Advantages of converting to doctest would:

• increase reproducibility
• each example can run as a stand-alone
• make testing on smaller units
• smaller test units simplify debugging

Additional context

• https://docs.python.org/3/library/doctest.html
• https://thomas-cokelaer.info/tutorials/sphinx/doctest.html
• https://stackoverflow.com/questions/361675/python-doctest-vs-unittest

[Borda]

I would say to use doctest for small functionalities where we want to use it also as examples in docs and unittest for extension testing, for example, catching fail cases or running multiple edge test-cases

[awaelchli]

I can keep working on this if you like.

[Borda]

just pls do it by small pieces as you started so it is easier to merge / less potential conflicts to resolve lol

[awaelchli]

Many of the examples in LightningModule are just a method, something like this:

def training_epoch_end(self, outputs):
    train_acc_mean = 0
    for output in outputs:
        train_acc_mean += output['train_acc']

    train_acc_mean /= len(outputs)

    # log training accuracy at the end of an epoch
    results = {
        'log': {'train_acc': train_acc_mean.item()}
    }
    return results

Should we convert this to doctest this or not? Basically it would only do syntax check in this case, nothing else. What do you think?

[awaelchli]

just adding a reminder here that we may want to enable doctest for .rst files. (comment)

make doctest currently raises many import errors