Pytorch-lightning: How to use the test function from the trainer?

[LuposX]

❓ Questions and Help

What is your question?

So when I testmy model with trainer.test() I get as output this:

 ----------------------------------------------------------------------------------------------------
TEST RESULTS
{}
----------------------------------------------------------------------------------------------------

I suppose in the empty bracket can stand sth. But I dont know what can stand there and where to do it. I can't seem to find it in the docs not under trainer nor under test set stands for what this is. If I am right and it doesn'T stand anywhere maybe add to the docs.

Code

trainer.test()

What have you tried?

Searching the docs and google it.

What's your environment?

• Windows 10
• conda
• lighning 0.7.1

[williamFalcon]

you have to return something in test_epoch_end. right now you return an empty dict

[LuposX]

@williamFalcon
I did return sth in test_end than I switched the method out to test_epoch_end. Thats my code:

    def test_epoch_end(self, outputs):
        avg_acc = 100 * self.test_correct_counter / self.test_total_counter
        avg_loss = torch.stack([x['test_loss'] for x in outputs]).mean()

        self.test_correct_counter = 0
        self.test_total_counter = 0

        tensorboard_logs = {'avg_acc': avg_acc, 'test_loss': avg_loss}
        return {'avg_test_loss': avg_loss, 'avg_acc': avg_acc, 'log': tensorboard_logs}

But it still returns:

```

TEST RESULTS

{}

```

[polars05]

@LuposX, I'm doing it as follows:

class CoolSystem(pl.LightningModule):

    def __init__(self, hparams):
        super(CoolSystem, self).__init__()

        ########## define the model ########## 
        ...
        ########## /define the model ########## 

    ...

    def test_step(self, batch, batch_idx):
        # OPTIONAL
        x, y = batch
        y_hat = self.forward(x)

        _, preds = torch.max(y_hat, 1)
        correct = torch.sum(preds == y.data)

        return {'test_loss': F.cross_entropy(y_hat, y), 'test_correct': correct}

    def test_end(self, outputs):
        # OPTIONAL
        avg_loss = torch.stack([x['test_loss'] for x in outputs]).mean()

        avg_acc = torch.stack([x['test_correct'].float() for x in outputs]).sum() / len(self.test_dataloader().dataset)

        logs = {'test_loss': avg_loss, 'test_acc': avg_acc}

        return {'avg_test_loss': avg_loss, 'avg_test_acc': avg_acc, 'log': logs, 'progress_bar': logs}

    ...

    @pl.data_loader
    def test_dataloader(self):
      transform = transforms.Compose([
                              transforms.Resize(256),
                              transforms.CenterCrop(224),
                              transforms.ToTensor(),
                              transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
                              ])

      test_set = torchvision.datasets.ImageFolder(os.path.join(self.data_dir, 'val'), transform)
      test_loader = torch.utils.data.DataLoader(val_set, batch_size=8, shuffle=False, num_workers=4)

      return test_loader

And I ran the test by:

model_test = CoolSystem.load_from_checkpoint('/path/to/checkpoint/filename.ckpt') 

trainer = pl.Trainer()

trainer.test(model_test)

And I get:

----------------------------------------------------------------------------------------------------
TEST RESULTS
{'test_loss': tensor(0.4956), 'test_acc': tensor(0.9020)}
----------------------------------------------------------------------------------------------------

Hopefully it helps you!

[LuposX]

@polars05
You shouldn't use test_end() instead you should use test_epoch_end() same with test_step().

test_end was deprecated in 0.7.0 and will be removed 1.0.0.  Use test_epoch_end instead.
validation_end was deprecated in 0.7.0 and will be removed 1.0.0  'Use validation_epoch_end instead.

[LuposX]

So you need to set 'progress_bar' in the return statement of test_epoch_end to get a return value.

[liebkne]

@LuposX I'm facing the same issue. Why inside 'progress_bar' instead of the return value itself?

[LuposX]

@isolet I dont know. Its just when I set it I get a return value and when not than not. :)

[tcwalther]

I would like to vote for reopening this issue, in particular because the Introduction Guide suggests that any value returned in the dictionary in test_epoch_end will be displayed in the summary. I assume that this was the case in the past, and that the Introduction Guide is simply outdated in this regard. Nevertheless, having to wrap it in a key called progress_bar seems counter-intuitive.

More generally speaking, it would help if there was a centralised documentation on what keys to return in the dictionary in each of these functions :-).

And of course, thanks so much for the work on this library. I'm just exploring it right now, but it looks really nice to use :-). I'm quite excited!