Pytorch-lightning: How to store test_step outputs to file?

[Ceceu]

Is there an approach to save to one file all the outputs during test_step?

    def test_step(self, batch, batch_idx):
        x1, x2 = batch["x1"], batch["x2"]
        r1, r2 = self(x1, x2)

        test_loss = self.loss_fn(predict)
        test_mrr = self.mrr(r1, r2)

        return {'test_loss': test_loss, 'test_mrr': test_mrr}

My LightningModule outputs two dense vectors for representation r1 and r2. They must be saved to be used in a downstream task and for the measurement of some metrics.

[rohitgr7]

EvalResult.write_dict()

[Ceceu]

EvalResult.write_dict()

I am sorry but I could not understand how to use TrainResult or EvalResult to save on file the outputs of the LightningModule.

It would be something like:

def test_step(self, batch, batch_idx):
        x1, x2 = batch["x1"], batch["x2"]
        r1, r2 = self(x1, x2)

        # saving prediction to file?
        predictions_to_write = {'r1': r1, 'r2': r2}
        result.write_dict(predictions_to_write)

        # loss and metric  
        test_loss = self.loss_fn(predict)
        test_mrr = self.mrr(r1, r2)

        return {'test_loss': test_loss, 'test_mrr': test_mrr}

[rohitgr7]

def test_step(self, batch, batch_idx):
        x1, x2 = batch["x1"], batch["x2"]
        r1, r2 = self(x1, x2)

        # saving prediction to file?
        predictions_to_write = {'r1': r1, 'r2': r2}
        result.write_dict(predictions_to_write, filename)

        # log or attach other metrics
        result.log('test_loss', test_loss)
        # or
        result.test_loss = test_loss
        return result

[stale[bot]]

This issue has been automatically marked as stale because it hasn't had any recent activity. This issue will be closed in 7 days if no further activity occurs. Thank you for your contributions, Pytorch Lightning Team!

[ydcjeff]

Closing it as it is resolved, feel free to reopen if needed

[Ceceu]

Hello @rohitgr7,

As EvalResult and TrainResult is not supported anymore (after pr 3938), how could someone save predictions to file?

[rohitgr7]

https://github.com/PyTorchLightning/pytorch-lightning/blob/471ca375babad9093abf60683a8d0647ac33d4a8/pytorch_lightning/core/lightning.py#L347-L352

This should work in validation and testing but for training its not supported yet.

[Ceceu]

Thank you @rohitgr7,
Then, could I use test_step to save predictions_dict, reload it from disk in test_epoch_end, and use reloaded predictions_dict to calculate a metric?

[rohitgr7]

@Ceceu I think it aggregates everything and saves it after epoch_end, but not sure if I am correct here. Maybe try and tell??