Fastai: Why `DataBunch` takes a `test_dl` if there are no function that uses it

[FrancescoSaverioZuppichini]

When training a model, usually we divide the dataset in train, val and test, these are exactly the three loaders you can pass to DataBunch. However, when we call learner.fit() only the train and val loaders are used. The only way I find out to compute the test score is to manually call

data = DataBunch(train_dl=train_dl, valid_dl=val_dl, test_dl=test_dl)
....
loss, acc = learner.validate(data.test_dl, metrics=[accuracy])

That is inconvinient because I need to pass again the loader and the metrics. Is there a more correct way to do so?

Thank you for the amazing library!

[sgugger]

You don't need to pass again the metrics is they are in your Learner object. Note that in fastai the test set is unlabeled (your labels, if you had some, were all thrown away), it is intended to make predictions on some new data (think test set as in a Kaggle competition). For this you would do

preds,_ = learn.get_preds(ds_type=DatasetType.Test)

If you want to validate on another set, you should create a second DataBunch that has it as its validation set and do

learn.data = data2
learn.validate()

[FrancescoSaverioZuppichini]

get_preds does inference. It is a shame that .fit does not call the test_dl, if exist, after all the epochs as I would expecting. What about implementing this feature, what do you think?

[sgugger]

Yes it does inference, which i also the point of the test set (again unlabeled data). fit can't call the test_dl since it is unlabeled, so loss/metrics on it would have no meaning.

[FrancescoSaverioZuppichini]

Understood, thank you :)