Parlai: Accuracy is not computed on my own TorchRankerAgent

[james-assiene]

Bug description
For some reasons that I really don't understand, accuracy is not computed in the eval_model script for one model that I'm developing but is during training (thanks to --train-predict True that now works). One consequence (I think) is that during the training, the validation crashes at the end because of a "Key error 'accuracy' " . However, with agents available in ParlAI by default like memnn, there is no problem. I have to precise that I use the exact same command line (with a difference of "-m name_of_the_model" of course) . I thought that therefore the problem is in my code so I compared my code and memnn one but I can't see any difference that can explain this behaviour.

The validation error during the training of my model:

[ eval completed in 1711.50s ]
valid:{'exs': 5129, 'lr': 1, 'num_updates': 45, 'examples': 5129, 'loss': 10330.0, 'mean_loss': 2.015, 'mean_rank': 4.157, 'mrr': 0.5497}
Traceback (most recent call last):
  File "/home/jupyter/ParlAI/parlai/scripts/train_model.py", line 798, in <module>
    TrainLoop(setup_args().parse_args()).train()
  File "/home/jupyter/ParlAI/parlai/scripts/train_model.py", line 756, in train
    stop_training = self.validate()
  File "/home/jupyter/ParlAI/parlai/scripts/train_model.py", line 531, in validate
    new_valid = valid_report[opt['validation_metric']]
KeyError: 'accuracy'

The output of eval_model for my model:

975s elapsed: {'exs': 1730, '%done': '33.73%', 'time_left': '1915s', 'lr': 1, 'num_updates': 6, 'examples': 1730, 'loss': 1018.0, 'mean_loss': 0.5884, 'mean_rank': 8.561, 'mrr': 0.3621}
Encoding text...
Done
Encoding text...
Done
980s elapsed: {'exs': 1740, '%done': '33.92%', 'time_left': '1909s', 'lr': 1, 'num_updates': 6, 'examples': 1740, 'loss': 1023.0, 'mean_loss': 0.5882, 'mean_rank': 8.55, 'mrr': 0.3619}
Encoding text...
Done
Encoding text...
Done
984s elapsed: {'exs': 1750, '%done': '34.12%', 'time_left': '1900s', 'lr': 1, 'num_updates': 6, 'examples': 1750, 'loss': 1029.0, 'mean_loss': 0.5881, 'mean_rank': 8.549, 'mrr': 0.3627}
Encoding text...
Done

The output of eval_model for memnn:

2s elapsed: {'exs': 350, '%done': '6.82%', 'time_left': '27s', 'accuracy': 0.1743, 'f1': 0.2258, 'hits@1': 0.174, 'hits@5': 0.517, 'hits@10': 0.7, 'hits@100': 1.0, 'bleu': 0.004728, 'lr': 1, 'num_updates': 1080, 'examples': 350, 'loss': 1325.0, 'mean_loss': 3.785, 'mean_rank': 18.26, 'mrr': 0.304}
4s elapsed: {'exs': 700, '%done': '13.65%', 'time_left': '25s', 'accuracy': 0.1757, 'f1': 0.2204, 'hits@1': 0.176, 'hits@5': 0.519, 'hits@10': 0.699, 'hits@100': 1.0, 'bleu': 0.008812, 'lr': 1, 'num_updates': 1080, 'examples': 700, 'loss': 2677.0, 'mean_loss': 3.824, 'mean_rank': 19.33, 'mrr': 0.2981}
6s elapsed: {'exs': 1060, '%done': '20.67%', 'time_left': '23s', 'accuracy': 0.167, 'f1': 0.2093, 'hits@1': 0.167, 'hits@5': 0.523, 'hits@10': 0.713, 'hits@100': 1.0, 'bleu': 0.01054, 'lr': 1, 'num_updates': 1080, 'examples': 1060, 'loss': 4062.0, 'mean_loss': 3.832, 'mean_rank': 18.88, 'mrr': 0.2945}
8s elapsed: {'exs': 1410, '%done': '27.49%', 'time_left': '21s', 'accuracy': 0.173, 'f1': 0.2172, 'hits@1': 0.173, 'hits@5': 0.525, 'hits@10': 0.73, 'hits@100': 1.0, 'bleu': 0.01076, 'lr': 1, 'num_updates': 1080, 'examples': 1410, 'loss': 5381.0, 'mean_loss': 3.816, 'mean_rank': 18.4, 'mrr': 0.2998}
10s elapsed: {'exs': 1780, '%done': '34.70%', 'time_left': '19s', 'accuracy': 0.1809, 'f1': 0.2242, 'hits@1': 0.181, 'hits@5': 0.537, 'hits@10': 0.737, 'hits@100': 1.0, 'bleu': 0.01078, 'lr': 1, 'num_updates': 1080, 'examples': 1780, 'loss': 6711.0, 'mean_loss': 3.77, 'mean_rank': 17.94, 'mrr': 0.3069}

Reproduction steps

Colab notebook showing the eval_model output of my model:
https://colab.research.google.com/drive/1uNMgjToI2QEwrcOYfivO-PFAZzOtNArL

Colab notebook showing the error during training in the validation phase:
https://colab.research.google.com/drive/1VPdT2cdsy6Ypl0lh7QSgNyc-tpYOu9zo

Colab notebook showing the correct behaviour with memnn (Section "With train-predict = True and checkpoint"):
https://colab.research.google.com/drive/14unkeYMswCDtz2iAwf0aj3KtRuT0pC14

Expected behavior
No "Key error 'accuracy'" and computation of the accuracy in eval_model

[stephenroller]

Perhaps your model isn't producing text output? Your model doesn't have f1 or bleu outputs either, suggesting that these metrics aren't being computed. Did you override train_step and/or eval_step?

[james-assiene]

Oh yes that was it. I overrided both of them but of course I called super().train_step and super().eval_step . But for eval_step() I didn't return the output of super().eval_step (but I did for train_step). So I replaced super().eval_step(batch) by return super().eval_step(batch) and the problem is solved.

Thank you very much