Parlai: Convai2 perplexity calculation might be slightly off

[Henry-E]

Ok so according to the FAQ the script will never evaluate the perplexity for predicting special tokens.

The separate eval_ppl script does a more careful job of evaluating (comparable across models) and doesn’t include these extra special tokens from the model.

However the dict_self generated in the convai2 home directory does contain these special tokens. So while their probabilities will never be used they are still part of the probability rebasing. Which presumably takes away relative probability from the correct token.

I guess it's worth making super clear to people not to assign probabilities to these tokens or manually setting them to zero. Otherwise they might be getting a perplexity point or two higher than they ought to.

[alexholdenmiller]

I see what you mean that you might get an advantage for spreading out the probability mass that could be assigned to those tokens.

Our baselines are also assigning a small mass to those tokens, and in fact running the baseline seq2seq model I got the same final perplexity (to two decimal points, I didn't check further) after setting removing these tokens from the dictionary.

So I think instead of rerunning all of the evaluations with the special tokens manually set to zero, we will leave these as is and leave it up to the entrant to have adjusted them to their own (possible negligible) advantage, especially since this only affects ppl and not the other evaluation metrics.

[Henry-E]

Cool, thanks for double checking. I noticed a little difference but I agree that it's probably negligible and not too bad since it's only for perplexity.

There was a couple of other things I wanted to note about the dictionary for future competitions.

One is that the validation set tokens are included and there's about 500 tokens that only appear in the validation set. This is my guess why there's such a difference between the perplexity of the validation set and the test set. Granted the test set is larger, so that might have something to do with it. It also provides a strong incentive to train on the validation set in case any of those 500 tokens show up in the test set.

The preprocessing does weird stuff with apostrophes. There's no versions of "its", "lets" or "were" with apostrophes i.e. "it's", "let's", "we're". There's also lots of odd concatenations based on the presence of apostrophes. The data from mechanical turk is obviously very messy to begin with and cleaning this up is quite tough but thought I'd mention it anyway just case.

Thanks for preparing a great dataset and running such an interesting competition

[emilydinan]

Thanks for pointing these things out. As far as the pre-processing/apostrophes go, the original persona-chat dataset was collected with pre-processing that got rid of apostrophes. For ConvAI2, we tried to reverse some of this pre-processing to make it a bit more natural. For the cases you pointed out ("its", "lets", "were"), we were unable to reverse the processing because they are all words on their own (without the apostrophes) so we would have had to examine them on a case by case basis.

[Henry-E]

Are you able share the processing code. It seems to make sense that for the interactive scripts they should be processed / tokenized in a similar way?

[emilydinan]

Unfortunately, we do not have it. The dataset was collected an intern who no longer has the code. As far as I am aware, the only processing was lowercasing, some spellchecking, and these apostrophe edits. For the human evaluations for the competition, we will certainly lowercase any human responses and automatically convert "its," "lets," "were" to feed to the models. If you find any other irregularities, please let us know as we would consider applying those changes, too.