Tensor2tensor: How to get real bleu score? [approx_bleu_score]

[ndvbd]

I see that only the approx_bleu_score is sent to the tensorboard. How can evaluate the real bleu?
What's the difference between approx_bleu_score to real bleu?

[martinpopel]

How can evaluate the real bleu?

use t2t-bleu

What's the difference between approx_bleu_score to real bleu?

The main difference is that approx_bleu is computed on the internal subwords instead of words, thus it is not replicable (not comparable with other models) and not suitable for reporting in publications.
Another problem is the autoregressive evaluation using gold previous tokens, which is a kind of cheating.
See #407, #522 and #436 for more details.

[stefan-it]

@nadavb Are there still questions left? Otherwise I think we could close that issue :)

[ndvbd]

@martinpopel thanks,

1. But how can we use t2t-bleu on the already existing model-data directory (the one that has all the train files, and one dev file) - so it will use the dev file for evaluation?

1. What script can we run in the command line to get the approx_bleu?