Fairseq: How can i add bleu validation in NAT task?

[sfxjh]

When train nat models, i have a question about how to eval the valid set correctly.
i think average last 5 checkpoint may not be very nice
So i add
--eval-bleu \
--eval-bleu-args '{"beam": 5, "max_len_a": 1.2, "max_len_b": 10}' \
--eval-bleu-detok moses \
--eval-bleu-remove-bpe \
--eval-bleu-print-samples \
--best-checkpoint-metric bleu --maximize-best-checkpoint-metric \
it didn't work...
and now i have a question about how to add bleu-eval in NAT task?

[lematt1991]

What do you mean it didn't work? Please follow the issue template. Provide a complete command and the full error that you are experiencing

[risingdhxs]

I think what @sfxjh meant is that, when training the NAT models using fairseq, adding the options he mentioned didn't add the BLEU plot to tensorboard (or as an evaluation metric) as expected. I have encountered this issue as well. I trained a NAT model on IWSLT dataset using the following script:

python train.py "data-bin/iwslt14_joined" \
--arch nonautoregressive_transformer_iwslt_en_de \
--optimizer adam \
--adam-betas '(0.9,0.98)' \
--criterion nat_loss \
--task translation_lev \
--label-smoothing 0.1 \
--noise full_mask \
--lr-scheduler inverse_sqrt \
--warmup-init-lr '1e-07' \
--lr 0.0005 \
--min-lr '1e-09' \
--warmup-updates 10000 \
--dropout 0.3 \
--weight-decay 0.01 \
--decoder-learned-pos \
--encoder-learned-pos \
--pred-length-offset \
--length-loss-factor 0.1 \
--apply-bert-init \
--max-tokens 8000 \
--max-epoch 200 \
--share-all-embeddings \
--fixed-validation-seed 7 \
--save-dir ./NATresults/IWSLT/checkpoints/NATtest_d3L5/ \
--tensorboard-logdir ./NATresults/IWSLT/tensorboard/NATtest_d3L5/ \
--eval-bleu \
--eval-bleu-args '{"beam": 4, "lenpen": 0.3}' \
--eval-bleu-remove-bpe \
--eval-bleu-detok moses \
--ddp-backend=no_c10d \

When I trained a regular Transformer with similar scripts, by adding the options with eval-bleu, I get a BLEU plot when I plot tensorboard. But not the case with NAT models: I still get best_loss, loss, and ppl plots, but not the BLEU plot.

Fairseq version: 0.9.0
python: 3.7.4
pytorch: 1.2.0
CUDA: 10.2

Let me know if there's any essential config info I missed.

Thanks in advance.