Gluon-nlp: Transformer en-de pretrained models are outdated

[JulianSlzr]

Description

Recent changes to the API (@leezu) have made the Transformer en-de pretrained models out of date. The models I tried were:

• The one listed in inference_transformer.py: https://github.com/dmlc/gluon-nlp/blame/3ce9995329fb0d18787019df541d4f229d7c9ded/scripts/machine_translation/inference_transformer.py#L169
• The one listed in the Model Zoo: https://gluon-nlp.mxnet.io/master/model_zoo/machine_translation/index.html

(also these models are not the same!)

To Reproduce

Download and load either .params file, and load via --model_parameter in inference_transformer.py

[sxjscience]

Another problem I find is the performance no longer matches our README. I trained the transformer on WMT2014 using the latest master version:

MXNET_GPU_MEM_POOL_TYPE=Round python train_transformer.py --dataset WMT2014BPE \
                       --src_lang en --tgt_lang de --batch_size 2700 \
                       --optimizer adam --num_accumulated 16 --lr 2.0 --warmup_steps 4000 \
                       --save_dir transformer_en_de_u512 --epochs 30 --gpus 0,1,2,3,4,5,6,7 --scaled \
                       --average_start 5 --num_buckets 20 --bucket_scheme exp --bleu 13a --log_interval

The result is

[Epoch 29] valid Loss=1.5213, valid ppl=4.5783, valid bleu=25.79
[Epoch 29] test Loss=1.3180, test ppl=3.7360, test bleu=26.11
Best model valid Loss=1.4877, valid ppl=4.4270, valid bleu=26.03
Best model test Loss=1.2844, test ppl=3.6124, test bleu=26.62

[eric-haibin-lin]

We should fix this issue before 0.9 release

[leezu]

@sxjscience did you confirm it's due #976 and not some change in MXNet? Should it be tracked separately?

The issue @JulianSlzr points out is due to the script not being checked on our CI and thus not being updated in #976.

[sxjscience]

@leezu I'm not sure about that. I feel that https://github.com/dmlc/gluon-nlp/pull/976 should not change the behavior because it passed the CI.

[sxjscience]

@leezu Let me rerun using the version before the refactor.

[sxjscience]

@leezu I trained using bfa5503e81ae53d26b9f202bce4fedcf09e47db4

The bleu scores are:

2019-12-14 16:35:52,264 - root - [Epoch 29] valid Loss=1.5258, valid ppl=4.5989, valid bleu=25.83
2019-12-14 16:39:04,154 - root - [Epoch 29] test Loss=1.3263, test ppl=3.7669, test bleu=26.20
2019-12-14 16:42:39,363 - root - Best model valid Loss=1.4925, valid ppl=4.4484, valid bleu=26.22
2019-12-14 16:45:49,673 - root - Best model test Loss=1.2831, test ppl=3.6077, test bleu=26.93

[leezu]

Could the difference be to an effective difference in the sequence of random values used during training? While we use the same seeds before and after bfa5503, parts of the computational graph are moved from the imperative API to hybrid API (run by CachedOp). Thus there may be a change in the sequence in which the random number generators are called?

What next steps do you recommend to take?

[szhengac]

Based on my previous experience on a version in Oct, the final bleu is generally 27.10 with std 0.1. So I think there should be some bugs in the master.

[leezu]

@szhengac @szha @sxjscience should this be a release blocker for MXNet 1.6?

[sxjscience]

@leezu Do you mean GluonNLP 0.9 or MXNet 1.6?

[leezu]

As already before bfa5503 you obtain 0.4 less than the result on https://gluon-nlp.mxnet.io/model_zoo/machine_translation/index.html#transformers and as @szhengac reports std 0.1 we need to check running the version before bfa5503 with MXNet 1.5. If this recovers the 0.4 bleu, it should be a MXNet release blocker. Or what do you think?

[sxjscience]

@leezu I think https://github.com/dmlc/gluon-nlp/commit/bfa5503e81ae53d26b9f202bce4fedcf09e47db4 has test BLEU equals to 26.93. Also, it might be related to the variance. The numpy version has BLEU ranging from 27.10 -- 27.40(the BLEU score reported here does not use the same evaluation method as in the ndarray version)

[leezu]

@sxjscience with 394e69a6dc9cea357d188797a949f678f6df7894 and a the following patch

--- a/scripts/machine_translation/train_transformer.py
+++ b/scripts/machine_translation/train_transformer.py
@@ -53,9 +53,9 @@ from bleu import _bpe_to_words, compute_bleu
 from translation import BeamSearchTranslator
 from utils import logging_config

-np.random.seed(100)
-random.seed(100)
-mx.random.seed(10000)
+np.random.seed(101)
+random.seed(101)
+mx.random.seed(10001)

I obtain with MXNet https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-15/dist/mxnet_cu100-1.6.0b20191215-py2.py3-none-manylinux1_x86_64.whl

2019-12-17 04:52:41,912 - root - [Epoch 29] valid Loss=1.5227, valid ppl=4.5845, valid bleu=25.44
2019-12-17 04:55:40,926 - root - [Epoch 29] test Loss=1.3274, test ppl=3.7712, test bleu=26.40
2019-12-17 04:58:49,950 - root - Best model valid Loss=1.4893, valid ppl=4.4340, valid bleu=26.03
2019-12-17 05:01:44,671 - root - Best model test Loss=1.2889, test ppl=3.6288, test bleu=26.73

You previously obtained (without the patch) test bleu=26.62. With bfa5503 you got 26.93.
I'll try with another seed and an older MXNet version as well.

[leezu]

@sxjscience with bfa5503 and a the following patch

--- a/scripts/machine_translation/train_transformer.py
+++ b/scripts/machine_translation/train_transformer.py
@@ -53,9 +53,9 @@ from bleu import _bpe_to_words, compute_bleu
 from translation import BeamSearchTranslator
 from utils import logging_config

-np.random.seed(100)
-random.seed(100)
-mx.random.seed(10000)
+np.random.seed(101)
+random.seed(101)
+mx.random.seed(10001)

I obtain with MXNet https://apache-mxnet.s3-us-west-2.amazonaws.com/dist/2019-12-15/dist/mxnet_cu100-1.6.0b20191215-py2.py3-none-manylinux1_x86_64.whl

2019-12-18 20:31:42,541 - root - [Epoch 29] valid Loss=1.5274, valid ppl=4.6062, valid bleu=25.87
2019-12-18 20:34:48,269 - root - [Epoch 29] test Loss=1.3206, test ppl=3.7458, test bleu=26.11
2019-12-18 20:38:05,245 - root - Best model valid Loss=1.4916, valid ppl=4.4444, valid bleu=26.22
2019-12-18 20:41:08,935 - root - Best model test Loss=1.2863, test ppl=3.6195, test bleu=26.67

You previously obtained (without the patch) test bleu=26.93 on bfa5503, which is better than the test bleu=26.62 you observed on 394e69a.

With the patch (ie. different seed) bfa5503 (test bleu=26.67) performs worse than 394e69a (test bleu=26.73).

[sxjscience]

@leezu I think it's due to the variance.

[leezu]

inference_transformer still needs to be added to CI test cases and updated.

[sxjscience]

@leezu Okay, I see, should not have closed it.

[leezu]

@JulianSlzr I can't reproduce the issue you report.
I believe you were using an updated script but didn't update your Gluonnlp installation. Or the other way around.

On master, I obtain the following output:

2020-01-15 17:54:58,450 - root - Use beam_size=4, alpha=0.6, K=5
2020-01-15 17:54:58,450 - root - Inference on test_dataset!
2020-01-15 17:54:58,455 - root - Test Batch Sampler:
FixedBucketSampler:
  sample_num=2737, batch_num=370
  key=[7, 8, 9, 10, 11, 13, 14, 16, 19, 22, 26, 30, 36, 42, 50, 59, 71, 84, 100]
  cnt=[13, 18, 29, 41, 39, 123, 81, 147, 239, 233, 324, 285, 392, 271, 206, 181, 87, 23, 5]
  batch_size=[35, 32, 28, 25, 23, 19, 18, 16, 13, 11, 10, 8, 7, 6, 5, 4, 3, 3, 2]
2020-01-15 17:55:13,160 - root - batch id=10, batch_bleu=32.0886
2020-01-15 17:55:22,247 - root - batch id=20, batch_bleu=32.7186
2020-01-15 17:55:31,011 - root - batch id=30, batch_bleu=26.0578
2020-01-15 17:55:40,384 - root - batch id=40, batch_bleu=23.3480
2020-01-15 17:55:49,569 - root - batch id=50, batch_bleu=31.0670
2020-01-15 17:55:58,953 - root - batch id=60, batch_bleu=28.4725
2020-01-15 17:56:08,257 - root - batch id=70, batch_bleu=25.6273
2020-01-15 17:56:17,991 - root - batch id=80, batch_bleu=24.1941
2020-01-15 17:56:26,178 - root - batch id=90, batch_bleu=24.3124
2020-01-15 17:56:34,284 - root - batch id=100, batch_bleu=30.3633

And on https://github.com/dmlc/gluon-nlp/commit/505920827eb97a34c58c7694461b5dea672d925a exactly the same results:

2020-01-15 17:50:13,486 - root - Use beam_size=4, alpha=0.6, K=5
2020-01-15 17:50:13,486 - root - Inference on test_dataset!
/home/ubuntu/gluon-nlp/src/gluonnlp/data/batchify/batchify.py:233: UserWarning: Padding value is not given and will be set automatically to 0 in data.batchify.Pad(). Please check whether this is intended
(e.g. value of padding index in the vocabulary).
  warnings.warn(
2020-01-15 17:50:13,491 - root - Test Batch Sampler:
FixedBucketSampler:
  sample_num=2737, batch_num=370
  key=[7, 8, 9, 10, 11, 13, 14, 16, 19, 22, 26, 30, 36, 42, 50, 59, 71, 84, 100]
  cnt=[13, 18, 29, 41, 39, 123, 81, 147, 239, 233, 324, 285, 392, 271, 206, 181, 87, 23, 5]
  batch_size=[35, 32, 28, 25, 23, 19, 18, 16, 13, 11, 10, 8, 7, 6, 5, 4, 3, 3, 2]
2020-01-15 17:50:28,319 - root - batch id=10, batch_bleu=32.0886
2020-01-15 17:50:37,401 - root - batch id=20, batch_bleu=32.7186
2020-01-15 17:50:46,119 - root - batch id=30, batch_bleu=26.0578
2020-01-15 17:50:55,485 - root - batch id=40, batch_bleu=23.3480
2020-01-15 17:51:04,610 - root - batch id=50, batch_bleu=31.0670
2020-01-15 17:51:13,958 - root - batch id=60, batch_bleu=28.4725
2020-01-15 17:51:23,202 - root - batch id=70, batch_bleu=25.6273
2020-01-15 17:51:32,925 - root - batch id=80, batch_bleu=24.1941
2020-01-15 17:51:41,173 - root - batch id=90, batch_bleu=24.3124
2020-01-15 17:51:49,363 - root - batch id=100, batch_bleu=30.3633

The difference in output (omitted here), is that logging.info(model) on master version prints the refactored model structure, and on the old version prints the old model structure..

To obtain the results I ran

wget http://apache-mxnet.s3-accelerate.dualstack.amazonaws.com/gluon/models/transformer_en_de_512_WMT2014-e25287c5.zip
unzip transformer_en_de_512_WMT2014-e25287c5.zip
python3 ./inference_transformer.py --model_parameter transformer_en_de_512_WMT2014-e25287c5.params --gpu 0 --dataset WMT2014BPE --src_lang en --tgt_lang de --batch_size 2700 --scaled --num_buckets 20 --bucket_scheme exp --bleu 13a

I'll close the issue now, but please reopen if needed