Gluon-nlp: awd-lstm training results on wikitext-2 do not match those in documentation

[dingran]

Listed on https://gluon-nlp.mxnet.io/scripts/index.html#language-model

[2] awd_lstm_lm_600_wikitext-2 (Val PPL 84.61 Test PPL 80.96)
python word_language_model.py --gpus 0 --emsize 200 --nhid 600 --epochs 750 --dropout 0.2 --dropout_h 0.1 --dropout_i 0.3 --dropout_e 0.05 --weight_drop 0.2 --tied --save awd_lstm_lm_600_wikitext-2

I'm getting near zero lr pretty early on and validation ppl is substantially higher (see below) than the published training log here: https://github.com/dmlc/web-data/blob/master/gluonnlp/logs/language_model/awd_lstm_lm_600_wikitext-2.log

My training log

[Epoch 169] time cost 306.39s, valid loss 6.06, valid ppl 429.32，lr 0.03
[Epoch 170 Batch 200/372] current loss 5.69, ppl 296.66, throughput 102.99 samples/s, lr 0.03
[Epoch 170] throughput 7052.65 samples/s
[Epoch 170] time cost 309.12s, valid loss 6.06, valid ppl 428.84，lr 0.03
Learning rate after interval update 0.003000
[Epoch 171 Batch 200/372] current loss 5.70, ppl 297.69, throughput 102.76 samples/s, lr 0.00
[Epoch 171] throughput 7029.03 samples/s
[Epoch 171] time cost 310.24s, valid loss 6.06, valid ppl 428.85，lr 0.00
[Epoch 172 Batch 200/372] current loss 5.69, ppl 295.60, throughput 106.35 samples/s, lr 0.00

Similar issue occurred for awd_lstm_lm_1150_wikitext-2, i.e. learning rate substantially lower and ppl substantially higher compared to published log.

My training log

[Epoch 95] time cost 540.07s, valid loss 5.50, valid ppl 243.47，lr 3.00
[Epoch 96 Batch 200/372] current loss 5.33, ppl 207.33, throughput 61.25 samples/s, lr 3.17
[Epoch 96] throughput 4152.54 samples/s
[Epoch 96] time cost 536.21s, valid loss 5.48, valid ppl 240.60，lr 3.00
[Epoch 97 Batch 200/372] current loss 5.31, ppl 202.84, throughput 60.55 samples/s, lr 3.43
[Epoch 97] throughput 4136.88 samples/s
[Epoch 97] time cost 538.99s, valid loss 5.47, valid ppl 238.03，lr 3.00
[Epoch 98 Batch 200/372] current loss 5.30, ppl 199.51, throughput 60.91 samples/s, lr 1.54

BTW "--gpus" flag in the documentation page should be changed to "--gpu"

The training progress of standard-lstm models appear to match pretty well to the published training log.

[szha]

Thanks for reporting, @dingran. @cgraywang, could you take a look?

[szha]

@dingran by the way, at least with respect to the document page, we've updated the document on master branch: http://gluon-nlp.mxnet.io/master/model_zoo/language_model/index.html

The change will be included in 0.4.1

[dingran]

@dingran by the way, at least with respect to the document page, we've updated the document on master branch: http://gluon-nlp.mxnet.io/master/model_zoo/language_model/index.html

The change will be included in 0.4.1

Cool, overall, very happy with this toolkit, looking forward to future updates!

[szha]

@dingran thanks for the kind words. Making the community productive is what we strive for. Keep letting us know how we can do better. If you are using the toolkit for something new and are willing to make it available to the community, let's work together and make it happen :)

[cgraywang]

@dingran Yes, the changes should be reflected in the new release. Thanks for proposing the issue and will take a look.

[szha]

@dingran @cgraywang 0.4.1 has been released. Does the aggressive learning rate decay issue still happen?

[dingran]

Reran training with the following setting
python word_language_model.py --gpu 0 --tied --ntasgd --lr_update_interval 50 --lr_update_factor 0.5 --save awd_lstm_lm_1150_wikitext-2

See (unfinished) training log below, it does not seem to match the log here very well

I'm using the repo, at commit

% git rev-parse --short HEAD
bd68aaa

Use AWDRNN
/home/rding/.pyenv/versions/anaconda3-5.0.0/lib/python3.6/site-packages/mxnet/gluon/block.py:521: UserWarning: All children of this Sequential layer 'awdrnn1_sequential0_' are HybridBlocks. Consider using HybridSequential for the best performance.
cld.hybridize(active, **kwargs)
AWDRNN(
(embedding): HybridSequential(
(0): Embedding(33278 -> 400, float32)
(1): Dropout(p = 0.65, axes=(0,))
)
(encoder): Sequential(
(0): LSTM(400 -> 1150, TNC)
(1): LSTM(1150 -> 1150, TNC)
(2): LSTM(1150 -> 400, TNC)
)
(decoder): HybridSequential(
(0): Dense(400 -> 33278, linear)
)
)
[Epoch 0 Batch 200/372] current loss 7.69, ppl 2193.76, throughput 63.75 samples/s, lr 29.57
[Epoch 0] throughput 4221.84 samples/s
[Epoch 0] time cost 527.67s, valid loss 6.98, valid ppl 1071.49，lr 30.00
[Epoch 0] test loss 6.91, test ppl 1001.03
[Epoch 1 Batch 200/372] current loss 7.13, ppl 1243.79, throughput 62.74 samples/s, lr 15.00
[Epoch 1] throughput 4241.81 samples/s
[Epoch 1] time cost 525.34s, valid loss 6.52, valid ppl 681.79，lr 30.00
[Epoch 1] test loss 6.45, test ppl 633.21
[Epoch 2 Batch 200/372] current loss 6.70, ppl 816.23, throughput 62.00 samples/s, lr 31.29
[Epoch 2] throughput 4220.45 samples/s
[Epoch 2] time cost 527.81s, valid loss 6.18, valid ppl 480.92，lr 30.00
[Epoch 2] test loss 6.10, test ppl 445.66
[Epoch 3 Batch 200/372] current loss 6.47, ppl 645.25, throughput 61.28 samples/s, lr 30.43
[Epoch 3] throughput 4240.88 samples/s
[Epoch 3] time cost 525.48s, valid loss 5.97, valid ppl 390.50，lr 30.00
[Epoch 3] test loss 5.89, test ppl 362.90
[Epoch 4 Batch 200/372] current loss 6.28, ppl 534.33, throughput 61.74 samples/s, lr 29.14
[Epoch 4] throughput 4216.72 samples/s
[Epoch 4] time cost 528.27s, valid loss 5.75, valid ppl 315.35，lr 30.00
[Epoch 4] test loss 5.67, test ppl 290.64
[Epoch 5 Batch 200/372] current loss 6.09, ppl 442.44, throughput 62.46 samples/s, lr 34.29
[Epoch 5] throughput 4240.43 samples/s
[Epoch 5] time cost 526.05s, valid loss 5.60, valid ppl 271.43，lr 30.00
[Epoch 5] test loss 5.52, test ppl 249.64
[Epoch 6 Batch 200/372] current loss 5.98, ppl 393.57, throughput 63.42 samples/s, lr 34.29
[Epoch 6] throughput 4213.51 samples/s
[Epoch 6] time cost 528.66s, valid loss 5.78, valid ppl 323.50，lr 30.00
Switching to NTASGD and avg_trigger is : 2604
[Epoch 7 Batch 200/372] current loss 6.05, ppl 424.55, throughput 63.13 samples/s, lr 31.29
[Epoch 7] throughput 4222.29 samples/s
[Epoch 7] time cost 527.64s, valid loss 5.56, valid ppl 261.00，lr 30.00
[Epoch 7] test loss 5.49, test ppl 241.81
[Epoch 8 Batch 200/372] current loss 5.98, ppl 394.48, throughput 61.71 samples/s, lr 32.14
[Epoch 8] throughput 4157.74 samples/s
[Epoch 8] time cost 535.18s, valid loss 5.53, valid ppl 252.51，lr 30.00
[Epoch 8] test loss 5.46, test ppl 234.14
[Epoch 9 Batch 200/372] current loss 5.93, ppl 375.90, throughput 60.95 samples/s, lr 29.14
[Epoch 9] throughput 4184.95 samples/s
[Epoch 9] time cost 531.95s, valid loss 5.51, valid ppl 246.08，lr 30.00
[Epoch 9] test loss 5.43, test ppl 228.21
[Epoch 10 Batch 200/372] current loss 5.90, ppl 363.26, throughput 61.44 samples/s, lr 13.71
[Epoch 10] throughput 4181.93 samples/s
[Epoch 10] time cost 532.27s, valid loss 5.49, valid ppl 241.11，lr 30.00
[Epoch 10] test loss 5.41, test ppl 223.60
[Epoch 11 Batch 200/372] current loss 5.87, ppl 353.10, throughput 61.56 samples/s, lr 31.29
[Epoch 11] throughput 4180.13 samples/s
[Epoch 11] time cost 532.59s, valid loss 5.47, valid ppl 236.93，lr 30.00
[Epoch 11] test loss 5.39, test ppl 219.71
[Epoch 12 Batch 200/372] current loss 5.84, ppl 344.08, throughput 60.93 samples/s, lr 30.00
[Epoch 12] throughput 4168.48 samples/s
[Epoch 12] time cost 533.90s, valid loss 5.45, valid ppl 233.43，lr 30.00
[Epoch 12] test loss 5.38, test ppl 216.47
[Epoch 13 Batch 200/372] current loss 5.82, ppl 335.64, throughput 61.52 samples/s, lr 31.29
[Epoch 13] throughput 4197.25 samples/s
[Epoch 13] time cost 530.44s, valid loss 5.44, valid ppl 230.41，lr 30.00
[Epoch 13] test loss 5.36, test ppl 213.64
[Epoch 14 Batch 200/372] current loss 5.80, ppl 329.18, throughput 61.88 samples/s, lr 29.14
[Epoch 14] throughput 4193.93 samples/s
[Epoch 14] time cost 530.96s, valid loss 5.43, valid ppl 227.77，lr 30.00
[Epoch 14] test loss 5.35, test ppl 211.15
[Epoch 15 Batch 200/372] current loss 5.78, ppl 322.57, throughput 60.93 samples/s, lr 34.29
[Epoch 15] throughput 4163.28 samples/s
[Epoch 15] time cost 534.49s, valid loss 5.42, valid ppl 225.37，lr 30.00
[Epoch 15] test loss 5.34, test ppl 208.88
[Epoch 16 Batch 200/372] current loss 5.77, ppl 319.99, throughput 60.87 samples/s, lr 28.71
[Epoch 16] throughput 4170.11 samples/s
[Epoch 16] time cost 534.20s, valid loss 5.41, valid ppl 223.21，lr 30.00
[Epoch 16] test loss 5.33, test ppl 206.84
[Epoch 17 Batch 200/372] current loss 5.75, ppl 314.95, throughput 60.90 samples/s, lr 31.29
[Epoch 17] throughput 4161.66 samples/s
[Epoch 17] time cost 534.67s, valid loss 5.40, valid ppl 221.21，lr 30.00
[Epoch 17] test loss 5.32, test ppl 204.98
[Epoch 18 Batch 200/372] current loss 5.74, ppl 311.62, throughput 61.84 samples/s, lr 14.14
[Epoch 18] throughput 4161.36 samples/s
[Epoch 18] time cost 534.73s, valid loss 5.39, valid ppl 219.37，lr 30.00
[Epoch 18] test loss 5.31, test ppl 203.28
[Epoch 19 Batch 200/372] current loss 5.73, ppl 307.36, throughput 61.56 samples/s, lr 27.43
[Epoch 19] throughput 4178.49 samples/s
[Epoch 19] time cost 532.71s, valid loss 5.38, valid ppl 217.70，lr 30.00
[Epoch 19] test loss 5.31, test ppl 201.71
[Epoch 20 Batch 200/372] current loss 5.71, ppl 302.62, throughput 61.27 samples/s, lr 30.00
[Epoch 20] throughput 4212.78 samples/s
[Epoch 20] time cost 528.56s, valid loss 5.38, valid ppl 216.23，lr 30.00
[Epoch 20] test loss 5.30, test ppl 200.33
[Epoch 21 Batch 200/372] current loss 5.71, ppl 301.11, throughput 61.71 samples/s, lr 27.86
[Epoch 21] throughput 4168.79 samples/s
[Epoch 21] time cost 533.82s, valid loss 5.37, valid ppl 214.85，lr 30.00
[Epoch 21] test loss 5.29, test ppl 199.06
[Epoch 22 Batch 200/372] current loss 5.70, ppl 297.83, throughput 61.07 samples/s, lr 27.00
[Epoch 22] throughput 4196.38 samples/s
[Epoch 22] time cost 531.02s, valid loss 5.36, valid ppl 213.59，lr 30.00
[Epoch 22] test loss 5.29, test ppl 197.88
[Epoch 23 Batch 200/372] current loss 5.68, ppl 293.88, throughput 61.53 samples/s, lr 27.86
[Epoch 23] throughput 4192.26 samples/s
[Epoch 23] time cost 531.04s, valid loss 5.36, valid ppl 212.01，lr 30.00
[Epoch 23] test loss 5.28, test ppl 196.41
[Epoch 24 Batch 200/372] current loss 5.66, ppl 287.28, throughput 61.07 samples/s, lr 28.71
[Epoch 24] throughput 4166.62 samples/s
[Epoch 24] time cost 534.08s, valid loss 5.35, valid ppl 210.60，lr 30.00
[Epoch 24] test loss 5.27, test ppl 195.12
[Epoch 25 Batch 200/372] current loss 5.66, ppl 288.26, throughput 62.01 samples/s, lr 32.14
[Epoch 25] throughput 4220.35 samples/s
[Epoch 25] time cost 527.70s, valid loss 5.34, valid ppl 209.36，lr 30.00
[Epoch 25] test loss 5.27, test ppl 193.96
[Epoch 26 Batch 200/372] current loss 5.65, ppl 283.33, throughput 61.02 samples/s, lr 30.43
[Epoch 26] throughput 4209.54 samples/s
[Epoch 26] time cost 529.02s, valid loss 5.34, valid ppl 208.18，lr 30.00
[Epoch 26] test loss 5.26, test ppl 192.86
[Epoch 27 Batch 200/372] current loss 5.64, ppl 280.85, throughput 61.68 samples/s, lr 15.43
[Epoch 27] throughput 4203.60 samples/s
[Epoch 27] time cost 529.69s, valid loss 5.33, valid ppl 207.05，lr 30.00
[Epoch 27] test loss 5.26, test ppl 191.81
[Epoch 28 Batch 200/372] current loss 5.63, ppl 277.28, throughput 60.57 samples/s, lr 29.57
[Epoch 28] throughput 4176.71 samples/s
[Epoch 28] time cost 532.89s, valid loss 5.33, valid ppl 206.00，lr 30.00
[Epoch 28] test loss 5.25, test ppl 190.84
[Epoch 29 Batch 200/372] current loss 5.63, ppl 280.02, throughput 60.73 samples/s, lr 30.00
[Epoch 29] throughput 4163.49 samples/s
[Epoch 29] time cost 534.45s, valid loss 5.32, valid ppl 205.05，lr 30.00
[Epoch 29] test loss 5.25, test ppl 189.97
[Epoch 30 Batch 200/372] current loss 5.62, ppl 276.64, throughput 60.92 samples/s, lr 32.57
[Epoch 30] throughput 4181.32 samples/s
[Epoch 30] time cost 532.30s, valid loss 5.32, valid ppl 204.16，lr 30.00
[Epoch 30] test loss 5.24, test ppl 189.14
[Epoch 31 Batch 200/372] current loss 5.62, ppl 275.06, throughput 62.41 samples/s, lr 34.71
[Epoch 31] throughput 4173.78 samples/s
[Epoch 31] time cost 533.25s, valid loss 5.31, valid ppl 203.33，lr 30.00
[Epoch 31] test loss 5.24, test ppl 188.35
[Epoch 32 Batch 200/372] current loss 5.61, ppl 272.50, throughput 60.88 samples/s, lr 30.00
[Epoch 32] throughput 4236.45 samples/s
[Epoch 32] time cost 525.82s, valid loss 5.31, valid ppl 202.54，lr 30.00
[Epoch 32] test loss 5.23, test ppl 187.62
[Epoch 33 Batch 200/372] current loss 5.60, ppl 271.70, throughput 61.28 samples/s, lr 29.57
[Epoch 33] throughput 4193.92 samples/s
[Epoch 33] time cost 530.86s, valid loss 5.31, valid ppl 201.80，lr 30.00
[Epoch 33] test loss 5.23, test ppl 186.93
[Epoch 34 Batch 200/372] current loss 5.60, ppl 270.66, throughput 62.13 samples/s, lr 31.71
[Epoch 34] throughput 4198.53 samples/s
[Epoch 34] time cost 530.28s, valid loss 5.30, valid ppl 201.09，lr 30.00
[Epoch 34] test loss 5.23, test ppl 186.25
[Epoch 35 Batch 200/372] current loss 5.60, ppl 269.95, throughput 61.38 samples/s, lr 33.00
[Epoch 35] throughput 4186.06 samples/s
[Epoch 35] time cost 531.75s, valid loss 5.30, valid ppl 200.43，lr 30.00
[Epoch 35] test loss 5.22, test ppl 185.63
[Epoch 36 Batch 200/372] current loss 5.60, ppl 269.15, throughput 61.59 samples/s, lr 30.43
[Epoch 36] throughput 4222.25 samples/s
[Epoch 36] time cost 527.48s, valid loss 5.30, valid ppl 199.82，lr 30.00
[Epoch 36] test loss 5.22, test ppl 185.06
[Epoch 37 Batch 200/372] current loss 5.59, ppl 267.33, throughput 62.36 samples/s, lr 30.86
[Epoch 37] throughput 4186.69 samples/s
[Epoch 37] time cost 531.62s, valid loss 5.29, valid ppl 199.25，lr 30.00
[Epoch 37] test loss 5.22, test ppl 184.52
[Epoch 38 Batch 200/372] current loss 5.59, ppl 266.57, throughput 61.69 samples/s, lr 32.14
[Epoch 38] throughput 4201.15 samples/s
[Epoch 38] time cost 529.94s, valid loss 5.29, valid ppl 198.71，lr 30.00
[Epoch 38] test loss 5.21, test ppl 184.01
[Epoch 39 Batch 200/372] current loss 5.57, ppl 262.90, throughput 63.15 samples/s, lr 29.14
[Epoch 39] throughput 4205.08 samples/s
[Epoch 39] time cost 529.50s, valid loss 5.29, valid ppl 198.18，lr 30.00
[Epoch 39] test loss 5.21, test ppl 183.50
[Epoch 40 Batch 200/372] current loss 5.56, ppl 260.27, throughput 61.26 samples/s, lr 31.29
[Epoch 40] throughput 4173.64 samples/s
[Epoch 40] time cost 533.22s, valid loss 5.29, valid ppl 197.58，lr 30.00
[Epoch 40] test loss 5.21, test ppl 182.93
[Epoch 41 Batch 200/372] current loss 5.54, ppl 255.56, throughput 61.98 samples/s, lr 30.43
[Epoch 41] throughput 4190.63 samples/s
[Epoch 41] time cost 531.22s, valid loss 5.28, valid ppl 197.02，lr 30.00
[Epoch 41] test loss 5.21, test ppl 182.40
[Epoch 42 Batch 200/372] current loss 5.54, ppl 254.97, throughput 61.09 samples/s, lr 27.86
[Epoch 42] throughput 4168.94 samples/s
[Epoch 42] time cost 533.78s, valid loss 5.28, valid ppl 196.49，lr 30.00
[Epoch 42] test loss 5.20, test ppl 181.90
[Epoch 43 Batch 200/372] current loss 5.54, ppl 255.29, throughput 60.51 samples/s, lr 27.43
[Epoch 43] throughput 4185.39 samples/s
[Epoch 43] time cost 532.36s, valid loss 5.28, valid ppl 195.99，lr 30.00
[Epoch 43] test loss 5.20, test ppl 181.42
[Epoch 44 Batch 200/372] current loss 5.54, ppl 254.05, throughput 61.38 samples/s, lr 11.57
[Epoch 44] throughput 4199.02 samples/s
[Epoch 44] time cost 530.18s, valid loss 5.28, valid ppl 195.52，lr 30.00
[Epoch 44] test loss 5.20, test ppl 180.97
[Epoch 45 Batch 200/372] current loss 5.54, ppl 253.44, throughput 61.04 samples/s, lr 29.57
[Epoch 45] throughput 4188.64 samples/s
[Epoch 45] time cost 531.43s, valid loss 5.27, valid ppl 195.06，lr 30.00
[Epoch 45] test loss 5.20, test ppl 180.54
[Epoch 46 Batch 200/372] current loss 5.52, ppl 250.33, throughput 60.30 samples/s, lr 27.86
[Epoch 46] throughput 4184.35 samples/s
[Epoch 46] time cost 531.93s, valid loss 5.27, valid ppl 194.62，lr 30.00
[Epoch 46] test loss 5.19, test ppl 180.12
[Epoch 47 Batch 200/372] current loss 5.52, ppl 249.60, throughput 61.83 samples/s, lr 28.71
[Epoch 47] throughput 4185.47 samples/s
[Epoch 47] time cost 531.80s, valid loss 5.27, valid ppl 194.19，lr 30.00
[Epoch 47] test loss 5.19, test ppl 179.71
[Epoch 48 Batch 200/372] current loss 5.52, ppl 249.75, throughput 60.91 samples/s, lr 26.57
[Epoch 48] throughput 4211.40 samples/s
[Epoch 48] time cost 528.74s, valid loss 5.27, valid ppl 193.78，lr 30.00
[Epoch 48] test loss 5.19, test ppl 179.33
[Epoch 49 Batch 200/372] current loss 5.52, ppl 250.43, throughput 62.46 samples/s, lr 30.43
[Epoch 49] throughput 4203.11 samples/s
[Epoch 49] time cost 529.77s, valid loss 5.26, valid ppl 193.38，lr 30.00
[Epoch 49] test loss 5.19, test ppl 178.96
[Epoch 50 Batch 200/372] current loss 5.51, ppl 247.50, throughput 62.89 samples/s, lr 33.43
[Epoch 50] throughput 4189.56 samples/s
[Epoch 50] time cost 531.36s, valid loss 5.26, valid ppl 192.99，lr 30.00
[Epoch 50] test loss 5.19, test ppl 178.60
[Epoch 51 Batch 200/372] current loss 5.51, ppl 246.73, throughput 61.06 samples/s, lr 29.14
[Epoch 51] throughput 4176.73 samples/s
[Epoch 51] time cost 533.40s, valid loss 5.26, valid ppl 192.62，lr 30.00
[Epoch 51] test loss 5.18, test ppl 178.26
[Epoch 52 Batch 200/372] current loss 5.51, ppl 247.17, throughput 61.23 samples/s, lr 28.71
[Epoch 52] throughput 4202.52 samples/s
[Epoch 52] time cost 529.78s, valid loss 5.26, valid ppl 192.24，lr 30.00
[Epoch 52] test loss 5.18, test ppl 177.89
[Epoch 53 Batch 200/372] current loss 5.50, ppl 244.85, throughput 61.88 samples/s, lr 30.00
[Epoch 53] throughput 4204.22 samples/s
[Epoch 53] time cost 530.07s, valid loss 5.26, valid ppl 191.86，lr 30.00
[Epoch 53] test loss 5.18, test ppl 177.54
[Epoch 54 Batch 200/372] current loss 5.49, ppl 241.59, throughput 61.18 samples/s, lr 27.43
[Epoch 54] throughput 4198.71 samples/s
[Epoch 54] time cost 530.24s, valid loss 5.25, valid ppl 191.51，lr 30.00
[Epoch 54] test loss 5.18, test ppl 177.21
[Epoch 55 Batch 200/372] current loss 5.48, ppl 239.52, throughput 60.79 samples/s, lr 26.57
[Epoch 55] throughput 4176.99 samples/s
[Epoch 55] time cost 532.85s, valid loss 5.25, valid ppl 191.17，lr 30.00
[Epoch 55] test loss 5.18, test ppl 176.90
[Epoch 56 Batch 200/372] current loss 5.48, ppl 240.30, throughput 61.15 samples/s, lr 31.71
[Epoch 56] throughput 4202.11 samples/s
[Epoch 56] time cost 529.84s, valid loss 5.25, valid ppl 190.86，lr 30.00
[Epoch 56] test loss 5.17, test ppl 176.61
[Epoch 57 Batch 200/372] current loss 5.48, ppl 239.55, throughput 61.30 samples/s, lr 27.00
[Epoch 57] throughput 4181.31 samples/s
[Epoch 57] time cost 532.29s, valid loss 5.25, valid ppl 190.55，lr 30.00
[Epoch 57] test loss 5.17, test ppl 176.33
[Epoch 58 Batch 200/372] current loss 5.48, ppl 238.67, throughput 61.73 samples/s, lr 26.14
[Epoch 58] throughput 4176.71 samples/s
[Epoch 58] time cost 533.34s, valid loss 5.25, valid ppl 190.26，lr 30.00
[Epoch 58] test loss 5.17, test ppl 176.06
[Epoch 59 Batch 200/372] current loss 5.49, ppl 243.07, throughput 61.44 samples/s, lr 30.43
[Epoch 59] throughput 4189.99 samples/s
[Epoch 59] time cost 531.30s, valid loss 5.25, valid ppl 189.98，lr 30.00
[Epoch 59] test loss 5.17, test ppl 175.80
[Epoch 60 Batch 200/372] current loss 5.47, ppl 238.20, throughput 62.05 samples/s, lr 30.43
[Epoch 60] throughput 4198.31 samples/s
[Epoch 60] time cost 530.30s, valid loss 5.25, valid ppl 189.71，lr 30.00
[Epoch 60] test loss 5.17, test ppl 175.55
[Epoch 61 Batch 200/372] current loss 5.46, ppl 236.07, throughput 61.01 samples/s, lr 32.57
[Epoch 61] throughput 4172.85 samples/s
[Epoch 61] time cost 533.31s, valid loss 5.24, valid ppl 189.45，lr 30.00
[Epoch 61] test loss 5.17, test ppl 175.30
[Epoch 62 Batch 200/372] current loss 5.47, ppl 236.53, throughput 62.64 samples/s, lr 32.57
[Epoch 62] throughput 4264.00 samples/s
[Epoch 62] time cost 522.62s, valid loss 5.24, valid ppl 189.20，lr 30.00
[Epoch 62] test loss 5.17, test ppl 175.06
[Epoch 63 Batch 200/372] current loss 5.47, ppl 236.68, throughput 61.63 samples/s, lr 26.14
[Epoch 63] throughput 4183.23 samples/s
[Epoch 63] time cost 532.08s, valid loss 5.24, valid ppl 188.96，lr 30.00
[Epoch 63] test loss 5.16, test ppl 174.83
[Epoch 64 Batch 200/372] current loss 5.46, ppl 235.66, throughput 62.04 samples/s, lr 28.71
[Epoch 64] throughput 4193.64 samples/s
[Epoch 64] time cost 530.86s, valid loss 5.24, valid ppl 188.72，lr 30.00
[Epoch 64] test loss 5.16, test ppl 174.61
[Epoch 65 Batch 200/372] current loss 5.46, ppl 235.30, throughput 61.32 samples/s, lr 30.43
[Epoch 65] throughput 4179.97 samples/s
[Epoch 65] time cost 532.47s, valid loss 5.24, valid ppl 188.50，lr 30.00
[Epoch 65] test loss 5.16, test ppl 174.40
[Epoch 66 Batch 200/372] current loss 5.46, ppl 234.36, throughput 60.48 samples/s, lr 33.86
[Epoch 66] throughput 4188.20 samples/s
[Epoch 66] time cost 531.48s, valid loss 5.24, valid ppl 188.28，lr 30.00
[Epoch 66] test loss 5.16, test ppl 174.20
[Epoch 67 Batch 200/372] current loss 5.45, ppl 232.14, throughput 62.43 samples/s, lr 28.71
[Epoch 67] throughput 4210.84 samples/s
[Epoch 67] time cost 529.33s, valid loss 5.24, valid ppl 188.07，lr 30.00
[Epoch 67] test loss 5.16, test ppl 174.00
[Epoch 68 Batch 200/372] current loss 5.46, ppl 234.81, throughput 61.49 samples/s, lr 31.29
[Epoch 68] throughput 4196.90 samples/s
[Epoch 68] time cost 530.96s, valid loss 5.24, valid ppl 187.86，lr 30.00
[Epoch 68] test loss 5.16, test ppl 173.81
[Epoch 69 Batch 200/372] current loss 5.45, ppl 233.77, throughput 61.78 samples/s, lr 27.43
[Epoch 69] throughput 4180.50 samples/s
[Epoch 69] time cost 532.94s, valid loss 5.23, valid ppl 187.64，lr 30.00
[Epoch 69] test loss 5.16, test ppl 173.60
[Epoch 70 Batch 200/372] current loss 5.44, ppl 231.22, throughput 61.10 samples/s, lr 25.71
[Epoch 70] throughput 4196.73 samples/s
[Epoch 70] time cost 530.52s, valid loss 5.23, valid ppl 187.43，lr 30.00
[Epoch 70] test loss 5.16, test ppl 173.40
[Epoch 71 Batch 200/372] current loss 5.44, ppl 230.08, throughput 60.31 samples/s, lr 29.57
[Epoch 71] throughput 4159.48 samples/s
[Epoch 71] time cost 535.01s, valid loss 5.23, valid ppl 187.23，lr 30.00
[Epoch 71] test loss 5.15, test ppl 173.21
[Epoch 72 Batch 200/372] current loss 5.43, ppl 228.41, throughput 60.43 samples/s, lr 13.71
[Epoch 72] throughput 4197.97 samples/s
[Epoch 72] time cost 530.81s, valid loss 5.23, valid ppl 187.04，lr 30.00
[Epoch 72] test loss 5.15, test ppl 173.02
[Epoch 73 Batch 200/372] current loss 5.44, ppl 229.47, throughput 61.41 samples/s, lr 28.29
[Epoch 73] throughput 4184.90 samples/s
[Epoch 73] time cost 532.33s, valid loss 5.23, valid ppl 186.85，lr 30.00
[Epoch 73] test loss 5.15, test ppl 172.84
[Epoch 74 Batch 200/372] current loss 5.44, ppl 230.18, throughput 61.39 samples/s, lr 30.86
[Epoch 74] throughput 4202.86 samples/s
[Epoch 74] time cost 530.24s, valid loss 5.23, valid ppl 186.67，lr 30.00
[Epoch 74] test loss 5.15, test ppl 172.67
[Epoch 75 Batch 200/372] current loss 5.43, ppl 228.35, throughput 61.35 samples/s, lr 28.71
[Epoch 75] throughput 4230.41 samples/s
[Epoch 75] time cost 526.54s, valid loss 5.23, valid ppl 186.49，lr 30.00
[Epoch 75] test loss 5.15, test ppl 172.50
[Epoch 76 Batch 200/372] current loss 5.42, ppl 225.97, throughput 63.02 samples/s, lr 15.86
[Epoch 76] throughput 4178.64 samples/s
[Epoch 76] time cost 532.66s, valid loss 5.23, valid ppl 186.31，lr 30.00
[Epoch 76] test loss 5.15, test ppl 172.33
[Epoch 77 Batch 200/372] current loss 5.42, ppl 226.64, throughput 60.79 samples/s, lr 29.14
[Epoch 77] throughput 4146.58 samples/s
[Epoch 77] time cost 536.46s, valid loss 5.23, valid ppl 186.13，lr 30.00
[Epoch 77] test loss 5.15, test ppl 172.17
[Epoch 78 Batch 200/372] current loss 5.42, ppl 225.94, throughput 61.30 samples/s, lr 29.57
[Epoch 78] throughput 4168.02 samples/s
[Epoch 78] time cost 533.94s, valid loss 5.23, valid ppl 185.96，lr 30.00
[Epoch 78] test loss 5.15, test ppl 172.01
[Epoch 79 Batch 200/372] current loss 5.42, ppl 226.26, throughput 61.59 samples/s, lr 27.00
[Epoch 79] throughput 4193.40 samples/s
[Epoch 79] time cost 530.85s, valid loss 5.22, valid ppl 185.80，lr 30.00
[Epoch 79] test loss 5.15, test ppl 171.86
[Epoch 80 Batch 200/372] current loss 5.42, ppl 226.30, throughput 59.65 samples/s, lr 29.57
[Epoch 80] throughput 4156.72 samples/s
[Epoch 80] time cost 535.28s, valid loss 5.22, valid ppl 185.65，lr 30.00
[Epoch 80] test loss 5.15, test ppl 171.72
[Epoch 81 Batch 200/372] current loss 5.42, ppl 226.76, throughput 61.31 samples/s, lr 32.57
[Epoch 81] throughput 4172.22 samples/s
[Epoch 81] time cost 533.48s, valid loss 5.22, valid ppl 185.50，lr 30.00
[Epoch 81] test loss 5.15, test ppl 171.57
[Epoch 82 Batch 200/372] current loss 5.42, ppl 224.84, throughput 61.62 samples/s, lr 28.29
[Epoch 82] throughput 4158.18 samples/s
[Epoch 82] time cost 535.10s, valid loss 5.22, valid ppl 185.36，lr 30.00
[Epoch 82] test loss 5.14, test ppl 171.44
[Epoch 83 Batch 200/372] current loss 5.41, ppl 224.43, throughput 60.65 samples/s, lr 27.43
[Epoch 83] throughput 4161.91 samples/s
[Epoch 83] time cost 534.68s, valid loss 5.22, valid ppl 185.21，lr 30.00
[Epoch 83] test loss 5.14, test ppl 171.30
[Epoch 84 Batch 200/372] current loss 5.41, ppl 223.18, throughput 61.94 samples/s, lr 26.57
[Epoch 84] throughput 4200.77 samples/s
[Epoch 84] time cost 529.99s, valid loss 5.22, valid ppl 185.08，lr 30.00
[Epoch 84] test loss 5.14, test ppl 171.17
[Epoch 85 Batch 200/372] current loss 5.42, ppl 225.40, throughput 61.73 samples/s, lr 32.57
[Epoch 85] throughput 4189.82 samples/s
[Epoch 85] time cost 531.34s, valid loss 5.22, valid ppl 184.93，lr 30.00
[Epoch 85] test loss 5.14, test ppl 171.03
[Epoch 86 Batch 200/372] current loss 5.40, ppl 222.07, throughput 61.17 samples/s, lr 30.86
[Epoch 86] throughput 4166.12 samples/s
[Epoch 86] time cost 534.18s, valid loss 5.22, valid ppl 184.79，lr 30.00
[Epoch 86] test loss 5.14, test ppl 170.90
[Epoch 87 Batch 200/372] current loss 5.40, ppl 220.61, throughput 61.96 samples/s, lr 29.14
[Epoch 87] throughput 4211.32 samples/s
[Epoch 87] time cost 529.28s, valid loss 5.22, valid ppl 184.66，lr 30.00
[Epoch 87] test loss 5.14, test ppl 170.77
[Epoch 88 Batch 200/372] current loss 5.40, ppl 221.30, throughput 61.63 samples/s, lr 33.00
[Epoch 88] throughput 4133.08 samples/s
[Epoch 88] time cost 538.16s, valid loss 5.22, valid ppl 184.52，lr 30.00
[Epoch 88] test loss 5.14, test ppl 170.64
[Epoch 89 Batch 200/372] current loss 5.39, ppl 219.46, throughput 60.63 samples/s, lr 27.86
[Epoch 89] throughput 4158.98 samples/s
[Epoch 89] time cost 535.04s, valid loss 5.22, valid ppl 184.40，lr 30.00
[Epoch 89] test loss 5.14, test ppl 170.52
[Epoch 90 Batch 200/372] current loss 5.40, ppl 221.82, throughput 61.29 samples/s, lr 28.71
[Epoch 90] throughput 4184.30 samples/s
[Epoch 90] time cost 532.02s, valid loss 5.22, valid ppl 184.27，lr 30.00
[Epoch 90] test loss 5.14, test ppl 170.40
[Epoch 91 Batch 200/372] current loss 5.39, ppl 219.74, throughput 61.16 samples/s, lr 15.43
[Epoch 91] throughput 4171.11 samples/s
[Epoch 91] time cost 533.53s, valid loss 5.22, valid ppl 184.15，lr 30.00
[Epoch 91] test loss 5.14, test ppl 170.29
[Epoch 92 Batch 200/372] current loss 5.39, ppl 220.19, throughput 61.55 samples/s, lr 28.71
[Epoch 92] throughput 4184.06 samples/s
[Epoch 92] time cost 532.01s, valid loss 5.22, valid ppl 184.04，lr 30.00
[Epoch 92] test loss 5.14, test ppl 170.17
[Epoch 93 Batch 200/372] current loss 5.57, ppl 261.89, throughput 57.11 samples/s, lr 13.29
[Epoch 93] throughput 4028.48 samples/s
[Epoch 93] time cost 551.32s, valid loss 5.22, valid ppl 184.15，lr 30.00
[Epoch 94 Batch 200/372] current loss 5.56, ppl 260.49, throughput 58.42 samples/s, lr 33.43
[Epoch 94] throughput 4068.54 samples/s
[Epoch 94] time cost 546.35s, valid loss 5.22, valid ppl 185.85，lr 30.00
[Epoch 95 Batch 200/372] current loss 6.21, ppl 497.44, throughput 56.49 samples/s, lr 28.29
[Epoch 95] throughput 3963.97 samples/s
[Epoch 95] time cost 559.82s, valid loss 5.26, valid ppl 191.72，lr 30.00

[cgraywang]

@dingran thanks for double checking the results. We are now aware of that this might due to weight dropout with gradient clipping due to the gluonnlp's dependencies' version change. We are looking into it and will update once we got this right. It should not take long. At the meantime, please feel free to try the AWD pretrained model by using
model, _ = nlp.model.get_model('awd_lstm_lm_1150', dataset_name='wikitext-2', pretrained=True)

[cgraywang]

@dingran It looks like the script yet similar results as reported in the repo.

I used the same command line:
python word_language_model.py --gpu 0 --tied --ntasgd --lr_update_interval 50 --lr_update_factor 0.5 --save awd_lstm_lm_1150_wikitext-2

The mxnet version I am using:

Name: mxnet-cu90
Version: 1.5.0b20190210

And the master branch of gluonnlp is used.

It hasn't finished training yet, but I pasted the some intermediate results here:

[Epoch 0 Batch 200/372] current loss 7.83, ppl 2515.23, throughput 276.81 samples/s, lr 29.57
[Epoch 0] throughput 18380.53 samples/s
[Epoch 0] time cost 118.68s, valid loss 6.41, valid ppl 608.57，lr 30.00
[Epoch 0] test loss 6.34, test ppl 567.69

[Epoch 10 Batch 200/372] current loss 5.35, ppl 211.57, throughput 261.67 samples/s, lr 13.71
[Epoch 10] throughput 17765.78 samples/s
[Epoch 10] time cost 122.64s, valid loss 4.94, valid ppl 140.29，lr 30.00
[Epoch 10] test loss 4.87, test ppl 129.93

[Epoch 100 Batch 200/372] current loss 4.26, ppl 70.86, throughput 279.14 samples/s, lr 27.00
[Epoch 100] throughput 19054.15 samples/s
[Epoch 100] time cost 114.58s, valid loss 4.37, valid ppl 79.30，lr 30.00
[Epoch 100] test loss 4.32, test ppl 75.29

[Epoch 200 Batch 200/372] current loss 4.09, ppl 59.74, throughput 285.04 samples/s, lr 30.86
[Epoch 200] throughput 19220.91 samples/s
[Epoch 200] time cost 113.69s, valid loss 4.31, valid ppl 74.10，lr 30.00
[Epoch 200] test loss 4.26, test ppl 70.64

[Epoch 300 Batch 200/372] current loss 4.02, ppl 55.84, throughput 281.34 samples/s, lr 28.29
[Epoch 300] throughput 19174.19 samples/s
[Epoch 300] time cost 113.88s, valid loss 4.28, valid ppl 72.03，lr 30.00
[Epoch 300] test loss 4.23, test ppl 68.82

[Epoch 476 Batch 200/372] current loss 3.96, ppl 52.47, throughput 282.90 samples/s, lr 31.29
[Epoch 476] throughput 19116.56 samples/s
[Epoch 476] time cost 114.33s, valid loss 4.25, valid ppl 70.36，lr 30.00
[Epoch 476] test loss 4.21, test ppl 67.31

Could you please try to run again using the same setting and let us know if there is more issue?

[eric-haibin-lin]

Is this issue resolved?

[dingran]

@eric-haibin-lin I just switched job and don't have access to gpu machines at the moment, let's consider it resolved given @cgraywang's experiment.

[szha]

@dingran thanks for the response. Feel free to ping us or open another issue if this is still a problem.

Best of luck with the new adventure!