Opennmt-py: Bidirectional encoder giving different loss even with same seed

Created on 5 Mar 2019  路  5Comments  路  Source: OpenNMT/OpenNMT-py

With -encoder_type rnn and with same seed, I am able to reproduce the same result each time. However, if I use encoder type as bidirectional encoder i.e., -encoder_type brnn, I am getting different log (different result) each time even with same seed.

Here are the details:

The OpenNMT-py code is same as the one in master branch. The dataset and preprocessing steps are same as : http://opennmt.net/OpenNMT-py/extended.html

Command I used to train:

python train.py \
    -data data/multi30k.atok.low \ 
    -save_model multi30k_model \
    -gpu_ranks 0 \
    -seed 7 \
    -encoder_type brnn

Log corresponding to first run

[2019-03-05 15:19:24,166 INFO]  * src vocab size = 10841
[2019-03-05 15:19:24,166 INFO]  * tgt vocab size = 18563
[2019-03-05 15:19:24,166 INFO] Building model...
[2019-03-05 15:19:26,292 INFO] NMTModel(
  (encoder): RNNEncoder(
    (embeddings): Embeddings(
      (make_embedding): Sequential(
        (emb_luts): Elementwise(
          (0): Embedding(10841, 500, padding_idx=1)
        )
      )
    )
    (rnn): LSTM(500, 250, num_layers=2, dropout=0.3, bidirectional=True)
  )
  (decoder): InputFeedRNNDecoder(
    (embeddings): Embeddings(
      (make_embedding): Sequential(
        (emb_luts): Elementwise(
          (0): Embedding(18563, 500, padding_idx=1)
        )
      )
    )
    (dropout): Dropout(p=0.3)
    (rnn): StackedLSTM(
      (dropout): Dropout(p=0.3)
      (layers): ModuleList(
        (0): LSTMCell(1000, 500)
        (1): LSTMCell(500, 500)
      )
    )
    (attn): GlobalAttention(
      (linear_in): Linear(in_features=500, out_features=500, bias=False)
      (linear_out): Linear(in_features=1000, out_features=500, bias=False)
    )
  )
  (generator): Sequential(
    (0): Linear(in_features=500, out_features=18563, bias=True)
    (1): Cast()
    (2): LogSoftmax()
  )
)
[2019-03-05 15:19:26,292 INFO] encoder: 8428500
[2019-03-05 15:19:26,292 INFO] decoder: 24339563
[2019-03-05 15:19:26,292 INFO] * number of parameters: 32768063
[2019-03-05 15:19:26,294 INFO] Starting training on GPU: [0]
[2019-03-05 15:19:26,294 INFO] Start training loop and validate every 10000 steps...
[2019-03-05 15:19:26,452 INFO] Loading dataset from data/multi30k.atok.low.train.0.pt, number of examples: 29000
[2019-03-05 15:19:29,612 INFO] Step 50/100000; acc:   6.78; ppl: 57617.48; xent: 10.96; lr: 1.00000; 12248/12784 tok/s;      3 sec
[2019-03-05 15:19:33,015 INFO] Step 100/100000; acc:   8.31; ppl: 8108.67; xent: 9.00; lr: 1.00000; 12600/13026 tok/s;      7 sec
[2019-03-05 15:19:36,246 INFO] Step 150/100000; acc:  11.10; ppl: 1858.45; xent: 7.53; lr: 1.00000; 12712/13305 tok/s;     10 sec
[2019-03-05 15:19:39,541 INFO] Step 200/100000; acc:  16.75; ppl: 526.31; xent: 6.27; lr: 1.00000; 12944/13368 tok/s;     13 sec
[2019-03-05 15:19:42,763 INFO] Step 250/100000; acc:  19.17; ppl: 314.52; xent: 5.75; lr: 1.00000; 12971/13447 tok/s;     16 sec
[2019-03-05 15:19:46,020 INFO] Step 300/100000; acc:  21.85; ppl: 234.78; xent: 5.46; lr: 1.00000; 12955/13426 tok/s;     20 sec
[2019-03-05 15:19:49,319 INFO] Step 350/100000; acc:  23.89; ppl: 165.90; xent: 5.11; lr: 1.00000; 12717/13095 tok/s;     23 sec
[2019-03-05 15:19:52,494 INFO] Step 400/100000; acc:  26.36; ppl: 126.13; xent: 4.84; lr: 1.00000; 13350/13855 tok/s;     26 sec
[2019-03-05 15:19:55,955 INFO] Step 450/100000; acc:  27.12; ppl: 107.84; xent: 4.68; lr: 1.00000; 12102/12597 tok/s;     30 sec

Log corresponding to second run

[2019-03-05 15:20:49,051 INFO]  * src vocab size = 10841
[2019-03-05 15:20:49,052 INFO]  * tgt vocab size = 18563
[2019-03-05 15:20:49,052 INFO] Building model...
[2019-03-05 15:20:51,171 INFO] NMTModel(
  (encoder): RNNEncoder(
    (embeddings): Embeddings(
      (make_embedding): Sequential(
        (emb_luts): Elementwise(
          (0): Embedding(10841, 500, padding_idx=1)
        )
      )
    )
    (rnn): LSTM(500, 250, num_layers=2, dropout=0.3, bidirectional=True)
  )
  (decoder): InputFeedRNNDecoder(
    (embeddings): Embeddings(
      (make_embedding): Sequential(
        (emb_luts): Elementwise(
          (0): Embedding(18563, 500, padding_idx=1)
        )
      )
    )
    (dropout): Dropout(p=0.3)
    (rnn): StackedLSTM(
      (dropout): Dropout(p=0.3)
      (layers): ModuleList(
        (0): LSTMCell(1000, 500)
        (1): LSTMCell(500, 500)
      )
    )
    (attn): GlobalAttention(
      (linear_in): Linear(in_features=500, out_features=500, bias=False)
      (linear_out): Linear(in_features=1000, out_features=500, bias=False)
    )
  )
  (generator): Sequential(
    (0): Linear(in_features=500, out_features=18563, bias=True)
    (1): Cast()
    (2): LogSoftmax()
  )
)
[2019-03-05 15:20:51,172 INFO] encoder: 8428500
[2019-03-05 15:20:51,172 INFO] decoder: 24339563
[2019-03-05 15:20:51,172 INFO] * number of parameters: 32768063
[2019-03-05 15:20:51,174 INFO] Starting training on GPU: [0]
[2019-03-05 15:20:51,174 INFO] Start training loop and validate every 10000 steps...
[2019-03-05 15:20:51,334 INFO] Loading dataset from data/multi30k.atok.low.train.0.pt, number of examples: 29000
[2019-03-05 15:20:54,514 INFO] Step 50/100000; acc:   6.91; ppl: 55073.98; xent: 10.92; lr: 1.00000; 12169/12701 tok/s;      3 sec
[2019-03-05 15:20:57,952 INFO] Step 100/100000; acc:   7.66; ppl: 8183.91; xent: 9.01; lr: 1.00000; 12469/12891 tok/s;      7 sec
[2019-03-05 15:21:01,209 INFO] Step 150/100000; acc:  10.18; ppl: 1639.56; xent: 7.40; lr: 1.00000; 12611/13199 tok/s;     10 sec
[2019-03-05 15:21:04,543 INFO] Step 200/100000; acc:  16.62; ppl: 560.21; xent: 6.33; lr: 1.00000; 12790/13210 tok/s;     13 sec
[2019-03-05 15:21:07,803 INFO] Step 250/100000; acc:  19.80; ppl: 316.12; xent: 5.76; lr: 1.00000; 12822/13292 tok/s;     17 sec
[2019-03-05 15:21:11,072 INFO] Step 300/100000; acc:  22.44; ppl: 212.44; xent: 5.36; lr: 1.00000; 12908/13377 tok/s;     20 sec
[2019-03-05 15:21:14,356 INFO] Step 350/100000; acc:  24.59; ppl: 156.75; xent: 5.05; lr: 1.00000; 12773/13153 tok/s;     23 sec
[2019-03-05 15:21:17,519 INFO] Step 400/100000; acc:  26.89; ppl: 119.32; xent: 4.78; lr: 1.00000; 13400/13907 tok/s;     26 sec
[2019-03-05 15:21:20,966 INFO] Step 450/100000; acc:  26.97; ppl: 103.17; xent: 4.64; lr: 1.00000; 12152/12649 tok/s;     30 sec


Log corresponding to third run

[2019-03-05 15:43:12,280 INFO]  * src vocab size = 10841
[2019-03-05 15:43:12,280 INFO]  * tgt vocab size = 18563
[2019-03-05 15:43:12,280 INFO] Building model...
[2019-03-05 15:43:14,412 INFO] NMTModel(
  (encoder): RNNEncoder(
    (embeddings): Embeddings(
      (make_embedding): Sequential(
        (emb_luts): Elementwise(
          (0): Embedding(10841, 500, padding_idx=1)
        )
      )
    )
    (rnn): LSTM(500, 250, num_layers=2, dropout=0.3, bidirectional=True)
  )
  (decoder): InputFeedRNNDecoder(
    (embeddings): Embeddings(
      (make_embedding): Sequential(
        (emb_luts): Elementwise(
          (0): Embedding(18563, 500, padding_idx=1)
        )
      )
    )
    (dropout): Dropout(p=0.3)
    (rnn): StackedLSTM(
      (dropout): Dropout(p=0.3)
      (layers): ModuleList(
        (0): LSTMCell(1000, 500)
        (1): LSTMCell(500, 500)
      )
    )
    (attn): GlobalAttention(
      (linear_in): Linear(in_features=500, out_features=500, bias=False)
      (linear_out): Linear(in_features=1000, out_features=500, bias=False)
    )
  )
  (generator): Sequential(
    (0): Linear(in_features=500, out_features=18563, bias=True)
    (1): Cast()
    (2): LogSoftmax()
  )
)
[2019-03-05 15:43:14,413 INFO] encoder: 8428500
[2019-03-05 15:43:14,413 INFO] decoder: 24339563
[2019-03-05 15:43:14,413 INFO] * number of parameters: 32768063
[2019-03-05 15:43:14,414 INFO] Starting training on GPU: [0]
[2019-03-05 15:43:14,414 INFO] Start training loop and validate every 10000 steps...
[2019-03-05 15:43:14,572 INFO] Loading dataset from data/multi30k.atok.low.train.0.pt, number of examples: 29000
[2019-03-05 15:43:17,733 INFO] Step 50/100000; acc:   7.48; ppl: 54075.01; xent: 10.90; lr: 1.00000; 12248/12783 tok/s;      3 sec
[2019-03-05 15:43:21,130 INFO] Step 100/100000; acc:   8.09; ppl: 4684.89; xent: 8.45; lr: 1.00000; 12623/13049 tok/s;      7 sec
[2019-03-05 15:43:24,351 INFO] Step 150/100000; acc:  12.49; ppl: 1212.96; xent: 7.10; lr: 1.00000; 12747/13342 tok/s;     10 sec
[2019-03-05 15:43:27,652 INFO] Step 200/100000; acc:  17.16; ppl: 508.54; xent: 6.23; lr: 1.00000; 12921/13344 tok/s;     13 sec
[2019-03-05 15:43:30,878 INFO] Step 250/100000; acc:  19.25; ppl: 308.93; xent: 5.73; lr: 1.00000; 12957/13432 tok/s;     16 sec
[2019-03-05 15:43:34,110 INFO] Step 300/100000; acc:  22.38; ppl: 201.66; xent: 5.31; lr: 1.00000; 13058/13532 tok/s;     20 sec
[2019-03-05 15:43:37,408 INFO] Step 350/100000; acc:  24.36; ppl: 153.67; xent: 5.03; lr: 1.00000; 12718/13096 tok/s;     23 sec
[2019-03-05 15:43:40,577 INFO] Step 400/100000; acc:  27.57; ppl: 113.65; xent: 4.73; lr: 1.00000; 13373/13879 tok/s;     26 sec
[2019-03-05 15:43:44,026 INFO] Step 450/100000; acc:  27.10; ppl: 102.62; xent: 4.63; lr: 1.00000; 12146/12643 tok/s;     30 sec

picture achaitanyasai

Most helpful comment

Can you try to run a minimal example on CPU only ?

With CPU, it's working fine - the results are same each time. The problem arises only with GPU.
The other thing I observed is that, the initialized weights are same each time. But during the course of training, the loss and so are the gradients are differing. I guess is this related to precision issue?

so unless there is an obvious reason in pytorch, it's difficult to debug

I guess there are no issues with pytorch implementation of brnn. I wrote a minimalistic framework for MT in pytorch-1.0.1 which uses Bi-LSTM. I don't see any issue with it. Any way, I will check once again with pytorch issues/documentation and get back.

picture achaitanyasai on 8 Mar 2019
👍3

All 5 comments

I can't find a reason why this is happening.
Can you try to run a minimal example on CPU only ?
the only difference between rnn and brnn is being passed to pytorch in the rnn itself with bidirectional=true, so unless there is an obvious reason in pytorch, it's difficult to debug.

vince62s picture vince62s on 8 Mar 2019

Can you try to run a minimal example on CPU only ?

With CPU, it's working fine - the results are same each time. The problem arises only with GPU.
The other thing I observed is that, the initialized weights are same each time. But during the course of training, the loss and so are the gradients are differing. I guess is this related to precision issue?

so unless there is an obvious reason in pytorch, it's difficult to debug

I guess there are no issues with pytorch implementation of brnn. I wrote a minimalistic framework for MT in pytorch-1.0.1 which uses Bi-LSTM. I don't see any issue with it. Any way, I will check once again with pytorch issues/documentation and get back.

achaitanyasai picture achaitanyasai on 8 Mar 2019
👍3

Did you end up finding the reason ?
closing for now, reopen if needed.

vince62s picture vince62s on 8 Apr 2019

Looks like the problem is in cuDNN. The following issue https://github.com/pytorch/pytorch/issues/18110 has already been reported in PyTorch.

achaitanyasai picture achaitanyasai on 13 Apr 2019
👍2

I'm having the same problem here. Been trying for awhile to figure out what is the possible cause. Thanks for this report!

mhn226 picture mhn226 on 26 Apr 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

kakkartushar1 picture kakkartushar1  路  5Comments
1130310223 picture 1130310223  路  4Comments
andreaferretti picture andreaferretti  路  5Comments
Bachstelze picture Bachstelze  路  3Comments
mrpega picture mrpega  路  3Comments