Opennmt-py: Torchtext v0.2 issue

[laubonghaudoi]

I followed the instructions in Quickstart to train the model, and after training some time it showed the following errors, it there any way to solve it ?

Epoch  1,    50/  157; acc:   3.77; ppl: 111362.48; 200 src tok/s; 201 tgt tok/s;    356 s elapsed
Epoch  1,   100/  157; acc:   6.40; ppl: 21387.26; 217 src tok/s; 214 tgt tok/s;    692 s elapsed
Epoch  1,   150/  157; acc:   7.70; ppl: 4699.85; 218 src tok/s; 217 tgt tok/s;   1021 s elapsed
Train perplexity: 20880.5
Train accuracy: 6.0594
Traceback (most recent call last):
  File "train.py", line 299, in <module>
    main()
  File "train.py", line 295, in main
    train_model(model, train, valid, fields, optim)
  File "train.py", line 160, in train_model
    valid_stats = trainer.validate()
  File "/home/user/Documents/OpenNMT-py/onmt/Trainer.py", line 155, in validate
    outputs, attns, _ = self.model(src, tgt, src_lengths)
  File "/home/user/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/Documents/OpenNMT-py/onmt/Models.py", line 417, in forward
    enc_hidden, context = self.encoder(src, lengths)
  File "/home/user/anaconda3/lib/python3.6/site-packages/torch/nn/modules/module.py", line 224, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/user/Documents/OpenNMT-py/onmt/Models.py", line 94, in forward
    packed_emb = pack(emb, lengths)
  File "/home/user/anaconda3/lib/python3.6/site-packages/torch/nn/utils/rnn.py", line 79, in pack_padded_sequence
    raise ValueError("lengths array has to be sorted in decreasing order")
ValueError: lengths array has to be sorted in decreasing order

[JianyuZhan]

Duplicate of #189, the new torchtext v0.2 has addressed this problem.

[laubonghaudoi]

@JianyuZhan I have upgraded my torchtext to 0.2.0 and this issue still exists, please reopen this issue.

[JianyuZhan]

@laubonghaudoi , seems we need some modification based on torchtext 0.2.0 as well.

I didn't encounter this problem in my environment, could you help test if this patch helps solve this problem? If you encounter any problem, please post the trackback.

diff --git a/onmt/IO.py b/onmt/IO.py
index f67ccdc..f635d0e 100644
--- a/onmt/IO.py
+++ b/onmt/IO.py
@@ -107,8 +107,8 @@ class ONMTDataset(torchtext.data.Dataset):

     @staticmethod
     def sort_key(ex):
-        "Sort in reverse size order"
-        return -len(ex.src)
+        "Sort by length of sentences."
+        return len(ex.src)

     def __init__(self, src_path, tgt_path, fields, opt,
                  src_img_dir=None, **kwargs):
diff --git a/train.py b/train.py
index eb102b4..7e25877 100644
--- a/train.py
+++ b/train.py
@@ -94,10 +94,12 @@ def make_train_data_iter(train_data, opt):
     ordered iterator strategy here, but more sophisticated strategy
     like curriculum learning is ok too.
     """
+    # Sort batch by decreasing lengths of sentence required by pytorch.
+    # sort=False means "Using dataset's sortkey instead of iterator's sortkey".
     return onmt.IO.OrderedIterator(
                 dataset=train_data, batch_size=opt.batch_size,
                 device=opt.gpuid[0] if opt.gpuid else -1,
-                repeat=False)
+                sort=False, sort_within_batch=True, repeat=False)


 def make_valid_data_iter(valid_data, opt):
diff --git a/translate.py b/translate.py
index 245c947..88d7f67 100644
--- a/translate.py
+++ b/translate.py
@@ -60,7 +60,7 @@ def main():
     test_data = onmt.IO.OrderedIterator(
         dataset=data, device=opt.gpu,
         batch_size=opt.batch_size, train=False, sort=False,
-        shuffle=False)
+        sort_within_batch=True, shuffle=False)

     counter = count(1)
     for batch in test_data:

[laubonghaudoi]

I run this patch and it works fine now, thank you very much.

[thammegowda]

I hit the same error today. This happened with torchtext (0.2.0a0) and torch (0.2.0.post4) with default installation

I just downgraded torchtext with pip install torchtext==0.1.1 as a temporary/quick fix, it works.

Going forward:

1. the code should be patched with the above patch https://github.com/OpenNMT/OpenNMT-py/issues/345#issuecomment-338145520
   OR
2. requirements.txt should force the torchtext==0.1.1

[jekbradbury]

Since we'd already made the decision to change the sort behavior in torchtext source, and some people had started to rely on that, we basically had to break things slightly for either 0.1 users/ONMT or source users. Sorry about that; hopefully the two issues from 0.1.1 -> 0.2 can be resolved quickly.

[donglixp]

Add ``sort_within_batch=False'' as follows (train.py) will fix this issue.

valid_iter = table.IO.OrderedIterator(
        dataset=valid_data, batch_size=opt.batch_size, device=opt.gpuid[0], train=False, sort=True, sort_within_batch=False)

[caozhen-alex]

@donglixp Thx for the answer. But, it seems no $"valid_iter = table.IO.OrderedIterator(
dataset=valid_data, batch_size=opt.batch_size, device=opt.gpuid[0], train=False, sort=True, sort_within_batch=False)"$ in the train.py file

[caozhen-alex]

@laubonghaudoi Excuse me, how can I run that patch?

[caozhen-alex]

@JianyuZhan Thx for your answer. But how can I run the patch you provided?

[JianyuZhan]

@caozhen-alex , copy the text and save it into a file like foo.patch, then put it in the root directory of the project and do patch -p1 < foo.patch.

[donglixp]

@caozhen-alex It's line 113 in train.py: https://github.com/OpenNMT/OpenNMT-py/blob/master/train.py#L113

[caozhen-alex]

@donglixp Thx Li Dong.

[caozhen-alex]

@JianyuZhan Thx, Jianyu. Thank you for your instrument. Heard this kind of stuff first time. Could you also tell me what this file will do? What knowledge can help me to understand this?

[JianyuZhan]

@caozhen-alex , the problem in this issue is due to some change in the new version of torchtext(our project depends on it), so our project need fix to accommodate this change.

The code is generated by git diff, which shows code difference between the base version and the modified version, and the diff output is usually called patch, and it can be patched into base version by the tool patch.

[caozhen-alex]

@JianyuZhan Thank you very much for your patience and clear explanation!

[JianyuZhan]

Hello, @jekbradbury , is the fix available on PyPI? If yes, we will resolve the problems on our side.