Opennmt-py: Preprocess failing on single token inputs

[andrewnc]

I am trying a simple translation problem from data that looks like

-2.0 x^ 1.0 + -1.0 x^ -1.0 + ... -> b j r q p o n m l k a i h g f e d c

When I run the preprocessing step, the vocab is

tgt vocab {'<unk>': 0, '<blank>': 1, '<s>': 2, '</s>': 3}
src vocab {'<unk>': 0, '<blank>': 1}

I have checked the source/target files pretty thoroughly and am confused about the issue. I would appreciate any guidance.

[andrewnc]

I additionally, I constructed my own vocab files and passed them with the preprocessing step. This also doesn't produce training data and the training step fails.

[pltrdy]

Could you provide:

• the exact command line your running (the preprocess)
• small data examples, (i.e. train.src.txt / train.tgt.txt)

[andrewnc]

onmt_preprocess -train_src ./data/train_src.txt -train_tgt ./data/train_tgt.txt -valid_src ./data/valid_src.txt -valid_tgt ./data/valid_tgt.txt -save_data ./data/model

train_tgt.txt:
```b j r q p o n m l k a i h g f e d c
b j k l m n o p q r a c d e f g h i
c k j r q p o n m l b a i h g f e d
c k j r q p o n m l a b i h g f e d
c k j l m n o p q r b a d e f g h i

train_src.txt:
```-2.0 x^ 1.0 + -1.0 x^ -1.0 + 0.0 x^ 2.0 + 1.0 x^ -2.0 + 2.0 x^ 2.0 + 3.0 x^ -2.0 + 4.0 x^ 2.0 + 5.0 x^ -2.0 + 6.0 x^ 2.0 + 7.0 x^ -2.0 + 8.0 x^ 2.0 + 9.0 x^ -2.0 + 10.0 x^ 2.0 + 11.0 x^ -2.0 + 12.0 x^ 2.0 + 13.0 x^ -2.0 + 14.0 x^ 1.0 + 15.0 x^ -1.0 + 16.0 x^ 1.0 
5.0 x^ 1.0 + 6.0 x^ -1.0 + 7.0 x^ 2.0 + 8.0 x^ -2.0 + 9.0 x^ 3.0 + 10.0 x^ -3.0 + 11.0 x^ 3.0 + 12.0 x^ -3.0 + 13.0 x^ 3.0 + 14.0 x^ -3.0 + 15.0 x^ 3.0 + 16.0 x^ -3.0 + 17.0 x^ 3.0 + 18.0 x^ -3.0 + 19.0 x^ 3.0 + 20.0 x^ -3.0 + 21.0 x^ 2.0 + 22.0 x^ -2.0 + 23.0 x^ 1.0 
-4.0 x^ 1.0 + -3.0 x^ -1.0 + -2.0 x^ 2.0 + -1.0 x^ -3.0 + 0.0 x^ 4.0 + 1.0 x^ -4.0 + 2.0 x^ 4.0 + 3.0 x^ -4.0 + 4.0 x^ 4.0 + 5.0 x^ -4.0 + 6.0 x^ 4.0 + 7.0 x^ -4.0 + 8.0 x^ 4.0 + 9.0 x^ -4.0 + 10.0 x^ 3.0 + 11.0 x^ -3.0 + 12.0 x^ 2.0 + 13.0 x^ -1.0 + 14.0 x^ 1.0 
-5.0 x^ 1.0 + -4.0 x^ -2.0 + -3.0 x^ 3.0 + -2.0 x^ -3.0 + -1.0 x^ 4.0 + 0.0 x^ -4.0 + 1.0 x^ 4.0 + 2.0 x^ -4.0 + 3.0 x^ 4.0 + 4.0 x^ -4.0 + 5.0 x^ 4.0 + 6.0 x^ -4.0 + 7.0 x^ 4.0 + 8.0 x^ -4.0 + 9.0 x^ 3.0 + 10.0 x^ -3.0 + 11.0 x^ 2.0 + 12.0 x^ -1.0 + 13.0 x^ 1.0 
2.0 x^ 1.0 + 3.0 x^ -1.0 + 4.0 x^ 2.0 + 5.0 x^ -3.0 + 6.0 x^ 4.0 + 7.0 x^ -4.0 + 8.0 x^ 5.0 + 9.0 x^ -5.0 + 10.0 x^ 5.0 + 11.0 x^ -5.0 + 12.0 x^ 5.0 + 13.0 x^ -5.0 + 14.0 x^ 5.0 + 15.0 x^ -5.0 + 16.0 x^ 4.0 + 17.0 x^ -4.0 + 18.0 x^ 3.0 + 19.0 x^ -2.0 + 20.0 x^ 1.0 

[francoishernandez]

Your src is longer than 50, which is the default value of -src_seq_length:
https://github.com/OpenNMT/OpenNMT-py/blob/46c04567ba425f1186a9e9d262fc8ec5764aa169/onmt/opts.py#L287
Hence every line is ignored in your dataset.

[andrewnc]

Thank you! I feel a little silly, but I definitely appreciate the help.

[francoishernandez]

No worries, some of the default values might not be the best depending on tasks.