Opennmt-py: Translation from English to Chinese, always repeating the same translation sentence.

Created on 30 Oct 2018  Β·  17Comments  Β·  Source: OpenNMT/OpenNMT-py

Recently, I used OpenNMT framework to train English to Chinese model just use the default settings. My training set is over one million English - Chinese translation sentences.
But my translation results got weird.
For example,
This is the original English text:
Send them to these foreign places.
the necessary excitation, even using crude natural crystals.
These are handcuffs, made for bad people.
Oh, I'd love to see that.
and Pozzuoli is very near to Baia, and near to Naples, and near to Pompeii and Herculaneum and so on
And I think that whatever he's attempting to dislodge...
What inning is it? - Bottom of the seventh.
So you don't have a date yet, either.
Oh, no, no, no. The kid's not gonna do any action without a contract.
And this is the Chinese Translation, I think I got something wrong in the training process. It would be very helpful if you can help me to dig into this issue.
是 ηš„ 。
是 ηš„ ， ζˆ‘ ηŸ₯道 。
δ½  ηŸ₯道 吗 ， ζˆ‘ 很 ε₯½ 。
ε₯½ ηš„ 。
是 ηš„ ， ζˆ‘ ηŸ₯道 。
是 ηš„ 。
是 ηš„ ， ζˆ‘ ηŸ₯道 。
是 ηš„ ， ζˆ‘ ηŸ₯道 。
δ½  在 θ―΄ δ»€δΉˆ ？
ζˆ‘ 是 θ―΄ ， ζˆ‘ 是 θ―΄ ，
哦 ， 倩 ε•Š ， ζˆ‘ 很 ε₯½ 。
是 ηš„ ， ζˆ‘ ηŸ₯道 。
δ½  ηŸ₯道 吗 ？
ε₯½ ηš„ 。
是 ηš„ 。
是 ηš„ ， ζˆ‘ ηŸ₯道 。
是 ηš„ ， ζˆ‘ ηŸ₯道 。
哦 ， 倩 ε“ͺ 。
是 ηš„ ， ζˆ‘ ηŸ₯道 。

question
Source
picture stanislashzc

All 17 comments

not reading Chinese so I can't help you with this.
Post your preprocess and training command lines if you want some help here.

vince62s picture vince62s on 30 Oct 2018

Sure.
preprocess command is:
python preprocess.py -train_src data/tgt-train-chn.txt -train_tgt data/src-train_1023.txt -valid_src data/tgt-val-chn.txt -valid_tgt data/src-val_1023.txt -save_data data/ai_1025
training command is:
python train.py -data data/ai_1025 -save_model ai_1025 -gpu_ranks 0 1
translate command is:
python translate.py -model ai_1025_step_70000.pt -src data/src-test.txt -output pred-chn-7.txt -replace_unk -verbose
some of the training logs:
[2018-10-27 10:47:24,262 INFO] Step 98450/100000; acc: 34.88; ppl: 23.89; xent: 3.17; lr: 0.00003; 18872/21004 tok/s; 5161 sec
[2018-10-27 10:47:26,351 INFO] Step 98500/100000; acc: 41.92; ppl: 16.15; xent: 2.78; lr: 0.00003; 28751/10934 tok/s; 5163 sec
[2018-10-27 10:47:28,541 INFO] Step 98550/100000; acc: 38.67; ppl: 17.49; xent: 2.86; lr: 0.00003; 20083/10857 tok/s; 5165 sec
[2018-10-27 10:47:30,708 INFO] Step 98600/100000; acc: 37.07; ppl: 19.58; xent: 2.97; lr: 0.00003; 18855/20794 tok/s; 5168 sec
[2018-10-27 10:47:32,868 INFO] Step 98650/100000; acc: 45.18; ppl: 15.49; xent: 2.74; lr: 0.00003; 15549/14751 tok/s; 5170 sec
[2018-10-27 10:47:34,971 INFO] Step 98700/100000; acc: 36.67; ppl: 23.24; xent: 3.15; lr: 0.00003; 2279/13329 tok/s; 5172 sec
[2018-10-27 10:47:37,186 INFO] Step 98750/100000; acc: 39.02; ppl: 17.79; xent: 2.88; lr: 0.00003; 26903/8643 tok/s; 5174 sec
[2018-10-27 10:47:39,319 INFO] Step 98800/100000; acc: 38.16; ppl: 19.81; xent: 2.99; lr: 0.00003; 20175/10966 tok/s; 5176 sec
[2018-10-27 10:47:41,504 INFO] Step 98850/100000; acc: 39.44; ppl: 20.30; xent: 3.01; lr: 0.00003; 6687/7657 tok/s; 5178 sec

stanislashzc picture stanislashzc on 30 Oct 2018
👍1

This is probably not your full command line.
did you set your optim to adam ? or used the default sgd ?
did you set a learning rate ?
also gpu_ranks 0 1 has to go with world_size 2

vince62s picture vince62s on 30 Oct 2018

Same here. I am also having the same issue and am using similar command lines from the quickstart/translation documentation.

python preprocess.py -train_src data/train.zh.atok -train_tgt data/train.en.atok -valid_src data/val.zh.atok -valid_tgt data/val.en.atok -save_data data/data.atok.low -lower"

python3 train.py -data data/data.atok.low -save_model models/data -gpu_ranks 0

python translate.py -model models -src data/src-test.txt -output pred.txt -replace_unk

@vince62s I will try the training again, being more specific in my command:

python3 train.py -data data/data.atok.low -save_model models/data -world_size 1 -gpu_ranks 0 -optim adam -adam_beta1 0.9 -adam_beta1 0.999 -learning_rate 0.0001

echan00 picture echan00 on 31 Oct 2018

Guys, if you want good results with Chinese you'll need to use the transformer.

read this:
https://github.com/OpenNMT/OpenNMT-py/blob/master/docs/source/FAQ.md

vince62s picture vince62s on 31 Oct 2018

Thanks. I am also trying out the transformer at the same time.

Do you have any suggested hyperparameters in addition to the ones in the FAQ?

python  train.py -data /tmp/de2/data -save_model /tmp/extra \
        -layers 6 -rnn_size 512 -word_vec_size 512 -transformer_ff 2048 -heads 8  \
        -encoder_type transformer -decoder_type transformer -position_encoding \
        -train_steps 200000  -max_generator_batches 2 -dropout 0.1 \
        -batch_size 4096 -batch_type tokens -normalization tokens  -accum_count 2 \
        -optim adam -adam_beta2 0.998 -decay_method noam -warmup_steps 8000 -learning_rate 2 \
        -max_grad_norm 0 -param_init 0  -param_init_glorot \
        -label_smoothing 0.1 -valid_steps 10000 -save_checkpoint_steps 10000 \
        -world_size 4 -gpu_ranks 0 1 2 3

Also, if I have 8 GPUS, does that mean I would use -world_size 8 -gpu_ranks 0 1 2 3 4 5 6 7?

echan00 picture echan00 on 31 Oct 2018

it should work fine and yes with 8 GPUs this is it.
However you may try to reduce warmup_steps to 6000 and see how it scores after 100000 or 130000 steps it should have converged already.

vince62s picture vince62s on 31 Oct 2018

Just received the result for my model trained with cmd line:
python3 train.py -data data/data.atok.low -save_model models/data2 -world_size 1 -gpu_ranks 0 -optim adam -adam_beta1 0.9 -adam_beta1 0.999 -learning_rate 0.0001

Getting the same issue with the predictions, the model is translating all sentences the same way:

SENT 3: (u'\u80a1\u4e1c\u5927\u4f1a_^_\u5206\u4e3a_^_\u5e74\u5ea6_^_\u80a1\u4e1c\u5927\u4f1a_^_\u548c_^_\u4e34\u65f6_^_\u80a1\u4e1c\u5927\u4f1a_^_\u3002_^_',)
PRED 3: In addition to the United States , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations and the United Nations , the United Nations and the United Nations .
PRED SCORE: -124.6664

SENT 4: (u'\u5e74\u5ea6_^_\u80a1\u4e1c\u5927\u4f1a_^_\u6bcf\u5e74_^_\u53ec\u5f00_^_\u4e00\u6b21_^_\uff0c_^_\u5e76_^_\u5e94\u4e8e_^_\u4e0a_^_\u4e00_^_\u4f1a\u8ba1\u5e74\u5ea6_^_\u5b8c\u7ed3_^_\u4e4b\u540e_^_\u4e4b_^_6_^_\u4e2a_^_\u6708_^_\u4e4b\u5185_^_\u4e3e\u884c_^_\u3002_^_',)
PRED 4: In addition to the United States , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations and the United Nations , the United Nations and the United Nations .
PRED SCORE: -124.6664

SENT 5: (u'\u4e34\u65f6_^_\u80a1\u4e1c\u5927\u4f1a_^_\u4e0d\u5b9a\u671f_^_\u53ec\u5f00_^_\uff0c_^_\u51fa\u73b0_^_\u300a_^_\u516c\u53f8\u6cd5_^_\u300b_^_\u548c_^_\u300a_^_\u516c\u53f8\u7ae0\u7a0b_^_\u300b_^_\u7b49_^_\u89c4\u5b9a_^_\u4e4b_^_\u5e94\u5f53_^_\u53ec\u5f00_^_\u4e34\u65f6_^_\u80a1\u4e1c\u5927\u4f1a_^_\u4e4b_^_\u60c5\u5f62_^_\u65f6_^_\uff0c_^_\u4e34\u65f6_^_\u80a1\u4e1c\u5927\u4f1a_^_\u5e94\u5f53_^_\u5728_^_2_^_\u4e2a_^_\u6708_^_\u5185_^_\u53ec\u5f00_^_\u3002_^_',)
PRED 5: In addition to the United States , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations and the United Nations , the United Nations and the United Nations .
PRED SCORE: -124.6664

SENT 6: (u'\u516c\u53f8_^_\u5728_^_\u4e0a\u8ff0_^_\u671f\u9650\u5185_^_\u4e0d\u80fd_^_\u53ec\u5f00_^_\u80a1\u4e1c\u5927\u4f1a_^_\u7684_^_\uff0c_^_\u5e94\u5f53_^_\u62a5\u544a_^_\u516c\u53f8_^_\u6240\u5728\u5730_^_\u4e2d\u56fd\u8bc1\u76d1\u4f1a_^_\u6d3e\u51fa\u673a\u6784_^_\u548c_^_\u516c\u53f8\u80a1\u7968_^_\u6302\u724c_^_\u4ea4\u6613_^_\u4e4b_^_\u8bc1\u5238_^_\u4ea4\u6613\u6240_^_\uff0c_^_\u8bf4\u660e_^_\u539f\u56e0_^_\u5e76_^_\u516c\u544a_^_\u3002_^_',)
PRED 6: In addition to the United States , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations and the United Nations , the United Nations and the United Nations .
PRED SCORE: -124.6664

SENT 7: (u'\u7b2c\u56db\u6761_^_\u80a1\u4e1c\u5927\u4f1a_^_\u662f_^_\u516c\u53f8_^_\u4e4b_^_\u6743\u529b_^_\u673a\u6784_^_\uff0c_^_\u4f9d\u6cd5_^_\u884c\u4f7f_^_\u4e0b\u5217_^_\u804c\u6743_^_\uff1a_^_',)
PRED 7: In addition to the United States , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations and the United Nations , the United Nations and the United Nations .
PRED SCORE: -124.6664

SENT 8: (u'\u4e00_^_\u3001_^_\u51b3\u5b9a_^_\u516c\u53f8_^_\u4e4b_^_\u7ecf\u8425_^_\u65b9\u9488_^_\u548c_^_\u6295\u8d44_^_\u8ba1\u5212_^_\uff1b_^_',)
PRED 8: In addition to the United States , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations , the United Nations and the United Nations , the United Nations and the United Nations .
PRED SCORE: -124.6664

Will also report with the transformer predictions once ready.

echan00 picture echan00 on 31 Oct 2018

can you post the end of training log (with validation too) ?

vince62s picture vince62s on 31 Oct 2018
😄1

Sorry, I did not log the training, I'm not sure I can salvage it anymore if it isn't saved by default.

I really appreciate your help and would be happy to run it again. Could you clarify what you mean by "with validation too"?

This is the command I plan to run (let me know if you want to make any changes):
python3 train.py -data data/data.atok.low -save_model models/data -world_size 1 -gpu_ranks 0 -optim adam -adam_beta1 0.9 -adam_beta1 0.999 -learning_rate 0.0001 -log_file log/log1.txt

echan00 picture echan00 on 31 Oct 2018

@vince62s, here is the log
log1.txt

echan00 picture echan00 on 2 Nov 2018

You seem to have 15M segments, so your network of 2x500 is too small.
But even with that are you 100% sure your data set is properly aligned and pre-processed ?
Because the ACC/PPL looks a bit erratic.
Try with the transformer you will know right away, because with 177 vlaidation PPL you won't have any good results for sure.

vince62s picture vince62s on 2 Nov 2018

But even with that are you 100% sure your data set is properly aligned and pre-processed?

Yes, I am fairly sure. I randomly sampled my training data and I see tokenized versions of the english and chinese text translations. Afterall, @stanislashzc also had the same problem with repeating translations.

Will keep you updated with transformer results...

echan00 picture echan00 on 3 Nov 2018

@vince62s Just finished training the transformer, the same problem exists, the predictions are being repeated:

SENT 2: (u'\u672c_^_\u89c4\u5219_^_\u9002\u7528_^_\u4e8e_^_\u516c\u53f8_^_\u80a1\u4e1c\u5927\u4f1a_^_\uff0c_^_\u5bf9_^_\u516c\u53f8_^_\u3001_^_\u80a1\u4e1c_^_\u3001_^_\u80a1\u4e1c_^_\u6388\u6743_^_\u4ee3\u7406\u4eba_^_\u3001_^_\u516c\u53f8_^_\u8463\u4e8b_^_\u3001_^_\u76d1\u4e8b_^_\u3001_^_\u4ee5\u53ca_^_\u5217\u5e2d_^_\u80a1\u4e1c\u5927\u4f1a_^_\u4e4b_^_\u7ecf\u7406_^_\u548c_^_\u5176\u4ed6_^_\u9ad8\u7ea7_^_\u7ba1\u7406\u4eba\u5458_^_\u5747_^_\u5177\u6709_^_\u7ea6\u675f\u529b_^_\u3002_^_',)
PRED 2: I can 't believe it .
PRED SCORE: -11.2765

SENT 3: (u'\u80a1\u4e1c\u5927\u4f1a_^_\u5206\u4e3a_^_\u5e74\u5ea6_^_\u80a1\u4e1c\u5927\u4f1a_^_\u548c_^_\u4e34\u65f6_^_\u80a1\u4e1c\u5927\u4f1a_^_\u3002_^_',)
PRED 3: I can 't believe it .
PRED SCORE: -11.2765

SENT 4: (u'\u5e74\u5ea6_^_\u80a1\u4e1c\u5927\u4f1a_^_\u6bcf\u5e74_^_\u53ec\u5f00_^_\u4e00\u6b21_^_\uff0c_^_\u5e76_^_\u5e94\u4e8e_^_\u4e0a_^_\u4e00_^_\u4f1a\u8ba1\u5e74\u5ea6_^_\u5b8c\u7ed3_^_\u4e4b\u540e_^_\u4e4b_^_6_^_\u4e2a_^_\u6708_^_\u4e4b\u5185_^_\u4e3e\u884c_^_\u3002_^_',)
PRED 4: I can 't believe it .
PRED SCORE: -11.2765

SENT 5: (u'\u4e34\u65f6_^_\u80a1\u4e1c\u5927\u4f1a_^_\u4e0d\u5b9a\u671f_^_\u53ec\u5f00_^_\uff0c_^_\u51fa\u73b0_^_\u300a_^_\u516c\u53f8\u6cd5_^_\u300b_^_\u548c_^_\u300a_^_\u516c\u53f8\u7ae0\u7a0b_^_\u300b_^_\u7b49_^_\u89c4\u5b9a_^_\u4e4b_^_\u5e94\u5f53_^_\u53ec\u5f00_^_\u4e34\u65f6_^_\u80a1\u4e1c\u5927\u4f1a_^_\u4e4b_^_\u60c5\u5f62_^_\u65f6_^_\uff0c_^_\u4e34\u65f6_^_\u80a1\u4e1c\u5927\u4f1a_^_\u5e94\u5f53_^_\u5728_^_2_^_\u4e2a_^_\u6708_^_\u5185_^_\u53ec\u5f00_^_\u3002_^_',)
PRED 5: I can 't believe it .
PRED SCORE: -11.2765

SENT 6: (u'\u516c\u53f8_^_\u5728_^_\u4e0a\u8ff0_^_\u671f\u9650\u5185_^_\u4e0d\u80fd_^_\u53ec\u5f00_^_\u80a1\u4e1c\u5927\u4f1a_^_\u7684_^_\uff0c_^_\u5e94\u5f53_^_\u62a5\u544a_^_\u516c\u53f8_^_\u6240\u5728\u5730_^_\u4e2d\u56fd\u8bc1\u76d1\u4f1a_^_\u6d3e\u51fa\u673a\u6784_^_\u548c_^_\u516c\u53f8\u80a1\u7968_^_\u6302\u724c_^_\u4ea4\u6613_^_\u4e4b_^_\u8bc1\u5238_^_\u4ea4\u6613\u6240_^_\uff0c_^_\u8bf4\u660e_^_\u539f\u56e0_^_\u5e76_^_\u516c\u544a_^_\u3002_^_',)
PRED 6: I can 't believe it .
PRED SCORE: -11.2765

SENT 7: (u'\u7b2c\u56db\u6761_^_\u80a1\u4e1c\u5927\u4f1a_^_\u662f_^_\u516c\u53f8_^_\u4e4b_^_\u6743\u529b_^_\u673a\u6784_^_\uff0c_^_\u4f9d\u6cd5_^_\u884c\u4f7f_^_\u4e0b\u5217_^_\u804c\u6743_^_\uff1a_^_',)
PRED 7: I can 't believe it .
PRED SCORE: -11.2765

Here is my log:
transformer-opennmt_2018-11-04t08-18-29-846489.log

For the sake of completeness, below are the commands used:

python preprocess.py -train_src data/train.zh.atok -train_tgt data/train.en.atok -valid_src data/val.zh.atok -valid_tgt data/val.en.atok -save_data data/data.atok.low -lower

python3 train.py -data data/data.atok.low -save_model models/data \
        -layers 6 -rnn_size 512 -word_vec_size 512 -transformer_ff 2048 -heads 8  \
        -encoder_type transformer -decoder_type transformer -position_encoding \
        -train_steps 150000  -max_generator_batches 2 -dropout 0.1 \
        -batch_size 4096 -batch_type tokens -normalization tokens  -accum_count 2 \
        -optim adam -adam_beta2 0.998 -decay_method noam -warmup_steps 6000 -learning_rate 2 \
        -max_grad_norm 0 -param_init 0  -param_init_glorot \
        -label_smoothing 0.1 -valid_steps 10000 -save_checkpoint_steps 10000 \
        -world_size 8 -gpu_ranks 0 1 2 3 4 5 6 7 -log_file log/log1.txt

python translate.py -gpu 0 -model models/data_step_150000.pt -src ./folder1/some_file.atok -output ./folder2/some_file.atok -replace_unk -verbose

Here is the validation data I used (renamed from .atok to .txt):
val.en.txt val.zh.txt
(training data is exactly the same format, and I have checked parallel corpus is aligned)

echan00 picture echan00 on 10 Nov 2018

There is one issue for sure which is that your dataset is not shuffled BEFORE preprocessing. see some other discussion on this.

The second point is your chinese tokenization.
I don't know what you used to prepare your chinese side, but you need space tokenization, where does this "_^_" come from ?

vince62s picture vince62s on 10 Nov 2018

"^" is a symbol representing spaces between tokens, I chose it to keep actual spaces as a separate token.

Would you mind pointing me to where I can find out more about shuffling before preprocessing?

echan00 picture echan00 on 10 Nov 2018

You can't do that unless you modify the code, and even if you change the splitter you wold need to pay attention between source and target.
start preprocessing correctly and train on a toy before trying to train huge dataset that requires shuffling.

vince62s picture vince62s on 10 Nov 2018
Was this page helpful?
0 / 5 - 0 ratings

Related issues

1130310223 picture 1130310223  Β·  4Comments
howardyclo picture howardyclo  Β·  6Comments
achaitanyasai picture achaitanyasai  Β·  5Comments
caozhen-alex picture caozhen-alex  Β·  5Comments
azureskyL picture azureskyL  Β·  3Comments