Nmt: How much time does it take to learn the Byte Pair Encoding?

[ArmageddonKnight]

Hi,

I am trying to work on the large-scale data (WMT 16 Englist-to-German). I spent the whole night running the command nmt/scripts/wmt16_en_de.sh /tmp/wmt16 but noticed that it seems to get "stuck" when learning the Byte Pair Encoding. I also noticed, from the top command, that the program learn_bpe.py is occupying around 80% of my memory (it is as if my computer just got frozen and I was unable to run any other program). Therefore, could you please kindly tell me:

• How much time does it usually take to complete the learn_bpe.py program?

• Is it possible to skip the Byte Pair Encoding step? If yes, how should we do this and what impact will it have on the training results?

Thank you.

[lmthang]

I can't recall how long does it take maybe a few hours ...
Yes, you can comment out that learn_bpe step and use this final result file (attached, bpe.32000.txt).

You will need to rename to bpe.32000 for apply_bpe as in
https://github.com/tensorflow/nmt/blob/master/nmt/scripts/wmt16_en_de.sh#L142

[ArmageddonKnight]

Thanks a lot.