Opennmt-py: Assertion Error when preprocessing large parallel corpus

[kakkartushar1]

Preprocessing is failing multiple times giving me the same error again and again. Last time it created 99 shards before throwing this, one time this occurred after creating 199 shards. Also, the combined size of these 97 shards is almost 2.5 times the size of the original corpus.

[2019-01-24 13:38:41,952 INFO] Building shard 97.
[2019-01-24 13:39:06,778 INFO]  * saving 97th train data shard to /data/opus/fr-en/data.train.97.pt.
Traceback (most recent call last):
  File "preprocess.py", line 166, in <module>
    main()
  File "preprocess.py", line 156, in main
    train_dataset_files = build_save_dataset('train', fields, opt)
  File "preprocess.py", line 66, in build_save_dataset
    assert len(src_shard) == len(tgt_shard)
AssertionError

Command that I am using is this -

python preprocess.py -train_src /data/opus/es-en/training-data/es.txt -train_tgt /data/opus/es-en/training-data/en.txt -valid_src /data/opus/es-en/validator-data/es.txt -valid_tgt /data/opus/es-en/validator-data/en.txt -save_data /data/opus/es-en/data -report_every 10000

[vince62s]

I just experienced the same thing this morning, I will check.

[vince62s]

As far as I am concerned, it was a classic issue in my dataset with one of these line break sequences:
https://stackoverflow.com/questions/13836352/what-is-the-utf-8-representation-of-end-of-line-in-text-file
search for E2 80 A8 or E2 80 A9 or C2 85
pretty sure this is a similar issue for you.

[kakkartushar1]

Thanks for the quick action

[F1nalFortune]

Kakkartushar1 - how did you fix the error ?

[girishnjha]

i get this AssertionError from the multi_headed_attn.py ONLY when i use the 'transformer' model in training
onmt_train
-data EMMT/demo
-save_model demo-model
-encoder_type transformer
-decoder_type transformer

Any help?