Opennmt-py: pickle.dump memory error

[mzeidhassan]

I am getting a memory error while ONMT is saving the demo.train.pt file.

Here is the output of the terminal:

Prepared 11062622 sentences (788855 ignored due to length == 0 or src len > 50 or tgt len > 50) Preparing validation ... Processing data/src-val.txt & data/tgt-val.txt ... ... shuffling sentences ... sorting sentences by size Prepared 5300 sentences (17 ignored due to length == 0 or src len > 50 or tgt len > 50) Saving source vocabulary to 'data/demo.src.dict'... Saving target vocabulary to 'data/demo.tgt.dict'... Saving data to 'data/demo.train.pt'... Traceback (most recent call last): File "preprocess.py", line 296, in <module> main() File "preprocess.py", line 292, in main torch.save(save_data, opt.save_data + '.train.pt') File "C:\Users\Anaconda3\envs\python35\lib\site-packages\torch\serialization.py", line 120, in save return _save(obj, f, pickle_module, pickle_protocol) File "C:\Users\Anaconda3\envs\python35\lib\site-packages\torch\serialization.py", line 186, in _save pickler.dump(obj) MemoryError

Is there a workaround for this issue?

I came across this issue and it could be helpful.
https://stackoverflow.com/questions/17513036/pickle-dump-huge-file-without-memory-error

Thanks,
mzeid

[srush]

Yes, we are aware this is an issue and are looking into it.

[laubonghaudoi]

I also have this issue and up to now have no fundamental solutions. A temporary work around is to divide your training set into several pieces, and use multi-fold training like cross-validation. Or you have to add more RAM to your machine.

[wolfshow]

Same issue with large training data

[srush]

Okay, so we now have a way to shard the training preprocessing. Let us know if this satisfies your issues.