Opennmt-py: can't save vocab, pickling error

Created on 10 Jan 2019 · 8Comments · Source: OpenNMT/OpenNMT-py

I am experiencing the following error when I run the preprocessor.py function

python preprocess.py -train_src data/morph/src.train -train_tgt data/morph/tgt.train -valid_src data/morph/src.valid -valid_tgt data/morph/tgt.valid -save_data data/morph

[2019-01-09 20:31:10,776 INFO] Extracting features...
[2019-01-09 20:31:10,778 INFO] * number of source features: 0.
[2019-01-09 20:31:10,778 INFO] * number of target features: 0.
[2019-01-09 20:31:10,778 INFO] Building Fields object...
[2019-01-09 20:31:10,778 INFO] Building & saving training data...
[2019-01-09 20:31:10,778 INFO] Reading source and target files: data/morph/src.train data/morph/tgt.train.
[2019-01-09 20:31:10,802 INFO] Building shard 0.
[2019-01-09 20:31:10,859 INFO] * saving 0th train data shard to data/amr.train.0.pt.
[2019-01-09 20:31:10,970 INFO] Building & saving validation data...
[2019-01-09 20:31:10,970 INFO] Reading source and target files: data/morph/src.valid data/morph/tgt.valid.
[2019-01-09 20:31:10,972 INFO] Building shard 0.
[2019-01-09 20:31:10,973 INFO] * saving 0th valid data shard to data/amr.valid.0.pt.
[2019-01-09 20:31:10,992 INFO] Building & saving vocabulary...
[2019-01-09 20:31:11,009 INFO] * reloading data/amr.train.0.pt.
[2019-01-09 20:31:11,045 INFO] * tgt vocab size: 31.
[2019-01-09 20:31:11,046 INFO] * src vocab size: 69.
Traceback (most recent call last):
File "preprocess.py", line 177, in
main()
File "preprocess.py", line 173, in main
build_save_vocab(train_dataset_files, fields, opt)
File "preprocess.py", line 118, in build_save_vocab
torch.save(fields, vocab_path)
File "/home/dhanush/Software/miniconda2/envs/py36/lib/python3.6/site-packages/torch/serialization.py", line 218, in save
return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
File "/home/dhanush/Software/miniconda2/envs/py36/lib/python3.6/site-packages/torch/serialization.py", line 143, in _with_file_like
return body(f)
File "/home/dhanush/Software/miniconda2/envs/py36/lib/python3.6/site-packages/torch/serialization.py", line 218, in
return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
File "/home/dhanush/Software/miniconda2/envs/py36/lib/python3.6/site-packages/torch/serialization.py", line 291, in _save
pickler.dump(obj)
_pickle.PicklingError: Can't pickle at 0x7f23213f5c80>: attribute lookup Field. on torchtext.data.field failed

I experienced the same error on my data aswell.

Any reason why this is happening ?

I am using python 3.6 and torch 1.0.0

Source

Dabuk

👍1

Most helpful comment

did you try to install from git+https://github.com/pytorch/text
let us know if it fixes it.
@bpopeters not sure about the wersion that allowed to pickle fields.

vince62s on 10 Jan 2019

👍3

All 8 comments

what is your torchtext version ?

vince62s on 10 Jan 2019

0.3.1

Dabuk on 10 Jan 2019

did you try to install from git+https://github.com/pytorch/text
let us know if it fixes it.
@bpopeters not sure about the wersion that allowed to pickle fields.

vince62s on 10 Jan 2019

👍3

I'm not sure exactly which version fixed it, but I have torchtext 0.4.0 and fields can be saved. If I recall, in previous versions of torchtext the default tokenizer for fields was a lambda expression for some reason and that was the only reason it couldn't be pickled. tl;dr try torchtext 0.4.0

Digression: @vince62s this reminds me, could we change the name of the `morph' dataset? It originates in some Serbo-Croatian grapheme-to-phoneme conversion experiments I did a couple years ago and somehow got labeled as morphology even though it isn't.

bpopeters on 10 Jan 2019

sure, I just dont' have access to hosted datasets (@ harvard NLP) but this name should not be in there.

vince62s on 10 Jan 2019

solved the issue. Thank you.
@vince62s your suggestion helped
@bpopeters yes, I had a lower version and hence the issue.

Dabuk on 10 Jan 2019

Same error after update the onmt modules.

File "/opt/anaconda3/envs/pytorch/lib/python3.6/site-packages/torch/serialization.py", line 292, in _save
pickler.dump(obj)
_pickle.PicklingError: Can't pickle at 0x7f24904688c8>: attribute lookup Field. on torchtext.data.field failed

I was using PyTorch 0.4.1. Than, I verified that the requirements.txt changed too. I had to remove the old version before to install PyTorch 1.0.0. However, the error still occurs when preprocess.py tries to save the vocab.

I think that threre is another problem when using ￭ (joiner), but is better to take one thing at time.