Text: Unable to load vocab objects - _default_unk_index issue

[dilettacal]

Exception:
Error while loading vocabularies: Can't get attribute '_default_unk_index' on <module 'torchtext.vocab' from 'home/user/project/env37/lib/python3.7/site-packages/torchtext/vocab.py'>

I have trained an encoder-decoder model for translation. For the model I have pickled the vocabularies.
Now, every time I launch the program, this problem occurs. So I have to reload and process the files, then recreate the objects, in order to use my model to translate from terminal or from a file.

To pickle the vocab objects I use the dill module. I load the vocabularies for the source and target language as follows:
SRC_vocab = torch.load(os.path.join(path_to_exp, "src.pkl")),
TRG_vocab = torch.load(os.path.join(path_to_exp, "trg.pkl"))

The problem occurs with python3.7, while with python3.6 it works fine.

Attached the dependencies file (python3.7: req_37.txt, python3.6: req_36.txt).
req_36.txt
req_37.txt

[mttk]

So, if I understood correctly, you store the objects wih dill, but load with torch.load (which uses regular pickle as the backend)? I'm not sure if this is guaranteed to work as the load/save methods are different.

Could you try storing your vocab with torch.save instead (Vocabs are picklable since #453)?

[dilettacal]

I store the objects with torch.save but using the pickle module dill:
torch.save(obj_dict, os.path.join(path_to_pkl), pickle_module=dill), and obj_dict is a torchtext Field object. Then I load the pickled object using torch.load as explained in the comment above

[mttk]

Ah, ok. That was changed in the meantime I guess. Could you try leaving the pickle_module arg as default and report what happens?

[zhangguanheng66]

I tried to torch.save vocab before with the default args and it worked.

[QDxiaoye]

I guess the issue is the version of the package.
when I use dill to save object at torchtext=0.3.1 and load object at torchtext=0.4.0, the issue happend

[duhaime]

I just downgraded to 0.3.1 and this issue was resolved:

pip install torchtext==0.3.1