Opennmt-py: preprocess.py gives TypeError: __reduce_ex__() takes exactly one argument (0 given)

Created on 12 Oct 2018 · 8Comments · Source: OpenNMT/OpenNMT-py

Error occurred running this command:

python OpenNMT-py/preprocess.py \
    -train_src /home/arya/OpenNMT-py/data/src-train.txt \
    -train_tgt /home/arya/OpenNMT-py/data/tgt-train.txt \
    -valid_src /home/arya/OpenNMT-py/data/src-val.txt \
    -valid_tgt /home/arya/OpenNMT-py/data/tgt-val.txt \
    -shard_size 0 \
    -save_data /export/c03/arya/bibles/data/onmt-models/German-English/data//data \
    -seed 1337

The log is here:

[2018-10-11 21:36:54,908 INFO] Extracting features...
[2018-10-11 21:36:54,910 INFO]  * number of source features: 0.
[2018-10-11 21:36:54,910 INFO]  * number of target features: 0.
[2018-10-11 21:36:54,910 INFO] Building `Fields` object...
[2018-10-11 21:36:54,910 INFO] Building & saving training data...
[2018-10-11 21:36:56,293 INFO]  * saving train dataset to /export/c03/arya/bibles/data/onmt-models/German-English/data//data.train.pt.
Traceback (most recent call last):
  File "/home/arya/OpenNMT-py/preprocess.py", line 235, in <module>
    main()
  File "/home/arya/OpenNMT-py/preprocess.py", line 225, in main
    train_dataset_files = build_save_dataset('train', fields, opt)
  File "/home/arya/OpenNMT-py/preprocess.py", line 183, in build_save_dataset
    torch.save(dataset, pt_file)
  File "/home/arya/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 209, in save
    return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
  File "/home/arya/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 134, in _with_file_like
    return body(f)
  File "/home/arya/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 209, in <lambda>
    return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
  File "/home/arya/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 282, in _save
    pickler.dump(obj)
  File "/home/arya/OpenNMT-py/onmt/inputters/dataset_base.py", line 39, in __reduce_ex__
    return super(DatasetBase, self).__reduce_ex__()
TypeError: __reduce_ex__() takes exactly one argument (0 given)

My ONMT version is 0.4.1, torchtext is 0.3.1, and PyTorch is 0.4.1.

Un-setting -shard_size made no difference.

Source

aryamccarthy

Most helpful comment

It looks like it's as simple as passing proto to DataetBase.__reduce_ex__:

class DatasetBase(torchtext.data.Dataset):
    """
    A dataset basically supports iteration over all the examples
    it contains. We currently have 3 datasets inheriting this base
    for 3 types of corpus respectively: "text", "img", "audio".

    Internally it initializes an `torchtext.data.Dataset` object with
    the following attributes:

     `examples`: a sequence of `torchtext.data.Example` objects.
     `fields`: a dictionary associating str keys with `torchtext.data.Field`
        objects, and not necessarily having the same keys as the input fields.
    """

    def __getstate__(self):
        return self.__dict__

    def __setstate__(self, _d):
        self.__dict__.update(_d)

    def __reduce_ex__(self, proto):
        "This is a hack. Something is broken with torch pickle."
        return super(DatasetBase, self).__reduce_ex__(proto) # <---- this seems to fix it

    # ...

I'm not sure why moving back to Python 3.6 would change this. 🤔

erip on 27 Nov 2018

👍10

All 8 comments

I don't have time to install python 3.7 and inquire but this is the same as this:
https://github.com/OpenNMT/OpenNMT-py/issues/937

vince62s on 12 Oct 2018

I have the same question. Do you have solved it?

zhangxinlu16 on 31 Oct 2018

~/code/OpenNMT-py$ python3 preprocess.py -data_type audio -src_dir data/speech/an4_dataset -train_src data/speech/src-train.txt -train_tgt data/speech/tgt-train.txt -valid_src data/speech/src-val.txt -valid_tgt data/speech/tgt-val.txt -shard_size 300 -save_data data/speech/demo
[2018-10-31 16:42:13,321 INFO] Extracting features...
[2018-10-31 16:42:13,321 INFO] * number of source features: 0.
[2018-10-31 16:42:13,321 INFO] * number of target features: 0.
[2018-10-31 16:42:13,321 INFO] Building Fields object...
[2018-10-31 16:42:13,321 INFO] Building & saving training data...
[2018-10-31 16:42:13,321 INFO] Reading source and target files: data/speech/src-train.txt data/speech/tgt-train.txt.
[2018-10-31 16:42:13,322 INFO] Splitting shard 0.
[2018-10-31 16:42:13,322 INFO] Splitting shard 1.
[2018-10-31 16:42:13,323 INFO] Splitting shard 2.
[2018-10-31 16:42:13,323 INFO] Splitting shard 3.
[2018-10-31 16:42:13,325 INFO] Building shard 0.
300it [00:03, 96.67it/s]
[2018-10-31 16:42:16,449 INFO] * saving 0th train data shard to data/speech/demo.train.0.pt.

Traceback (most recent call last):
File "preprocess.py", line 239, in
main()
File "preprocess.py", line 229, in main
train_dataset_files = build_save_dataset('train', fields, opt)
File "preprocess.py", line 157, in build_save_dataset
opt)
File "preprocess.py", line 128, in build_save_in_shards_using_shards_size
torch.save(dataset, pt_file)
File "/home/ncj/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 209, in save
return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
File "/home/ncj/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 134, in _with_file_like
return body(f)
File "/home/ncj/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 209, in
return _with_file_like(f, "wb", lambda f: _save(obj, f, pickle_module, pickle_protocol))
File "/home/ncj/anaconda3/lib/python3.7/site-packages/torch/serialization.py", line 282, in _save
pickler.dump(obj)
File "/home/ncj/code/OpenNMT-py/onmt/inputters/dataset_base.py", line 39, in reduce_ex
return super(DatasetBase, self).reduce_ex()
TypeError: reduce_ex() takes exactly one argument (0 given)
This is a very urgent question.Can you explain it?please

T-Rebecca on 31 Oct 2018

Python 3.7 may cause the question. You can try to replace 3.7 with 3.6, the opennmt can work well.

zhangxinlu16 on 31 Oct 2018

👍1

I can confirm that Python 3.6 fixes this issue.

manuelciosici on 22 Nov 2018

👍1

It looks like it's as simple as passing proto to DataetBase.__reduce_ex__:

class DatasetBase(torchtext.data.Dataset):
    """
    A dataset basically supports iteration over all the examples
    it contains. We currently have 3 datasets inheriting this base
    for 3 types of corpus respectively: "text", "img", "audio".

    Internally it initializes an `torchtext.data.Dataset` object with
    the following attributes:

     `examples`: a sequence of `torchtext.data.Example` objects.
     `fields`: a dictionary associating str keys with `torchtext.data.Field`
        objects, and not necessarily having the same keys as the input fields.
    """

    def __getstate__(self):
        return self.__dict__

    def __setstate__(self, _d):
        self.__dict__.update(_d)

    def __reduce_ex__(self, proto):
        "This is a hack. Something is broken with torch pickle."
        return super(DatasetBase, self).__reduce_ex__(proto) # <---- this seems to fix it

    # ...

I'm not sure why moving back to Python 3.6 would change this. 🤔

erip on 27 Nov 2018

👍10

There is a PR doing this but it breaks tests on 2.7

vince62s on 28 Nov 2018

Just comment the __reduce_ex__ function and everything works.