Text: [Discussion] Saving the field object

[PetrochukM]

Usage:
The field object is critical to checkpointing as it provides:

• tokenization
• padding
• numericalize

Having the ability to save the field object allows the user, given arbitrary text, to preprocess the text. The preprocessed text is then used with a checkpointed model. Then the output is predicted and interpreted without the output dictionary.

Problem:
torch.save is implemented with pickle. The field object accepts lambdas for tokenization, preprocessing and postprocessing; therefore, cannot be pickled.

Key Points:

• The vocab object needs to be pickled because the output of the model is uninterpretable without it.

Discussion:
What is the right abstraction here?
Should the vocab object be saved and the field object discarded?
Is it appropriate to have the field object and the vocab object closely bound?

[nelson-liu]

see #41 --- most of my problems were solved by passing dill to the pickle_module argument of torch.save, so I haven't really pushed on making it work with pickle further.

I just ended up saving the whole field, which is convenient for preprocessing test data for use in the data in a real system or something.

[jekbradbury]

In addition the vocab construction should be entirely replicable between runs if you provide the same dataset. Let us know if this isn't the case.

[SmartAI]

@nelson-liu then how to saving dataiters, since the save dataset can be used to many experiments without load raw data each time?

[nelson-liu]

I generally don't save the dataiters -- loading raw data isn't too big of an overhead for what I'm working with (and the train time >> load time, so it doesn't kill me to read data each time). It looks like it should be possible though, using #63.

[PetrochukM]

@nelson-liu @jekbradbury

Stumbled upon another use case for support pickle:
Torch.multiprocessing is a fork of multiprocessing. It runs on pickle exclusively I believe.

Do you think this could be solved easily by having get_state_dict like some objects in the PyTorch API support?

[jekbradbury]

One problem is that pickle itself doesn't actually use state_dict; that's an extension that PyTorch added rather than dealing with the fairly complicated __getstate__/__setstate__/etc. infrastructure that pickle uses for custom objects. So adding state_dict to Iterator seems to have solved the PyTorch serialization question, but doing the same thing for Field likely wouldn't solve the multiprocessing serialization issue.

[dfdx]

I tried passing pickle_module=dill, but it doesn't seem to work with PyTorch 0.4 and torchtext 0.3.0 (I haven't tried on previous versions though):

import dill
torch.save(my_field, "/path/to/file", pickle_module=dill)

gives:

...
~/miniconda3/envs/test/lib/python3.6/pickle.py in save(self, obj, save_persistent_id)
    494             reduce = getattr(obj, "__reduce_ex__", None)
    495             if reduce is not None:
--> 496                 rv = reduce(self.proto)
    497             else:
    498                 reduce = getattr(obj, "__reduce__", None)

TypeError: can't pickle torch.dtype objects

Am I doing something wrong or internals have changed since this discussion?

[mttk]

This error is something relatively new with regard to pickling torch.[dtype] attributes, such as torch.int64. #453 will enable fields to be saved by torch.save or pickle.dump

[leimao]

It is nearly 2020, so what is the official solution now? Thanks.

[PetrochukM]

This repository has stagnated since the main contributors @jekbradbury and @nelson-liu left the project. Due to the lack of contribution and stagnation in torchtext, the NLP open source community fragmented. There are now multiple NLP tools that are competing with each other:

• https://github.com/PetrochukM/PyTorch-NLP
• https://github.com/facebookresearch/pytext
• https://github.com/allenai/allennlp

With regard to me, I split off two years ago and started my own project: https://github.com/PetrochukM/PyTorch-NLP. I wanted to simplify the coupled torchtext objects into something more functional like PyTorch. This has greatly helped my personal productivity as a Deep Learning engineer at Apple, University of Washington, AI2, and WellSaid Labs.

Here is a comparison of the various tools: https://github.com/facebookresearch/pytext/issues/110

[mttk]

@leimao you can save fields by using torch.save since #453 was merged.

[leimao]

@leimao you can save fields by using torch.save since #453 was merged.

torch.save and torch.load seem to be working. Thank you.

[abhigenie92]

This doesn't work.

Train time:

vectors = vocab.FastText()
self.text.build_vocab(train_data, vectors=vectors, max_size=35000, unk_init=torch.Tensor.normal_)
torch.save(self.text, "vocab")

Test time:

self.text = torch.load( "vocab")

I get this error though while working arr = [[self.vocab.stoi[x] for x in ex] for ex in arr] AttributeError: 'Field' object has no attribute 'vocab'

This even though it has vocab, I am not able to figure why.

[zhangguanheng66]

@mttk Do you think those save and load func work for Vector?

[yipliu]

@leimao you can save fields by using torch.save since #453 was merged.

torch.save and torch.load seem to be working. Thank you.

Hi leimao,
Could you show an example to use torch.save for fields?

[zhangguanheng66]

Just to confirm. The transforms and components (e.g. Vocab, Vectors) in torchtext.experimental now support serialization/deserialization.