Text: Can we save the processed fields or datasets?

[wlhgtc]

I’m currently using torchtext, but I found that creating Dataset object and calling Field’s build_vocab takes quite a long time, especially when the tokenizer is complicated. However, I failed saving them with pickle. Is there a way that we can save the processed dataset and fields, so that we can speed up data loading?

Thanks.

[taineleau-zz]

What's the error message?

[wlhgtc]

It doesn't mean an error. It‘s just a problem that it takes a long time to deal with data before training the model. I wonder can the Dataset or Field can be save just like tensor?

[dasguptar]

@wlhgtc pickle might not work, since some of these TorchText objects use lambdas.
You can either use dill instead of pickle, or replace your lambdas with functions.

[wlhgtc]

@dasguptar Thanks for your help, but it seems that I can't understand the conception of lambdas? Can you show me your demo for saving an object like Filed or Dataset ?
E.G : I have "train and test" dataset an "Text" field.

[dasguptar]

@wlhgtc
Here is an example of a lambda being used for tokenization using Spacy in TorchText

Here is an additional thread discussing the challenges with serializing fields

Hopefully this should give you a clearer idea regarding the challenges with saving processed fields.

[wlhgtc]

@dasguptar Thanks for your help!

[bxclib2]

Hi For me I think the solution could be:
Save the tokenized data first and use an extremely easy tokenizer in torch.text
Such as you could preprocess the data with spacy and save the sentence as tokenized one in the csv file.
Such as "I am the king" to "Iamtheking", and in the torchtext you just give a tokenizer by split(""), I think this will be way faster than use the spacy to tokenize it every time