Text: Story: Serializing datasets

[srush]

Hi Torchtext,

It would be great to have a story for saving datasets. Things are currently not in a great place, and I would like to know where it might head.

1) Things are not serializable. In opennmt-py, we are hacking around this issue by serializing Dataset/Field objects. This doesn't really work out of the box because of the usage of defaultdict. However we can get around that issue by monkeypatching the __getstate__ of Vocab. Maybe this could be built in.

2) Datasets take a ton of memory. I like that datasets are so clean, but their internal storage is not cheap, storing the field names as strings along with all of the string data itself. It's cute that conversion/batching happens on the fly, but it might be nice to be able to turn that off, i.e. convert to tensors if you want.

3) It requires loading everything into memory. Dataset objects are currently monolithic. They assume that the universe stored directly in them. Ideally, datasets may require being stored on disc as shards. It would be great if the loading and usages of these shards could invisible to the user.

Thanks guys. As always great work. Cheers!
Sasha

[jekbradbury]

We'd absolutely appreciate a PR for 1 -- currently I've been serializing Datasets by storing their examples attribute and then rebuilding using the default Dataset constructor.

For 2, do you mean that you would want to be able to serialize several epochs' worth of prebuilt batches (i.e., the output of an Iterator)? Or a tensor version of a Dataset that hasn't been batched at all?

Also I don't know much about sharding or disk-based formats; our only existing support for datasets that can't fit in memory is that you can define a Dataset that is based around a Python generator and whose examples attribute is never actually instantiated as a list (the generator could do things like lazily load from disk but I'm not sure if anyone's actually done this).

[srush]

Hmm. My concern is that many NLP datasets are just too large to be kept in memory at all times. Particularly since python representations of dicts and string are large memory-wise. I don't feel like torchtext acknowledges this. We can hack our own thing, but I would prefer for the design of the library to reflect the following use case:

1) Stream data to construct fields.
2) Write data to disk in some compressed form.
3) Stream data out again to convert to batches.

[soumith]

This is something that needs to be fixed and Sasha's last comment is pretty good.

Since this is tagged as help wanted, I want to jump-in:

• @srush , do you have a API in mind, I can help implement something efficient
• if not, should i think about an API from the ground up to stream from disk / serialize efficiently?

[srush]

Haha nice. I haven't thought about this too much, but I guess, ideally the whole thing would be some sort of buffered streaming.

1) Preprocessing: Stream through the files (maybe with multprocessing), create fields and dictionaries, save to disk.
2) Train: At training time, shuffle file, stream through text data, and create batches on the fly.

The downside is (a) you might not be able to fully group mini-batches by length, (b) you might not know exactly how many batches are remaining. Maybe you can do some lookahead buffering so that you can find roughly length sentences.

[mttk]

@srush @soumith any progress on this? I can help out or start a basic implementation to give some push to the idea.

[srush]

I haven't had a chance to work on this. We did hack around it in OpenNMT-py, by implementing our own sharding.

[mttk]

Ok, I'll start working on this along the guidelines you suggested a couple replies ago. I've checked out the OpenNMT-py implementation as a pointer, thanks!

This seems to be a sizeable chunk of work -- so shouldn't expect anything too fast.

[mongoose54]

Not sure if this relevant but MxNet has a nice binarization method for large data: https://mxnet.incubator.apache.org/faq/recordio.html The problem is that it is C/C++ but it might be helpful to convert it into python (?)

[srush]

Hmm, I'm not sure we need to go that far, I'm okay with serializing python objects, but they just can't all be in memory at once.

However, it is true that python objects take a ton of disk space. Text data gets 10x larger.

[mttk]

Coming back to this with a general idea of a workflow and some issues:

Workflow:

1. First pass through dataset, preprocess, build vocabs for fields, store example lengths
2. Fix vocabs according to vocab_size, min_freq for each vocab, perhaps do some bucketing of examples according to their lengths
3. Second pass through dataset, numericalize (but don't pad), pickle dataset in blocks, save vocabs & fields
4. Train time, load blocks from serialized file, pad and convert into batches

Issues:

• Dataset blocks are going to have elements of uneven lengths (as is the standard case with text). Which is the most efficient way of storing lists of (lists? numpy arrays? tensors?)?
  
  
  
  • I'll try out each of these and see what works fastest and takes the least disk size
  
  
  • This can be somewhat alleviated by bucketing (1.), but it seems like overkill
  
  
• Should the preprocessed dataset be stored? If no, the preprocessing (essentially, splitting) is going to be done twice (in 1. and 3.). I see the argument for yes (faster) and no (more disk space used). This can be an optional argument to the implementation.
• 1 & 3 can be done in one pass without the nice frequency-sorted vocab indices. This could also be an optional argument.

I'll get an alpha PR running in about two weeks time, and I'd appreciate some feedback (esp. with regard to efficient block storage)

[srush]

Actually, I don't think you need to be this complicated. Block storage is hard, and doesn't buy us that much. The following would solve most of use cases and be more easily debugged:

1.2. exactly as you say.

1. Skip second pass entirely.

2. Train time. Load data by reading from text files. However, never construct all the examples in memory. Simply buffer enough examples from a file to construct reasonably sized batches. (This should be user controllable. Note this buffering line is bad: magic number 100):

https://github.com/pytorch/text/blob/master/torchtext/data/iterator.py#L271

The main contribution is that datasets should be inherently lazy. It should be easy to make one, but it shouldn't default to loading everything in. When you call build_vocab it should go line by line. When you make an iterator it should automatically buffer enough examples for tight batching. Currently both these operations take up a ton of memory.

[mttk]

Ok, that will speed things up.

I plan to simply add StreamingDataset / StreamingField classes which would essentially build everything required online. Thanks!

[srush]

Any progress here?

[mttk]

Yeah, although I got sidetracked a bit since Repl4NLP got extended. Will get something up soon.

[abhinavkashyap92]

Is someone actively working on this still??

[mttk]

@abhinavkashyap92 yep, you can check out the current status here: #282