Datasets: MemoryError when loading German Wikipedia

[gregburman]

Hi, first off let me say thank you for all the awesome work you're doing at Hugging Face across all your projects (NLP, Transformers, Tokenizers) - they're all amazing contributions to us working with NLP models :)

I'm trying to download the German Wikipedia dataset as follows:

wiki = nlp.load_dataset("wikipedia", "20200501.de", split="train")

However, when I do so, I get the following error:

Downloading and preparing dataset wikipedia/20200501.de (download: Unknown size, generated: Unknown size, total: Unknown size) to /home/ubuntu/.cache/huggingface/datasets/wikipedia/20200501.de/1.0.0...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/ubuntu/anaconda3/envs/albert/lib/python3.7/site-packages/nlp/load.py", line 520, in load_dataset
    save_infos=save_infos,
  File "/home/ubuntu/anaconda3/envs/albert/lib/python3.7/site-packages/nlp/builder.py", line 433, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/ubuntu/anaconda3/envs/albert/lib/python3.7/site-packages/nlp/builder.py", line 824, in _download_and_prepare
    "\n\t`{}`".format(usage_example)
nlp.builder.MissingBeamOptions: Trying to generate a dataset using Apache Beam, yet no Beam Runner or PipelineOptions() has been provided in `load_dataset` or in the builder arguments. For big datasets it has to run on large-scale data processing tools like Dataflow, Spark, etc. More information about Apache Beam runners at https://beam.apache.org/documentation/runners/capability-matrix/
If you really want to run it locally because you feel like the Dataset is small enough, you can use the local beam runner called `DirectRunner` (you may run out of memory). 
Example of usage: 
    `load_dataset('wikipedia', '20200501.de', beam_runner='DirectRunner')`

So, following on from the example usage at the bottom, I tried specifying beam_runner='DirectRunner, however when I do this after about 20 min after the data has all downloaded, I get a MemoryError as warned.

This isn't an issue for the English or French Wikipedia datasets (I've tried both), as neither seem to require that beam_runner be specified. Can you please clarify why this is an issue for the German dataset?

My nlp version is 0.2.1.

Thank you!

[lhoestq]

Hi !

As you noticed, "big" datasets like Wikipedia require apache beam to be processed.
However users usually don't have an apache beam runtime available (spark, dataflow, etc.) so our goal for this library is to also make available processed versions of these datasets, so that users can just download and use them right away.

This is the case for english and french wikipedia right now: we've processed them ourselves and now they are available from our google storage. However we've not processed the german one (yet).

[gregburman]

Hi @lhoestq

Thank you for your quick reply. I thought this might be the case, that the processing was done for some languages and not for others. Is there any set timeline for when other languages (German, Italian) will be processed?

Given enough memory, is it possible to process the data ourselves by specifying the beam_runner?

[lhoestq]

Adding them is definitely in our short term objectives. I'll be working on this early next week :)

Although if you have an apache beam runtime feel free to specify the beam runner. You can find more info here on how to make it work on Dataflow but you can adapt it for Spark or any other beam runtime (by changing the runner).

However if you don't have a beam runtime and even if you have enough memory, I discourage you to use the DirectRunner on the german or italian wikipedia. According to Apache Beam documentation it was made for testing purposes and therefore it is memory-inefficient.

[lhoestq]

German is [almost] done @gregburman

[lhoestq]

I added the German and the Italian Wikipedia to our google cloud storage:
First update the nlp package to 0.3.0:

pip install nlp --upgrade

and then

from nlp import load_dataset
wiki_de = load_dataset("wikipedia", "20200501.de")
wiki_it = load_dataset("wikipedia", "20200501.it")

The datasets are downloaded and directly ready to use (no processing).

[gregburman]

Hi @lhoestq

Wow, thanks so much, that's really incredible! I was considering looking at creating my own Beam Dataset, as per the doc you linked, but instead opted to process the data myself using wikiextractor. However, now that this is available, I'll definitely switch across and use it.

Thanks so much for the incredible work, this really helps out our team considerably!

Have a great (and well-deserved ;) weekend ahead!

P.S. I'm not sure if I should close the issue here - if so I'm happy to do so.

[lhoestq]

Thanks for your message, glad I could help :)
Closing this one.