Datasets: Huggingface NLP, Uploading custom dataset

[Nouman97]

Hello,

Does anyone know how we can call our custom dataset using the nlp.load command? Let's say that I have a dataset based on the same format as that of squad-v1.1, how am I supposed to load it using huggingface nlp.

Thank you!

[richarddwang]

What do you mean 'custom' ? You may want to elaborate on it when ask a question.

Anyway, there are two things you may interested
nlp.Dataset.from_file and load_dataset(..., cache_dir=)

[lhoestq]

To load a dataset you need to have a script that defines the format of the examples, the splits and the way to generate examples. As your dataset has the same format of squad, you can just copy the squad script (see the datasets forlder) and just replace the url to load the data to your local or remote path.

Then what you can do is load_dataset(<path/to/your/script>)

[lhoestq]

Also if you want to upload your script, you should be able to use the nlp-cli.

Unfortunately the upload feature was not shipped in the latest version 0.2.0. so right now you can either clone the repo to use it or wait for the next release. We will add some docs to explain how to upload datasets.

[lhoestq]

Since the latest release 0.2.1 you can use

nlp-cli upload_dataset <path/to/dataset>

where <path/to/dataset> is a path to a folder containing your script (ex: squad.py).
This will upload the script under your namespace on our S3.

Optionally the folder can also contain dataset_infos.json generated using

nlp-cli test <path/to/dataset> --all_configs --save_infos

Then you should be able to do

nlp.load_dataset("my_namespace/dataset_name")