Datasets: [Question] Create Apache Arrow dataset from raw text file

[mrm8488]

Hi guys, I have gathered and preprocessed about 2GB of COVID papers from CORD dataset @ Kggle. I have seen you have a text dataset as "Crime and punishment" in Apache arrow format. Do you have any script to do it from a raw txt file (preprocessed as for BERT like) or any guide?
Is the worth of send it to you and add it to the NLP library?
Thanks, Manu

[lhoestq]

We store every dataset in the Arrow format. This is convenient as it supports nested types and memory mapping. If you are curious feel free to check the pyarrow documentation

You can use this library to load your covid papers by creating a dataset script. You can find inspiration from the ones we've already written in /datasets. Here is a link to the steps to add a dataset

[IbtihalFerwana]

Hello @mrm8488 and @lhoestq

Is there a way to convert a dataset to Apache arrow format (locally/personal use) & use it before sending it to hugging face?

Thanks :)

[lhoestq]

Is there a way to convert a dataset to Apache arrow format (locally/personal use) & use it before sending it to hugging face?

Sure, to get a dataset in arrow format you can either:

• load from local files (txt, json, csv)
• OR load from python data (dict, pandas)
• OR create your own dataset script