Datasets: [Feature request] Add Google Natural Question dataset
Would be great to have https://github.com/google-research-datasets/natural-questions as an alternative to SQuAD.
elyase
All 7 comments
Indeed, I think this one is almost ready cc @lhoestq
thomwolf
on 17 May 2020
I'm doing the latest adjustments to make the processing of the dataset run on Dataflow
lhoestq
on 18 May 2020
Is there an update to this? It will be very beneficial for the QA community!
ronakice
on 22 May 2020
Still work in progress :)
The idea is to have the dataset already processed somewhere so that the user only have to download the processed files. I'm also doing it for wikipedia.
lhoestq
on 22 May 2020
Super appreciate your hard work !!
I'll cross my fingers and hope easily loadable wikipedia dataset will come soon.
richarddwang
on 23 May 2020
Quick update on NQ: due to some limitations I met using apache beam + parquet I was not able to use the dataset in a nested parquet structure in python to convert it to our Apache Arrow format yet.
However we had planned to change this conversion step anyways so we'll make just sure that it enables to process and convert the NQ dataset to arrow.
lhoestq
on 28 Jun 2020
NQ was added in #427 馃帀
lhoestq
on 23 Jul 2020
Related issues
astariul
路
3Comments
sshleifer
路
4Comments
NielsRogge
路
3Comments
gregburman
路
7Comments
jplu
路
6Comments
Most helpful comment
Still work in progress :)
The idea is to have the dataset already processed somewhere so that the user only have to download the processed files. I'm also doing it for wikipedia.