Datasets: NotADirectoryError while loading the CNN/Dailymail dataset

[arc-bu]

Downloading and preparing dataset cnn_dailymail/3.0.0 (download: 558.32 MiB, generated: 1.28 GiB, post-processed: Unknown size, total: 1.82 GiB) to /root/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0/0128610a44e10f25b4af6689441c72af86205282d26399642f7db38fa7535602...

---

NotADirectoryError Traceback (most recent call last)

in ()
22
23
---> 24 train = load_dataset('cnn_dailymail', '3.0.0', split='train')
25 validation = load_dataset('cnn_dailymail', '3.0.0', split='validation')
26 test = load_dataset('cnn_dailymail', '3.0.0', split='test')

5 frames

/root/.cache/huggingface/modules/datasets_modules/datasets/cnn_dailymail/0128610a44e10f25b4af6689441c72af86205282d26399642f7db38fa7535602/cnn_dailymail.py in _find_files(dl_paths, publisher, url_dict)
132 else:
133 logging.fatal("Unsupported publisher: %s", publisher)
--> 134 files = sorted(os.listdir(top_dir))
135
136 ret_files = []

NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'

[lhoestq]

Looks like the google drive download failed.
I'm getting a Google Drive - Quota exceeded error while looking at the downloaded file.

We should consider finding a better host than google drive for this dataset imo
related : #873 #864

[arc-bu]

It is working now, thank you.

Should I leave this issue open to address the Quota-exceeded error?

[lhoestq]

Yes please. It's been happening several times, we definitely need to address it

[gchhablani]

Any updates on this one? I'm facing a similar issue trying to add CelebA.

[lhoestq]

I've looked into it and couldn't find a solution. This looks like a Google Drive limitation..
Please try to use other hosts when possible

[gchhablani]

The original links are google drive links. Would it be feasible for HF to maintain their own servers for this? Also, I think the same issue must also exist with TFDS.

[lhoestq]

It's possible to host data on our side but we should ask the authors. TFDS has the same issue and doesn't have a solution either afaik.
Otherwise you can use the google drive link, but it it's not that convenient because of this quota issue.

[gchhablani]

Okay. I imagine asking every author who shares their dataset on Google Drive will also be cumbersome.