Vision: ImageNet Dataset 404 Not Found ARCHIVE_DICT URLs

[awwong1]

All three URLs listed in the ARCHIVE_DICT are returning 404 not found errors.

$ wget http://www.image-net.org/challenges/LSVRC/2012/nnoupb/ILSVRC2012_img_train.tar
--2019-10-12 16:38:39--  http://www.image-net.org/challenges/LSVRC/2012/nnoupb/ILSVRC2012_img_train.tar
Resolving www.image-net.org (www.image-net.org)... 171.64.68.16
Connecting to www.image-net.org (www.image-net.org)|171.64.68.16|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2019-10-12 16:38:39 ERROR 404: Not Found.

$ wget http://www.image-net.org/challenges/LSVRC/2012/nnoupb/ILSVRC2012_img_val.tar
--2019-10-12 16:38:51--  http://www.image-net.org/challenges/LSVRC/2012/nnoupb/ILSVRC2012_img_val.tar
Resolving www.image-net.org (www.image-net.org)... 171.64.68.16
Connecting to www.image-net.org (www.image-net.org)|171.64.68.16|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2019-10-12 16:38:51 ERROR 404: Not Found.

$ wget http://www.image-net.org/challenges/LSVRC/2012/nnoupb/ILSVRC2012_devkit_t12.tar.gz
--2019-10-12 16:39:01--  http://www.image-net.org/challenges/LSVRC/2012/nnoupb/ILSVRC2012_devkit_t12.tar.gz
Resolving www.image-net.org (www.image-net.org)... 171.64.68.16
Connecting to www.image-net.org (www.image-net.org)|171.64.68.16|:80... connected.
HTTP request sent, awaiting response... 404 Not Found
2019-10-12 16:39:03 ERROR 404: Not Found.

Current ImageNet dataset class is no longer functioning as expected.

[pmeier]

From the ImageNet blog:

While conducting our study, since January 2019 we have disabled downloads of the full ImageNet data, except for the small subset of 1,000 categories used in the ImageNet Challenge.

Thus, we should be able to download. Maybe they closed the links since they were accessible without their authorization. I'm not sure how to proceed here, @fmassa?

[fmassa]

Yes, they seem to have removed download access without filling the form http://image-net.org/download-images.

In this case, I think there is no better alternative than removing the download link and the download functionality, and requesting the user to download the images themselves.

cc @soumith for awareness

[pmeier]

I'll send a PR later.

[pmeier]

This is closed by #1457

[fmassa]

Thanks for fixing it @pmeier !

[jph00]

FYI the Imagenet competition dataset is also available thru Kaggle.

[fmassa]

@jph00 can we consider it as an official version of the dataset?

[jph00]

I don't know what your guidelines are for that @fmassa . It's officially shared with Kaggle by the Imagenet consortium AFAIK, and it certainly has the same contents (I've checked).

[RylanSchaeffer]

@pmeier , is there a way to download the 1000-class ImageNet subset through PyTorch?

@fmassa maybe you know?

[pmeier]

No, there isn't. As the disclaimer of torchvision states:

This is a utility library that downloads and prepares public datasets. We do not host or distribute these datasets, vouch for their quality or fairness, or claim that you have license to use the dataset. It is your responsibility to determine whether you have permission to use the dataset under the dataset's license.

IMO we were already in a grey area before, since we were hosting download links that should have not been publicly accessible. Since the authors decided to close these links, there is nothing we can do.