Flux.jl: Verify dataset downloads

[KristofferC]

Trying to run a model from the model-zoo that downloads MNIST I got:

julia> include("conv.jl")
[ Info: Downloading MNIST dataset
[ Info: Downloading MNIST dataset
[ Info: Downloading MNIST dataset
ERROR: LoadError: EOFError: read end of file
Stacktrace:
 [1] read at .\iobuffer.jl:175 [inlined]
 [2] imageheader(::Base.GenericIOBuffer{Array{UInt8,1}}) at C:\Users\Kristoffer\.julia\packages\Flux\U8AZD\src\data\mnist.jl:44
 [3] images(::Symbol) at C:\Users\Kristoffer\.julia\packages\Flux\U8AZD\src\data\mnist.jl:93
 [4] images() at C:\Users\Kristoffer\.julia\packages\Flux\U8AZD\src\data\mnist.jl:91
 [5] top-level scope at none:0
 [6] include at .\boot.jl:326 [inlined]
 [7] include_relative(::Module, ::String) at .\loading.jl:1038
 [8] include(::Module, ::String) at .\sysimg.jl:29
 [9] include(::String) at .\client.jl:403
 [10] top-level scope at none:0
in expression starting at C:\Users\Kristoffer\MachineLearning\model-zoo\vision\mnist\conv.jl:7

It seems the download was corrupted. Trying again, just gives

julia> include("conv.jl")
ERROR: LoadError: EOFError: read end of file

so the faulty download is cached.

Would be nice to verify that the correct file is downloaded so one doesn't end up in a broken state.

[oxinabox]

Doing things like data download and verification consistantly and nicely is what DataDeps.jl is for.
(Of course you can reimplement that, is you want, one off verification isn't rocket science.)

But furthermore, why not just use MLDatasets.jl,
which has these datasets already and already uses DataDeps.jl for hash-checking etc.

[MikeInnes]

I think we (a) wanted something in a slightly different format and (b) wanted the model zoo dependencies to just be Flux. But now we have manifests we're less worried about that, so maybe we could just switch over there and drop MNIST from flux itself.

[oxinabox]

Indeed now that we have sane package management,
I think it would be a good enhancement to the model zoo to actually depend on as much of the ecosystem as possible,
to demonstrate that flux does not stand alone, but can fully integrate with everything.
Unlike other frameworks where everything has to be included and made special just for that framework.

One might think datasets would be trivial, and ovious that you could use any.
but actually IIRC in python tensorflow and I think pytorch,
you have to use there inbuilt data loaders (because python is too slow),
and they thus have their own dataset repositories setup using them.

[MikeInnes]

Feel free to open an issue on the model zoo, and I can mark it as help needed.