Cupy: Implementing `tofile` and `fromfile` to use GPUDirect

[jakirkham]

Recently GPUDirect Storage added support for reading and writing directly using the GPU. It would be useful if tofile and fromfile were implemented to use this functionality (possibly falling back to their current behavior).

[leofang]

@jakirkham @awthomp What's the hardware and software requirement to use this feature? Is there a public API or documentation for it?

Related: Is it possible to use GPUDirect Storage to load hdf5 data? It's much more relevant for our need (and thus easier to justify if I spend time on it 😁).

[kdatta]

API information here: https://docs.nvidia.com/gpudirect-storage/design-guide/index.html

[cnewburn]

And the Design Guide Kushal pointed to describes system support and what GDS is validated on.
A third party at NERSC has been experimenting with HDF5 here: https://github.com/hpc-io/hdf5/tree/cu_dev.
Please let us know what other readers you have interest in.

[leofang]

Thanks @kdatta @cnewburn for quick reply and pointer! For our purposes I think a hdf5 reader is a good enough starter!

Looks like it's unlikely to test this on our system...But I happen to have access to NERSC. Do you know if this can already run on the Cori GPU nodes there? I was in the Cori GPU User Call earlier and asked about GPUDirect Storage, and they thought it's possible but not 100% sure. If you know a point of contact who we can talk to, that'd be great! 🙂