H5py: .flat iterator

[ax3l]

Something I would really love to use is a .flat member function on a h5.Dataset that returns a numpy.flatiter object for multi-dimensional data sets.

One application I have is (and someone might now an alternative solution to it): I read a very large nd-array that can not fit in RAM at once. I now want to iterate it 1D chunk-wise (ideally in a local manner; the order of access does not matter but if it is deterministic I could parallelize the processing.) Copy and .flatten() is due to local memory constrains not possible.

(P.S.: nditer is a generalized version of np.flatiter)

[aparamon]

For h5py, I believe it makes most sense to iterate by chunks, not single elements, due to efficiency reasons. While efficient read-only per-element iteration is feasible to achieve, write-mode (op_flags having readwrite or writeonly) should definitely operate on chunks.

Somewhat related: https://github.com/h5py/h5py/issues/975

[takluyver]

Datasets now have an iter_chunks() method which can facilitate efficient access to the underlying data where it's in chunked storage. This should be more efficient in general than treating the data as a 1D array, because the flat interpretation might interleave different chunks.

There's still room to improve for accessing datasets with contiguous storage (not in chunks) and virtual datasets (might be chunked, but we can't easily get the chunk shape, and it could even vary). E.g. we could allow the caller to pass in a chunk shape for iter_chunks(), or maybe even try to guess one.

I'm going to remove the 'pull request welcome' tag on this, because I think it wrongly suggests that there's an idea ready to implement. Ideas, discussion and experimentation are more welcome at this stage.

[ax3l]

Thank you for the update, I think this sounds great.

Your proposal to provide a fallback or hint chunk to contiguous storage sounds great to me as a future update, I am ok to close this issue and document this separately if you prefer that.