Libertem: Allow specifying navigation and signal shape for HDF5 files

[uellue]

Since HDF5 is a widely used container format, LiberTEM should allow a maximum of flexibility to load and interpret the data.

In particular, it should be possible to "fold" the data in a HDF5 dataset into appropriate navigation and signal dimensions, since not all software saves data with the correct shape.

Particular example: I tried unsuccessfully to open a file that contains data with a flattened navigation dimension.

[sk1p]

Particular example: I tried unsuccessfully to open a file that contains data with a flattened navigation dimension.

Agreed, and that's a valid use-case! Thanks for the issue!

[AnandBaburajan]

I'm interested in working on this issue (Project 4: Allow reshaping datasets into a custom shape) as part of GSoC 2020.

[sk1p]

See also: https://github.com/LiberTEM/LiberTEM/issues/553

[uellue]

As a comment, this is also relevant for the EMPAD dataset. Right now, the scan_size parameter is overwritten by information from the XML file if an EMPAD dataset is opened from XML. Users might want to open data with their own scan_size, contrary to information in the XML file.

[AnandBaburajan]

For HDF5, I tried different methods including accessing indices and slices & h5py's read_direct. Reading with indices was too slow. Using slices and h5py, I was finished with nav_shape reshaping, sig_shape reshaping and -ve sync_offset (check this commit). But the solution doesn't look maintainable and sig_shape reshaping is slow. So I'll try memory mapping next.

Edit: the commit mentioned above was lost in a squash-and-rebase but I've the files locally to continue working on it

[AnandBaburajan]

See also: #984

[sk1p]

See also: #984

Indeed - let me know if you have questions about the changes in that PR - right now, it's still fresh in my mind :)

[AnandBaburajan]

Indeed - let me know if you have questions about the changes in that PR - right now, it's still fresh in my mind :)

@sk1p I see lots of interesting and important discussions there so I'll get back to you after trying my best to understand it. By the way, I'm happy to share with you and @uellue that I got a job offer at @frappe! :D They build and maintain a few open source products for small businesses. So I would like to thank you two again for the GSoC opportunity, helpful feedback and guidance! I hope and look forward to spending some time during weekends working on interesting projects at LiberTEM. I also started @RocketMeet recently (under dev with a team of 6) which is a meeting scheduling app and I was inspired to build one due to the lack of good open source meeting scheduling apps. So thanks to you for introducing me to Doodle which gave me the idea! :D Looking forward to meet you two one day.

[sk1p]

By the way, I'm happy to share with you and @uellue that I got a job offer at @frappe! :D They build and maintain a few open source products for small businesses. So I would like to thank you two again for the GSoC opportunity, helpful feedback and guidance! I hope and look forward to spending some time during weekends working on interesting projects at LiberTEM.

Congrats on your new position!

I also started @RocketMeet recently (under dev with a team of 6) which is a meeting scheduling app and I was inspired to build one due to the lack of good open source meeting scheduling apps. So thanks to you for introducing me to Doodle which gave me the idea! :D Looking forward to meet you two one day.

Nice! There is definitely high demand for this kind of application, especially in corona times.

[uellue]

@AnandBaburajan congratulations! This sounds exciting. Wishing you lots of success with these endeavours! 🚀