Libertem: File Format

[sk1p]

This is the overview/summary for file format issues. If you create a new issue regarding the file format, please comment below!

Projects / Formats:

• EMD: #3
• Hyperspy HSpy format: #2
• The file format of pycroscopy: #1
• CERN ROOT file format: #4
• MSA / MAS / AMAS HyperDimensional Data File: #6
• NeXus #12

Currently a HDF5-based solution is the strongest contenter (the first three formats are all HDF5-based); if you know of any other relevant projects/file formats, please create an issue and comment below!

[magnunor]

There is also the MSA / MAS / AMAS HyperDimensional Data File (HMSA): http://www.csiro.au/luminescence/HMSA/index.html

Discussion in the HyperSpy project about adding support for this: https://github.com/hyperspy/hyperspy/issues/1251

[magnunor]

One thing to keep in mind with the HDF5 formats is that they are not really different "file formats", as for example DM3 and TIFF would be. The underlying data structure for HDF5 is (usually) exactly the same, what differs between them is the internal "folder" and metadata structure.

This means that it was a lot easier to add an EMD-reader to HyperSpy, while getting support for Bruker’s bcf file format was a real pain.

Also, there already exist a HDF5-reader/writer for Digital Micrograph, which would make it easy to import these types of files into DM: https://github.com/niermann/gms_plugin_hdf5

[jan-car]

In the hyperspy gitter, people mentioned katai-struct as a nice way to define parsers for binary formats. I think this might be worth looking into.

[uellue]

@jan-car That works on a very low level, right? I.e. the HDF5 people could write their specs with that, and it would generate code that you could use to read HDF5? Or does it work on a higher abstraction level as well, i.e. describe how things should be arranged in a HDF5 file, "HDF5 Schema" so-to-speak?

[jan-car]

As far as I can see at first glance, it is byte-by-byte, so no tree-structure like in HFD5. Maybe it's still valuable for import/export of other binary formats into the one we develop.

[uellue]

@jan-car Makes sense! :-)

[magnunor]

One thing which should be included in whatever is chosen, is easy access to navigation images. For example, when having a 4 dimensional pixelated STEM dataset there should ideally also be a 2 dimensional virtual BF/ADF, and a 2 dimensional sum of the diffraction patterns.

Having this makes it so much easier to explore these datasets, since one does not have no calculate the navigation images when trying to visualize the data. For our internal pixelated STEM datasets, these navigation images are generated when converting the data from the medipix binary format to HDF5s.

[uellue]

@magnunor 👍 Excellent point, thx!

[magnunor]

Example of one of our EMD-style HDF5 files, which we've converted from the Merlin software binary file (Medipix3). The data itself is just zeros.

example_file.zip

It can be loaded in HyperSpy using:

import hyperspy.api as hs
s = hs.load("example_file.emd")

This will return a list of signals, which includes the pixelated STEM dataset itself, navigation images for both the image and diffraction dimensions, and various metadata as signals. The data itself is accessed in the fpd_data signal, which has to be picked out manually from the list. So this is currently not very streamlined in HyperSpy, but works much more smoothly in my fpd_data_processing library (which I might make publicly available during the weekend, if I find the time...).

To plot the data in HyperSpy:

s_fpd = s[5].transpose(signal_axes=(0, 1))
s_fpd.plot()

[magnunor]

If HDF5 is used, one very important aspect is the data chunking: https://support.hdfgroup.org/HDF5/doc/Advanced/Chunking/index.html . This has major effects on loading times for the HDF5-files, especially the large ones which we're routinely getting.

[uellue]

The question of chunking brought @sk1p and me on an interesting consideration. In general, we are limited by bus and memory transfers for the simple algorithms, not processing.

For each processing run of a data set with a given mask, this mask is a constant. The mask can be up to 32 MB large. That is much larger than typical CPU caches. If we process frame by frame, we'd have to stream both data and mask from main memory each time. I'm imagining all frames stacked on each other like sheets of paper. If we divide the mask into portions that fit into the L1 cache (typically 32 kB for data on modern CPUs) and then process the data set as "columns" of that size, we could keep the relevant portion of the mask in L1 cache and at least double our throughput. Our operation is associative for each frame and completely parallel, that means we can reorder it in that way.

The same applies, in principle, for GPUs, but with different sizes. We could even think about loading a new set of shaders with the relevant portion of the mask as constants for each column to process. Again, that saves us memory transfers.

In order to reap the full benefits of such an arrangement, it should result in linear access patterns both in memory and on disk. @sk1p will develop a small demo to test if this consideration is true and what arrangement results in the best performance.

[uellue]

I've opened issue #11 to separate questions of execution order and chunking in memory from the file format.

[uellue]

Unified format for x-ray, neutron and muon science: http://www.nexusformat.org/ #12

[uellue]

Mailing list message about file format. Summary: We recommend NeXus https://groups.google.com/forum/#!topic/libertem/-jY0ufrQNxg

[uellue]

With respect to issue #11 about execution order and chunking: A file format optimized for the fast processing order would store stacks of tiles, not whole frames. That is particularly important for large frames where data and mask don't fit into the L3 cache anymore. The hierarchy would be stacks->tiles->frames. This is much more efficient to process for our typical algorithm.

Ideally, the file format should hide this underlying complexity. A naive application can just load frames by index and not worry about caches and whatnot, as long as frames fit into memory. A smart application should have an API available that allows loading subsets that are good to process in one chunk. The file format should provide information on what subsets are native to the particular file and therefore efficient, but allow loading arbitrary subsets as well.

HDF5 seems to support this with chunking https://support.hdfgroup.org/HDF5/doc/Advanced/Chunking/index.html and hyperslabs https://support.hdfgroup.org/HDF5/Tutor/select.html

The task is to set the optimal chunking when creating a data set in terms of write speed during acquisition and performance of later offline processing, to get the particular chunking of a file, and to read the data in optimal hyperslabs that are both good to process and good to read with the given chunking from a particular file on a particular system. As far as I can tell, HDF5 provides the necessary API calls for this. Now our task is to make good use of that API!

[uellue]

We can discuss the details on how to map pixelated STEM data onto NeXus in issue #15.

[sk1p]

Interesting for later: Single Writer Multiple Readers mode/pattern for HDF5. This can be used to reliably process data while it is being written. Maybe useful for live processing?