Libertem: Electron Microscopy Dataset (EMD)

[sk1p]

https://emdatasets.com/

EMD stands for “Electron Microscopy Dataset.” It is a subset of the open source HDF5 wrapper format. N-dimensional data arrays of any standard type can be stored in an HDF5 file, as well as tags and other metadata. EMD files typically have a “.emd” extension, though they may also have “.h5″ or other extensions. All generic HDF5 file viewers can open EMD files; in addition we provide a viewer for EMD files that allows n-dimensional data to be easily viewed along any combination of axes, as well as integrating along any number of axes. Our viewer can open and convert standard proprietary formats such as Gatan’s digital micrograph “.dm3” datasets and FEI’s STEM “.ser” datasets.

[uellue]

From what it looks to me, this is a bit similar to the #2 Hyperspy format, but more generic. Hyperspy supports this as an input format. From what I see in the format definition https://emdatasets.com/format/ it is probably too generic and therefore difficult to use for data exchange in an automated workflow. A valid EMD file can become quite complex and would need human help to figure out what is what when using the data.

Probably it would benefit from clearly defined, strict schemas for different types of experiments. We could provide that for pixelated STEM. It would be very cool if any LiberTEM file is a valid OpenEMD and/or a valid Hyperspy file!

Unless there are technical reasons not to do that, I'd look if it is possible to go this way.

[woozey]

@uellue Fair statement. Though my understanding is that there is a conceptual difference between the two formats:

HS file is a container of one multidimensional signal (including metadata), whereas EMD can contain several experiments and even simulations which are equivalent to the signals. For example having EDS, pixelated-STEM and secondary electron STEM map acquired in one go, HS would use 3 files for storing each signal separately; EMD can store all in one file.

Please correct me if I'm wrong.

[jan-car]

I think there were discussions about having several datasets/signals in one HyperSpy file, but currently, there can be only one per file (as far as I now).

[uellue]

@woozey I think you are right about the difference!

I'd strongly prefer to have one dataset / one signal in one file. First, this scales better because parallel writes are not coordinated by HFS5, but by the file system which I'd expect to be MUCH better at that.

Second, it makes the file format easier to support when using it in different software packages. Less complexity in the data means less complexity in the application supporting it.

[jan-car]

A whole while back, I created overviews of the HDF5 file structures of EMD and HyperSpy, which I'll attach here, maybe they are helpful.

If we go with the NeXus format (is this fixed by now?), we can not have a file format which satisfies all three (NeXus, HyperSpy, EMD) structure requirements, right?

[uellue]

@jan-car That depends a bit on how restrictive these structure requirements are. NeXus is a bit of a hybrid in that sense. In general, it seems to be quite flexible and can accommodate a lot of different types of data. However, if you tie it to a NeXus Application Definition, it is by design very restrictive in what data goes where and which fields are required.

I'd say that we can definitely define a NeXus Application Definition to include all the relevant information that both an EMD and a Hyperspy file usually contain. Furthermore, one can add additional "rogue" data that is not part of the Application Definition.

However, this NeXus file will of course not work out of the box as an EMD or Hyperspy file and vice-versa, because they follow different naming and structure conventions.

Another point is that EMD, if I understand correctly, allows arbitrarily nested "directory" structures, while NeXus is in tendency flat. In my understanding, one NeXus file is supposed to contain data from one measurement run, i.e. one frame, scan or time series where one or several parameters are varied in a specific fashion. That means such a nested, complex EMD file with several measurements in one file would have to be split into several NeXus files. That should be desirable because it reduces complexity in any software that reads the file, because you just specify file paths and not on top of that paths within the HDF5 file to identify your data.

TL;DR: It should be possible to map EMD, Hyperspy and NeXus onto each other with a simple conversion. However, opening the same file as NeXus, EMD and Hyperspy file will not work so well because the same information is saved under different names in different places.

[jan-car]

I agree, even though conversion should be easily possible, one unified format is out of scope and also not necessary in my opinion.

EMD being able to hold multiple data sets complicates things, but I also think that this might not the most desirable way to go, as it creates unnecessary complexity, as you said. One data set per file seems good to me :-)!