Libertem: hyperspy

[sk1p]

http://hyperspy.org

HyperSpy is an open source Python library which provides tools to facilitate the interactive data analysis of multi-dimensional datasets that can be described as multi-dimensional arrays of a given signal (e.g. a 2D array of spectra a.k.a spectrum image).

HyperSpy aims at making it easy and natural to apply analytical procedures that operate on an individual signal to multi-dimensional arrays, as well as providing easy access to analytical tools that exploit the multi-dimensionality of the dataset.

I/O, file format(s)

[woozey]

Here are the file format specs:
http://hyperspy.org/hyperspy-doc/current/user_guide/io.html#hspy-hyperspy-s-hdf5-specification

Here is the metadata structure:
http://hyperspy.org/hyperspy-doc/current/user_guide/metadata_structure.html

[woozey]

From my experience with HS file format:

• I don't know much about performance, obviously as any HDF5 it can be read in memory in parts which is crucial for large datasets such as 4D-STEM.
• I like the idea of having very defined metadata since it facilitates easy access to the parameters that are stored inside
• I generally like the metadata structure of HS, though it is certainly incomplete.

[magnunor]

There is also the HyperSpyUI: http://hyperspy.org/hyperspyUI/ , which is a GUI interface for working with HyperSpy, and other python libraries.

[woozey]

Regarding experience with Hyperspy as a package for data analyses, the way I see it:
PROS:

• Very good platform which is suitable for data with arbitrary dimensionality and arbitrary source. It separates dimensions in two categories: signal dimension and navigation dimensions. This helps to discriminate over which axes operation should be performed and which axes are iterable.
• Python-based - easy scripting, cross-platform, access to large number of science related libraries and user community, Jupyter notebooks to keep track on analyses and reporting.
• Low entry barrier for developers: one of major programming languages, modular structure, good documentation, permissive administration.
• Well maintained code and high standard for code quality. (At least it is high comparing to typical mess created by scientists :)) )
• Great tutorials which show real data analyses, which are done in Jupyter notebooks and ready to use with user's data.
• Good advertising strategy which involves tutorials at significant microscopy conferences.
• GPL license.
• Relatively large developer and user community.

CONS:

• Python-based - compromised performance, I guess...
• In my opinion it has high entry barrier for users. First of all, due to it scripting nature. Second, installation is involved primarily since it needs specific versions of some libraries resulting in dependency conflicts which should be sorted out (pyqt=4 -> qt=4 -> py<=3.5). Here I have to admit I have never checked the UI project, which probably main function is to overcome all limitations mentioned above and drastically reduce the barrier.
• Lack of resources: it looks like have grown in something bigger HS needs a full time developer for the maintenance of the project.

[uellue]

I absolutely love the Hyperspy project. For me "best practice". The file format definition is incomplete, but generally looks sane to me and works, i.e. has a working implementation and people who use it! :-) Definitely a good project to look into.

I agree that we will likely run into performance bottlenecks if we only use Hyperspy, or at least complicate things quite a bit if we want to make it work for us. The main task in LiberTEM will be to shovel LOADS of data into GPUs to shrink it down into useful information. Our bottleneck is memory and bus transfer, and I/O when reading from files. Sounds like a C/C++ job to me.

Perhaps the other direction is more promising: make any high-throughput processing backends that we might adopt or implement available in Hyperspy as a convenient working environment?

Generally, I think that Hyperspy is perfect as an experimental/prototyping environment for processing algorithms. It would be very nice to make things written for Hyperspy easily available in LiberTEM and vice-versa.

[woozey]

Perhaps the other direction is more promising: make any high-throughput processing backends that we might adopt or implement available in Hyperspy as a convenient working environment?

For me this sound as one of the best ways to go. I don't know how implementable it is and which drawbacks it will introduce...

Generally, I think that Hyperspy is perfect as an experimental/prototyping environment for processing algorithms. It would be very nice to make things written for Hyperspy easily available in LiberTEM and vice-versa.

Totally agreed here!!!

[uellue]

Actually, the metadata definition seems to have grown since the last time I looked at this a few months ago! 👍 IMO the best data format in terms of being practical, handling metadata and being well-defined.

A few points to work on to make it usable for other projects:

• Move from "whatever Hyperspy does" to a more formal definition how data and metadata are mapped onto HDF5
• Spin off the development process for the format from Hyperspy development -- wider scope
• Release libraries and toolchain under a permissive license -- MIT and such. :-)
• C/C++ libraries for projects that don't use Python.

[magnunor]

One way to offset the sometimes "slowness" of Python is either using optimized libraries like numpy, or implementing things using cython, which is used in HyperSpy for loading BCF-files: https://github.com/hyperspy/hyperspy/blob/RELEASE_next_minor/hyperspy/io_plugins/unbcf_fast.pyx

Another important functionality in HyperSpy is the lazy loading support using dask, which I use extensively to process pixelated STEM datasets: http://hyperspy.org/hyperspy-doc/current/user_guide/big_data.html

I've been implementing a library for analysing pixelated STEM data, which is essentially an extension of HyperSpy. Which includes things like radial integration, virtual detectors, centre of mass, and other utility functions. It is currently not publicly available, but will be soonish.

[jan-car]

Sounds great, in my experience the integration of faster code (e.g. C/C++) is not that difficult in Python.

[uellue]

@magnunor I'm looking forward to your release of your software! Is it using GPUs? What kind of throughput are you getting? That would be quite an interesting shootout in comparison with other implementations!

Personally, I've tried to write an optimized Python/numpy implementation of the mandelbrot benchmark https://benchmarksgame.alioth.debian.org/u64q/mandelbrot.html (soon disqualified for using numpy and doing things in a weird way). I got it in the region of being perhaps 10x slower than an optimized code in a compiled language, which I was quite proud of! But that solution became quite messy because it needs to manipulate bits and bytes for the final output as binary format and can't efficiently be described as a simple element-wise operation or scipy library call because each cell has a different time to terminate the iterations. That's where my Python-related skepticism is coming from.

[magnunor]

@uellue, my fpd_data_processing library just builds upon HyperSpy. For example, I made PixelatedSTEM class which inherits the Signal2D class, and then adds stuff like radial_integration, virtual bright field, virtual annular dark field, center of mass, .... The processing itself is using the map function in HyperSpy, which automatically "multiprocesses" the calculations. Its purpose isn't really to do things very quickly, but being able to do advanced processing of 50+ GB datasets on a computer with limited.

With regards to Numpy performance, I don't think Mandelbrot is a good measure, since the most common type of processing the pixelated STEM datasets require is simple array operations. And there Numpy is highly optimized.

An a broader point, the reason I'm a bit skeptical about the C/C++ route is that:

• It usually takes a lot more development time to make equivilant functionality, especially for people not familiar with C/C++.
• It is very easy to get bogged down in implementing highly optimized C/C++ code, while the developer time could be better spent elsewhere.
• The majority of people doing open source software development in the TEM community (seems) to be Python-people, and using C/C++ will drastically increase the barriers of entry, making the possible developer base smaller.

Note, this does not extend to implementing smaller, computationally expensive functions in for example Cython, where there obviously can be pretty big gains. For example currently the _parse_detector_data function in fpd_live_imaging is the main limitation for running 2000+ fps, so in that case it could be useful to make a highly optimized function.

[uellue]

@magnunor Sounds great! One way or the other, I'd be happy if this work can be combined. Maybe implementing such optimized functions could be a goal for LiberTEM? Even if some parts of LiberTEM would in the end be pure C/C++ without any Python, the code could be structured in such a way that the low-level functions can easily be made available within Hyperspy. At the same time, your implementation could be a nice prototype for an optimized version!

[sk1p]

@magnunor I agree that Python will enable more people to contribute, and that it generally leads to higher programmer productivity. So there should at least be a Python interface to functions we implement in a lower level language. How would you judge the barrier of entry for contributing to Cython code?

I could also imagine keeping pure-Python code as a reference implementation, and judging the correctness of optimized versions using the pure-Python implementations.

For CUDA later it would be possible to use the numba jit, if the result is fast enough.

Regarding numpy/Cython/... performance, maybe this is a more realistic benchmark: https://people.duke.edu/~ccc14/sta-663/Optimization_Bakeoff.html

[magnunor]

Those things sound good! Especially first making a pure python implementation, which makes it easier to see exactly which processing steps are the most computationally heavy.

Making any high-throughput library importable as a library would make it very easy to integrate it into other projects. This is basically how HyperSpy works: large number of utility and convenience functions, which relies on a variety of different libraries to do the heavy processing.

I made the pixStem library publicly available during the weekend:

• http://pixstem.org/
• https://pypi.python.org/pypi/pixstem

Easiest way to install: pip3 install pixstem or pip install pixstem.

Edit: the name recently got changed from fpd_data_processing to pixStem, so this has been updated here.

[uellue]

@magnunor Awesome! :-) 👍

[dnjohnstone]

You may also like to note this: https://github.com/pyxem/pyxem which also builds on hyperspy for scan and record a diffraction pattern at every probe position type experiments.