Libertem: Architecture for high throughput

[uellue]

In the appended file I've written down some ideas on getting throughput way past the GB/s region that @sk1p and I had discussed. What do you think?
Ideas for scalable architecture.docx

[uellue]

@magnunor, in relation to your comments to issue #2, @sk1p and I were thinking if one could use some optimized algorithms in a C/C++ + GPU or even FPGA solution for the acquisition nodes. Here, speed and soft real time response are critical to get the fastest scan possible and avoid dropping frames.

The offline processing nodes could be based on hyperspy. We'd work to integrate the optimized algorithms into hyperspy, but we are not limited to those, but can get creative. Here, people can "play" with the friendly and extensible programming interface of hyperspy and still get a VERY decent performance. If one or the other operation takes a bit longer, that is not critical.

For the control system we can use pretty much whatever we like since that one has to deal with WAY less data.

What do you think?

[tjof2]

This looks really interesting!

There are some creative applications or ideas out there - the online robust PCA (OR-PCA) and NMF algorithms implemented by myself and @to266 in HyperSpy (e.g. https://github.com/hyperspy/hyperspy/blob/RELEASE_next_minor/hyperspy/learn/rpca.py) should be easy enough to adapt.

Back when I was writing my thesis, my long-term idea was to use ORPCA for initial denoising and segmentation (foreground/background separation), assuming the data meets the low-rank requirement (which time-based EM acquisition does) all while sat at the microscope. If you wanted to e.g. quickly denoise the data to check you're looking in the right area of the specimen (for more in-depth analysis later on) then ORPCA is perfect. Plus it handles sparse corruptions and outliers (it can actually inpaint pixels up to 25% corruption) all in real-time and with only a few parameters.

@woozey can chip in here too perhaps. I optimized it a bit further in Python compared to what's currently in HyperSpy (or in the Matlab code @woozey has), and I got it to the point of processing a 1024x1024 frame in about 100us with numpy. Shift that to C++ and I reckon you can go a fair bit quicker.

The other benefit is that for low-rank data you can just store the L and R matrices, rather than the full data, and reconstruct later doing L.R^T. You could get your 12 GB/s down to 200 MB/s for a rank-4 or rank-5 matrix with that approach I think.

I also implemented a stochastic gradient descent method that nicely handles sample drift among other things - there's some good images and discussion in my thesis. O-NMF would be a bit slower I think. Happy to discuss further - I'll post my thesis chapter here when home. Caveat applying it from "2D+time" videos to further/additional dimensions might be difficult and mean other approaches work better.

[uellue]

I really like these ideas. Perhaps some of the operations can even be applied to the L and R matrices instead of the full one?

[uellue]

Idea: Control interface is web application?

[uellue]

Idea: Processing nodes available as docker image for easy rollout? Control system as well if it is web-based?

[sk1p]

@uellue on Docker: I like it! Are servies like Docker/kubernetes/openshift/... commonly available on HPC clusters?

[uellue]

@sk1p If not, we will MAKE them available. EUDAT, move processing to the data and such... ;-)

[magnunor]

One idea could be to use a widely used messaging library, this would allow any program to "hook into" the data stream. This could be ZeroMQ (http://zeromq.org/), which runs on pretty much anything, and has bindings for a large variety of programming languages: http://zeromq.org/bindings:_start

It also works over TCP, so it would be easy to implement a distributed live processing system.

(Thanks to @gwpaterson for this idea!).

[magnunor]

Another idea: instead of implementing the GPU code directly, use some kind of abstraction layer. For example https://arrayfire.com/ . This could greatly reduce the developer time.

I think @matkraj is familiar with this, so he might have some input on this.

[sk1p]

I really like ZeroMQ as a communication layer - that's what I actually had as possible IPC method in my initial presentation for my interview! But maybe there is a more highlevel distributed computing framework that we can use?

[sk1p]

Another GPU abstraction layer is numba which does some JIT magic to run Python(ish?)-code on the GPU. Does anyone have experience with that?

[magnunor]

I'm a little bit hesitant with using solutions which only work with CUDA, since that necessitates a Nvidia GPU. I think this is especially relevant with AMD releasing pretty competitive GPUs recently, where they're open sourcing more and more of their driver stack, potentially making it much more future proof.

[matkraj]

As Magnus mentioned I am using arrayfire for most of heavy data processing. This library allows you to write parallel GPU code with a very little involvement. It supports a unified compiler backend - the program itself will chose the most efficient backend available (CUDA, openCL, intelMP/boost/fftw).

To give an example, the following python code shows number of calls needed to find a centroid in 4D dataset (32-bit 256x256x256x256):

[uellue]

Numba seems to have experimental support for HSA devices, i.e. AMD: http://numba.pydata.org/numba-doc/0.36.1/hsa/overview.html About numba I like that it seems to integrate much better with numpy, while arrayfire uses its own data type. That means it should be easier with numba than with arrayfire to accelerate existing scipy/numpy code. I'm wondering if there are differences in performance, though?

[uellue]

Another point to consider is that "our" nodes at JURECA have _four_ GPUs and 24 cores (48 threads afaik). http://www.fz-juelich.de/ias/jsc/EN/Expertise/Supercomputers/JURECA/Configuration/Configuration_node.html

I'm wondering if/how abstraction layers like arrayfire or numba distribute the work over all available resources?

[sk1p]

From a quick Google: numba / arrayfire-python

(sorry for closing, that should be a reminder for me to not comment on github using my mobile :stuck_out_tongue:)

[matkraj]

@uellue I have not tried any multi GPU processing. However it seems that the last version of ArrayFire had some extensive work done in thread safety https://github.com/arrayfire/arrayfire/pull/1706.
Arrayfire and Numpy can by used fairly simply:
arrayAF = af.np_to_af_array(arrayNP) and versa
arrayNP = np.array(arrayAF)
I think the difference is that arrayAF will only be stored on a device which performs the computation.
I like one additional feature of this library - it comes with openGL visualisation http://arrayfire.org/docs/forge_visualization.htm

[magnunor]

dask is another parallel computing library: https://dask.pydata.org/en/latest/

I haven't used it directly myself, but HyperSpy uses it for lazy processing.

[uellue]

The XFEL uses a very similar architecture: https://in.xfel.eu/readthedocs/docs/data-analysis-user-documentation/en/latest/overview.html#data-flow-cal

[sk1p]

For more details of the Karabo framework of XFEL see also: DATA ANALYSIS SUPPORT IN KARABO AT EUROPEAN XFEL - some points:

• Separation into scenarios: Rapid feedback, online analysis, offline analysis (This roughly corresponds to our: live / interactive / offline processing, I think)
• Pipeline processing: create arbitrary data stream processing networks
• Extension via ZeroMQ sockets - either using pub/sub (drops 'frames' if the processing does not keep up with the data rate, used in the case of rapid feedback) or request/response (for offline processing - the processing dictates the speed)
• They report latencies of 1.5 to 2.5 seconds for their pipeline
• Offline processing is either done using data streams (ZeroMQ req/resp) or by reading from HDF5 files. The former can be used to run processing using the same interface as live processing, so you can test your methods using data from HDF5 files.
  
  
  
  • Remote data analysis using jupyter notebooks
  
  
  • Flexible analysis by running arbitrary software via containers
  
  
  • GUI for live feedback while running experiments
  
  

[sk1p]

A new draft of the architecture is now available in the repository. It is currently focused on data processing, though we haven't forgotten about acquisition and live processing. Feedback welcome!

[uellue]

I'm closing this now since we are discussing architecture for live acquisition and embedding in #123.