Libertem: Handle live acquisition and offline processing as a back-end for other software

[uellue]

For many existing TEM software packages it would be attractive to handle live acquisition and processing of pixelated STEM data without having to implement all the cluster, high-performance and cloud computing details that LiberTEM has and will continue to expand.

For that purpose, we should work on a setup where LiberTEM is deployed to handle the data streams and number crunching, while being controlled by an external software and feeding reduced data to that software. In this issue we can collect the requirements and data flows for such a setup.

Possible candidates:

• Hyperspy
• Swift
• Digital Micrograph
• Velox
• ...?

[woozey]

This sounds like a brilliant idea! I've been recently experimenting with LiberTEM installation on an acquisition machine on our Holo microscope.

For instance for STEM-diffraction experiments with electromagnetic field mapping the data flow could be:
DM acquisition->LiberTEM ADF and COM calculation -> Hyperspy ADF and COM alignment and subtraction -> Hyperspy or DM visualisation

[uellue]

Here at IMC I've had a lot of encouraging discussions about workflows like that. This should probably go to high priority for the future.

[uellue]

In #135 I've started assembling workflows and functionality. Feedback and improvement are highly appreciated!

[uellue]

With respect to the recent success in automating the microscope with Tango: Perhaps Tango can be used to control the LiberTEM acquisition server? I.e. tell it what kind of data to receive from where?

Would Tango be suitable to submit processing jobs and receive data, both for online and offline processing? I.e. sth. like the Web API, just through Tango? That could be a convenient solution when designing Tango-based control solutions since it would present a unified view on the entire system through one API.

This could be a nice plus for integrating LiberTEM in synchrotron and neutron beamline systems!

Addendum: As far as I remember, the "synchrotron guys" were facing limitations of Tango when handling data from very fast detectors since Tango is not too efficient in transmitting large volumes of data quickly. Perhaps delegating the low-level data handling to LiberTEM could be an elegant solution? The detector, LiberTEM cluster and storage handle the full rate data stream and processing among each other, and provide control interfaces and monitoring signals through Tango?

[sk1p]

With respect to the recent success in automating the microscope with Tango: Perhaps Tango can be used to control the LiberTEM acquisition server? I.e. tell it what kind of data to receive from where?

Good idea! That should work well, actually.

Would Tango be suitable to submit processing jobs and receive data, both for online and offline processing?

Addendum: As far as I remember, the "synchrotron guys" were facing limitations of Tango when handling data from very fast detectors since Tango is not too efficient in transmitting large volumes of data quickly. Perhaps delegating the low-level data handling to LiberTEM could be an elegant solution? The detector, LiberTEM cluster and storage handle the full rate data stream and processing among each other, and provide control interfaces and monitoring signals through Tango?

I think we have to try - especially for monitoring high data rate live streams, we would have to benchmark the system (we can fake this by streaming from a K2 file, for example). Also, I don't know if this kind of 'stream' data type exists in Tango, that's something to find out. If this doesn't work, one could use a zeromq socket for this, with connection control happening via tango.

That could be a convenient solution when designing Tango-based control solutions since it would present a unified view on the entire system through one API.

Agreed.

[sk1p]

See also: https://software.pan-data.eu/software/65/nexdatas

They have some old links to google code wiki and repo, project currently seems to live here: https://github.com/nexdatas/ and is still active

[sk1p]

Probably most interesting:

• https://nexdatas.github.io/project_goals.html
• https://nexdatas.github.io/design.html#sequence-diagrams