Libertem: Support for Norpix Seq files

[uellue]

Required for Direct Electron cameras (Benjamin Bammes)

[sk1p]

See also:

• C impl for reading 8bit norpix seq files https://github.com/nclack/seq (weird license though)
• https://www.norpix.com/support/Norpix2MATLAB.m matlab implementation by norpix, nonfree license as far as I can see?

Using those as a reference, we should be able to implement a reader once we have sample data.

[sk1p]

See also: https://github.com/soft-matter/pims

[uellue]

It should support dark reference and gain maps, refs #154

[sk1p]

As a note: @jhgee mentioned that some data was taken without having proper synchronization, this means having frames before and/or after the actual acquisition in the file. So this is another use case where more complicated "re-shaping" of data is needed /cc @AnandBaburajan

On another note, the seq files don't contain any information about the scan, so there was a question if we could read this from a "sidecar" file, like we do with other formats with their proprietary implementations. One idea would be to have a simple file, maybe in .ini format, or .yml, which contains that information. This idea could even be extended to other formats that need additional information, like raw. Something like this:

my_dataset.yml:

format: SEQ
path: something.seq
scan_size: [256, 256]
dark_frame: dark.tif

my_other_dataset.yml:

format: RAW
path: other.bin
dtype: u2
scan_size: [256, 256]
detector_size: [128, 128]
dark_frame: dark.tif

These yml files could then provide one-click opening in LiberTEM, even if the underlying raw dataset doesn't provide enough information.

[sk1p]

The SEQ support will shortly be merged. Thanks to @jhgee for the sample data and pointers! For now, we don't support compression or "weird" bit depths (i.e. other than 8/16/32/64), but both should be possible to add. Let us know if this is interesting to you!

[jhgee]

Thanks a lot!

[uellue]

CC @bdalevin we now have native SEQ support in LiberTEM thanks to @sk1p and @jhgee!

[jhgee]

Thanks again for the implementation! We have already used it a lot and it makes data analysis much easier!
We talked about including dark and gain correction of the raw frames. Is that easy to implement, @sk1p ?

[sk1p]

This is on our agenda for 0.6 (#720, #154) - the correction facilities are already in place, only the interface is still missing. I may get some time this or next week to implement this, fingers crossed!