Libertem: Implement big-endian raw import

[unsimulatable]

Another field to select between little- and big-endian raws in the open data menu would be helpful.

[uellue]

@unsimulatable which camera or file format for pixelated STEM uses big endian raw data? Implementing this option makes sense if there's a specific use case.

[unsimulatable]

The case is a more or less homemade electron camera based on a Lumenera Lu170 chip (https://www.lumenera.com/lu170.html), thus a quite special case. I can understand if it is deemed not useful enough to implement this change.

[sk1p]

Implementation should be more or less straight-forward, reading big-endian files will have some performance impact though. Scheduling this for the 0.2 milestone, as we are very close to releasing 0.1.

Let me know if you'd like to have a try at implementing it yourself!

[uellue]

@unsimulatable @sk1p Sounds good! :+1:

[uellue]

By the way, fast endian conversion could be a nice first issue to try some performance engineering. The operation is simple enough and should be very, very fast if it is implemented right. The approach would be to test if built-in methods are fast enough, and if not build a faster one with numba.

[sk1p]

Hi @unsimulatable - we just realized that this is already possible given the current raw reader! You can use the standard numpy way to denote endianess, for example for big-endian 16bit unsigned ints, use >u16. See the numpy documentation for details on dtype specification. Note that there was a bug in an unrelated code path that prevented endian-annotated dtypes from working, which I just fixed in 1645904231f36671d47fc15d95ff1582996695dd, so in practice this only works with the latest development release.

So, the point now would be to add this to our documentation so that it doesn't get lost.

The performance overhead should not be so bad, but I didn't benchmark it in any comprehensive way.

Please, let me know if this works for you or if you need any more assistance!

[unsimulatable]

Hi @sk1p, I tried it with the latest release version (the one I obtained by updating via pip) and using the correct expression for big-endian unsigned 16bit (which actually seems to be >u2) and I got an error (the message read just '>u2'). In comparison, if I just type something that is not a data type into the data type field, like xyz, the error message reads 'data type "xyz" not understood'.

[sk1p]

Yes, >u2 is correct, sorry. Installing via pip (pip install libertem) gives you the latest stable release, 0.1, which does not yet have this bug fixed. You need the current development version for that to work, which means either installing via git (as described in our documentation), or using the continuously built AppImage (Linux only). If you'd like to keep up-to-date with current developments, keeping a local git clone is preferrable. Let me know if this works for you!

[unsimulatable]

Yes, it works using the git clone. Thanks, problem solved!