Libertem: K2IS: read tags from gtg file

[sk1p]

See #63 for related K2 reader discussion. Mostly interested for scanning parameters.

Maybe wait for hyperspy IO split and use their DM reader?

[sk1p]

Using the hyperspy reader would be nice, and is quite easily implemented:

            from hyperspy.io_plugins.digital_micrograph import DigitalMicrographReader
            with open(_get_gtg_path(path), "rb") as f:
                reader = DigitalMicrographReader(f)
                reader.parse_file()
            size = reader.tags_dict['SI Dimensions']
            params['scan_size'] = (size['Size Y'], size['Size X'])

But, as hyperspy is GPL, and we want to keep our libertem.io package MIT-licensed, we currently can't do this.

[sk1p]

There is another DM reader in python-mrcz, which is BSD licensed. I may have a go at integrating that one. May need some convincing to read tag-only .gtg files, but let's see...

[ercius]

I have two versions of a DM reader. One in ncempy (
https://github.com/ercius/openNCEM/blob/master/ncempy/io/dm.py) and another
that I wrote in pure python for ImageJ (unreleased). I'd be happy to share
or help integrate either of those.

On Tue, Dec 4, 2018, 8:04 AM Alexander Clausen [e-mail]
wrote:

There is another DM reader in python-mrcz
https://github.com/em-MRCZ/python-mrcz/blob/master/mrcz/ioDM.py, which
is BSD licensed. I may have a go at integrating that one. May need some
convincing to read tag-only .gtg files, but let's see...

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/LiberTEM/LiberTEM/issues/109#issuecomment-444154389,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AO7olEwylPu8vEOgBxbuvMa3TBfC2pnWks5u1p0kgaJpZM4Wa7Hj
.

[sk1p]

@ercius Thanks! I'll have a look at the ncempy implementation tomorrow. What is the difference between that and the one implemented for ImageJ?

[ercius]

@sk1p I actually erred. I wrote a plugin for EMI/SER files for ImageJ/Fiji (not DM3/DM4). Sorry for the pre-coffee confusion.

Ive used the DM class in ncempy/io/dm.py to read GTG files before. They are just a set of DM tags. You can install ncempy via pip and then:

import ncempy.io as nio
aa = nio.dm.fileDM('filename.gtg') #open the file and parse the header
print(aa.allTags)

It is set up to skip the binary data and only parse the tags. Then you can load the data (if it exists) later using aa.getDataset() or aa.getSlice(). GTG files are just tags though.

Also, this will read 4D DM4 files properly in case that would be useful for integrating DM files into LiberTEM.

[sk1p]

I had a look, and it works really well! The only problem: it is GPLv3 licensed. We want to keep our io package under a permissive license, MIT in this case (actually dual-licensed MIT+GPLv3). The reason behind this is, we want to avoid the situation where people have to re-write I/O code just for licensing reasons, and there is an effort underway to create a common I/O library for electron microscopy (see https://github.com/hyperspy/hyperspy/issues/1978). To really share a common I/O layer between many projects, the license needs to be as permissive as possible.

Right now, there are at least three Python DM reader implementations that I'm aware of:

• hyperspy (GPLv3)
• ncempy (GPLv3)
• python-mrcz (BSD3)

Both the hyperspy and the ncempy readers work well with .gtg files, python-mrcz sadly doesn't out of the box (it only reads ImageList tags from the root tag list, cc @robbmcleod).

Would you be willing, and able, to re-license the I/O part of ncempy to a permissive license like MIT or BSD? We would need the permission of all authors of those parts, so that would be you, @gonzalorodrigo and maybe @fniekiel ?

[ercius]

Sorry for the late reply. Im open to changing the license to make it more
permissive. Ill check with the other authors to make sure that is ok.

On Wed, Dec 5, 2018 at 5:06 AM Alexander Clausen [e-mail]
wrote:

I had a look, and it works really well! The only problem: it is GPLv3
licensed. We want to keep our io package under a permissive license, MIT
in this case (actually dual-licensed MIT+GPLv3). The reason behind this is,
we want to avoid the situation where people have to re-write I/O code just
for licensing reasons, and there is an effort underway to create a common
I/O library for electron microscopy (see hyperspy/hyperspy#1978
https://github.com/hyperspy/hyperspy/issues/1978). To really share a
common I/O layer between many projects, the license needs to be as
permissive as possible.

Right now, there are at least three Python DM reader implementations that
I'm aware of:

• hyperspy (GPLv3)
• ncempy (GPLv3)
• python-mrcz (BSD3)

Both the hyperspy and the ncempy readers work well with .gtg files,
python-mrcz sadly doesn't out of the box (it only reads ImageList tags
from the root tag list, cc @robbmcleod https://github.com/robbmcleod).

Would you be willing, and able, to re-license the I/O part of ncempy to a
permissive license like MIT or BSD? We would need the permission of all
authors of those parts, so that would be you, @gonzalorodrigo
https://github.com/gonzalorodrigo and maybe @fniekiel
https://github.com/fniekiel ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/LiberTEM/LiberTEM/issues/109#issuecomment-444478376,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AO7olJkMBEI241fW4bYJKNwlrbS2DCRZks5u18TPgaJpZM4Wa7Hj
.

[gonzalorodrigo]

(sorry, I thought that I had answered already).

For me it is also fine. Actually, as UC regents directive, any lab related open-source work is always prefered to be under BSD (modified).

[fniekiel]

For me it is ok and I would like to leave the decision to Peter. I only remember that we moved from LGPL to GPL because I needed to use PyQt. But the io part should be pretty free from any dependencies.

So should we release ncempy.io as a separate package under a different license? Or should Peter just contribute his work to you project (which then leaves two copies of the code at two different place impeding maintainability)?
Personally, seeing so many different projects I would prefer the TEM community to join forces and unite all repositories at some day. I mean, it would still be possible to have a wild variety of forks where everyone does what he wants.

[uellue]

@fniekiel full agreement regarding joining forces! We are working with Hyperspy to have a common IO library for Hyperspy and LiberTEM: https://github.com/hyperspy/hyperspy/issues/1978

Thank you for considering releasing it under a permissive license! 😃 Probably @sk1p is the best person to discuss how exactly that should be structured.

[ercius]

Excellent. I read through hyperspy/hypersp#1978 and that seems very interesting. Ill try to get involved with that discussion. Its been a long running problem in the EM community to make IO simple and useful for novice to expert users. Working at a user facility makes that extremely apparent. File reading is our users' biggest issue! That is why Im trying to support ncempy.io and will happily make it as open as possible.

@sk1p What are the next steps to making our IO library permissive? Are you still interested in our code if you are going to work with hyperspy on IO? Should we try to take the best of all the projects' IO libraries and make one repo? Let me know what you think.

[sk1p]

Excellent. I read through hyperspy/hypersp#1978 and that seems very interesting. Ill try to get involved with that discussion. Its been a long running problem in the EM community to make IO simple and useful for novice to expert users. Working at a user facility makes that extremely apparent. File reading is our users' biggest issue! That is why Im trying to support ncempy.io and will happily make it as open as possible.

:+1:

@sk1p What are the next steps to making our IO library permissive? Are you still interested in our code if you are going to work with hyperspy on IO? Should we try to take the best of all the projects' IO libraries and make one repo? Let me know what you think.

In the long run, integrating the EM I/O libraries into one definitely makes sense. But I think it will take some time, especially since they are also planning to re-license from GPL to a more permissive license, as far as I understood. So I'm definitely still interested in ncempy.io!

I think there are (at least) two possibilities for re-licensing:

1) Release the io part as a separate package.
2) Just license the io part by putting a LICENSE file into the directory.

Option 1) is "clean" but means some overhead, as you have to manage a separate package, releases, CI, ..., and option 2) can seem a bit messy, as it may not be 100% clear for users what parts are licensed in what way (I just added a note about that to our README to clarify it a bit).
In LiberTEM the main part is also GPL, with the io part dual-licensed. We chose 2) and have a second LICENSE file in the subdirectory: https://github.com/LiberTEM/LiberTEM/blob/master/src/libertem/io/LICENSE. This works fine, as long as your io code doesn't depend on any other modules.

[ercius]

I just updated ncempy to version 1.4.0. This includes a new licensing of the io portion as MIT. I just copied how libtertem implemented it. Please let me if there is anything you need in ncempy to work better with libertem. Im very interested in this project and want to support it any way I can.