Cellprofiler: Explore using ImageJ's REST server for updated ImageJ integration

[ctrueden]

We have now completed an initial working prototype of the RESTful image server we discussed back in May.

https://github.com/imagej/imagej-server

Please check it out and let us know what you think. We are greatly looking forward to pursuing this further so that CP and ImageJ continue to integrate well together.

[AnneCarpenter]

So excited to see this, thanks for following through, Curtis.

See #1994 for background info and context.

[0x00b1]

@ctrueden 😮

I just spent some time with this. I’m super impressed!

How can we help? 😎 Are you looking for feedback?

Would it make sense to start writing a Python package to wrap the API?

[ctrueden]

@0x00b1 Great! I'd like to invite @lnyng, the primary developer of this project, to chime in here. I have currently tasked him with the following goals:

1. A working demo of the ImageJ server in action from Python—e.g., from the REPL. Spin up an ImageJ, do some work, spin it down when finished.
   
   
   
   • In order to achieve this, we need some convenience code to wrap things up on the Python side. We probably want to avoid some autogenerated API monster across all available ImageJ modules, in favor of a lean API that lets you query and invoke the modules in a general way. Then e.g. CP can simply do that and expose them all through some UI.
   
   
   • As part of said convenience code, I asked Leon to research the existing best practices regarding cross-language RESTful services. Surely there are some nice tools that have already been created to expose REST endpoints in a nice way for various languages? Do you have any suggestions there? I'd hate for us to reinvent this presumably very common wheel.
   
   
2. A solid test framework so we can ensure the endpoints remain functional across future changes of the infrastructure. This will probably include both unit and integration testing, but especially the latter for key common modules, as well as upload/download of common data structures.

As for how you can help: we are hoping you can start trying to integrate this on the CellProfiler side, as your time allows, and come back with feedback regarding any pain points we can address. I understand that it will be very helpful to have the aforementioned client-side Python layer. If this is something you know how to create easily, then help from you in prototyping that would be most welcome, but we are also OK doing it ourselves as needed.

[lnyng]

I am currently working on integrating it into imagej, so we can spin up the server by just calling imagej --server. Once it starts to work, I will work on creating a Python client package prototype, so that you can play around with it. Hope that it won't take too long! Any feedback will be awesome. 😄

[ctrueden]

@0x00b1 @AnneCarpenter @eliceiri
Status update: @lnyng has pushed some minimal-but-working Python client code to the repository. He is currently working on making the documentation sexier, but really everything is working well from the Python REPL already. Please give it a try (just be aware that that linked README describes a custom REPL which @lnyng developed, which we are phasing out in favor of calling the API from the built-in Python REPL). It might be enough for a reasonable CP/ImageJ2 integration now, without further effort on the Python side. But it's really up to you: if you feel a fancier Python client would make your life a lot easier, it could be created. Please let us know—@lnyng is with us at LOCI for ~10 hours a week through the end of this semester, and that's it. So the project will slow down a lot after that. @lnyng will post another update here when the docs have been updated.

[lnyng]

@0x00b1 @AnneCarpenter @eliceiri @ctrueden
The documentations for the project is updated and ready to go. Please check it out when you have time. Any feedback would be awesome. Thanks!

[ctrueden]

One thing that is still missing is the ability to easily spin up an ImageJ server from Python. For now, for development & testing, you can spin it up manually beforehand. You can simply run mvn -Pexec from the toplevel of a clone of imagej/imagej-server. Later, we can add a Server update site to ImageJ's list of update sites, which ships imagej-server along with dependencies, and adds support for a --server CLI flag that spins up ImageJ in server mode.

@0x00b1 We eagerly await your feedback. :grin:

[eliceiri]

thanks Curtis and Leon. Anne and I met about this briefly and is also quite excited about this plan. We both would love to get Allen's feedback as soon as possible as it would help drive the direction. I've cced both of them.

best

kevin

Kevin W Eliceiri, PhD

PI and Director

Laboratory for Optical and Computational Instrumentation

University of Wisconsin at Madison

Investigator, Morgridge Institute for Research

---

From: Curtis Rueden [e-mail]
Sent: Wednesday, February 8, 2017 1:53:31 PM
To: CellProfiler/CellProfiler
Cc: KEVIN W ELICEIRI; Mention
Subject: Re: [CellProfiler/CellProfiler] Explore using ImageJ's REST server for updated ImageJ integration (#2394)

One thing that is still missing is the ability to easily spin up an ImageJ server from Python. For now, for development & testing, you can spin it up manually beforehand. You can simply run mvn -Pexec from the toplevel of a clone of imagej/imagej-server. Later, we can add a Server update site to ImageJ's list of update sites, which ships imagej-server along with dependencies, and adds support for a --server CLI flag that spins up ImageJ in server mode.

@0x00b1https://github.com/0x00b1 We eagerly await your feedback. 😁

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/CellProfiler/CellProfiler/issues/2394#issuecomment-278442274, or mute the threadhttps://github.com/notifications/unsubscribe-auth/APPjumr1aENwxHMfySi7up4ZyTC2hnNYks5rah07gaJpZM4KZRFr.

[mcquin]

Hi @ctrueden @lnyng @eliceiri,

The API/Python wrapper look great! I was able to spin up an ImageJ server and interact with it using the Python wrapper. It was very easy and worked right out of the box, which I always appreciate. ;)

A few remarks:

1. The server host is hardcoded. I am curious about the behavior when we can't connect to the host or we point to something goofy like "https://www.google.com". It doesn't look like any validation is done in the imagej.IJ constructor. Maybe a ping/pong call and response could validate that we are able to communicate to an ImageJ server.

1. Is there a way to retrieve the inputs for a module given its ID?

2. Have you, or are you planning to publish the Python API wrapper to PyPI? We highly recommend Python wheels for Linux/OS X and Windows.

3. Which versions of Python are supported?

4. How will the ImageJ server be packaged? We likely won't include it in the CellProfiler build itself. We can add documentation to help users install the server.

Very cool!!!!

[ctrueden]

The server host is hardcoded.

Haha, good catch. @lnyng went to all the trouble of allowing customization of the host... except you still can't. 😁

Is there a way to retrieve the inputs for a module given its ID?

That is what IJ.detail(id) is for, no? This method corresponds to the server's GET /modules/{id} endpoint.

Have you, or are you planning to publish the Python API wrapper to PyPI?

Good idea.

Which versions of Python are supported?

It looks like the dependencies (requests and Pillow) both support 2.6, 2.7 and 3.x. @lnyng which version(s) of Python were you testing with?

How will the ImageJ server be packaged?

As an ImageJ update site, probably called Server. So installation will be:

1. Download ImageJ (or Fiji).
2. Enable the Server update site.
3. Point CellProfiler at the ImageJ installation somehow.
4. Profit.

It would be nice if CellProfiler could bundle the ImageJ Python client, since it is very small with few dependencies. Does that make sense?

[mcquin]

That is what IJ.detail(id) is for, no? This method corresponds to the server's GET /modules/{id} endpoint.

Ah, I misunderstood the documentation. I see that it works. :)

As an ImageJ update site, probably called Server. So installation will be: [...] 3. Point CellProfiler at the ImageJ installation somehow.

At its most basic, I think it will be sufficient to require users to start the ImageJ server before running a CellProfiler pipeline which requires ImageJ. I don't think that CellProfiler should be responsible for discovering ImageJ Server, nor starting and stopping an ImageJ server.

We will have the ability to specify the server host within CellProfiler.

It would be nice if CellProfiler could bundle the ImageJ Python client, since it is very small with few dependencies. Does that make sense?

Once a package is available on PyPI we can add it as a project dependency which will be bundled with CellProfiler distributions. (We're mega stoked about the few dependencies!!)

[ctrueden]

I think it will be sufficient to require users to start the ImageJ server before running a CellProfiler pipeline which requires ImageJ. I don't think that CellProfiler should be responsible for discovering ImageJ Server, nor starting and stopping an ImageJ server.

That is fair!

However, it would still be nice if you could configure the URL of the server from within CellProfiler, right? I mean, defaulting to http://localhost:8080 is great, but some users might want to change it, or run one remotely?

[mcquin]

Yep. I realized that was what you were asking and added a last minute edit. My bad! We'll definitely support a configurable host. :)

[lnyng]

@mcquin @ctrueden
Sorry for the delay! Since your feedback, I made a couple fixes and improvements to the repo.

1. The host is designed to be configurable, but there was a bug, which is fixed now. You can actually change the host after the client is created, and it only succeeds if the client can get a list of modules from the host.
2. Yes, but unfortunately this functionality is not implemented in a very user friendly way. You call detail(<module_id>) to see the details of a modules, which has its inputs/outputs. But this only shows the names of those inputs/outputs. No type information exists for now.
3. Checkout imagej on pypi. This is completely experimental. Should work the same as running the one you clone in imagej-server repo.
4. There was some issues with Python3, but now they are also fixed. The client was developed under Python 2.7, and I also ran it with Python 3.4 and it works. Ohter Python versions should work as well as the client is very simple now.
5. I guess Curtis has already answer this.

[ctrueden]

We're mega stoked about the few dependencies!!

@lnyng Could you briefly comment on those dependencies? Pillow is only needed for display, so it is not even really a dependency here for CellProfiler's use case, right? So the only real dependency is requests, no? Is there a way to declare Pillow as "optional" somehow? What is the Python best practice here?

Checkout imagej on pypi. This is completely experimental.

:+1: :+1: :+1: @lnyng I am creating a PyPI account now; once I do, I hope there is a way to add me as a co-maintainer of this project. Edit: my PyPI account name is ctrueden. Let me know how it goes.

[lnyng]

For the dependency, we can use the extras_require to make pillow optional. I am working on it.

[ctrueden]

@mcquin Any news?

FYI, @lnyng is finishing up his work on imagej-server this week; any final requests?

Also, I want to draw your attention to another exciting Java+Python project being developed in the ImageJ/SciJava community: imglyb. It lets you wrap numpy arrays as ImgLib2 data structures on the Java side. It lets you visualize numerical data from Python via BigDataViewer without copying chunks of memory. See also imagey, which embeds a CPython REPL inside ImageJ. All of this is accomplished thanks to pyjnius. Maybe all of this is somehow interesting or useful on the CellProfiler side.

[eliceiri]

Thanks Curtis. Be curious what the Cell Profiler thinks

best

k

Kevin W Eliceiri, PhD

PI and Director

Laboratory for Optical and Computational Instrumentation

University of Wisconsin at Madison

Investigator, Morgridge Institute for Research

---

From: Curtis Rueden [e-mail]
Sent: Thursday, May 11, 2017 1:50:16 PM
To: CellProfiler/CellProfiler
Cc: KEVIN W ELICEIRI; Mention
Subject: Re: [CellProfiler/CellProfiler] Explore using ImageJ's REST server for updated ImageJ integration (#2394)

@mcquinhttps://github.com/mcquin Any news?

Also, I want to draw your attention to another exciting Java+Python project being developed in the ImageJ/SciJava community: imglybhttps://github.com/hanslovsky/imglib2-imglyb. It lets you wrap numpy arrays as ImgLib2 data structures on the Java side. It lets you visualize numerical data from Python via BigDataViewer without copying chunks of memory. See also imageyhttps://github.com/hanslovsky/imagey, which embeds a CPython REPL inside ImageJ. All of this is accomplished thanks to pyjniushttps://github.com/kivy/pyjnius. Maybe all of this is somehow interesting or useful on the CellProfiler side.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/CellProfiler/CellProfiler/issues/2394#issuecomment-300882910, or mute the threadhttps://github.com/notifications/unsubscribe-auth/APPjuvI3hN6QZow-TTEKdbP261H3ZIgtks5r41hogaJpZM4KZRFr.

[mcquin]

Hi! We've had some time to review the latest work. Thanks for making the host configurable and publishing the code to PyPi. That's a huge help! We've come up with a few further suggestions, which we've filed as issues on the project:

• Python 2 and 3 interoperability
• Module details should return type information

Returning type information is necessary for creating the ImageJ CellProfiler module.

[ctrueden]

Thanks @mcquin. I commented on the issues!

[ctrueden]

Just a heads up that the ImageJ Forum currently has some discussion going on about ways of combining Python + ImageJ. There are several different ways now. See this thread.

[eliceiri]

thanks

k

Kevin W Eliceiri, PhD

PI and Director

Laboratory for Optical and Computational Instrumentation

University of Wisconsin at Madison

Investigator, Morgridge Institute for Research

---

From: Curtis Rueden [e-mail]
Sent: Monday, June 5, 2017 9:38:16 AM
To: CellProfiler/CellProfiler
Cc: KEVIN W ELICEIRI; Mention
Subject: Re: [CellProfiler/CellProfiler] Explore using ImageJ's REST server for updated ImageJ integration (#2394)

Just a heads up that the ImageJ Forum currently has some discussion going on about ways of combining Python and Java. There are several different ways now. See this threadhttp://forum.imagej.net/t/jupyter-notebook-for-imagej/5421/7.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/CellProfiler/CellProfiler/issues/2394#issuecomment-306204307, or mute the threadhttps://github.com/notifications/unsubscribe-auth/APPjur9hPMgBd-2IxjdAS8MzJS_nR3Hgks5sBBLYgaJpZM4KZRFr.

[ctrueden]

I was discussing this project with @eliceiri yesterday—and Kevin also with @AnneCarpenter recently—and we were wondering how things are going? Any roadblocks, questions, concerns?

[burgerga]

So... no chance import imagej.imageprocessor as ijiproc is working anytime soon I guess (I have some old plugins that use this)?
Am I correct in assuming imagej was removed completely from CP3?

[mcquin]

Am I correct in assuming imagej was removed completely from CP3?

Yes. The RunImageJ module isn't supported in CellProfiler 3.0. As you can see from this discussion, we're investigating new ways for CellProfiler to interact with ImageJ.

[ctrueden]

@burgerga Would you be able/willing to share some of these plugins? And/or describe your use cases for combining CellProfiler and ImageJ? We'd love to have a better understanding of the community's needs along these lines, so that we can pursue the most effective avenues of integration!

[burgerga]

@ctrueden It is a collection of segmentation algorithms developed by @kuandannyan that we use in our department. IIRC they are imagej plugins with an add-on cellprofiler plugin to set the parameters in the cellprofiler gui, you can find an example here: https://github.com/kuandannyan/cp_pipeline/blob/master/hysteresis_segment.py

Our current workaround is to do our segmentation in whatever program seems suitable (eg Ilastik) and then import the binary images or probability maps in cellprofiler, but this results in a lot of extra steps that are hard to do for some less tech-savvy colleagues

[kuandannyan]

@ctrueden @burgerga
Thanks for mentioning my works. I am also curious about CellProfiler 3.0 and its ImageJ server solution with RESTful service.

By any chance, can the CP ImageJ RESTful service provides a method which allows end user to dynamically loading a jar-file containing customized ImageJ plugins into the current instance behind service (or create a new instance of ImageJ). I assume behind the RESTful service, there is either a singleton wrapping of ImageJ 1.X or a port-based access to ImageJ 2.X's new instance interface. So basically, the work around will be allowing reflection creation of customized plugins with the uploaded jar file.

But I guess dynamic class loader means potential injection hacking as well.

Kuan

[ctrueden]

can the CP ImageJ RESTful service provides a method which allows end user to dynamically loading a jar-file containing customized ImageJ plugins into the current instance behind service

@kuandannyan It would be doable to include a module on the ImageJ server side capable of executing arbitrary script code fed as a string. As you point out, this would be a big security risk, but also very convenient, and probably OK if the server is running over localhost for the local user only. This might be something like the following script, placed into the ImageJ server installation at scripts/Plugins/Scripting/Run_Script.groovy:

#@ ScriptService scriptService
#@ String language
#@ String script
#@ Map inputs
#@output Map outputs
lang = scriptService.getLanguageByName(name)
ext = lang.getExtensions()[0]
outputs = scriptService.run("script." + ext, script, true, inputs).get().getOutputs()

Untested though, so there are probably wrinkles.

Dynamic classpath modification is also possible—see e.g. the scijava-grab project. We will be developing that project further to support reproducible scripting including reproducible Jupyter notebooks.

But before we sink a lot of effort into classloader shenanigans, it would be helpful to hear more about why you feel you need specifically to upload JARs to a running server. Why can't the server be restarted? Why can't the sysadmin (in many cases, the user herself) install the JAR before spinning up the server? Why does the client-side code need to be a JAR file with Java bytecode specifically, rather than an interpreted source executed as a script as above?

[kuandannyan]

@ctrueden

Yes, I have tried the run script before but it is not script or macro I am interested in. I am particular in dynamic loading of ImageJ/FIJI plugins into CellProfiler's instance of ImageJ.

A quick answer to dynamic classloader is basically for my own interests. For our design, the server with ImageJ instance maybe run on a HPC and shared among end users for heavy segmentation algorithm. Rather than sharing plugins and functions, we also share computation power. So restarting server constantly is not an option during production since you never know when people are finished with their analyses. Another issue is how to keep consistency between analysis if it maybe interrupted by a plugin update which requires a server restart. I am basically doing a comparison with for example a redeployment of WAR-file.

Another common issues I received from my end users are that they often find some new ImageJ or fiji plugins out of somewhere which they are want to use in the pipeline. But I simply cannot just restart the server every time someone wants a new plugin installed.

Hope the information is sufficient.

Kuan

[ctrueden]

@kuandannyan Thanks for the additional information, and apologies for delay in response.

It sounds like what you might want to do, then, is to create a "dynamic plugin" module: something like my example script above, but specifically geared toward loading bytecode instead of executing scripts. In principle there is nothing stopping the ImageJ RESTful server, or ImageJ in general, from doing this via a custom classloader. You can register additional plugins to the system via the addPlugin method of PluginService. With such a module in place, your users could upload whatever compiled Java code they want and execute it.

Implementing such a module would be a significant effort to puzzle out the details, and it is not currently on our roadmap. But it would be great to make something like that available to users.

The dynamic approach I do want to support in the short term is the scijava-grab project I mentioned above: you put e.g. #@dependency("net.imagej:imagej:2.0.0-rc-61") at the top of a script and that Maven artifact is resolved from a remote repository and put onto the classpath, so that those classes can be immediately used. This is great for writing reproducible scripts. For Java code (even snapshots) published to a Maven repository somewhere, this can be used to load new plugins into an existing runtime without restarting.

[kuandannyan]

@ctrueden

Thank you for the following up. Interesting idea of PluginService concept, I will try it out a bit.

[ctrueden]

Last week, @mcquin and I had a mini-hackathon in Madison to work on this project, and we made quite a bit of progress! 😁

We pursued two main goals:

1. Jupyter notebooks combining use of ImageJ and CellProfiler.
2. A CellProfiler module for invoking ImageJ modules.

Both goals were achieved, with functional proof-of-concept code, which we can now further refine and polish.

We created a GitHub repository at CellProfiler/notebooks.

There are two different integration mechanisms we explored: A) interprocess; and B) in-process.

Interprocess solution

The interprocess approach uses the ImageJ Server. This approach provides a nice layer of security between CP and IJ, such that even if IJ crashes, CP still does not. See the ImageJ Server demo notebook for an example.

In-process solution

The in-process approach uses a tool called pyjnius for calling Java from Python over JNI. This approach is similar to python-javabridge, but more seamless: it wraps Java classes completely as Python classes, which makes it very easy to use them (almost) as though they were written in Python. See the ImageJ pyjnius demo notebook for an example.

The in-process notebook successfully imports the entire ImageJ application gateway into Python, which was a first.

Integrating CellProfiler + ImageJ in Python

We developed a notebook demonstrating execution of a CellProfiler pipeline from Jupyter.

Then we wrote a combined notebook utilizing ImageJ Server + CellProfiler together.

We also developed a combined notebook utilizing ImageJ pyjnius + CellProfiler, but it is not 100% functional yet.

@kkangle is will continue to improve these notebooks, and get the CP+IJ pyjnius notebook fully working.

A new RunImageJ module for CellProfiler

We created an initial version of a new RunImageJ module for CellProfiler. For the moment it is still on a branch. But it already works!

Next steps

• Continue refining the notebook demos
  
  
  
  • Generalize and streamline the setup code cells
  
  
  • Make both ij-server & ij-pyjnius approaches fully work
  
  
  • Make it run in Binder
  
  
  • Improve user-friendliness w.r.t. ImageJ Server
  
  
• Create an ImageJ update site for the ImageJ Server

Later

We decided not to pursue a RunImageJ pyjnius module yet. We will do so later once scyjava (a generalization of imglyb, for using ImageJ from Python via pyjnius) is farther along.

Eventually, we want to enable calling CP pipelines from a JVM. But it is more involved, requiring either: A) launching ImageJ from inside Python, and/or leveraging JyNI; or B) exposing CellProfiler functionality via a cross-language API. We decided to wait until ImageJ's Python support is better before attempting this direction of integration.

[Skepe]

@ctrueden

Is the new runimagej module still available and usable with CP3.1.8?
I wanted to try, but the link to the branch you showed lead to a 404 message.
Is the branch "https://github.com/CellProfiler/CellProfiler-plugins/tree/runimagej" still alive?

best regards