Cellprofiler: A new module to integrate imaging flow cytometry workflow

[holgerhennig]

Description of the "montage" module:
The CP montage module could be based on our Matlab code cif_reader.m (file attached) which reads a .cif file and generates and saves the n-by-n image montages. No need to save the image montages though (displaying would be nice), as we have the CP module save images. Test files called testfile_.cif (and Matlab files from bioformats to access the cif file) are here:
https://www.dropbox.com/s/ubtm77b4gnpv3bf/cif_reader_implementation.zip?dl=0
Parameters: size of the n-by-n grid, default: 32x32 grid yielding 32_32=1024 images per image montage (if you estimate that we reach the sweet spot in terms of computational speed with a larger/smaller grid then pls pick that value as default)

Background:

• A compensated image file (.cif file) is a proprietary file format of the imaging flow cytometers developed by Millipore. Typically, a .cif file contains hundreds of thousands of small images, each image has only ~7KB, and metadata such as channel number along with each image. Loading/reading a .cif file already works nicely in CP, Lee Kamentsky has written an open-source reader for cif files available on BioFormats

Why we would love to have a montage module in CP:

1. Open-source/user-friendly: We currently use Matlab for the image tiling and then load the image montages into CP, but we would like to provide an open-source and user-friendly imaging flow cytometry workflow also suitable for everyday biologists
2. speed: Running 100,000 small images in CP (segmentation, extracting features) would e.g. take about 3-4 days on my laptop (4 cores), while running the same number of images on the same machine but with image montages (currently 225 images per montage) takes 2 hours. Thus the overhead in CP is huge- with 225 images per tile we are about 30-50 times faster using the tiling as compared to handling single images. The tiling itself is fast (~seconds for typical .cif file sizes).

cif_reader.m.zip

[0x00b1]

@holgerhennig Would you mind posting an example montage?

[bethac07]

Hi Holger,
Is there a reason that CP's Tile module doesn't work for what you want to do?

[AnneCarpenter]

Good Q - CP is super super slow at performing this task.

[0x00b1]

@AnneCarpenter Are you responding to @bethac07’s question?

[AnneCarpenter]

Yep.

[0x00b1]

@AnneCarpenter should this issue be “the tile module is slow, make it faster?"

[AnneCarpenter]

More like, image-leading is slow, make it faster.

[AnneCarpenter]

• loading

[AnneCarpenter]

But Holger should clarify before proceeding.

[holgerhennig]

Here's an example montage. The current montage Matlab script does the following
(1) pad images (with noise) so that all images have the same number of pixels and dimensions
(2) tile image. The images are read from a .cif file, a container for images and metadata. Image tiles are placed on each montage until it's full, then the next montage is filled.
(3) save all montages to disk (not needed in CP montage module)

The whole process (padding, tiling and saving to disk) takes a few secs per channel for a cif file with 100,000 images with the current matlab script.

example_montage_G2_cells.tif.zip

[AnneCarpenter]

Can you confirm why you don't load the CIF file in CellProfiler then use the Tile module to make the montages? Is it indeed speed?

[holgerhennig]

Yes, it's speed and we need a certain functionality explained below.

Speed: Just reading the metadata from 100,000 images takes longer than I've ever waited. I've tried with small .cif test files and waited minutes. With the montage module we don't need to read all the metadata. We only extract the total number of channels and total number of images and then loop over number of channels and images

Functionality: Unfortunately, we can't distinguish masks from images with the metadata provided by the .cif file. Our script takes care of that by skipping every other "image" (because every other image is an image mask). DavidL and I have tried to come up with a workaround using the metadata extracted by CP but we haven't found anything.

[0x00b1]

Speed: Just reading the metadata from 100,000 images takes longer than I've ever waited. I've tried with small .cif test files and waited minutes. With the montage module we don't need to read all the metadata. We only extract the total number of channels and total number of images and then loop over number of channels and images

@holgerhennig Unfortunately, I don’t think this is possible unless you use a module that reads the images, stitches them, and adds them to your ImageSetList (circumventing the Images, Metadata, NamesAndTypes, and Groups modules). If this is needed, then I’d recommend using a pre-processing script (or ImageMagick) outside of CellProfiler since it’d further complicate CellProfiler’s already confusing input/output situation. I’d be more than happy to write this. @mcquin too.

[AnneCarpenter]

Yeah, so think this through with me and see if this is a generalizable solution: we have these cases where a script is obviously better for a particular task, given CP's slow input but also sometimes because it doesn't follow the one-image-per-cycle rule. We have several situations like this where we perform some function that is sort of a one-off specialized case that doesn't fit CP's pipeline structure well.

The current MATLAB solution is bad because to explain to a computationally-unsavvy person how to get matlab, start it up, run a script, etc. is too overwhelming plus many don't have matlab licenses; we could compile but that's an ongoing maintenance issue. Obviously we could make the script in an open-source language but asking the computationally-unsavvy to run something at the command line is also intimidating. And means that @bethac07 has to answer befuddled Qs on the forum. And again, we are maintaining some stand-alone blob of code.

I am starting to wonder whether stuff like this should just be a data tool in CP (or alternately, some new category of thing, that is accessible from the menu perhaps). Each one is totally discrete so doesn't mess w anything else in CP, it lets people do the one thing.

Any downsides to this?

[0x00b1]

Any downsides to this?

A substantial downside is feature creep. Especially since feature creep quickly becomes code and software bloat. And code and software bloat quickly becomes software rot. If the counter-argument is @bethac07 answering questions on the forum, this feels like an easy decision (sorry, @bethac07).

[0x00b1]

Moreover, this isn’t technically sensible.

Each one is totally discrete so doesn't mess w anything else in CP, it lets people do the one thing.

A major problem of CellProfiler is that everything touches everything.

[bethac07]

The immediately-doable middle ground between keeping it locked behind a paywall in matlab and adding it to CP would be to script it in FIJI; a script like that would be trivial (I have some that do very similar things so editing one for your purposes would take ~30 minutes tops). The user has to download the .py file into their FIJI install directory but after that it just acts like any normal ImageJ menu function; not ideal since they have to find their FIJI install directory but anyone doing image flow cytometry probably has that minimal level of savvy.

[0x00b1]

@bethac07 If it’s just a Python script, why use Fiji? Just run the Python script. (I don’t know anything outside of what you’ve mentioned in the past about Fiji’s scripting abilities.)

[AnneCarpenter]

Yeah, it's not just about forum Qs, it's about serving the audience we intend to (esp given Holger's paper whose whole point is converting scripts into point-and-click software). Our goal here was to make a user-friendly solution that doesn't require installing or using multiple software packages. Beth's solution is indeed a reasonable middle ground because at least the software we are requiring them to install is point and click.

When we have more bandwidth we should talk about this - I don't think my proposed solution differs much from allowing CP to have a plugin folder that people can throw their own custom modules into. It's just that we want to allow custom data tools rather than modules. But I understand now is not the time.

[AnneCarpenter]

(And I'd agree w @0x00b1 - in fact, running a python script from Terminal might be a more concise set of instructions than downloading Fiji? Beth is the best judge of which is more intimidating or error-prone)

[bethac07]

I should've specified, this is a python script run from inside ImageJ that calls various ImageJ functions (aligning, etc) on the images currently loaded. Apparently you can modify things to run headless, but apparently its likelihood of succeeding headless depends on exactly what's in it and doesn't work at all for jython scripts on Windows (thus me never having tried it, since that's the only kind of macro-ing I do).

I'd conservatively say 60+% of people who do microscopy have used ImageJ in one format (stock ImageJ, MicroManager, FIJI, or one of the others) or another; my guess for command line is much much lower (like 5%). I think most of the ImageJ variants support scripting but would have to look into it; if not, they all support macros, so I/someone can learn enough java to figure it out. Either way, it's at max 2 steps (download ImageJ, download script into a folder in ImageJ) and then you have a clickable menu option that generates a friendly GUI that asks where your folders with images are, etc. Never underestimate a biologist's love of a friendly GUI :smile: .

This obviously isn't actually my project, so I'm happy to yield to the collective wisdom of others, but if this issue was a blocker for Holger's project this would be one way of bypassing it in the short term.

[0x00b1]

Yeah, it's not just about forum Qs, it's about serving the audience we intend to (esp given Holger's paper whose whole point is converting scripts into point-and-click software). Our goal here was to make a user-friendly solution that doesn't require installing or using multiple software packages. Beth's solution is indeed a reasonable middle ground because at least the software we are requiring them to install is point and click.

I’ve never seen the advantage of a monolithic applications, but what about a point-and-click application that exists outside of CellProfiler?

When we have more bandwidth we should talk about this - I don't think my proposed solution differs much from allowing CP to have a plugin folder that people can throw their own custom modules into. It's just that we want to allow custom data tools rather than modules. But I understand now is not the time.

I don’t know if this is the best comparison—I’ve never successfully used the plugins directory since I’ve worked on CellProfiler. 😳

If you have some time, I think you should create an issue. Regardless of my personal opinion, I think this is worth discussing. I think it’d be especially helpful if you defined “data tools.” What do you mean by this? For example, what would a data tool input and what would a data tool output? How are they different than a CellProfiler module?

[0x00b1]

Another question that’s worth asking, would you still be interested in “data tools” is CellProfiler’s messy input/output problems were resolved? What about CellProfiler’s performance issues?

[0x00b1]

This obviously isn't actually my project, so I'm happy to yield to the collective wisdom of others, but if this issue was a blocker for Holger's project this would be one way of bypassing it in the short term.

Your advice has been especially helpful! 😎

[AnneCarpenter]

Three points:

1. Beth is indeed awesome and helpful.
2. A point-and-click application that exists outside of CellProfiler would be okay (I assumed maintenance would be annoying for you). This would be my preference vs the FIJI/ImageJ solution but even that is okay under the circumstances.
3. It's a good question - fixing the input/output speed would solve a fair number of these problems we've encountered. Let's raise that question separately some day when heads are not exploding.

[AnneCarpenter]

@holgerhennig IIUC, you have provided example input files and the matlab script, can you also provide the expected output from that file to ensure we know what we're aiming at?

[mcquin]

@AnneCarpenter It looks like expected output is included in the zip file from Dropbox: https://www.dropbox.com/s/ubtm77b4gnpv3bf/cif_reader_implementation.zip?dl=0

[mcquin]

@holgerhennig I've started working on a Python script for this, based off your Matlab implementation.

I've been unable to load testfile_CRF118_large.cif. I think there is an issue with the metadata because I am getting the follow error when I load it:

JavaException: Channel count (1) does not match number of channel names (6) in string "C0|C1|C21|C3|C4|C5"

The other image (testfile_CRF118_Eosinophil_small.cif) also has six channels and loads just fine. However, it appears that the outputs in tiff_montages were generated from the large file. I'm assuming this from some of the code I've read and the difference between these images:

Expected (first image, first channel from montage):

Actual (first image, first channel from testfile_CRF118_Eosinophil_small.cif:

Let me know if I am not correct in this assumption, and am getting the wrong result. :)

One of these two things will help us move forward:

1. Corrected metadata for testfile_CRF118_large.cif.
2. Tiled outputs from testfile_CRF118_Eosinophil_small.cif.

[holgerhennig]

@mcquin @AnneCarpenter
Yes, the expected output is included in the zip file.

Regarding the error message: Yes, there are two slightly different "types" of .cif files generated by the IDEAS software, that seem to have different numbers of columns in the metadata. The BioFormats reader currently can read one .cif file type but not the other. The .cif file that is initially created by the IDEAS software is the one we cannot read, however, if we select a subpopulation in IDEAS and export it as .cif file (which is our usual workflow), then we obtain the .cif file type that Lee's .cif reader can read. Hmm, can we fix this somehow in the .cif reader? I can also talk to Millipore next week, maybe it helps to know more about how they generate the .cif files in order to develop a more robust .cif reader to facilitate advanced data analysis (which Millipore might be interested in too...). So for now, best is to work with testfile_CRF118_Eosinophil_small.cif

I just reran the matlab cif_reader.m using testfile_CRF118_Eosinophil_small.cif, attached are the tiled outputs
tiff_montages_tmp.zip

so something seems to be wrong with the actual image you're getting... in Matlab it's the function reader.getIndex that we're calling, maybe that's different somehow in python?

[mcquin]

@holgerhennig thanks for the info and images! I realized I was reshaping the array incorrectly, and am now able to produce an image with the correct shape.

It is my opinion that the CIF output by the IDEAS software should have correct metadata. It is not our responsibility to support reading files with incorrect metadata. It is my understanding that the number of channels must match the length of channel names.

I have not been able to get Lee's CIF reader to work, yet. I can load an image, but when I run queries to get the number of images or channels the value -1 is always returned.

What I've written so far (and your MatLab files) is entirely code Java. This is not ideal. We should reconsider whether or not this functionality makes sense in CP/outside Java-based applications. Does this make more sense as an FIJI/IntelliJ plugin, or similar?

[holgerhennig]

I fully agree, the no. of channels should match the length of channel names. And reverse engineering the different .cif file metadata structures is also a bit cumbersome.. I'll try to discuss the .cif metadata issue with Millipore next week..
If you can get the tiling for testfile_CRF118_Eosinophil_small.cif to work (and ignore testfile_CRF118_large.cif), that'd be awesome (it works with the Matlab script, but that's not open source and not very user-friendly..)

[mcquin]

@holgerhennig how are you padding the images? i'm using zeros but it looks like you have another approach? are zeros ok?

[holgerhennig]

we use noise. zeros may lead to false segmentation in identify object (because zeros can introduce edges at the transition from the zeros to the actual image)

[AnneCarpenter]

Can you point her to the details on how your noise is defined? I guess it's in both papers?

[holgerhennig]

sure, will do asap

[holgerhennig]

Generating the noise:
1) take a small sample of 100 data points from the cell image (a square of 10 by 10 points in the lower left corner- where most likely there's no intensity from an actual cell)
2) calculate mean m and standard deviation s for the 100 data points
3) generate normal distributed noise with mean m and standard deviation s

Pls make sure that noise is not added to the actual image of the cell.

Here's the Matlab function.

---

Function [ image_scaled ] = scale( image_raw, made_up_size )

sample = double(reshape(image_raw(1:10,1:10),100,1));   
mu = mean(sample);
sigma = sqrt(var(sample));

image_scaled = uint16(abs(round(random('norm',mu,sigma,[made_up_size,made_up_size]))));

image_raw_cut = cut(image_raw,made_up_size);

ind1 = findind(size(image_raw_cut,1),made_up_size);
ind2 = findind(size(image_raw_cut,2),made_up_size);

image_scaled(ind1,ind2) = image_raw_cut;

end

---

[mcquin]

@holgerhennig do you care if the images are stacked vertically (as they are now) or horizontally?

[holgerhennig]

Hmm, vertical stacking would be better- so to obtain the same results with the python and matlab tiling

[mcquin]

bummer, horizontal is much easier ;)

btw, @holgerhennig you have an off-by-one error in your grid size. you are producing 33x33 grids.

[0x00b1]

@mcquin use skimage.util.montage.montage2d, @holgerhennig update your slides. 😝

[mcquin]

[0x00b1]

nice

[holgerhennig]

love it

[AnneCarpenter]

@mcquin Have you done a check to ensure the final pixel output is identical to that produced by the MATLAB script? (I'm not sure whether the step of adding noise is non-identical such that we actually don't expect identical output, though?)