Cellprofiler: Rescale intensity relative to maximum intensity in image.

[mcquin]

Currently, when loading an image, intensities are rescaled based on the following criteria:

1. A value provided by the user*.
2. The maximum intensity supported by the image, as defined in the image metadata.
3. The maximum value supported by the image data type**.

Instead of defaulting to rescaling by supported maximum intensity (options 2 or 3), CellProfiler should default to rescaling intensities by actual maximum intensity. That is, rescale intensities between 0 and 1 such that the actual minimum intensity value is mapped to 0 and the actual maximum intensity value is mapped to 1 (and all other intensities are adjusted accordingly):

image = (image - image.min())/(image.max() - image.min())

Rescaling intensities in this way has the following benefits:

1. It works.
2. It's consistent and not dependent on image metadata.
3. It won't rescale all values to be teeny-tiny if metadata is missing and the data type is 32-bit integer (in this case, all values would be rescaled by dividing by 65535. Ew!).
4. We won't be dependent on python-bioformats to load images and rescale values (this is very important for 3D work).

---

* Kind of but not really. It's first scaled by supported maximum intensity (option 2 or 3) and then scaled again by the user specified value. This boolean rescale is not passed into the call to read. Nonzero values evaluate to True, so the image is first rescaled to its supported maximum intensity and then rescaled again according to the provided value.

** This isn't entirely accurate either, because these data type maximums are not correct.

[bethac07]

I disagree pretty strongly on this; it's important that you all the images in a given experiment are scaled in the same way (so that the relative intensity values are comparable), and you can't do that if you are scaling each image individually. Unless I'm misunderstanding what you plan to do here?

[0x00b1]

@bethac07 Sure, but that isn’t happening now …

[bethac07]

???? If I load in a set of images with the same bit depth now, they don't all scale in the same way? If so that's a massive, massive problem.

[0x00b1]

I guess this is a symptom of https://github.com/CellProfiler/CellProfiler/issues/2272

(i.e. we shouldn’t implicitly rescale anywhere)

[AnneCarpenter]

Yeah the current situation is messy but each option results from there being limitations in the other existing options.

One example weirdness: many cameras are 12-bit but the images are saved in 16-bit format leaving a ton of empty space. But the metadata doesn't provide this information. This is why we offer the user the option to enter their own information to tell CP to use 12-bits as the top for their image set.

It's a fair question - in such a case why not just use 16-bit (the max for the file format)? I don't recall the answer here. I don't know if the image processing we do starts getting weird when all the image data is in a super-low range or if it's related to display (i.e., images appear completely black). I'm sure someone remembers and I suppose it's important to figure that out in order to know what we can adjust.

But I'm with @bethac07 that we can do whatever we like as long as it doesn't affect the relative intensities from one image to the next within the set.

[0x00b1]

???? If I load in a set of images with the same bit depth now, they don't all scale in the same way? If so that's a massive, massive problem.

Yeah, e.g. see @mcquin’s first note in the original issue’s message. If this is correct, then images are not correctly rescaled across the set.

I don’t know if it’s a “massive, massive problem.” But it’s certainly a problem.

[mcquin]

If only it were easier to pre-process images... we could find the maximum intensity across the image set. :(

Maybe we should remove implicit rescaling.

Unfortunately, some external libraries (e.g., skimage) work best for images scaled between 0 and 1. Dunno how much would be impacted by removing implicit rescaling.

[0x00b1]

Unfortunately, some external libraries (e.g., skimage) work best for images scaled between 0 and 1. Dunno how much would be impacted by removing implicit rescaling.

OpenCV too.

[AnneCarpenter]

Can you explain what you mean by "implicit rescaling" - which step is that? Do you mean the rescaling that happens by virtue of just loading an image into CP?

[0x00b1]

Can you explain what you mean by "implicit rescaling" - which step is that? Do you mean the rescaling that happens by virtue of just loading an image into CP?

From a computer vision perspective this is odd behavior for the aforementioned reasons.

I’d expect the behavior to be using the RescaleIntensity module explicitly. However, I understand the usability issues (i.e. it’s tricky business).

[0x00b1]

@bethac07 and I just spoke with @mcquin.

@bethac07 shared some useful examples that she should share here too. 😎

[0x00b1]

Unfortunately, some external libraries (e.g., skimage) work best for images scaled between 0 and 1. Dunno how much would be impacted by removing implicit rescaling.

Hopefully, and I believe this is the case, image processing modules appropriately rescale.

[0x00b1]

Anyway, my opinion (and you’re welcome to ignore), don’t rescale but enforce a reasonable dtype (e.g. 64-bit floating-point). You’ll need to use your judgment to determine a reasonable dtype. 😝

[bethac07]

@AnneCarpenter Implicit rescaling happens when the input image is rescaled to 0-1 right at read-in; CP usually does this by the bit depth or the metadata, but apparently both of those are much more fragile and corruptable than I had any idea about. (I'm honestly a bit scared right now).

We discussed and rejected rescaling at read-in on a per-image basis (because then intensities aren't comparable image to image) and on a per-all-images-currently-being-loaded basis (because then intensities aren't comparable from CP run to CP run); as Allen alluded above, leaning towards leaving the image unscaled unless the user specifically invokes RescaleImage may end up being the way to go. It'll break direct comparability between values obtained from CP2 and CP3, but I think most people can probably live with that.

[AnneCarpenter]

It sounds exciting if leaving unscaled really would work, certainly it's way simpler.

Awesome summary of exactly the issues I was worried about @bethac07 .

I think people will survive with a sudden change in intensity values from one version to the next, so I am not too worried about that (possibly painful transition but if it's what we need we will survive).

Some gotchas to watch out for:
1) image displays: do images look normal/decent/not entirely black and/or saturated? IIUC, we would have no choice but to contrast-stretch each image (in a relative way from image to image) prior to displaying in a window, otherwise data might not be visible. This kind of puts us back where we started, unfortunately, with users not being able to compare the brightness of one image to the next. Also, when a particular channel of a color image has low signal throughout, it gets stretched such that the entire image shows bright noise acrost it, which is bad and confusing too.
2) module settings that expect a [0, 1] range: I suppose threshold correction factors and the like have some expectations, so do we need to re-think default values and adjust documentation.
3) image displays: in cases where we show pixel values, do we have enough real estate to show values in the tens of thousands? I suspect not a problem.
4) Integrated intensities will be in the bajillions, does that cause any problems? I suspect not.
5) actual module processing behavior: do we have any hard-coded settings within algorithms that are expecting a [0,1] range?
6) Illumination correction especially... I can't quite think through what issues might arise here but worth pondering.
7) And a task: adjusting documentation throughout CP because I imagine we patiently explain that images are in a [0, 1] range in a bunch of places.

I am hoping some long-time contributors to CP can comment on any other foreseeable problems so we can be sure of what we are getting into. @thouis ?

[bethac07]

1) We pretty often rescale the images for display anyway (I'd say >50% of the time)- there is an option for showing the raw values, but given that all but the cheapest cameras these days are 12bit or higher and monitors are 8bit we pretty frequently are in "Normalized" or "LogNormalized" territory for display. So I don't see this being a huge problem.

[AnneCarpenter]

Wow, so you already have no idea if one image is pretty bright and the next image is dim, and whatnot?

8) Saving images. Seems like this could be very painful if we don't know the range of the image that is incoming.

[0x00b1]

Wow, so you already have no idea if one image is pretty bright and the next image is dim, and whatnot?

It depends (on the matplotlib configuration).

[0x00b1]

1) image displays: do images look normal/decent/not entirely black and/or saturated? IIUC, we would have no choice but to contrast-stretch each image (in a relative way from image to image) prior to displaying in a window, otherwise data might not be visible. This kind of puts us back where we started, unfortunately, with users not being able to compare the brightness of one image to the next. Also, when a particular channel of a color image has low signal throughout, it gets stretched such that the entire image shows bright noise acrost it, which is bad and confusing too.

Was this the original motivation for rescaling by default?

[AnneCarpenter]

Was this the original motivation for rescaling by default?

I am not 100% sure but I suspect indeed display and save might be two of the biggest motivations.

[0x00b1]

2) module settings that expect a [0, 1] range: I suppose threshold correction factors and the like have some expectations, so do we need to re-think default values and adjust documentation.

Yeah, I think that’s the case. We’ll also need to rethink settings.

[0x00b1]

I am not 100% sure but I suspect indeed display and save might be two of the biggest motivations.

Yeah, this issue makes a lot more sense with that context.

[bethac07]

Wow, so you already have no idea if one image is pretty bright and the next image is dim, and whatnot?

You get a pretty good guess by how the background looks usually.

[AnneCarpenter]

You get a pretty good guess by how the background looks usually.

Wowee, that is a non-obvious-to-new-users workaround!

[AnneCarpenter]

Yeah, this issue makes a lot more sense with that context.

It seems like this issue gets you either coming or going. You can ignore it upon loading but then you're going to have to deal with it later. I'm amazed at Beth's commentary that usually they stretch display intensities anyway. Which would argue not to worry about the display issue but this is HUGELY worrisome to new users who wouldn't have a clue why their first image shows nice nuclei with a dark background and image number two shows as weird gray-static (because it has no nuclei at all). Really seems that this is a dealbreaker to me.

[bethac07]

You get a pretty good guess by how the background looks usually.

Wowee, that is a non-obvious-to-new-users workaround!

Yeah, but it's not a CP specific one- using "Auto" in ImageJ has the same effect, or for that matter the autoscaling built into the capture sortware of just about any microscope. I think it's something you pick up pretty fast when you start taking images.

'm amazed at Beth's commentary that usually they stretch display intensities anyway. Which would argue not to worry about the display issue but this is HUGELY worrisome to new users who wouldn't have a clue why their first image shows nice nuclei with a dark background and image number two shows as weird gray-static (because it has no nuclei at all).

The alternative is that all their images look black, which I'd argue is more worrisome. My $0.02

[AnneCarpenter]

The alternative is that all their images look black, which I'd argue is more worrisome. My $0.02

Hopefully this doesn't happen often because we catch it and handle it when loading (ie the 12 bit to 16 bit thing) . But I trust your judgment, you've been looking at images from a lot more varied sources lately!

[karhohs]

I'm working with image sets that have a wide range of intensity values between samples. Samples in this case are tissue sections from different patients that are stained and imaged days, weeks, or months apart from each other. When using CPA to find rules that filter objects, I found that if I want to apply the same rules across the samples that I must normalize the images within each sample; that is to say images are rescaled based on parameters unique to each tissue section. Otherwise, I must train CPA for each sample. After reading the github thread, I realize I don't have a good sense about when normalization is appropriate or not. Does it sound like my use case is a proper application of normalization?

Also, I noticed that CP does not auto-scale 32-bit float images. (I think this default behavior is related to illumination correction images being stored as 32-bit floats). In any case, I've been using the RescaleIntensity module for normalization purposes.

[braymp]

Which would argue not to worry about the display issue but this is HUGELY worrisome to new users who wouldn't have a clue why their first image shows nice nuclei with a dark background and image number two shows as weird gray-static (because it has no nuclei at all). Really seems that this is a dealbreaker to me.

Meh, I don't think it's as bad as you'd guess; a noisy, empty image is not too hard to ID as such when contrast-stretched.

The more common confusion I receive is when an image has a few saturated pixels, but the rest is normal and the contrast is in normalized mode. In that case, it's hard to figure out that the dark image you're seeing actually isn't blank.

[mcquin]

We're decided on not rescaling intensities on image load, unless explicitly set?

[AnneCarpenter]

No, not at all... I think it could unleash a world of unintended consequences that would plague you. I think we should take time to digest.

[AnneCarpenter]

Do you need help brainstorming a way around whatever problem initiated this discussion, itI was something related to 3D? we could think on that for a while.

[braymp]

No, not at all... I think it could unleash a world of unintended consequences that would plague you. I think we should take time to digest.

Could not agree more.

[tischi]

May I add a user perspective?!

I would strongly favor not re-scaling!

• It is really annoying if the numbers in CP do not match the numbers in IJ nor the numbers on the microscope.
• I am training people a lot to actually look at the numbers; i.e. an image is a matrix of numbers and not a pretty picture. It is difficult for me to get this message across is the numbers are different in different software packages.
• I think our attitude as bio-image analysts should be: "The gray value of a pixel is holy! Who- or what-ever changes it unnecessarily should be condemned ."
• The 12-bit saved as 16-bit issue is another reason for why sensible re-scaling is probably difficult anyway.
• When I do image background subtraction I have to enter values like -0.0015 in ImageMath; 50% of the cases I miss one zero and subtract the wrong number.

Regarding image display in CellProfiler:

• I think CP would greatly benefit from a more sophisticated Brightness and Contrast adjustment.
• It is probably enough to be able to enter(slider) min and max values which are mapped to dark and bright, as sometimes neither the log-scale nor the auto-contrast work.

[AnneCarpenter]

Thanks, @tischi , I definitely understand your rationale!

Even so, my opinion is that this convenience you describe would not compensate for the fact that users would no longer be able to rely on the defaults in CellProfiler and have the image analysis, saving images, etc. itself just work nicely. If someone shares a pipeline file with a colleague and the colleague's images have a different bit depth, the pipeline will need manual adjustment. I've not heard arguments about how to solve these kinds of issues nicely for the user.

I will admit I'm partially also influenced by the incredible amount of work it would take to go through the codebase systematically in order to make adjustments related to this and/or assess whether there are more gotchas than what I originally listed. As well, we would need to adjust all existing example pipelines accordingly and users would need to update their pipelines to transition over. So there is a lot of momentum against making a change, in addition to the actual merits of the decision.

I've entered your suggestions about image display in CP as #2446, we've all wanted this and somehow I don't see an issue yet on it!

[tischi]

Hi Anne,
I totally understand that it would be a lot of work, and thus I would also understand if you did not do it in the end.

Regarding your argument: "If someone shares a pipeline file with a colleague and the colleague's images have a different bit depth, the pipeline will need manual adjustment. I've not heard arguments about how to solve these kinds of issues nicely for the user."

I actually think that is a good thing! People should know what bit depth there images have! I have seen several projects go wrong where people did some 16-bit to 8-bit conversion without knowing what they were doing and of course got nonsense in the end. The bad thing is that you often don't notice that it is nonsense unless a picky guy like me closely examines every single gray value ;-) So, if the thresholding in CellProfiler would totally not work anymore after the change of bit-depths, people might at least have a hint that there is something strange with their data.

I guess in general in my point is the following: It is very dangerous to give users an image analysis tool that hides too much of the actual data and mathematics, because the users stop thinking and might get random results. Thus, forcing users to think about the bit-depth of their data is imho a good thing, because it raises "data awareness".

[tischi]

Hi Anne,

May I cite you: "Generally speaking, CellProfiler rescales images to be 0-1 based on the max possible allowed intensity for your image format. So just multiply the values by 255 and you should be all set (maybe 256, but perhaps you don't need such precision!)"

In fact, this is exactly the kind of annoying issues you have to deal with when you want to compute back to the real values, i.e. whether to multiply by 255 or 256.

May I cite from our user guidelines:

• Our Olympus ScanR microscopes produce tiff files with an integer 16 bit/pixel container.
• The actual image data only has 4096 grey levels. However, as the images are saved as "signed" 16bit images these gray values have an offset of 2^15, i.e. they range from 2^15 (32768) to 2^15+4095 (=36863).
• CellProfiler expects pixel data in the range from 0.0 to 1.0 floating point.
• Upon loading the raw ScanR pictures, CellProfiler already rescales them such that the range from 0 to 2^16-1 is mapped to 0.0 to 1.0. This means that the useful image range is rescaled to a range from 2^15/(2^16-1) to (2^15+4095)/(2^16-1) which is 0.50000762951 to 0.56249332417.

Now, all our CP pipelines have one module where we check how many saturated pixels are in each cell, because we use this later as a cell-based QC. To do this properly one has to seriously start thinking about stuff like whether 0.56249332417 is exact or rounded, and about how much 1 gray value is in CP units (1/65535=... or 1/65536=...). This is not much fun for me, and certainly too much for or most of our users.

I admit, the problem starts with the signed 16bit, but the resulting values in CP surely don't help.

[AnneCarpenter]

You make some interesting and useful points and it's great to have someone care about this issue, @tischi !

My thinking is that pixel values are arbitrary fluorescence units to begin with. So it is not in fact critical that users understand the original source image's data range (as long as it is loaded properly - both you and @mcquin point out some cases where this is not true! and we should fix those cases - Feel free to file an issue about ScanR if you have example images; I've not encountered that.).

Thousands of biologists have accomplished great things without having any clue about bit depth. It is certainly useful for them to look at pixel values - I think that is a great educational step - but I don't see any harm in those values being in the [0,1] range within CellProfiler.

My opinion is partly based on not seeing any compelling reason to convert them back to any other range - I'm not sure why you would bother to multiply again by 255 but maybe you have cases where this is helpful. (even for saving, having images in a standard [0,1] range is very useful and allows users to store images in whatever bit depth they like rather than worrying whether their data will saturate in the bit depth they've chosen).

I think that requiring that users learn about bit depth before they can use a pipeline (whether from a colleague, a published paper, or an example pipeline) would be a major usability catastrophe. The biologist would see their results as zero objects identified, or zillions identified, but have no idea why, and we can't very well pop up a dialog at the beginning saying "please read this tutorial on bit depth prior to using CellProfiler".

We are aiming for "it just works", and one of the reasons things just work in CellProfiler is because when we load images they are in a range that is (to some degree) standard so that the default values have some hope of working.

A counter argument to the above: an ideal pipeline avoids hard-coded pixel values such as thresholds; they ought to generally be automatically calculated in which case everything should work just fine with a different pixel depth image as input (and hard-coded pixel values are likely to fail on someone else's images even if they ARE the same bit depth!). But hard-coded thresholds are an option that nevertheless do exist in real pipelines and there are likely lots of other cases (like illumination calculation maybe?) that would not automatically adjust to images presented in a vastly different range than the original. We haven't really thought through to catalog them all, but it would require a serious effort to do this and figure out how to handle each case in a way that is likely to not fail on images that can be in very different ranges.

[0x00b1]

This is a useful conversation. However, I think we’re discussing two separate issues:

• How should we type intensities?
• How should we normalize (i.e. rescale intensities)?

A few comments about typing:

Typing is a common issue for computer vision libraries. Insight Segmentation and Registration Toolkit (ITK), OpenCV, scikit-image, etc. are inconsistent across operations and manual casting between a 2-bit, 8-bit, 16-bit or 32-bit integer and 32-bit floating point is required. While this is annoying from a user experience perspective, it’s a necessary trade-off from a performance perspective.

I think CellProfiler’s typing is fine. CellProfiler expects:

• binary masks are 2-bit integer (numpy.bool) arrays;
• images are 32-bit floating point (numpy.float32) arrays; and
• labels are 16-bit integer (numpy.int16) arrays.

By pushing casting onto modules, CellProfiler permits module authors to make sensible user experience and performance trade-offs. Moreover, it permits additional flexibility when using the aforementioned libraries that are inconsistent between operations.

Finally, while I didn’t make the decision to standardize 32-bit floating point arrays for images, I think it was the right decision from a user experience perspective since the minimum and maximum values are evident to the user without any consideration of the underlying bit depth.

I have one comment about normalization:

It should be explicit.

I believe this for many of the reasons @tischi enumerated, but it’d have the additional benefit of simplifying CellProfiler first four modules. 😆

[0x00b1]

The actual image data only has 4096 grey levels. However, as the images are saved as "signed" 16bit images these gray values have an offset of 2^15, i.e. they range from 2^15 (32768) to 2^15+4095 (=36863).

🤔

[AnneCarpenter]

That is helpful commentary, @0x00b1

I have one comment about normalization:
It should be explicit.

Can you describe what you mean by 'explicit'? Are you saying the user should specify it rather than CP trying to guess the right answer for each image?

[thouis]

I think it's possible to be explicit (= visible as a setting?) and still have CP guess the right thing to do, especially if it remembers that made a guess, and throws an exception if that guess later turns out to be wrong. This could be extended to applying existing pipelines to new images by other users.

This of course assumes there's some ability for CP to inspect a few test images before it actually tries to run the pipeline. I've been away from the code so long I don't know if that's actually a thing.

[tischi]

"Thousands of biologists have accomplished great things without having any clue about bit depth."
That is true! On the other hand, there is big discussions out there concerning the notion that ~90% of biological papers are not reproducible. My feeling is that it would help somewhat if biologists did have a clue about bit depth.

[tischi]

Hi again,

just wanted to mention that I do not want to be annoying. I of course LOVE CellProfiler (even with 0-1 scaling) and I teach biologists all over Europe and recently also in India what a GREAT software it is!
I am just seeing here an opportunity to (in my view) improve it even more, so I am being kind of passionate about it.

So please forgive me if I add a few more arguments in favor of not rescaling:

1) At least in my experience ~80% of the people using CellProfiler want only one or two numbers as readout, one of them typically being an intensity. Of course, as Anne pointed out, the intensity values are meaningless, so they need to be normalized at some point, for instance to intensities measured in control samples, on the same day, on the same microscope, with the same settings. However, before one can do this normalization there are typically a few steps involving some sort of background subtraction or illumination correction. During these pre-processing steps one has to be super careful that everything goes well in order to avoid artifacts. One simple issue me and our users have here is that the small numbers after rescaling are simply harder for the human brain to process: 0.005 and 0.0005 are more difficult to relate and to distinguish than 500 and 5000; maybe it has to do with the fact that most people think about money when they see numbers ;-)

2) Just an anecdote: We are running CP on a compute cluster and at some point had a problem that some of the nodes did not find the java, making CP fall back on PIL to load the images. I took us a while to figure out that all the images from these nodes had different rescaling than the others. I guess this might be fixed by now, but it also illustrates again that different software packages might have different opinions on how to rescale, making it just a bit harder to conduct reproducible science.

[thouis]

IIRC, scikit-image went through much discussion some years ago about how to rescale images at loading time and in processing. @stefanv might remember the thread and be able to point us to it.

FWIW, the original intent in all of the rescaling was to always use the same scaling for every image, though the factors that controlled that might be estimated from the images themselves. If we somehow managed to do something else, it was a bug relative to the intended functionality.

@tischi - do you happen to have one those images (from your anecdote) handy? Seems like a good candidate for a unit test, assuming CP is still using multiple possible backends for loading.

[tischi]

@thouis Here are the files that we typically deal with. But, in fact the first thing we do these days is to run the signed integer images through an own "transfer tool" that makes then unsigned to alleviate some of the mentioned downstream hassle.
signedIntTiff.zip
unsignedIntTif.zip

[stefanv]

In scikit-image, we never implicitly rescale data (I'm pretty sure most of our scientific users would find such behavior upsetting). Instead, we expect images to comply to our data type conventions.

Recently, we've had more and more users who want to process data with different ranges, so many of our functions now accept a preserve_range argument. E.g., if we simply compute a block mean, there's no need to know the scaling. For other methods, like histogramming, range is important.

We also make available a set of tools so that users can easily rescale their data as they see fit. E.g. rescale intensity allows you to scale to 0-1 using either the input range, the data type, or an explicitly specified range.

[mcquin]

Great discussion! Closing this issue since there seem to be no achievable outcomes at this time.