Darktable: Shadows and Highlights with negative values on shadow unexpected result

[mtedaldi]

When pulling the "shadows" slider into the negative the shadows become bricht suddenly on some values as if some kind of overflow condition occurs.
(See this short video clip for demonstration https://youtu.be/Sq2mscTcugI )

1. Reduce brightnes of image in exposure so parts of the image are below the black level
2. go to "shadow and highlights" and slowly pull the slider to the left while observing the shadows in the image

Expected behavior
The shadows shold stay dark and more parts of the image should become dark or black

Screenshots
Video on https://youtu.be/Sq2mscTcugI

Platform:

• Darktable 3.1.0+2136~g9a8da0352
• OS: debian 10.4
• Tested with OpenCL and without OpenCL. Result is the same

[hatsunearu]

Can't repro--mind posting the image you have, with the sidecar?

[mtedaldi]

Thanks for checking. I'll attach the raw file and the xmp here.

I've done a few more tests and found out the the important step to reproduce the issue is, that you really need to get parts of the image below the black level (I'll test, by how much. I quess at least -1EV or -2EV) with the "exposure" module.

dt_shadow_highlight_issue5625.zip

[hatsunearu]

I think I figured it out. You're not supposed to push the "black level correction" in the exposure module to the right--ever. View the hover tooltip for more info. I'm not even sure what you're supposed to use that tool for. But this bug is still interesting because it might highlight (hah) some underlying bug. I'm gonna look further into this.

[mtedaldi]

You're not supposed to push the "black level correction" in the exposure module to the right--ever. View the hover tooltip for more info. I'm not even sure what you're supposed to use that tool for.

I know this is not the right place to discuss this. But I'm doing this all the time! I don't use the exposure tool directly but "correct" the black level with the mouse wheel on the histogram at the top of the right panel on almost every image. In most cases to add more density to the blacks... I've never seen that tooltipp because I almost always do it on the histogram..

[hatsunearu]

You should be doing that with levels if you wanna work with the legacy toolset, or via new tools things like tone equalizer. Those work a lot better. Tone eq should actually be a good replacement for S&H as well.

Basically when you push the black level correction way too far to the right, a lot of pixel values end up becoming negative, which don't have physical meaning (that means having "negative light", whatever that means). Some tools handle it okay, some tools kinda freak out. I think the current policy (from what I discussed on IRC) around negative pixels is to gracefully fail, usually by clipping. The current behavior with S&H is kinda weird even considering this low bar and I'm looking into it. I'm gonna try to avoid clipping, but if it happens it happens.

tl;dr: don't abuse black level correction, but there is a legitimate bug (imo) with S&H.

[hatsunearu]

So I've done a really deep dive on this. First of all, when you do this, you push the picture WAYYYY out of the Lab* gamut. This seems to cause a lot of issues down the pipe. I think the error is not with S&H, but with the way the output picture is rendered.

My evidence for this is in the "input color profile", if you switch on "gamut clipping", the weird non-monotonicity disappears. My guess is that there is some sort of overflow behavior further down in the pipe (negative numbers wrapping around to be positive, etc).

[mtedaldi]

Wow. Thank you! Awesome that you go to so much effort to fix something that only occurs because I used the tools in a completely wrong way!

[hatsunearu]

Wow. Thank you! Awesome that you go to so much effort to fix something that only occurs because I used the tools in a completely wrong way!

No problem. That was an insane deep dive. Don't close this yet though. When the PR is merged it will close automatically.

[github-actions[bot]]

This issue did not get any activity in the past 30 days and will be closed in 365 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.