Rawtherapee: SIGSEGV in LUT.h

[Beep6581]

While preparing a control group to test PR #5562 I encountered this 100% reproducible crash in dev. Oddly, if I process any of these images via the queue individually, they save fine, but when I add them all to the queue and run it, it crashes.

LUT<float>::operator[]<float, float, void> (this=0x55555642e120 <rtengine::Color::cachefy>, index=-nan(0x7147a2)) at /home/morgan/programs/code-rawtherapee/rtengine/LUT.h:453
453             T p1 = data[idx];

Could be related to capture sharpening @heckflosse

#3  0x0000555555f76689 in rtengine::RawImageSource::captureSharpening(rtengine::procparams::CaptureSharpeningParams const&, bool, double&, double&) [clone ._omp_fn.11] (this=0x55555642e120 <rtengine::Color::cachefy>, sharpeningParams=..., showMask=214, conrastThreshold=<error reading variable>, radius=@0x555555c705e8: -1.2018730611837227e+29) at /home/morgan/programs/code-rawtherapee/rtengine/capturesharpening.cc:828

Full GDB log: log.txt
Test files (add them all to the queue and run it, I'm saving as 16-bit TIFF uncompressed): https://filebin.net/yv7oruoe7v094oz5/test.zip?t=248q31is

Reproducible in release and debug.

Version: 5.7-297-g0edbdc886
Branch: dev
Commit: 0edbdc886
Commit date: 2019-12-12
Compiler: cc 8.2.0
Processor: Intel(R)\ Core(TM)\ m3-6Y30\ CPU\ @\ 0.90GHz
System: Linux
Bit depth: 64 bits
Gtkmm: V3.24.1
Lensfun: V0.3.2.0
Build type: release
Build flags:  -std=c++11 -march=native -Werror=unused-label -Werror=delete-incomplete -Wall -Wuninitialized -Wcast-qual -Wno-deprecated-declarations -Wno-unused-result -fopenmp -Werror=unknown-pragmas -O3 -DNDEBUG -ftree-vectorize
Link flags:  -march=native
OpenMP support: ON
MMAP support: ON

[Beep6581]

I can also reproduce in review_demosaicers 5.7-301-gd91894518

[Floessie]

FYI: Cannot reproduce on Debian 10 AMD64. Now trying on Debian 9...

[Floessie]

Yep, crash on Debian 9.

@heckflosse Ingo, shall I bisect?

[heckflosse]

@Floessie Yes, please bisect. I can not reproduce

[heckflosse]

@Floessie
How is this possible?

#3  0x0000555555f76689 in rtengine::RawImageSource::captureSharpening(rtengine::procparams::CaptureSharpeningParams const&, bool, double&, double&) [clone ._omp_fn.11] (this=0x55555642e120 <rtengine::Color::cachefy>, sharpeningParams=..., showMask=false, conrastThreshold=<error reading variable>, radius=@0x555555c705e8: -1.2018730611837227e+29) at /home/morgan/programs/code-rawtherapee/rtengine/capturesharpening.cc:828
        i = 48
        xyz_rgb = {{0.0721689984, 0.0721689984, 0.0721689984}, {0.0721689984, 0, 0}, {0, 0, -nan(0x346ce)}}

xyz_rgb is this:

    constexpr float xyz_rgb[3][3] = {          // XYZ from RGB
                                    { 0.412453, 0.357580, 0.180423 },
                                    { 0.212671, 0.715160, 0.072169 },
                                    { 0.019334, 0.119193, 0.950227 }
                                    };

[Floessie]

Memory corruption or debugger problem? Odd, indeed...

[heckflosse]

I think the culprit is calling capture sharpening on predemosaiced dng files....

[heckflosse]

I will post a fix asap

[heckflosse]

Does this fix it?

diff --git a/rtengine/capturesharpening.cc b/rtengine/capturesharpening.cc
index 910de6fd3..281a2ad71 100644
--- a/rtengine/capturesharpening.cc
+++ b/rtengine/capturesharpening.cc
@@ -700,6 +700,10 @@ namespace rtengine

 void RawImageSource::captureSharpening(const procparams::CaptureSharpeningParams &sharpeningParams, bool showMask, double &conrastThreshold, double &radius) {

+    if (!(ri->getSensorType() == ST_BAYER || ri->getSensorType() == ST_FUJI_XTRANS || ri->get_colors() == 1)) {
+        return;
+    }
+
     if (plistener) {
         plistener->setProgressStr(M("TP_PDSHARPENING_LABEL"));
         plistener->setProgress(0.0);

[Floessie]

I bisected it to c7400459f056125cf5072cbec2d4a697c331c85f.

Now testing your fix...

[Floessie]

Positive. :+1:

[Beep6581]

Fix confirmed.