Rawtherapee: SIGSEGV (null lumcalc[][] deref) in rtengine::ImProcFunctions::RGB_denoise [5.8 tarball] when tiled

[mandree]

Hi,

I am the packager of RawTherapee for FreeBSD. I've recently received pkg-fallout@ mail meaning issues with our package, and chances are that due to GCC or library updates, circumstances have changed and I can crash rawtherapee with command line options -q -s -Y -c but not with -q -f -c.

• Preprocessing parameters: https://people.freebsd.org/~mandree/_DSC3041.ARW.pp3
• Encrypted raw data if needed: https://people.freebsd.org/~mandree/_DSC3041.ARW.gpg - I'll share the password for decryption on request (m-a on IRC) if I recognize you as RT developer because I cannot make the image public.

Now, this either appears to be racey or somehow not reliably reproducible, and under gdb I sometimes need to re-run things. On 2nd or 3rd run, I get
1st denoise pass failed due to insufficient memory, starting 2nd (tiled) pass now...
and then it crashes.

The crash happens in rtengine::ImProcFunctions::RGB_denoise(int, rtengine::Imagefloat*, rtengine::Imagefloat*, rtengine::Imagefloat*, float*, float*, float*, bool, rtengine::procparams::DirPyrDenoiseParams const&, double, rtengine::NoiseCurve const&, rtengine::NoiseCurve const&, float&, float&) [clone ._omp_fn.3] - with the command line and image given below, and RawTherapee 5.8 from tarball, it crashes when dereferencing lumcalc, which is a null pointer. Similar code is around l. 1000.

This happens on FreeBSD 12.1 on amd64 or i386, traces below from:
FreeBSD 121amd64-svn 12.1-RELEASE-p4 FreeBSD 12.1-RELEASE-p4 amd64
gcc9 (FreeBSD Ports Collection) 9.3.0

effective compiler command line from cmake/ninja:

[00:01:34] [61/284] /usr/local/bin/ccache /usr/local/bin/g++9  -DAUTO_GDK_FLUSH=0 -DMYFILE_MMAP -DNDEBUG -DSTRICT_MUTEX=1 -DTRACE_MYRWMUTEX=0 -D_DNDEBUG -Irtengine -I/usr/local/include/glib-2.0 -I/usr/local/lib/glib-2.0/include -I/usr/local/include/glibmm-2.4 -I/usr/local/lib/glibmm-2.4/include -I/usr/local/include/sigc++-2.0 -I/usr/local/lib/sigc++-2.0/include -I/usr/local/include/gtkmm-3.0 -I/usr/local/lib/gtkmm-3.0/include -I/usr/local/include/atkmm-1.6 -I/usr/local/include/atk-1.0 -I/usr/local/include/giomm-2.4 -I/usr/local/lib/giomm-2.4/include -I/usr/local/include/pangomm-1.4 -I/usr/local/lib/pangomm-1.4/include -I/usr/local/include/cairomm-1.0 -I/usr/local/lib/cairomm-1.0/include -I/usr/local/include/cairo -I/usr/local/include/pixman-1 -I/usr/local/include/freetype2 -I/usr/local/include/libdrm -I/usr/local/include/libpng16 -I/usr/local/include/pango-1.0 -I/usr/local/include/fribidi -I/usr/local/include/harfbuzz -I/usr/local/include/gtk-3.0 -I/usr/local/include/gdk-pixbuf-2.0 -I/usr/local/include/gio-unix-2.0 -I/usr/local/include/libepoll-shim -I/usr/local/lib/libffi-3.2.1/include -I/usr/local/include/at-spi2-atk/2.0 -I/usr/local/include/at-spi-2.0 -I/usr/local/include/dbus-1.0 -I/usr/local/lib/dbus-1.0/include -I/usr/local/include/gtk-3.0/unix-print -I/usr/local/include/gdkmm-3.0 -I/usr/local/lib/gdkmm-3.0/include -I/usr/local/include/libiptcdata -I/usr/local/include/lensfun -I/usr/local/include/librsvg-2.0 -pipe  -I/usr/local/include -fPIC -flto=10 -ggdb3 -g -fstack-protector-strong -Wl,-rpath=/usr/local/lib/gcc9  -nostdinc++ -isystem /usr/include/c++/v1 -Wl,-rpath=/usr/local/lib/gcc9 -isystem /usr/local/include -std=c++11 -mtune=generic -Werror=unused-label -Werror=delete-incomplete -Wall -Wuninitialized -Wcast-qual -Wno-deprecated-declarations -Wno-unused-result -fopenmp -Werror=unknown-pragmas   -ftree-vectorize -MD -MT rtengine/CMakeFiles/rtengine.dir/FTblockDN.cc.o -MF rtengine/CMakeFiles/rtengine.dir/FTblockDN.cc.o.d -o rtengine/CMakeFiles/rtengine.dir/FTblockDN.cc.o -c /wrkdirs/usr/ports/graphics/rawtherapee/work/rawtherapee-5.8/rtengine/FTblockDN.cc

Backtrace:

(gdb) bt
#0  0x000000000077e165 in rtengine::ImProcFunctions::_ZN8rtengine15ImProcFunctions11RGB_denoiseEiPNS_10ImagefloatES2_S2_PfS3_S3_bRKNS_10procparams19DirPyrDenoiseParamsEdRKNS_10NoiseCurveESA_RfSB_._omp_fn.3(void) () at /wrkdirs/usr/ports/graphics/rawtherapee/work/rawtherapee-5.8/rtengine/FTblockDN.cc:964
#1  0x0000000801d13192 in GOMP_parallel () at /usr/local/lib/gcc9/libgomp.so.1
#2  0x000000000077d9b5 in rtengine::ImProcFunctions::_ZN8rtengine15ImProcFunctions11RGB_denoiseEiPNS_10ImagefloatES2_S2_PfS3_S3_bRKNS_10procparams19DirPyrDenoiseParamsEdRKNS_10NoiseCurveESA_RfSB_._omp_fn.2(void) () at /wrkdirs/usr/ports/graphics/rawtherapee/work/rawtherapee-5.8/rtengine/FTblockDN.cc:941
#3  0x0000000801d1bfbd in  () at /usr/local/lib/gcc9/libgomp.so.1
#4  0x000000080250a736 in  () at /lib/libthr.so.3

Variables:

(gdb) info locals
j1 = 0
R_ = 46.3144913
G_ = 12.7827358
B_ = 13.8042622
j = 860
i1 = 0
i = 0
src = 0x1ef5b00
useNoiseCCurve = false
useNoiseLCurve = false
lumcalc = 0x0
ccalc = 0x0
noisevarL = 0
gam = 1.70000005
gamcurve = {maxs = 65534, maxsf = 65534, data = 0xe010000, clip = 1, size = 65536, upperBound = 65535, owner = 1, maxsv = {65534, 65534, 65534, 65534}, sizev = {65535, 65535, 65535, 65535}, sizeiv = {281470681808895, 281470681808895}, dirty = true}
gain = 2.46472931
numTries = 2
wpfast = {{0.827291906, 0.140211254, 0.0325175263}, {0.288040191, 0.711874127, 8.56999977e-05}, {0, 0, 1.00037587}}
noisevarlum = 0xe8ee000
noisevarchrom = 0xe9d0000
tiletop = 0
tileleft = 860
tileright = 1848
tilebottom = 932
width = 988
width2 = 494
labdn = 0x1698f60
maxNoiseVarab = 9.60698986

Code location:

959                                         if (((i1 | j1) & 1) == 0) {
960                                             if (numTries == 1) {
961                                                 noisevarlum[(i1 >> 1) * width2 + (j1 >> 1)] = useNoiseLCurve ? lumcalc[i >> 1][j >> 1] : noisevarL;
962                                                 noisevarchrom[(i1 >> 1) * width2 + (j1 >> 1)] = useNoiseCCurve ? maxNoiseVarab * ccalc[i >> 1][j >> 1] : 1.f;
963                                             } else {
964  >>                                             noisevarlum[(i1 >> 1) * width2 + (j1 >> 1)] = lumcalc[i >> 1][j >> 1];
965                                                 noisevarchrom[(i1 >> 1) * width2 + (j1 >> 1)] = ccalc[i >> 1][j >> 1];
966                                             }
967                                         }

Screen trace:

[00:05:26] + /usr/bin/time /usr/bin/env 'LANG=' 'LC_ALL=C' 'LANGUAGE=' 'DISPLAY=' 'HOME=/tmp/rawtherapee.hqLafWDs' 'RT_SETTINGS=/wrkdirs/usr/ports/graphics/rawtherapee/work/rt-config' 'RT_CACHE=/tmp/rawtherapee.hqLafWDs/rtcache' /wrkdirs/usr/ports/graphics/rawtherapee/work/stage/usr/local/bin/rawtherapee-cli -o /tmp/rawtherapee.hqLafWDs -q -s -Y -c /usr/ports/graphics/rawtherapee/files/../testimages
[00:05:26] Error: the user's processing profile path doesn't point to a directory or doesn't exist!
[00:05:26] 
[00:05:26] Could not open camera constants file "/usr/local/share/rawtherapee/camconst.json": No such file or directory
[00:05:26] 
[00:05:26] The bundled profile "${G}/Auto-Matched Curve - ISO Low" could not be found!
[00:05:26] Your installation could be damaged.
[00:05:26] Default internal values will be used instead.
[00:05:26] 
[00:05:26] RawTherapee, version 5.8, command line.
[00:05:26] "/usr/ports/graphics/rawtherapee/files/../testimages/_DSC3041.ARW.pp3"  is not one of the selected parsed extensions. Image skipped.
[00:05:26] "/usr/ports/graphics/rawtherapee/files/../testimages/DSC01029.ARW.pp3"  is not one of the selected parsed extensions. Image skipped.
[00:05:26] Output is 8-bit integer.
[00:05:26] Processing: /usr/ports/graphics/rawtherapee/files/../testimages/_DSC3041.ARW
[00:05:26]   Merging sidecar procparams.
[00:05:31] time: command terminated abnormally
[00:05:31]         5.29 real        50.19 user         8.89 sys
[00:05:31] Segmentation fault
[00:05:31] + echo 139

[mandree]

Perhaps I should add that the code to set up the directories I am referencing lives in the post-install: target here: https://svnweb.freebsd.org/ports/head/graphics/rawtherapee/Makefile?revision=538522&view=markup#l168

[heckflosse]

Thanks for reporting. Bug confirmed.

[mandree]

FreeBSD's rawtherapee 5.8_4 port and package are shipping this patch as cherry-pick (both on head/ as well as on the 2020Q2 quarterly branch)

[heckflosse]

@mandree I also fixed another one. Sorry for not mentioning the fix in this issue.
https://github.com/Beep6581/RawTherapee/commit/6c1055f985a202e1b1a87f8d33b482748f41a167

[mandree]

@heckflosse thanks for the heads-up. Now preparing rawtherapee-5.8_5 for FreeBSD...

[mandree]

Oh, and patch wasn't really helpful in combining these two patches that were so similar in content... had to do it manually, sequential patching or combinediff both failed.

[mandree]

It's in FreeBSD 5.8_5, as of r538813 (head/) and r538814 (quarterly branches/2020Q2) ports, packages to follow with next rebuild turn.