Rawtherapee: Development version: Problems using queue -> segmentation fault.

[jade-nl]

After editing this image posted on pixls.us I put it in the queue and activated it. This results in a segmentation fault. Restarting doesn't help, it keeps seg-faulting.

No other messages in the terminal besides: Segmentation fault

pp3 for the above: foggy.woods.cr2.pp3.zip

After deactivating the CIE Colour Appearance Model 2002 (and activating Contrast by detail.. to finish the edit) I was able to process the image but this did show up in the terminal:

Premature end of JPEG file
JPEG datastream contains no image

pp3 for this situation: foggy.woods.cr2.pp3.zip

A jpeg was produced an I could not see any problems with it, result is posted in the pixl link posted above.

Not sure if this might be relevant: Although this situation is mentioned in the tooltip, it is the first time I encountered the demosaicing to end op with a 0 (zero) Contrast threshold.

Version

• 5.8-2660-gf7f527d95

Platform

• OS : Linux (kernel 5.8.10-1)
• Distro : Debian Buster (rel. 10.6)

Additional context

• Self compiled : Yes.
• Compiler : gcc 8.3.0
• Cmake : 3.13.4

I did build with relwithdebinfo and WITH_PROF="ON", no dump was created, but I do have a gmon.out file that might be useful: gmon.out.zip

Just ask for info that I did not mentioned and might help.

[heckflosse]

Can not reproduce the crash with

Version: 5.8-2655-gccf1cc0a2
Branch: dev
Commit: ccf1cc0a2
Commit date: 2020-11-10
Compiler: gcc 10.2.0
Processor: undefined
System: Windows
Bit depth: 64 bits
Gtkmm: V3.24.2
Lensfun: V0.3.2.0
Build type: Release
Build flags: -Wno-parentheses -std=c++11 -march=native -Werror=unused-label -Werror=delete-incomplete -fno-math-errno -Wall -Wuninitialized -Wcast-qual -Wno-deprecated-declarations -Wno-unused-result -Wunused-macros -fopenmp -Werror=unknown-pragmas -O3 -DNDEBUG -ftree-vectorize
Link flags:  -march=native
OpenMP support: ON
MMAP support: ON

Not sure if this might be relevant: Although this situation is mentioned in the tooltip, it is the first time I encountered the demosaicing to end op with a 0 (zero) Contrast threshold.

RT searches for a flat non overexposed area to calculate the contrast threshold. If it finds none, it will set the contrast threshold to zero.

[jade-nl]

I just rebuild RT with build type debug instead of relwithdebinfo and running the process queue while replying to you.

This takes very, very long....

• Starting time RawTherapee is decent but noticeably slower.
• Loading the image in question takes very long. About 30-45 seconds per individual edit action. Total load time for this image: about 5 1/2 minutes
• The process queue has been running for close to 15 minutes now.

All cores are being utilized and this is on a AMD Ryzen 5 3600 6-Core Processor (12 hyper-threaded) and 32 GB

Damn! The image was just processed (17 mins) _without seg-faulting_ (I do get the premature end... messages in the terminal).

Just rebuild with build type release: Also _does not seg-fault_.

And back to relwithdebinfo: And this one _does_ crash!

OK: Not sure if this is something that needs to be looked at or not. A normal release build works. Feel free to close this if no action is wanted/needed.

PS: @heckflosse: Thanks for the demosaic explanation!

[heckflosse]

(I do get the premature end... messages in the terminal)

Do you export your jpeg to the same folder as your raw resides?

[Thanatomanic]

FWIF I cannot reproduce on Windows and the resulting JPEG looks fine, also when exported to the same folder.

[jade-nl]

@heckflosse : Yes, in the same directory, as I almost always do. I did remove the previous generated jpegs when I retested this with the different build types.

@Thanatomanic : You're on a relwithdebinfo build? That's the only build type that triggers the seg-fault, the other 2 work fine (not counting the terminal messages).

[heckflosse]

@jade-nl Then the premature end messages occur because RT has opened this folder and periodically scans it for new files. If a file is not written completly, you will get this message in terminal

[jade-nl]

OK, thanks for clearing that up and that makes it a non-issue in relation to the relwithdebinfo thingy.

[Floessie]

@jade-nl Since you're on Debian and self-compiling, would you mind to run RT in GDB? RawPedia has a nice HOWTO to get you started. This will give you debugging super-powers and will eventually help us to trace down the bug.

HTH,
Flössie

[Thanatomanic]

You're on a _relwithdebinfo_ build? That's the only build type that triggers the seg-fault, the other 2 work fine (not counting the terminal messages).

@jade-nl Yes, I also tried with a relwithdebinfo build and it runs and exports normally and without any suspicious delays or console messages. Similarly, running through gdb gives no seg fault.
Edit: so I'm inclined to say this is either a Linux specific thing, or related to using LTO.

[jade-nl]

@Floessie - Find the log file from gdb at the bottom. But first this piece of info that might also help:

I rebuild RawTherapee with relwithdebinfo and could initially not reproduce the seg-fault. I remembered also setting WITH_PROF back to the default (OFF). If I turn that one back ON, it generates a seg-fault.

So: relwithdebinfo + WITH_PROF=ON -> seg-fault. Haven't found another combo that seg-faults.

rt.segfault.log.txt

[Floessie]

@jade-nl Of all things, the backtrace of thread 288 is missing in your log. :slightly_frowning_face:

[jade-nl]

Mmmm....

OK. Just to make sure that it isn't something on my end of things.

• Forced a clean clone (we're still on 5.8-2660-gf7f527d95),
• Rebuild with -DCMAKE_BUILD_TYPE="relwithdebinfo" and -DWITH_PROF="ON",
• Ran RT using gdb as described on RawPedia,
• Lot of stuff in the gbd environment, last message being Segmentation fault

Here's the resulting log file: rt.segfault.log.txt

One thing that might be worth mentioning: gdb quits and RT disappears after the
thread apply all bt full step in gdb. No need/way to use q and y to exit cleanly. Maybe this is normal, don't have that much experience with running software from within gdb.

[Thanatomanic]

Hmm gdb shouldn't just die on you... Not sure what is going on.

[Floessie]

@jade-nl That's strange indeed. My first thought was the OOM killer, but with 32GB that's unlikely unless GDB is tricked into a recursion.

[jade-nl]

I moved the set xyz parts before running rawtherapee, did a little digging and it seems that the default max-value-size is too small (65535) for this debug session, set it to unlimited with set max-value-size unlimited. Still no cigar, though. gdb still quits/dies.

Serious question: How important is this seg-fault issue?

You do need a specific combo to trigger it and I'm not even sure if WITH_PROF="ON" is of any value for debugging purposes. And it does not trigger a seg-fault when using debug instead of relwithdebinfo.

I do understand that developers want to know why this happens, but is it a good idea to spend much time on this? The fast majority of people will, unknowingly, run with _release_ and _with_prof=off_ and that combo does not trigger this.

I don't mind doing more digging/tests on this one if you want/need me to, but I do need to be pointed in the right direction to make any progress.

EDIT: It is also specific to this image. Haven't had any problems with other images.

[Floessie]

Tested on Debian Testing AMD64: Takes some time but no crash.

[Floessie]

Tested on Debian Buster AMD64: No crash.

[jade-nl]

@Floessie / @Thanatomanic :

I suggest to let this be for the moment seen from your side of things. This looks more and more to be a fringe case that might even be a local problem on my side of things. However strange though that it only happens with this one specific image combined with one specific combo of build options.

I'll do some more tests on my side, including setting up my laptop and give it a try on that platform. Might take a bit of time though, but I'll report back if I have news. I'll leave this issue open for now.

Thanks for the invested time (and patience) up to now!

[Thanatomanic]

Hi @jade-nl, this one will certainly be hard to pin down. I just compiled RT with LTO in a virtual machine with Linux Mint 20 and had no trouble exporting the image.

[Thanatomanic]

fwiw

Version: 5.8-2660-gf7f527d95
Branch: dev
Commit: f7f527d95
Commit date: 2020-11-12
Compiler: cc 9.3.0
Processor: x86_64
System: Linux
Bit depth: 64 bits
Gtkmm: V3.24.2
Lensfun: V0.3.2.0
Build type: Release
Build flags: -std=c++11 -march=native -Werror=unused-label -Werror=delete-incomplete -fno-math-errno -flto -Wall -Wun>
Link flags: -march=native -flto
OpenMP support: ON
MMAP support: ON
Build OS: Linux 5.4.0-53-generic x86_64
Build date: Mon, 16 Nov 2020 15:52:12 +0000 UTC
Build epoch: 1605541932
Build UUID: 1964624a-652e-456e-ae09-55f9a339475c

[jade-nl]

@Thanatomanic :

I see you build with Release, which won't trigger the seg-fault, and not relwithdebinfo.

I'm going to assume that the other info about the platform is what you want to show.

Your compiler is newer (9.3 vs 8.3), my kernel is newer (5.8 vs 5.4). There is the gtk issue: I'm on 3.24.5 which is known to cause problems in certain cases (drop down menu related only if I'm not mistaken), you are on a safe version.

Anyway, like stated before; I'm going to do some more testing when time permits.

EDIT: Found where you got that info, here's mine:

Version: 5.8-2660-gf7f527d95
Branch: dev
Commit: f7f527d95
Commit date: 2020-11-12
Compiler: cc 8.3.0
Processor: x86_64
System: Linux
Bit depth: 64 bits
Gtkmm: V3.24.0
Lensfun: V0.3.2.0
Build type: relwithdebinfo
Build flags: -std=c++11 -march=native -Werror=unused-label -Werror=delete-incomplete -fno-math-errno -flto -pg -Wall -Wuninitialized -Wcast-qual -Wno-deprecated-declarations -Wno-unused-result -Wunused-macros -fopenmp -Werror=unknown-pragmas -O2 -g -DNDEBUG -ftree-vectorize
Link flags: -march=native -flto -pg
OpenMP support: ON
MMAP support: ON
Build OS: Linux 5.8.0-0.bpo.2-amd64 x86_64
Build date: Mon, 16 Nov 2020 17:12:03 +0000 UTC
Build epoch: 1605546723
Build UUID:

[Thanatomanic]

@jade-nl You're right, I forgot about the relwithdebinfo thing again 😄
In any case, also that build does not crash for me:

Version: 5.8-2660-gf7f527d95
Branch: dev
Commit: f7f527d95
Commit date: 2020-11-12
Compiler: cc 9.3.0
Processor: x86_64
System: Linux
Bit depth: 64 bits
Gtkmm: V3.24.2
Lensfun: V0.3.2.0
Build type: relwithdebinfo
Build flags: -std=c++11 -march=native -Werror=unused-label -Werror=delete-incomplete -fno-math-errno -flto -Wall -Wuniniti
alized -Wcast-qual -Wno-deprecated-declarations -Wno-unused-result -Wunused-macros -fopenmp -Werror=unknown-pragmas -O2 -g
-DNDEBUG -ftree-vectorize
Link flags: -march=native -flto
OpenMP support: ON
MMAP support: ON
Build OS: Linux 5.4.0-53-generic x86_64
Build date: Mon, 16 Nov 2020 17:11:28 +0000 UTC
Build epoch: 1605546688
Build UUID: 7b17c305-5d83-47fc-95cb-3b5b9e5c3c96

[jade-nl]

Had to know more, so spend time on this now instead of later....

Just spend the last hour or so building RT on my laptop (4 cores, 4Gb LOL) and I am not able to reproduce it on that machine.

The hardware is different, it's an old MacBook Pro. But the software is roughly the same: a fully up-to-date Debian (10.6 / Buster). Only big difference is I'm not running any backported stuff on the laptop. I do need the latest versions for, mainly, my NVidia card on my desktop, though.

I did notice that if I build (on the desktop) with LTO disabled RT does not crash either.

Anyway: I'm going to close this issue, it is all but obvious that this is something to do with my setup and not something in RawTherapee that will affect "the masses". I found 2 ways to get around it during all this, so it won't hinder me in any way or form. No need for you guys (and gals?) to spend any more time on this one.

Thanks!