Rawtherapee: It takes 4 minutes to start RT (due to regex call?)

[kuromimi]

Hello,
when I ran RawTherapee_5.2 on my Win7 environment, RT didnot start immediately, it took 4 min to start. Here is summary.
Environment:

• Windows7 Pro SP1-64bit [email protected] 11GB memory
• RawTherapee_5.2_WinVista_64.exe downloaded from http://rawtherapee.com/downloads

Symptom:
After running Rawtherapee.exe, no screen comes up and RT starts allocating and freeing 1-2GB of memory repeatedly.

4 minutes later, all of suddon RT application window is displayed, and everything starts working (I could browse/edit photos). I tried RT 5.1, but the symptom still occurred. Tried RT 5.0-r1-gtk3, it worked without the delay.

What I tried:
I was able to build RT binary from source (based on the detailed instruction, thanks!), and checked what is going on inside RT. Here is build version info (I used CMAKE_BUILD_TYPE="debug"):

-- Git checkout information:
--      Commit description:     5.2-95-gae9afd3
--      Branch:                 dev
--      Commit:                 ae9afd3
--      Commit date:            2017-09-03
--      Commits since tag:      95
--      Commits since branch:   95
--      Version (unreliable):   5.2.95

I figured out RT got stuck right after calling line#33:

rtgui/xtransprocess.cc
32:    for( size_t i = 0; i < procparams::RAWParams::XTransSensor::numMethods; i++) {
33:        static const std::regex what ("[() -]");
34:        const std::string langKey = std::regex_replace (procparams::RAWParams::XTransSensor::methodstring[i], what, "");

This is gdb's console log at the delayed case:

$ gdb ./rawtherapee
GNU gdb (GDB) 7.12
Copyright (C) 2016 Free Software Foundation, Inc.
 :
Reading symbols from ./rawtherapee...done.
Breakpoint 1 at 0x756580: xtransprocess.cc:33. (3 locations)
(gdb) r
Starting program: C:\msys64\home\ksh\repo-rt\build\debug\rawtherapee.exe
[New Thread 904.0x9cc]
[New Thread 904.0x470]
[New Thread 904.0xdfc]
[New Thread 904.0x11a0]
[New Thread 904.0xbfc]
[New Thread 904.0xd98]
[New Thread 904.0x92c]
[New Thread 904.0x10d4]
[New Thread 904.0x894]
[New Thread 904.0x4cc]
[New Thread 904.0x540]
[New Thread 904.0x1150]
[New Thread 904.0x858]
[New Thread 904.0x117c]
[New Thread 904.0x3b4]
[Thread 904.0x117c exited with code 0]
[Thread 904.0x3b4 exited with code 0]
[New Thread 904.0x124c]
[New Thread 904.0x8c8]
[New Thread 904.0x478]
[Thread 904.0x8c8 exited with code 0]
[New Thread 904.0x8c]
[New Thread 904.0x794]
[New Thread 904.0x480]
[New Thread 904.0xeb8]
[New Thread 904.0xf5c]
[New Thread 904.0x137c]
[New Thread 904.0x10b0]
[New Thread 904.0x13b0]
[New Thread 904.0x10ec]
[Thread 904.0x10ec exited with code 0]
[Thread 904.0x137c exited with code 0]
[Thread 904.0x13b0 exited with code 0]
[Thread 904.0x478 exited with code 0]
[Thread 904.0xf5c exited with code 0]
[Thread 904.0x8c exited with code 0]
[Thread 904.0x794 exited with code 0]
[Thread 904.0x480 exited with code 0]
[Thread 904.0xeb8 exited with code 0]
[New Thread 904.0xc24]
[New Thread 904.0x49c]
[New Thread 904.0xa6c]
[New Thread 904.0x10c8]
[New Thread 904.0xe4]
[New Thread 904.0x1304]
[New Thread 904.0x1338]
[New Thread 904.0xc38]
[New Thread 904.0xecc]

Thread 1 hit Breakpoint 1, XTransProcess::XTransProcess (this=0x103b3ca0,
    __in_chrg=<optimized out>, __vtt_parm=<optimized out>)
    at C:/msys64/home/ksh/repo-rt/rtgui/xtransprocess.cc:33
33              static const std::regex what ("[() -]");
(gdb) bt
#0  XTransProcess::XTransProcess (this=0x103b3ca0, __in_chrg=<optimized out>,
    __vtt_parm=<optimized out>)
    at C:/msys64/home/ksh/repo-rt/rtgui/xtransprocess.cc:33
#1  0x0000000000705a93 in ToolPanelCoordinator::ToolPanelCoordinator (
    this=0x9707090) at C:/msys64/home/ksh/repo-rt/rtgui/toolpanelcoord.cc:83
#2  0x000000000041788e in BatchToolPanelCoordinator::BatchToolPanelCoordinator
    (this=0x9707090, parent=0x9691600)
    at C:/msys64/home/ksh/repo-rt/rtgui/batchtoolpanelcoord.cc:28
#3  0x000000000054a23e in FilePanel::FilePanel (this=0x9691600,
    __in_chrg=<optimized out>, __vtt_parm=<optimized out>)
    at C:/msys64/home/ksh/repo-rt/rtgui/filepanel.cc:55
#4  0x00000000006bb5b7 in RTWindow::RTWindow (this=0x9458b20,
    __in_chrg=<optimized out>, __vtt_parm=<optimized out>)
    at C:/msys64/home/ksh/repo-rt/rtgui/rtwindow.cc:178
#5  0x00000000005ce03d in (anonymous namespace)::create_rt_window ()
    at C:/msys64/home/ksh/repo-rt/rtgui/main.cc:353
#6  0x00000000005cfeaf in main (argc=1, argv=0x67a5080)
    at C:/msys64/home/ksh/repo-rt/rtgui/main.cc:662
(gdb) n
[Thread 904.0x1338 exited with code 0]
[Thread 904.0xc38 exited with code 0]
[Thread 904.0xa6c exited with code 0]
[Thread 904.0xe4 exited with code 0]
[Thread 904.0x10c8 exited with code 0]
[Thread 904.0x10b0 exited with code 0]
[Thread 904.0x1304 exited with code 0]
[Thread 904.0xc24 exited with code 0]
[Thread 904.0x49c exited with code 0]
warning: Invalid parameter passed to C runtime function.
warning: Invalid parameter passed to C runtime function.
[New Thread 904.0x4ac]
warning: Invalid parameter passed to C runtime function.
warning: Invalid parameter passed to C runtime function.
warning: Invalid parameter passed to C runtime function.
warning: Invalid parameter passed to C runtime function.
warning: Invalid parameter passed to C runtime function.
 :
(The same message displayed repeatedly)

Just an experiment I modified the code as below, and run RT.

--- xtransprocess.cc.org        2017-09-04 17:12:45.491743200 +0900
+++ xtransprocess.cc    2017-09-04 17:13:02.448973000 +0900
@@ -30,7 +30,7 @@ XTransProcess::XTransProcess () : Foldab
     method = Gtk::manage (new MyComboBoxText ());

     for( size_t i = 0; i < procparams::RAWParams::XTransSensor::numMethods; i++) {
-        static const std::regex what ("[() -]");
+        static const std::regex what ("aaaaa");
         const std::string langKey = std::regex_replace (procparams::RAWParams::XTransSensor::methodstring[i], what, "");
         method->append(M("TP_RAW_" + Glib::ustring(langKey).uppercase()));
     }

Now gdb could step over line#33.

 :
[New Thread 5024.0x1384]
[New Thread 5024.0x9d0]
[New Thread 5024.0x160]
[New Thread 5024.0x13d4]

Thread 1 hit Breakpoint 1, XTransProcess::XTransProcess (this=0x10294c50,
    __in_chrg=<optimized out>, __vtt_parm=<optimized out>)
    at C:/msys64/home/ksh/repo-rt/rtgui/xtransprocess.cc:33
33              static const std::regex what ("aaaaa");
(gdb) n
[Thread 5024.0x9d0 exited with code 0]
[Thread 5024.0x160 exited with code 0]
34              const std::string langKey = std::regex_replace (procparams::RAWParams::XTransSensor::methodstring[i], what, "");
(gdb) n
[Thread 5024.0x133c exited with code 0]
[Thread 5024.0x1384 exited with code 0]
35              method->append(M("TP_RAW_" + Glib::ustring(langKey).uppercase()));
(gdb)

Actually there is one more regex call in rtgui/rtwindow.cc ; I modified the call like this:

--- rtwindow.cc.org     2017-09-04 17:38:47.608520900 +0900
+++ rtwindow.cc 2017-09-04 17:37:57.002032000 +0900
@@ -316,7 +316,7 @@ void RTWindow::on_realize ()
     // Display release notes only if new major version.
     // Pattern matches "5.1" from "5.1-23-g12345678"
     std::string vs[] = {versionString, options.version};
-    std::regex pat ("(^[0-9.]+).*");
+    std::regex pat ("1122222");
     std::smatch sm;
     std::vector<std::string> vMajor;

After those two modifications, RT started working immediately. regex is an external library not directly related to RT, but I think this can be a potential bug and maybe someone is encountering the same issue; posting here.
Please let me know if I need to post some other info like msys2 package info, etc.

[Floessie]

Hi @kuromimi,

Your bug report is a delight to read. Very well done! 👍

regex is an external library not directly related to RT, but I think this can be a potential bug and maybe someone is encountering the same issue

I can only agree with you: std::regex is part of the STL and may use another library under the hood. Strange enough, you're the first to report that issue. I'm pretty confident, it relates to your locale setting and the fact, that you're on Windows, which might not use UTF-8. Character ranges and character classes are locale-aware, so removing them from the regex healed it.

I'd suggest, you experiment a bit with a radically stripped down test program (just a main() which uses std::regex) to see what helps. If there's no working solution with std::regex, I'd implement a different solution for you (which is easy in both cases). Okay?

Best,
Flössie

[Floessie]

@kuromimi Here's the patch:

diff --git a/rtgui/rtwindow.cc b/rtgui/rtwindow.cc
index 9af387567..80e481315 100644
--- a/rtgui/rtwindow.cc
+++ b/rtgui/rtwindow.cc
@@ -18,7 +18,6 @@
  */

 #include <gtkmm.h>
-#include <regex>
 #include "rtwindow.h"
 #include "options.h"
 #include "preferences.h"
@@ -315,37 +314,27 @@ void RTWindow::on_realize ()

     // Display release notes only if new major version.
     // Pattern matches "5.1" from "5.1-23-g12345678"
-    std::string vs[] = {versionString, options.version};
-    std::regex pat ("(^[0-9.]+).*");
-    std::smatch sm;
+    const std::string vs[] = {versionString, options.version};
     std::vector<std::string> vMajor;

-    for (const auto &v : vs) {
-        if (std::regex_match (v, sm, pat)) {
-            if (sm.size() == 2) {
-                std::ssub_match smsub = sm[1];
-                vMajor.push_back (smsub.str());
-            }
-        }
+    for (const auto& v : vs) {
+        vMajor.emplace_back(v, 0, v.find_first_not_of("0123456789."));
     }

-    if (vMajor.size() == 2) {
-        if (vMajor[0] != vMajor[1]) {
-
-            // Update the version parameter with the right value
-            options.version = versionString;
+    if (vMajor.size() == 2 && vMajor[0] != vMajor[1]) {
+        // Update the version parameter with the right value
+        options.version = versionString;

-            splash = new Splash (*this);
-            splash->set_transient_for (*this);
-            splash->signal_delete_event().connect ( sigc::mem_fun (*this, &RTWindow::splashClosed) );
+        splash = new Splash (*this);
+        splash->set_transient_for (*this);
+        splash->signal_delete_event().connect ( sigc::mem_fun (*this, &RTWindow::splashClosed) );

-            if (splash->hasReleaseNotes()) {
-                splash->showReleaseNotes();
-                splash->show ();
-            } else {
-                delete splash;
-                splash = nullptr;
-            }
+        if (splash->hasReleaseNotes()) {
+            splash->showReleaseNotes();
+            splash->show ();
+        } else {
+            delete splash;
+            splash = nullptr;
         }
     }
 }
diff --git a/rtgui/xtransprocess.cc b/rtgui/xtransprocess.cc
index a663ac7c5..87db6ff17 100644
--- a/rtgui/xtransprocess.cc
+++ b/rtgui/xtransprocess.cc
@@ -19,7 +19,7 @@
 #include "xtransprocess.h"
 #include "options.h"
 #include "guiutils.h"
-#include <regex>
+
 using namespace rtengine;
 using namespace rtengine::procparams;

@@ -30,8 +30,29 @@ XTransProcess::XTransProcess () : FoldableToolPanel(this, "xtransprocess", M("TP
     method = Gtk::manage (new MyComboBoxText ());

     for( size_t i = 0; i < procparams::RAWParams::XTransSensor::numMethods; i++) {
-        static const std::regex what ("[() -]");
-        const std::string langKey = std::regex_replace (procparams::RAWParams::XTransSensor::methodstring[i], what, "");
+        const std::string langKey = [i]() -> std::string
+            {
+                const std::string str(procparams::RAWParams::XTransSensor::methodstring[i]);
+
+                std::string res;
+                for (const auto& c : str) {
+                    switch (c) {
+                        case '(':
+                        case ')':
+                        case ' ':
+                        case '-': {
+                            continue;
+                        }
+
+                        default: {
+                            res += c;
+                            break;
+                        }
+                    }
+                }
+
+                return res;
+            }();
         method->append(M("TP_RAW_" + Glib::ustring(langKey).uppercase()));
     }

HTH,
Flössie

[kuromimi]

Flössie,
Thank you for the comment and the patch! I tried the patch, NOW RT WORKED without delay!!
I agree that the cause can be the difference of locale on Windows (mine is Japanese version). Hope this topic helps others.
Just FYI, I tried a simple program using std::regex (on MSYS2), it worked without any error message or hang.

#include <iostream>
#include <regex>
int main()
{
        static const std::regex what ("[() -]");
        std::cout << regex_match("abcd", what);
        return 0;
}

I am new to this project. Should I close the topic? (by pressing "Close and comment" button), or should I wait for the commit of the patch?
Anyway, thank you again for your keen insight about the cause and patch!!

Thanks,
kuromimi

[Floessie]

@kuromimi

Just FYI, I tried a simple program using std::regex (on MSYS2), it worked without any error message or hang.

Well, that's indeed interesting. Maybe @heckflosse has a clue, which precondition we're missing. It could be the way the locale is globally set in RT and/or GTK3...

I am new to this project. Should I close the topic? (by pressing "Close and comment" button), or should I wait for the commit of the patch?

Please keep it open. We'll close it when there's a consensus on how to proceed.

@Beep6581 From my POV the patch is beneficial, as the std::regex was a bit like cracking a nut with a sledgehammer, but I could sleep better, if we find the real cause of the problem. Apart from that, there are places using Glib::regex which don't seem to have the same problems.

Best,
Flössie

[Floessie]

@kuromimi

Could you add a setlocale(LC_ALL, ""); as the first line to your main() (you'll also need to include <locale.h>) and test again?

[kuromimi]

Flössie,
OK I will keep opening the bug, thanks.
Tried your change. The sample got hung. Here is gdb log:

(gdb) r
Starting program: C:\msys64\home\ksh\a.exe
[New Thread 1276.0x960]
warning: Invalid parameter passed to C runtime function.
warning: Invalid parameter passed to C runtime function.
warning: Invalid parameter passed to C runtime function.
warning: Invalid parameter passed to C runtime function.
warning: Invalid parameter passed to C runtime function.
warning: Invalid parameter passed to C runtime function.

[Floessie]

@kuromimi Okay, so you're able to reproduce the problem with a minimal example. 👍 Next up: File an issue over at msys2, as it needs to be solved there. Please keep us informed about the status (simply paste a link to the msys2 issue here and ping us, when they've found something), so that we'll know if there's something we should do differently in RT.

Best,
Flössie

[kuromimi]

@Flössie
OK, filed an issue as https://github.com/msys2/msys2/issues/80
Will post the status.

Thanks,
kuromimi

[Beep6581]

I wish all bug reports were like this... Good work @kuromimi and @Floessie :)

[Floessie]

@kuromimi I've pushed the patch. I hope the msys people will solve your problem. You couldn't have helped them better. 👍

[kuromimi]

Flössie,
Thank you, seems it takes time to solve the regex issue on msys2; not using regex in RT should be a practical way so far (I will keep tracking the msys2 bug).
I checked the Twitter's tweets, a couple of RT users in Japan are encountering this not-starting-RT symptom since May 2017, after RT5.1 release. That is great misfortune they cannot use this really-really useful software. I hope we have a new release with this fix shortly.

[Floessie]

@kuromimi

I checked the Twitter's tweets, a couple of RT users in Japan are encountering this not-starting-RT symptom since May 2017, after RT5.1 release. That is great misfortune they cannot use this really-really useful software. I hope we have a new release with this fix shortly.

The point in time is strange, as the regex was applied only after the 5.1 release (in June, see #3905). As to the release plans, see #3989. Shouldn't be too far away...

Best,
Flössie

[kuromimi]

Hmm, it is strange. However, it is true exactly the same symptom occurred to the official RT5.1, too (I just verified). Some other users in Japan also reported the same symptom on 5.1.
I can find "[() -]" regex pattern in 5.1's rawtherapee.exe binary. There might be some misconception.

[Floessie]

@kuromimi Yep, you're right. I only looked at the first occurrence, but the one in xtransprocess.cc you are referring to is six month old. Puzzle solved. 😉

[iliasg]

I had the same slow start problem (more than 5mins and during this, the PC looked frozen) with my PC (win10-64 - Greek language) which is now solved with the https://github.com/Beep6581/RawTherapee/commit/099e6e9f678c46b3fd047a3b833f25e12b156445 . The problem started at some point after RawTherapee dev 5.0-r1-gtk3-111-g06137b02 (last version that I tested to work normally.. )
(*edit: corrected version *)

Many many thanks.

[RuralHunter]

Thanks for the bug report! I have the same problem on Win10 Chinese version and I had no clue!

[gaaned92]

@RuralHunter
while waiting for stable 5.3, you can find RawTherapee_dev_5.2-113-gd1fe99f8_WinVista_64.zip at https://drive.google.com/open?id=0B2q9OrgyDEfPS2FpdDAtMVI1RG8

[RuralHunter]

This build works for me. Thanks a lot!

[hyyz17200]

@gaaned92 Dev build works!