Darktable: Darktable from Master does not open / hangs

[YAFU]

My system: Kubuntu 20.04 (KDE) - GTX 960

When I try to open my CR2 files as usual, Darktable just ends up showing only a small empty window. I must kill Darktable process. The problem occurs even with empty "~ /.config/darktable/" folder.
I have built Darktable with:
./build.sh --prefix /home/yafu/.darktable_master --buildtype RelWithDebInfo

Then when I start darktable from the terminal:
./darktable -d all IMG_4537.CR2

this is part of the output I get until it hangs:
https://pastebin.com/RLWdvNnQ

I'm not sure why this makes so much reference to "/media/TEMPORAL/@@Darktable_Git/darktable/" location, which is the path where I have the sources, not the binaries.

Darktable opens correctly if I run darktable binary. The problem occurs only if I want to open image files "with" darktable. It is always reproducible. I think the problem occurred more than a month ago, I really didn't have time before making the report. The last known version that works correctly on my system is: 3.1.0+1795~g0c7527f8c

Edit:
Just in case my bad English:
when I said "I think the problem occurred more than a month ago", it was actually "I think the problem started happening for about a month, and still the problem is present in current master on my system"

[TurboGit]

Works for me on current master. This is reassuring as it probably means a build or context setup and not something broken on the release which is very close now.

[YAFU]

Just in case my bad English:
when I said "I think the problem occurred more than a month ago", it was actually "I think the problem started happening for about a month, and still the problem is present in current master on my system"

[ptilopteri]

what manner are you using to "open an image with darktable"

try to open an image with following command-line:
darktable --library :memory:

[YAFU]

I have associated image files to be opened with Darktable built by myself in KDE. Anyway the problem also occurs from the terminal with the order that I show in the first message. Also the problem is present with the order you show:

• With empty "~ /.config/darktable/" folder I get:

$ ./darktable --library :memory: IMG_4537.CR2
[defaults] found a 64-bit system with 24574004 kb ram and 8 cores (0 atom based)
[defaults] setting very high quality defaults

(darktable:3289552): GLib-CRITICAL **: 23:00:48.189: g_variant_unref: assertion 'value != NULL' failed
ERROR in lualib, couldn't find a position for `lua scripts installer', this should never happen
[image_cache_allocate] failed to open image 4294967295 from database: no more rows available

At this point only the next files are created in "~ /.config/darktable/" : data.db, data.db-pre-3.1.0 (0 B), data.db.lock, keyboardrc, keyboardrc_default

• After all files are created in "~ /.config/darktable/" by running darktable binary and then trying again:

$ ./darktable --library :memory: IMG_4537.CR2
ERROR in lualib, couldn't find a position for `lua scripts installer', this should never happen
[image_cache_allocate] failed to open image 4294967295 from database: no more rows available

(I have copied "IMG_4537.CR2" image to the same path of darktable binary just so I don't have to write the path in each test)
Darktable was hung and never finished opening in both cases from the terminal.

[ptilopteri]

@YAFU appears you have displayed your problem,
ERROR in lualib, couldn't find a position for `lua scripts installer', ...

but I cannot advise further. I believe "William Ferguson" is the LUA expert.

[YAFU]

Okay.
Another question. Doing tests I just accidentally deleted binaries of my latest functional version that I had (3.1.0+1795~g0c7527f8c)
How do I "git checkout" that version? I have tried "git checkout g0c7527f8c", but a message tells me that there is no such version.
EDIT:
Ok, I checkout by date:
https://stackoverflow.com/questions/6990484/how-to-checkout-in-git-by-date

{2020-06-26 18:30:00} -> still doesn't work.

{2020-06-20 18:30:00} -> this version worked.

[YAFU]

I have done a google search related to: ERROR in lualib, couldn't find a position for

Seems most of the time this happens in KDE?
What could be causing the problem?

This is my cmake-gui config related to LUA:

[ptilopteri]

doesn't happen to me in plasma5/kde5

[YAFU]

Ok, thanks for testing.
I have been able to shorten to one day the period where the problem started to occur:
{2020-06-22 18:30:00} -> still doesn't work.

{2020-06-21 18:30:00} -> this version worked.

What commit could be related to the problem within those 24 hours?

[ptilopteri]

close darktable
try moving /home//.config/darktable/luarc to a different filename, ie:
/home//.config/darktable/xxluarc
and test again where you see dt failing

[YAFU]

In that location I have data, library and keyboard files, but I don't have that file /home/.config/darktable/luarc
That file was never created there, even with working Darktable version.

Anyway the problem/hang occurs even deleting all files there (Empty /.config/darktable/)

[ptilopteri]

Then one more test. Remove your build of dt and install one of the precompiled packages provided at:
https://www.darktable.org/install/

if that works, there is a problem with your compile procedure or your system.

[YAFU]

I have downloaded and installed: darktable_3.3.0~git2.c94b5fda3_amd64.deb
https://software.opensuse.org/download.html?project=graphics:darktable:master&package=darktable

It also hangs at startup when I try to open images with it.
I think the key to my problem is in commits that occurred between June 21-22. I have no idea how to see it.

[ptilopteri]

try running dt as a different/new user

[johnny-bit]

@YAFU - if you know how and can compile your own install, then there's git bisect tht could aid you in tracing from which commit the issue came.

As for commits - remove "g" in front of version in order to checkout given ver, eg: instead of g0c7527f8c use 0c7527f8c, see: 0c7527f8c

[YAFU]

Hi. I am just an old Linux user, I know how to compile and some other things related to Git, but mainly following recipes. I have searched about that "bisect" thing, but really it is much more beyond my knowledge and capabilities. Later I will try to do my best to try to understand about it.
Thanks.

[johnny-bit]

It's really easy! All tutorials make it seems complicated. Here's my mini
tutorial:
We're gonna assume you are in director where you compile darktable, version
A is one you remember working, version on which you're now doesn't work. So
you do following:

1. Git bisect start
2. Git bisect bad
3. Git bisect good A

Now compile and try to run darktable. If it works you write 'git bisect
good' if it doesn't - 'git bisect bad'

Repeat till it spills first bad commit and report it here 😃 after that do
'git bisect reset' to go back to normal git operations.

niedz., 2 sie 2020, 01:01 użytkownik YAFU [e-mail]
napisał:

Hi. I am just an old Linux user, I know how to compile and some other
things related to Git, but mainly following recipes. I have searched about
that "bisect" thing, but really it is much more beyond my knowledge and
capabilities. Later I will try to do my best to try to understand about it.
Thanks.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/darktable-org/darktable/issues/5840#issuecomment-667597569,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/ACRRKFKVXN5W6S6X4IEHSWTR6SNGLANCNFSM4PON7YKQ
.

[YAFU]

Wow! That is much easier than what I had read out there. Thank you! Only you just should correct the capital "G" in 1,2,3 steps in case any Linux newbie falls for this mini tutorial when doing internet searches.
Here what I did:

My good known version is: 0c7527f8c
and bad known version is: 3f399bdf8
I have checkout bad version: git checkout 3f399bdf8
Just built in this instance to check that Darktable hangs when opening image, and indeed Darktable hangs/did not start.

Then:
$ git bisect start
$ git bisect bad
$ git bisect good 0c7527f8c
Biseccionando: faltan 275 revisiones por probar después de esto (aproximadamente 8 pasos)
[f6842cc0d100eb44079394566ab07558259fd9c6] Merge pull request #5392 from dtorop/histogram-fixups

I have built and Darktable still fails. Then:

$ git bisect bad
Biseccionando: faltan 139 revisiones por probar después de esto (aproximadamente 7 pasos)
[0acaf66c4450abe5408d3e1247cf1dc7d579234b] utility: add const to parameter.

I have built and Darktable works. Then:

$ git bisect good
Biseccionando: faltan 69 revisiones por probar después de esto (aproximadamente 6 pasos)
[b05487b63b14a562078f6ec7d7a316c76eb0c7a4] filmicrgb: fix typos in labels

Darktable works

$ git bisect good
Biseccionando: faltan 34 revisiones por probar después de esto (aproximadamente 5 pasos)
[61fa7204b412b9de85da8dc96331e47fe0920c0e] histogram: reverse resize scroll direction

Darktable hangs

$ git bisect bad
Biseccionando: faltan 17 revisiones por probar después de esto (aproximadamente 4 pasos)
[b1d8484da6cca3b90a4e08c003b6fc62351e004d] Merge branch 'po/fix-masks'

Darktable works

$ git bisect good
Biseccionando: faltan 7 revisiones por probar después de esto (aproximadamente 3 pasos)
[4a13314950b164f03ec22946683664d87b247d1f] Merge pull request #5354 from AlicVB/over_margins

Darktable Works

$ git bisect good
Biseccionando: faltan 3 revisiones por probar después de esto (aproximadamente 2 pasos)
[4cd2733f5dcab5f9c832f2150f213b0935c4884e] dynamic tagging module buttons

Darktable Works

$ git bisect good
Biseccionando: falta 1 revisión por probar después de esto (aproximadamente 1 paso)
[e5723be0a925b8cd51c36c4270b4ca94c0b17dd7] Removed script_manager/luarc choice. The default installation is a luarc file that loads script_manager.

Darktable hangs

$ git bisect bad
Biseccionando: faltan 0 revisiones por probar después de esto (aproximadamente 0 pasos)
[bbea7a11a17f79e4dac320b6e552c50323467bfe] Added a module to the default luarc file to provide a point and click interface for installing the lua scripts repository and configuring it for use. The module only shows if the scripts aren't installed.

Darktable hangs

$ git bisect bad
bbea7a11a17f79e4dac320b6e552c50323467bfe is the first bad commit
commit bbea7a11a17f79e4dac320b6e552c50323467bfe
Author: Bill Ferguson [e-mail]
Date: Sat Jun 6 21:25:56 2020 -0400

Added a module to the default luarc file to provide a point and
click interface for installing the lua scripts repository and
configuring it for use.  The module only shows if the scripts aren't
installed.

RELEASE_NOTES.md | 3 +-
data/luarc | 184 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 186 insertions(+), 1 deletion(-)

$ git bisect reset

So tell me if I have done things right, and if the information is useful.

Edit:
Oh, terminal messages are in Spanish

[johnny-bit]

Continue till you get no revisions remaininig :) it'll even show what change caused problem!

[YAFU]

Oh ok. I'll report later.

[YAFU]

I have edited the above message. So finally the problem was related to LUA as @ptilopteri had noticed in my error messages?

And my concern would be, why does it hang darktable on my system when I try to open an image "with" darktable? And why this does not create problems in your system.

[johnny-bit]

Well, the issue's with @wpferguson commit, however I think since that point he did fix it... Maybe the fix wasn't "full"?

[wpferguson]

@johnny-bit thanks for tagging me. I saw this when it first opened, but didn't think it applied to me.

The hang problem I fixed was trying to create a GUI element when darktable-cli was started, which has no GUI. This is a different issue.

I started out trying to test for this condition and this is what I found:

I have several different versions of darktable, for testing, development, etc, installed in /opt. I run these with scripts located in ~/bin. One of those is ~/bin/dttest1. It's contents are
/opt/dttest/bin/darktable -d lua --cachedir $HOME/.cache/dttest-1 --configdir $HOME/.config/dttest-1

I cleared $HOME/.config/dttest-1 and randttest <somefile>.CR2 and everything worked fine.

I ran /opt/dttest/bin/darktable -d lua --cachedir $HOME/.cache/dttest-1 --configdir $HOME/.config/dttest-1 <somefile>.CR2 and it hung with the ERROR in lualib, couldn't find a position for lua scripts installer, this should never happen. I also tried bash -c ... and that hung too.

So, there's good news and there's bad news.

First the bad news. This is a bug. If you start darktable in a certain way, with certain conditions, it will hang. It's going to take some research to figure this out.

Now the good news. There are lots of workarounds for this.

If you have the lua scripts already installed, you wont see this.

If you don't have the lua scripts installed you can

• from a terminal cd ~/.config/darktable;echo lua/_scripts_install/dont_show=TRUE >> darktablerc
  or
• from a terminal cd ~/.config/darktable;touch luarc;mkdir lua
  or
• start darktable without an argument and either install the scripts or select dont show again
  or
• put the darktable start command in a script and run it that way

@YAFU the workarounds listed above should fix your problem until we get this sorted out.
@TurboGit it's a bug. It takes a specific set of circumstances to trigger it, i.e. starting darktable with an image argument and not having the lua scripts installed or the installer disabled. There are workarounds.

[YAFU]

I have opted for this workaround "cd ~/.config/darktable;touch luarc;mkdir lua" and it worked. Thank you!

[wpferguson]

I figured it out. When you start darktable from the command line with an image specified darktable switches to darkroom mode on startup. The scripts installer module needs to be placed in lightroom, so when the switch is made there is no place for the module and it hangs. Fix is on the way....

I still don't understand why it works when you start darktable from a script though....

[TurboGit]

Will be in 3.2, thanks!