Darktable: [FR] new pluggable input (not for drawing!) system for darktable

[johnny-bit]

This sat on my inbox and back burner for far too long :/

On the legs of some comments left in #3722 it became apparent that current "input system" darktable has is a bit "hacky" and requires loads of boilerplate code + isn't very extensible as shown by extent of work needed for MIDI support.

After a bit of discussion there was another one on IRC to get the feel of what needs to be done in what way. It became apparent that the extent of work is rather large and will span months of work.

Currently darktable has: vimkeys, lua integration callbacks, key accels and dbus bindings, all of them being separate. Unification is needed in a pluggable and generic way. Suggested approach to coming up with solution was to sorta follow vimkeys approach of registering every control and then having access to it exposed for all known inputs, however multi-modules etc come a bit in way and do mess with vimkeys as of now. the job for input system would be to handle registering and unregistering of "action targets" (eg sliders, combos etc) and "input modules" (think keyboard shortcuts, midi, vimkeys etc) and handling communications between them.

From modules & control side, input system must handle:

1. multiple instances
2. getting & setting of all possible values/controlls
3. shortcuts discoverability (eg tooltips with shortcut info are easy example)
4. "same" shortcut for separate dt mode (eg tethering and lighttable are guaranteed to never receive same input, so could technically use same shortcuts)

From input modules side, input system must handle:

1. plugging and unplugging of input module
2. conflicts between input modules (eg vimkeys vs keyboard)
3. back communication to device (in MIDI world there are sliders that can be moved mechanically, so if user moves slider using mouse, an attached slider on midi device should move by itself)

From iop/contributor's side input system must guarantee:

1. less boilerplate code (no more signal connections needed to get benefits of shortcuts, so any known slider, gets to have shortcuts, no need for any additional coding)
2. respect set limits (prefferably soft limits), so that input system is guaranteed to never let slider go "up to eleven" if soft-limit is 10. hard limits are to be inputed manually via bauhaus input in right click over slider
3. be completely transparent code-wise from IOP standpoint (similarly to IOP introspection)

In order to create generic input system couple things must be kept in mind:

1. Various inputs, which can be different than what one would expect, so possible "input module" may not seem like input at all, eg:
2. Keyboard
3. Mouse (with extra keys)
4. Pressure sensitive touchpad
5. Remote
6. Game controller (joysticks, pads, etc)
7. MIDI Device (those are VERY varied)
8. Pedals
9. Lua subsystem
10. vimkeys
11. emacs mode
12. Dbus interface
13. Com
14. Voice
15. device with SDK/API like eg. Loupedeck (post LD/LD+ which are MIDI)
16. basic module replacement, where controls in it are actually shortcuts to actual modules. (basic module replacement could be thus a favourite-controll thingie, sorta like favourite-modules, but for controlls) - thanks @dterrahe
17. Modificators/specials for physical inputs such as:
18. Double/triple/long click
19. Toggle
20. Hold
21. Modifiers across inputs (eg shift+sliding, shift + midi knob, pedal + mouse, pedal + knob etc)
22. Key sequences (such as emacs-mode)
23. various inputs darktable has:
24. sliders (most known)
25. combo boxes(dropdowns)
26. buttons/checkboxes (there was suggestion to change checkboxes to yes/no dropdowns for better UI flow)
27. complex widgets (eg color zones, wavelets editors... those need to be taken into account)
28. UI/UX stuff needed (or good to have) to make input system beneficial to users (couple things already mentioned):
29. predictable multi-module handling
30. being able to handle masking stuff
31. help screens (like current shortcuts window, maybe things like device image with layered on it functions etc)
32. tooltips & general discoverability (users can and do forget keys they set ;) having easy tooltips and helpscreens would alleviate problems)
33. point&click configuration (so instead of going to settings one could point to selected ui element and attach inputs that way. Sorta like press and hold mouse on slider and at the same time turn knob on midi device to attach knob to slider... or maybe some "attach shortcut to this element" window?)
34. device/input module hot plugging, default configs for known devices and possibly easy sharing of configs for custom devices
35. events aggregation when dt is busy/can't process and then apply once possible, cancelling of aggregated events. input system shouldn't aggregate events, instead it's a job for job control system - @parafin

There are couple of use-cases that might be interesting to tackle with input system:

1. Active-module dependent control, eg set a knob (or kb shortcut) to control set slider in active module, regarding of what module it is, eg active module is exposure - control exposure slider, active module is white balance - control temperature slider and so on
2. Instance-name dependent control, eg: knob 1 controls exposure in active exposure instance, knob 2 controls exposure in exposure instance named "subject", knob 3 controls exposure slider in exposure instance named "background"
3. Open view-dependent control, eg control1 setst different map source on map view, same controll changes paper size on print view etc :)
4. Direct shortcut to combo/dropdown value, eg "click gamepad X" and it'll set specific combo/dropbox value on active module, eg select "Fluorescent" preset on white balance or something like this.

Pinging @dterrahe and @elstoc since much of their work and talk is embeded in discussions that led to above wall of text. Also pinging @houz for input

I hope I haven't missed anything :)

[elstoc]

Wow. And I thought it sounded big _before_ you wrote it all down.

[parafin]

events aggregation when dt is busy/can't process and then apply once possible, cancelling of aggregated events.

BTW https://github.com/darktable-org/darktable/issues/2724 is related to what you mention here
and I'm not sure if it's a job for input system or job control. Right now it's spread over several subsystems. Similarly there's some code for smooth scrolling that aggregate events - that one is more suitable for input system. But I think input system shouldn't know about pipeline latencies and such - its job is to deliver events IMHO. Caching them inside input subsystem will only create latency. Instead job control should do these decisions.

[dterrahe]

In list 1 I forgot at the time to add "basic module replacement #3648", i.e. somehow we could imagine a user defined module, or multiple modules, that is just a collection of shortcuts to elements in other modules. Sort of like a favorite list but at the individual control level.

[dterrahe]

Event aggregation can happen at two (well, several) levels. My midi code does some aggregation, because the input event loop can already cause some midi messages to back up (if a knob is turned quickly). So I aggregate those (at least the first series of the same type) before updating the bauhaus slider. Then that slider may, I guess, send several jobs to the queue? And some of them could then become irrelevant and should get kicked out? I haven't seen a problem with that. But if it is necessary to implement something at the jobs stage and if that works really well, then the earlier stages could decide not to bother with their own aggregation.

But the main advantage of the aggregation I do in midi (and this might also go for the scroll wheel aggregation) is that if you can _see_ the queue and _know_ there is a back-log, then you can wait before kicking off the first job and aggregate what you already have. If you _don't_ see the queue, you would kick off after you've been given the first tiny move and then draw again when you're finished and see that you've accumulated a lot of changes in the mean time. _Or_ you'd have to always wait a little to see if your input module is going to send you a burst. This might work well enough, but would be slightly less responsive (depending on the delay you build in).

[github-actions[bot]]

This issue did not get any activity in the past 30 days and will be closed in 365 days if no update occurs. Please check if the master branch has fixed it and report again or close the issue.