Rack: Enhance touch-screen / rack interaction

[alectron]

I was absolutely thrilled reading VCV rack can be controlled via touch screens.
Unfortunately after testing this via:

• Added "allowCursorLock" to settings.json (set to "false" for touch screen support).

the experience doing this is somewhat patchy/frustrating as far as I have experienced.
There's probably many ways to tackle this, but as I would think there's not much requirement for this on the large scale of the VCV community, I'll try to focus on a suggestion for a easy/quickest route to enhance this.

As for the frustrations:

• when interacting with VCVRack on a touch-screen, the biggest frustration for me was that the module 'whitespace' (i.e. whatever is not a plug/button/encoder/fader) usually makes up 60-90% of the module UI/front-plate, and one more often than not ends up selecting, and therefore dragging the module about in the rack, and not the plug/button/encoder/fader one was actually after.
• especially cable ports/plugs seem to have a very small selection areas - i.e. just the ports themselves, but I blame my sausage fingers & general touch-screen approximations (since it's never pixel-perfect as a mouse pointer), which makes dragging a cable from one to the other very finicky, and on touch screen, I usually need multiple attempts to even get a cable connector to appear

As for suggestions:

• would it be easy enough to introduce a 'module lock' state - where one can not move modules, i.e. this:
  https://github.com/VCVRack/Rack/issues/492
• maybe the above would get rid of most of the frustrating miss-clicks via touch, but from how it feels, it would be great if there was a preference setting to 'scale' selection areas around the interactive parts of the racks (?) - so it's easier to correctly grab a control via touch
• additionally, this might be over the top though as it would require some fair bit of dev time I would think and probably is a niche requirement... multi-touch compatibility would be neat to have to be able to control more than 1 parameter at the same time

Churs.
c.

[AndrewBelt]

Increasing the hitboxes for things like knobs would be a monumental task and probably won't happen ever.

I tried Rack on a friend's touchscreen computer and hated it, but even if everything was fixed, I would still prefer using a mouse. I just don't like the trend of rubbing your fingers on glass to interact with computers while we have dozens of muscles in our hand. (see A Brief Rant on the Future of Interaction Design by Bret Victor)

So I will discourage anyone to attempt to use Rack with a touch screen. It will never reach my quality standards so there's no point in trying to nudge it in that direction.

[alectron]

Thanks for the reply & interesting article that!
And I hear you on some parts - and I would add my own negatives that touchscreen computers are generally frustrating & ill-suited for audio in itself due to non-audio needs (battery life, small components, etc) taking priority over the things audio folks care about, i.e. DPC latency, DAC quality, etc.
But as someone who fell in love with the old Lemur as a midi device, for me the abundance & easy access to touch devices nowadays is a plus and there is value in touch-screen as a UX/interaction-method for those that like it (but, this is highly subjective). And for me, there are clear upsides compared to mouse input, especially with multi-touch, which I do prefer over using a mouse (which, it needs to be said for the sake of the argument/feature request :P, is also a non tactile abstraction, just one some people are more comfy/natural with than nudging their fingers around on a glass surface... ;)
At least in the interim touch will remain as a mainstay program/device interaction method... until a new glass-less UX approach will manifest into popular actual devices, eg something that doesn't exist yet, or something alike Leap Motion's AR/interaction propotypes:
https://twitter.com/keiichiban/status/976850288949809154
Whereas & alas, this will be even less tactile than glass in the beginning.... until people also figure out to stimulate nerves directly from virtual interactions... which I am not sure I would want...
Interesting times, either way.

I'll just keep poking VCV for fun & learning, and hope for the Module-lock from #492 to manifest at some point which should remove some of the biggest frustrations, and coupled with display zoom/scale, I might be able to get something workable for me needs/funs for now ;)
Thanks again for your VCVing!
Churs.
c.

[ajbmorris]

I tried this on my laptop before reading this and it wasn't practical at all. Would be nice to be able to use a pen stylus as well as a mouse though (didn't work when trying it on a Surface Pro1).

[AndrewBelt]

At the bare minimum, https://github.com/glfw/glfw/issues/42 would need to be solved for multi-touch.

[alectron]

@AndrewBelt - would this Wayland & Windows implementation mentioned:
https://github.com/glfw/glfw/issues/42#issuecomment-604945436
that has been done push this forward to be usable for VCV potentially ? Or would you need the mentioned x11 & Cocoa implementations too...?
Churs.

[AndrewBelt]

To add true touch behavior to Rack, I'd need a stable touch API added to GLFW and implemented on all platforms with first-class support.