Retroarch: [Request] Display controller input via Onscreen Overlay [BOUNTY] [$80]

[HeyItsLollie]

The premise for this bounty is fairly straight-forward: If the player has an Onscreen Controller Overlay enabled and is using a physical controller, allow RetroArch to listen to the user's input binds, and display them through the Onscreen Controller Overlay.

I see this being primarily useful for recording/streaming and speedrunning, where controller input is more useful for an audience. That said, this could also make it easier to test the visual side of Onscreen Controller Overlays on PC environments.

Given that RetroArch's overlays already provide visual feedback for touch-input (buttons light up, joysticks move around) and custom overlays can be made fairly easily, it seems like the perfect fit for controller input display. I hope it'd give onscreen controller overlays more purpose on PC/Mac/Linux.

---

There is a $80 bounty on this request.

[Ryunam]

This has been talked about in the past and I fully agree that it would be an amazing feature. First of all, it would provide a practical - and not purely aesthetical - use to overlays for non-touchscreen users. Secondly, like you said speedrunners would benefit immensely from a real-time display of inputs while playing.

Implementing a listening mode through overlays seems like the ideal choice simply because that way you could customize your view for each system, create a border / frame with only the inputs you want to monitor and have them react to your controller in real time. I tried to look into creating an option like this myself, but admittedly I would need more directions and I am not nearly competent enough to do it.

[Ryunam]

See also https://github.com/libretro/RetroArch/issues/3314 about this.

[HeyItsLollie]

See also # 3314 about this.

I saw that issue — the use-case seemed to be far more specific, so I decided not to revive it. I wasn't sure if unixunion was talking about displaying gamepad input, or displaying a prepared info-graphic as a _border_ overlay. (If it was the latter, a new Hotkey Bind to toggle overlays on/off would serve their purpose.)

Still, I'm glad to see that others have brought this up in the past! I know it's a pretty niche feature, but I hope that people would find it useful enough to consider implementing.
(also- updating my original post with what I wrote on the forums, better description)

[Ryunam]

I would suggest editing the title of this issue here by including a reference to your $50 bounty, in order to hopefully increase its visibility and attract potential contributors.

[twinaphex]

Added it to the title for him.

I also posted it on Twitter.

[Ryunam]

Just reporting that I have added $25 towards this request. It'd be cool to update the title and hopefully some contributors can chime in the not-so-distant future!

[twinaphex]

Updated the title!

[HeyItsLollie]

I'm not sure how Bountysource shares updates, but I'd just like to state here that @Denu8thell has started working on this bounty! https://github.com/Denu8thell/RetroArch/tree/overlay_controller_response

[Ryunam]

Hi, @Denu8thell! I am currently testing your code that was just merged with PR #5269. All buttons appear to light up properly and react to controller inputs as intended. The only aspect I am having some issues with is the D-Pad.

On the vast majority of the overlays that are distributed through the libretro servers (the ones you can update through Retroarch itself) D-Pad directions do not seem to light up, save for the PS1-styled variants. I am actually wondering if it's more of a quirk with the overlays themselves, such as the Nintendo-styled D-Pads that come with these overlays possibly not being programmed to detect individual directions.

[BinaryFissionGames]

@Ryunam Yeah, a lot of the overlays are set up so the D-pad is just a big button. The flat variants work well, I think the overlays just need to be set up so that each d-pad direction is it's own image.

Edit: The obstacles with fixing the overlays in the program itself are much greater than changing/creating new overlays. The problems are:

1. No connection between the D-Pad image and the button. The D-Pad image is its own overlay component, without any input assigned to it. The inputs are actually invisible pads placed on top of the image. Fixing this would require some change to the file format of the overlays, as well as the way it's represented in code, which would take a bit of work.

2. No generalized way to light up D-Pad buttons of a single image. All the D-Pads are different, and finding a way to change them without added information to the overlay is pretty much impossible.

Both of these problems require editing the overlays themselves, so I think the easiest way would be to edit those overlays so that each D-Pad is split into sections. The overlays could look and function the same, other than lighting up specific directions instead of the whole D-Pad when pressed through the pointer.

The only problem with that approach is the center of the D-Pad will light up when pressed with the pointer, so there would need to be a way to specify that certain overlay parts shouldn't change opacity when hit with the pointer.

[HeyItsLollie]

I'm happy to report that the solution for this bounty seems to be working just fine on my end. Thank you for working on this! I've been building a couple overlays designed for input display - here's one based on the Dualshock controller, everything should light up correctly.
input-display-ps1.zip

The easiest way to ensure that certain parts of an overlay _don't_ light up, is to have the following lines in the overlay's config:
overlay0_alpha_mod = 2.0
This controls the alpha modifier for all elements in the overlay. value can be any amount.

overlay0_desc#_alpha_mod = 1.0
Ensures that the element's alpha is multiplied by 1, effectively doing nothing. Use it for any element that shouldn't light up.

Overlays do need to be specifically designed for light-up action, but I found that gamepadsdual-shockdual-shock.cfg does it out of the box. I'm going to be working over a few new overlays for input display, see if I can't fill this new void up a little.

[HeyItsLollie]

Development-wise, is there anything that still needs to be worked on? As far as I can tell, input display seems to be totally functional - it just requires overlays that can actually take full advantage of the feature. I'd be more than happy to see @Denu8thell claim the bounty.

[BinaryFissionGames]

As soon as this issue is closed, I'll be able to put in a claim. If everyone is satisfied, then the issue just needs to be closed!

[HeyItsLollie]

I was waiting in case @Ryunam or @twinaphex had anything else to say on the matter — felt a bit presumptuous for me to close the issue, not being part of the RetroArch team and all. But if everyone's happy/has nothing else to add, I'll close the issue this evening!

[Ryunam]

As far as I am concerned, I would more than glad to award the bounty to @Denu8thell. The work done for this request is absolutely satisfying and the explanation provided about the D-Pads on the Nintendo-style overlays is perfectly clear. Thanks a lot for contributing so quickly to this!

[twinaphex]

Alright, I'll close the issue then.

[hizzlekizzle]

This option seems to be working great and I've used it to make some overlays that use opacity changes for fake animations on input. However, it seems only a single button can be detected at a time. If you push a second button, it cancels out the first.

Is there any way to extend this to cover all of the buttons that are being registered?

[HeyItsLollie]

Weird, displaying all given inputs (for a single controller) at any moment is how the feature is supposed to work. I've just updated to the latest nightly to check, and the feature has seemingly been busted in RetroArch 1.7.1.

Show Inputs In Overlay was last working correctly in the Stable 1.7.0 release.
1.7.0 can be found here: https://buildbot.libretro.com/stable/1.7.0/

[hizzlekizzle]

@HeyItsLollie
Aha! So it used to work properly? I hadn't used it during that time, so I assumed it was always this way. I'll try to bisect it.

EDIT: looks like this is the one that broke it:
56389466aaff5e2e196031e3f6b4d468bd35607c

EDIT2: Twinaphex reverted the commit, so it should be back to normal now. I'll do some testing and report back if not.