Bevy: Editor-Ready UI

[cart]

_This is a Focus Area tracking issue_

Before we can start work on the Bevy Editor, we need a solid UI implementation. Bevy UI already has nice "flexbox" layout, and we already have a first stab at buttons and interaction events. But Bevy UI still needs a lot more experimentation if we're going to find the "right" patterns and paradigms. Editor-Ready UI has the following requirements:

• Embraces the Bevy architecture: Bevy ECS, Bevy Scenes, Bevy Assets, Bevy Events
• A Canvas-style API for drawing widgets with shapes and anti-aliased curves
• Define a consistent way to implement widgets
• A core set of widgets: buttons, inputs, resizable panels, etc
• Theme-ability
• "Interaction" and "focus" events
• Translation-friendly. We can't be anglo-centric here

Active Crates / Repos

_No active crates or repos. Feel free to make one. Link to it in this issue and I'll add it here!_

Sub Issues

_No active issues discussing subtopics for this focus area. If you would like to discuss a particular topic, look for a pre-existing issue in this repo. If you can't find one, feel free to make one! Link to it in this issue and I'll add it to the index._

Documents

• Collection of thoughts from Discord discussions: https://hackmd.io/lclwOxi3TmCTi8n5WRJWvw?view

[Kleptine]

I wanted to share a few quick thoughts on UI systems in games. I'm a gamedev professionally, and I read this list of requirements, and had a bit of a knee-jerk reaction. These are good features from a basic point of view, but sort of miss the boat on what I would consider are the harder questions around UI:

• Structure: Immediate Mode vs. Retained Mode
• Performance: How fast is a Noop render (where nothing in the UI changes)?
• Styling: CSS-like, or in-code?
• Flexibility: How easy is it to build any UI, vs. just 'widgets'

I'm happy to elaborate on any of these, but primarily I'd recommend trying to learn from where Unity is at the moment. They started with an immediate mode UI engine for both the Editor and the Runtime. Later, because of performance and flexibility, they adopted a retained mode UI for the Runtime. Now, as of two years ago, they've finally been building a (really fantastic) unified UI system for both the Editor and Runtime, open source.

This is a great talk on the new design (UIElements/UIToolkit): https://www.youtube.com/watch?v=zeCdVmfGUN0

I'd highly recommend taking as much learning as you can from their path. Frankly, both previous systems were a total mess. They seemed great in the moment, but just collapsed under any sort of complexity.

What makes UI challenging is that it's very hard to build incrementally, you have to plan a lot of these things in from the start. I don't think Bevy UI should be nearly as complicated as the latest Unity UI, but it's really worth learning, and not repeating their mistakes.

[Kleptine]

And also, if the intention for Bevy is to start off as a hobbyist engine, incremental is perfectly fine. I would just expect to have to rewrite the UI a few times as you want to attract larger projects.

[thefuntastic]

Some early distillation of thoughts is happening here:
https://hackmd.io/lclwOxi3TmCTi8n5WRJWvw?view

At the moment anyone is free to add their particular domain experience. This is stopgap - suggestions are that Cart takes ownership of the hackmd / it's used as ingest for a github wiki/markdown page

[tektrip-biggles]

Would it be worth also looking as SwiftUI as an example of prior art?

[erlend-sh]

Would it be worth also looking as SwiftUI as an example of prior art?

Add your write-up of it to the doc if you’d like 👍

[stefee]

Can we add "screen-reader friendly" and "multiple input types" (keyboard, mouse) as hard requirements for any UI framework / UI primitives?

[cart]

At the risk of being _very_ ignorant, is screen-reader friendliness a high priority for a visual editor? I have a feeling visually impaired folks will have difficulty getting use out of the editor. Making a visual scene editor friendly to visually impaired users sounds like an entire research project. Mouse + keyboard navigation is a clear win for accessibility so I think I'd want to focus efforts there, given our limited resources.

I'm _certainly_ not saying we shouldn't make ourselves screen reader friendly, as that will absolutely be useful in other apps, I'm just questioning if it should be an Editor-UI priority. I'm curious what the use cases would be for screen readers + visual editors.

_Please_ tell me if I'm wrong here.

[stefee]

Yeah no worries, it was a genuine question. I'm not sure how well screen readers are supported in other editors, I am coming from a context of developing on the open web and it is really important in that context. I imagine native software is a lot more complicated.

The extent to which it is important comes down to how visual the editor is. If there are lots of text inputs (e.g. a way to change transform values of an entity numerically rather than just using click + drag) then I think screen reader support would be something to consider. If at least in the short-term we are looking at a more purely graphical interface then it is less important.

[stefee]

Also for the record, I don't use screen readers so I don't actually know what users would prefer in this case. Maybe folks would rather just edit the RON files directly. Let's not worry about it too much. :-)

[tektrip-biggles]

Have been looking at SwiftUI and one of the nice things about the declarative approach is that it does mean your data is well structured for adding things like that in the future.

The general concepts they use are really elegant and I suspect would translate well to Rust, although it may be an "all or nothing" commitment to go that route.

Main principle is defining a tree of very lightweight units composed together to create consistent and predictable results with surprisingly little boilerplate and small number of primitive/leaf components. Data dependencies are all declared up front so the framework knows what to update as & when the underlying data changes and there's an "environment" concept so that data can be passed down the hierarchy without needing to be passed through every hop.

I quite like the bounds calculation process too, where parent gives its children size "suggestions", they respond with their desired size and parent determines ultimate positioning (but has to respect child's desired size)

Worth watching the WWDC '19 & '20 talks on it all for ideas.

[regexident]

@tektrip-biggles: FYI, @raphlinus has done a bunch of great research in this whole topic, as well as in comparison to existing systems like SwiftUI:

https://raphlinus.github.io/rust/druid/2019/10/31/rust-2020.html

[Kleptine]

A few thoughts:

Screen Reading: If the goal is for Bevy UI to support in-game UI as well as the editor, that means the UI will need to support gamepad-only and keyboard-only navigation of UI from the beginning. So in a lot of ways, you're already halfway there to basic screen-reading support. The bigger question is one of UX design (ie. does the Editor require the mouse?), rather than implementation.

FRP: Modern FRP-based approaches work fantastically for the web. I think they're really strong when you have a flat UI that is structured as a hierarchy. Game UI can be much more complex. In-Game UI often isn't a flat plane, nor a tree. There might be a myriad of text cards, health-bars, holographic displays, etc. Depending on the game, it may be hard to treat this as a single tree.

Additionally, there are entire projects currently figuring out FRP in Rust (Yew, etc). It's been a massive undertaking, and most require lots of macros, templates, generics, etc. And that's without having to build a renderer. So I worry about complexity, scope here.

What I'd favor is a general, high performance UI rendering engine, integrated with the ECS. An FRP crate could be built on top of it, but wouldn't be explicitly baked into the solution. That would allow UI-heavy and 2D games to use FRP as needed, but not require jamming all other use cases inside of it.

[ShalokShalom]

I think Godot can serve as a nice project to look at, while developing the BevyUI

[ncallaway]

@Kleptine

It's been a massive undertaking, and most require lots of macros, templates, generics, etc. And that's without having to build a renderer. So I worry about complexity, scope here.

What I'd favor is a general, high performance UI rendering engine, integrated with the ECS. An FRP crate could be built on top of it.

I think that's right. Especially the part about complexity and scope. I really _really_ like developing UI in the FRP style and would love if Bevy supported it, but it's a mountain of work that we don't need to take on right now.

I think really nailing a UI rendering engine and ECS would us a good foundation to build different higher level UI experiments on. I could see a React-like UI system someday that treats the ECS UI the way React treats the DOM.

[stefee]

I 100% agree with this @ncallaway and I've had similar thoughts over the last few days. The high-level API is something that can - perhaps should - be developed _only once_ the underlying system has been established and is found to be performant and fit for purpose.

If it means writing very verbose or repetitive code for the time-being, that is a worthwhile trade-off IMO.

[stefee]

Also agree with @Kleptine that the high-level stuff can be user-land crates for the time being.

[stefee]

One point @Kleptine

• Styling: CSS-like, or in-code?

Why not start in-code and then we can add some optional CSS-like solution later (I'm thinking something like how CSS-in-JS works, i.e. it mostly compiles to target language at build time so everything is static at runtime).

[Kleptine]

I could see a React-like UI system someday that treats the ECS UI the way React treats the DOM.

Precisely my thoughts as well. The ECS would be a great way to store this information in a flat structure. You should take a look at the way Unity stores their UI components in a raw buffer (in the UIElements design talk). It's fairly similar.

Why not start in-code and then we can add some optional CSS-like solution later.

I think that could work out. Unity used in-code styling for their IMGUI implementation. One of the challenges was just that it was cumbersome to feed through all of the different styles through all parts of your program. It might be better, though, if styles can be stored as constants, somehow and exported statically.

So I think some more succinct form of styling would be nice. A CSS-like alternative could be added as a crate, although it might make the API a little more challenging to design. But I agree it's a fairly big task, probably too big for now. Personally, I would be averse to anything other than a CSS subset. There's enough UI styling languages out there that there's no need to reinvent the wheel.

Edit: Another downside of code-only styling is that you need to recompile to see changes. Bevy is already fast to recompile, but you might still have to spend a few minutes replaying the game to get back to the UI you were looking at. It'd be ideal if styling were an asset that could be hot-reloaded in-place, just like any other asset.

[stefee]

It'd be ideal if styling were an asset that could be hot-reloaded in-place, just like any other asset.

Anyone please correct me if I'm wrong, but I _think_ styling as it currently works in Bevy UI can be saved/loaded from scene assets at runtime.

It might be better, though, if styles can be stored as constants, somehow and exported statically.

I was assuming that this would be the case, or at least that UI styling would be built up out of composable functions (i.e. "mixins"). I think this would already be quite easy to do with Bevy UI, but I haven't actually tried it.

[stefee]

I wrote in Discord about the possibility of adopting the Every Layout (https://every-layout.dev/layouts/) primitives as our "building blocks" for layout. I think we could potentially just copy the CSS from the Every Layout components into Bevy UI as some sort of "style mixins".

I'm a big fan of their composable approach to flexbox layout primitives, and since Bevy UI currently uses the flexbox model anyway this would be a good fit: https://every-layout.dev/rudiments/composition/

P.S you have to pay for access to the book, but the CSS itself is not patented, so we can use it.

[ndarilek]

As to screen reader support in the editor/UI, I can speak a bit to that.

I'm a blind game developer working on adding accessibility to Godot. It may not be as relevant here as it is in Godot since I gather that more code-based workflows are first-class here in a way they aren't with Godot, but a few use cases I have in mind:

• Working alongside sighted developers who would prefer a visual editor. Right now we've got a handful of sighted developers building audio-only experiences for us, but I think the feedback loop could be drastically increased if they could work alongside blind developers vs. filtering feedback through slow playtest cycles. Right now hiring me as a Unity developer is probably impossible since Unity itself is inaccessible, but my work makes Godot accessible enough that I'm able to build a game with a hybrid editor/CLI workflow. It'd be less hybrid if I started my accessibility work earlier.
• Some tasks are hard to get right in text. I'm specifically thinking of tilemap editors, and have some PoC code that replaces Godot's tilemap editor with an alternate accessible interface. Screen reader access to the editor would let me do something similar with Bevy.
• Accessible modding tools. Godot's system for DLCs/mods offers using the editor as one option. If I build a moddable game, being able to ship a cross-platform accessible editor would be nice.

I'm running up against some Godot limits that make accessibility challenging to implement, so tentatively pencil me in as willing to help with Bevy UI accessibility. My big condition for helping out is that it be baked directly into the UI (I.e. separate components are fine, but I'd want to ship it as a UI requirement that someone might choose to disable for whatever reason, than as a third-party crate.) I'd also like for it to be as integrated with the repo as is the UI crate, such that CI failures breaking accessibility are blockers. IOW, I'm fine with people not launching the screen reader system if they'd rather not, but I'd want UI consumers to automatically have it and be assured that it works with the most recent UI. Hope that's acceptable.

In terms of making my job easier, here are two bits of advice:

1. Make keyboard navigation first-class. Godot has this problem right now. The tree widget is absolutely broken for keyboard navigation, and fixing it isn't a priority. I'd do it myself, but since I can't see what I'm doing, it's like coding with a strip of cloth between me and the keyboard. Not saying it needs to be exhaustive and platform-specific, but keyboard/gamepad support should be consistent, and fixing breakage should be prioritized.
2. Send events for just about everything. Focus enters/leaves a widget. Text is added to, or removed from, a TextEdit. Focus/selection moves around an editor. I need to intercept just about everything and provide a speech representation. Godot has some lacks here too, and there's been a bit of resistence to adding signals I need, meaning accessibility becomes more and more hacked-on.

Anyhow, hope that helps. Sorry for going long. I'm about to launch an accessible Godot game, but am happy to try Bevy for my next title and work on accessibility. One good aspect of audio-only games is that they don't exactly require lots of resources--a blank screen and good audio are usually enough. :)

[ncallaway]

I'd vote for accessibility / screen-reader friendly support as part of Editor Ready UI too. I think @ndarilek lays out great reasons why accessibility is important in the Editor itself.

The other reasons why I'd vote to tackle it as early as possible (even if it does expand scope somewhat) are:

• It's much easier to build-in early when you don't need it than bolt-on late when you do (similar to i18n).
• If accessibility support is baked into the core of the UI, then it's much more likely that games using Bevy will be more accessible by default.

I don't necessarily think we'll be able to get the first version of the editor to be in a place where it integrates with popular screen-readers on all platforms, but I would really like the core of the UI system to at least have all the pieces in place so that if someone wanted to add screen-reader support the UI system doesn't get in the way.

[ndarilek]

To clarify, as much as I don't like this route, I don't believe screen
reader support is the way to go. Instead, I'd advocate for the approach
I took with godot-accessibility--build a screen reader into the engine.
Each platform has its own accessibility API, so screen reader support
means supporting that entire wide surface area. Additionally, I'm not
sure that natively-rendered UIs can export accessibility information on
Android. It's certainly possible to improve the accessibility of custom
widgets, though I'm not sure that support extends to arbitrarily-drawn
pixels on a canvas. By contrast, the custom screen reader approach just
requires platform-specific text-to-speech, for which I've already
developed the tts crate which supports all major platforms except for
Android.

I don't advocate this approach generally, nor would I oppose eventually
putting in the work to bridge to platform-specific accessibility APIs.
Most of the challenge in game accessibility seems to involve changes to
the game rules and world itself (UI-heavy games not withstanding) and I
don't perceive that game UIs are necessarily as expansive as, say, a web
browser or office suite. An embedded screen reader might not get us 100%
of the way there, but it'd get us close enough for all intents and
purposes, and would be much easier to implement.

[stefee]

Thank you _so much_ @ndarilek!! This is really insightful.

Ideally, and especially because we have prior art in the form of Godot, we shouldn’t need to rely on blind/partially sighted contributors to implement this. So sign me up as well.

[stefee]

Copying @ncallaway's comment from Discord.

@ncallaway:
From the todomvc I've been working on the missing pieces that'd be useful from the UI system:

• [ ] Bug: Despawned Node Entity Continues to Affect Layout. #351
• [x] ~Bug: Despawned Node Entities are not removed from their parent Node's Children. #352~
• [ ] Bug: Move Interaction::Clicked to Release not Press (or have both available).
• [ ] Bevy UI Feature: pixel-ratio scaling. #195
• [ ] Bevy UI Feature: Additional styling (e.g. border, box-shadow, text-decoration, etc). [CSS / border] [CSS / box-shadow] [CSS / text-decoration]
• [ ] Bevy UI Feature: Focus model / events (related to #54 ?).
• [ ] Bevy UI Feature: Interaction::DoubleClicked (related to #389 ?).
• [ ] Bevy UI Feature/Research: Animation (related to #91 ?).
• [ ] UI Primitive: Scrolling container.
• [ ] UI Primitive: Text Input component (single-line first, but eventually textarea too).

[stefee]

Based on discussion on Discord, I think #195 should be a priority right now. I'm keen to get going on some of the styling stuff but I don't want to have to re-do too many bits for DPI scaling.

[j16r]

There is an OpenGL accelerated GUI framework in Makepad: https://github.com/makepad/makepad which looks incredible, though the code doesn't seem super well documented nor easily usable outside of makepad.

[follower]

@cart
At the risk of being very ignorant, is screen-reader friendliness a high priority for a visual editor?

Really appreciate @ndarilek took the time to share their own experience with you.

I can also recommend reading the "Why?" section of @ndarileksgodot-accessibility` README. (Also I would like to acknowledge @cart's willingness to admitting not having knowledge around this issue & being open to learning. )

Examples like "Working alongside sighted developers who would prefer a visual editor" really highlight to me the importance that the tools we create not exclude people from being part of a development team by being inaccessible.

And, the reality/virtue of developers being "lazy" means that, as suggested by @ndarilek, the best approach is to have the support included in the base system, enabled by default.

When @cart announced Bevy it was only a few days after Godot had received significant (and justified) criticism in relation to accessibility in this thread on "Making Advanced GUI Applications with Godot" and I'd intended to drop a link to the comments (which I'd recommend reading for more context/perspectives) but hadn't so I'm glad @stefee raised accessibility as a consideration early on.

For people on the Twitters, I can also recommend following \@ianhamilton_ ("Accessibility specialist, helping studios avoid excluding gamers with disabilities") which I've found helpful for gaining further insight into both the positive & negative sides of the current state of accessibility in relation to games.