Nannou: GUI - Conrod? Or roll something nannou-specific?

[mitchmindtree]

It would also be really nice to have a GUI framework that was built upon the components that come with nannou for consistency, such as event handling, graphics, colours, text layout, math (points, vectors, matrices), etc.

Conrod

Seeing as so much work has already gone into it, it would be worth experimenting with how tightly we can integrate conrod so that it feels this transparent and tightly integrated. I can foresee there being some limitations though:

• Conrod will always be 2D-only. Might be able to hack out a way of having a conrod 2D interface on an angled 3D surface though, for example, though might be a bit hackish.
• Conrod uses many of its own types that duplicate functionality from nannou and might feel inconsistent, e.g.
  
  
  
  • [f64; 2] for Point types
  
  
  • geometry types (Range, Rect, etc)
  
  
  • graphics primitives (Rectangle, Triangles, etc)
  
  
  • color module (nowhere near as comprehensive as the palette crate which we will probs use).
  
  

Nannou-specific

If we decide to roll our own, we could still likely take a lot of logic directly from conrod to make the process easier (e.g. text layout, scroll logic, widget graph, etc).

It could be worth experimenting with a design that allowed for both Retained and Immediate style GUI too. I can imagine a retained-mode GUI providing an optional immediate-mode layer on top, where the immediate layer acted as a cache for retained widgets (this is pretty much how conrod works internally anyways, despite only providing an immediate API).

If we're to take this route, it would be worth fleshing out the graphics API first so that this can be built on top.

[mitchmindtree]

Another issue worth considering if we're to go the conrod route is that we'd have to synchronise conrod's glium dependency version with nannou's. This could get a bit annoying if e.g. we want to update to the latest glium but conrod is still a version behind, or if we want to update to the latest conrod but we haven't yet updated to use the same version of glium. This probably wouldn't be a huge issue in practise but is worth noting.

[mitchmindtree]

Dynamic GUI?

One major problem that has frequently occurred when using conrod for GUI in our last few projects is the need for quick iteration. I've slowly come to the realisation that it is simply not practical to manually hard-code entire GUIs with Rust - or really any form of heavily compiled language - purely due to the inevitable compilation time. Even a few seconds is far longer than one should have to wait when such a vast amount of trivial tweaking, iteration and refactoring of GUI layout is involved, especially in significantly complex GUI projects like the spatial audio server. It can make tasks that should take minutes become hours, hours become days, days become weeks, etc.

One thing I'd like to start thinking about is how to best approach a fully dynamic (yet type-safe) GUI system - one that would allow creating entirely new widgets capable of producing near arbitrary events, all at runtime. Currently, this is not trivially possible with Conrod. While the immediate mode approach of GUI is very well suited for writing GUIs in code, it makes it particularly difficult to treat widgets as data and programatically compose them together to create new ones at runtime. If instead we can come up with a system that does easily allow for this in a scalable, efficient fashion, I think it could become a standout feature within Nannou.

Requirements

Here are some basic requirements that would massively increase flexibility compared to our situation with conrod today:

• Create new widgets that may produce unique events at run-time.
• Serialization of both widgets and entire GUIs at run-time.
• Serialization of widget events at run-time.
• Serialization and modification of styling and themes at run-time.
• Modify the entire layout and ordering of a GUI at run-time.

Performance

Inevitably, this will mean moving more work from compile time to run-time. Composition of widgets that would previously be hard-coded and statically compiled will now require some sort of representation in memory. GUI layouts that would previously be described by builder methods and the ordering of hard-coded widget instantiation will now need to be described in a way that can be changed while the program runs.

Fortunately however, none of these requirements ever come close to showing up as a bottleneck within GUIs of any significant size. Currently, there are two primary bottlenecks that show up in conrod GUIs.

1. Rendering the GUI to the screen. There are many optimisations implemented (and even more yet to be implemented) that revolve around the idea of avoiding drawing anything that does not have to be drawn when the system is ready for a new frame to be displayed. The translation of the GUI state into a buffer of vertices and the delivery of those vertices to the GPU for display - these steps almost always show up as the most significant hit during profiling and need the most attention.
2. The set_widgets or "update" pass. One of the downsides of immediate mode GUI is that, at least to some extent, all widgets that should currently be viewable within a window must be instantiated each time an update occurs. Most often, widget implementations will cache as much expensive computation and resources as possible in order to make this instantiation cost minimal, however in GUIs that involve thousands of widgets or more this cost can eventually reach a point of significance. The cost becomes even more noticeable when the GUI is required to update every frame, perhaps in response to some event generated within the program itself as is commonly the case in games or real-time applications.

In the proposed dynamic GUI system, the first bottleneck described above would be no more or less optimisable than it currently is. This is because the process of rendering the GUI is already a dynamic, run-time process. If the GUI system can be designed in a way that is general enough to allow for the requirements mentioned above, then it might also allow for a more generalised analysis of exactly what elements can be cropped / re-used / optimised away.

The second bottleneck described above would become a non-issue with a more run-time oriented GUI approach. This is because this entire bottleneck is a side-effect of the immediate-mode design, one of the purposes of which is to ease the burden of manually writing out GUIs in code - a problem that doesn't exist if we are moving towards a run-time oriented approach.

Design

Like conrod, the GUI would need a fundamental set of "special", primitive widgets, from which all other widgets may be composed. These might include:

• Geometry and shapes (lines, circles, points, spheres, etc).
• Colour areas (triangulate-able areas where each vertex can be coloured uniquely to create arbitrary gradients, etc).
• Texture areas (triangulate-able areas where each vertex can be aligned with co-ordinates upon some texture).
• Interactive areas (areas that emit some event when scrolled, clicked, touched, key pressed, etc).
• Crop areas (that limit interaction and visibility).
• Text (perhaps supported by a combination of geometry and texture areas, but probably want optimised GPU cache-able rendering by something like rusttype).

This list should aim to be the absolute minimal set of widgets needed, from which all other widgets can be created in an efficient manner. A simple example would be a Button composed of a Rectangle, ColourArea and an InteractiveArea.

The Unsolved

One slightly trickier design problem is how to allow for this dynamic GUI representation to interact with the logic of the rest of the program. E.g. What is the ideal way for widgets to receive unique events from the program, emit certain events that can be handled by the program and to alter each other's layout at run-time. When implementing a List widget, how should we allow for the widget to specify that all child elements should shift upwards when a user scrolls down but only until the bottom element reaches the bottom of the list area? How can we allow for a custom Slider to both emit values upon interaction as well as receive and snap to certain values upon certain program events? This is something that conrod and the immediate approach in general is very good at. It's easy to code complex behaviours and interactions between widgets when it's just code, and the immediate mode design means that the GUI is always a reflection of the application state (though at the cost of requiring widgets to be frequently instantiated). Going to have to think more on this.