minetest 🚀 - Formspec replacement

Using HTML/CSS, QT, or GTK would probably be too much for a game.

I suggest something like tgui or SFGUI as a backing library. We could reflect their API in Lua, and serialize and send to the client using JSON.

rubenwardy on 13 Oct 2017

i agree with that, those aren't exactly lightweight GUI's
however wxwidgets might be a possibility as it is automatically cross-platform.
but idk how much it is supported

ThomasMonroe314 on 13 Oct 2017

i have a partially working XML format in a branch: https://gitlab.com/project-demeter/core/tree/form_xml
The only missing patch is catching events from XML to core, other parts are managed and just need more and more components, it's XML with HTML-like attributes)
It's using libxml2 which is portable and very performant, used by many softwares

nerzhul on 13 Oct 2017

what about this: http://www.clutter-project.org/
it seems to be very lightweight and runs well on Android

ThomasMonroe314 on 13 Oct 2017

nuklear looks good, supports theming and it's very lightweight compared to Qt/GTK.

weqqr on 13 Oct 2017

👍5

@weqqr that looks like exactly what openarena uses, and its GUIs look pretty nice
but does it support android?

ThomasMonroe314 on 13 Oct 2017

@ThomasMonroe314 nuklear itself only receives user input and generates a list of shapes to draw, so in theory it can run on everything from a supercomputer to an electric kettle. You need to write a frontend (like this) to make it work with Irrlicht though.

weqqr on 13 Oct 2017

Wanting a CSS themeable, extendable (by CSM) GUI using a completely new GUI toolkit is aiming way too high IMO.
It pretty much guarantees that it won't ever be fully implemented and we'll be stuck on formspecs for a longer time.

Also having something like minetest.register_window removes quite a lot of dynamicness from the gui system.

(The rest of this post is just throwing my ideas in here.)

Instead I would fix "just" these issues:

coordinate system
proper layouting, meaning: containers with children
use nested tables to represent these (not exposed to Lua, should have object-oriented API)
proper event system: wide variety of events for each element, all optional

This retains the following from formspecs:

Toolkit-independent
dynamic ("made up on the spot", no pre-registered layouts)
events are produced in certain conditions, these can be individually listened for
uses Irrlicht GUI (probably)

There is likely no existing solution for this, but that's fine.
Since this is quite hard to put into words, some example code:

local window_name = minetest.get_current_modname() .. ":respawn"
local pad = 50

local function show_respawn_dialog(playername)
    local root = gui.Window:new(window_name, 100 + 2*pad, 200 + 2*pad)
    root:listenForEvent("close")

    local stack = gui.Stack:new()
    stack.padding = {pad, pad, pad, pad}
    root:add(stack)

    local button1 = gui.Button:new("Respawn", 100, 100)
    button1.id = "b_respawn" -- must be unique
    button1:listenForEvent("click")
    stack:add(button1)

    if creative.is_enabled_for(playername) then
        local button2 = gui.Button:new("Respawn at same position", 100, 100)
        button2.id = "b_respawn2"
        button2:listenForEvent("click")
        stack:add(button2)
    end

    minetest.show_gui_dialog(playername, root)
end

local function respawn_normally(player, state)
    assert(state["b_respawn"].pressed == true) -- just an example for state
end

minetest.register_gui_event(window_name, "b_respawn/click", respawn_normally)
minetest.register_gui_event(window_name, "close", respawn_normally)

minetest.register_gui_event(window_name, "b_respawn2/click", function(player, state)
    -- do something else
    return true
end)

internally/when sent via network it would look like this (no modder would ever see this though):

{
    "type": "window",
    "id": "testmod:respawn",
    "events": {"exit": true},
    "children": [
        {"type": "stack", "children": [
            {"type": "button", "id": "b_respawn", "events": {"click": true}},
            {"type": "button", "id": "b_respawn2", "events": {"click": true}}
        ]}
    ]
}

To achieve backward compatibility you would then need to:

Convert the formspec string to tables (duh)
Translate the stupid coordinate system to the new one during conversion
Give each element a default set of listened events (e.g. buttons always respond to clicks)
Convert the new events to the old system and call minetest.register_on_player_receive_fields callbacks

sfan5 on 13 Oct 2017

👍10

nuklear has lua bindings, a thing that i haven't seen anywhere else, and something that is straightforward, simple to use and should be simple to implement

ThomasMonroe314 on 13 Oct 2017

👍1

nuklear has lua bindings, a thing that i haven't seen anywhere else, and something that is straightforward, simple to use and should be simple to implement

Doesn't help when we need SSM to be able to make windows/dialogs

@sfan5, all of that sounds good to me. I actually prefer it to mine, so updated first post

It's fine to leave stuff out for an initial implementation, but it needs to be thought about to allow future additions. Yours does however, you could add theming without breaking

rubenwardy on 13 Oct 2017

pardon my ignorance, but what is SSM?

ThomasMonroe314 on 13 Oct 2017

pardon my ignorance, but what is SSM?

Sorry, server-side mods or server-side modding.

rubenwardy on 13 Oct 2017

ah ok thanks, but why is that a problem? cant it be used for that?

ThomasMonroe314 on 13 Oct 2017

ah ok thanks, but why is that a problem? cant it be used for that?

Their API will likely be directly manipulating elements, where as we need to send commands over the network to the client.

I suggest keeping both the new API and formspecs in parallel whilst developing, then making a server-side formspec to new GUI converter when stable enough

rubenwardy on 13 Oct 2017

hmm true, but as weqqr pointed out, we would need to write a frontend, so couldnt we add some that functionality to the front-end?

ThomasMonroe314 on 13 Oct 2017

I really like sfan5's idea, but not the API, so here's my concept.

local function say_hello(window_state, player)
    print("Hello, " .. window_state.name.text)
end

local function show_window(player_name)
    local window = minetest.window("mymod:window", function(w) w
        :size(200, 200)
        :field("name", function(f) f
            :default_text("someone")
            :position(50, 50)
        end)
        :container("my_container", function(c) c
            :position(60, 60)
            :padding(10, 10, 10, 10)
            :button("hello", function(b) b
                :text("Say hello!")
                :position(50, 50)
                :dimensions(100, 100)
                -- Callbacks can be set in-place
                :on_press(say_hello)
            end)
        end)
    end)

    minetest.show_window(player_name, window)
end

-- Complex callbacks can be handled outside of the window definition
minetest.register_on_gui_event("mymod:window/my_container/hello", "press", function(window_state, player)
    print("Your name is " .. player:get_player_name())
end)

minetest.register_on_gui_event("mymod:window", "close", function(window_state, player)
    print("Window closed")
end)

weqqr on 13 Oct 2017

❤1 👍1

I feel 'themeable' is low priority and should not count against a good, simple, practical solution.
~~MTs neutral grey boxes are visually fine and no problem.~~

paramat on 13 Oct 2017

👎12

MTs neutral grey boxes are visually fine and no problem.

MT's current GUI is ugly. The new GUI can be tweaked to look better without breaking things, and margins will be better as you would be able to specify exact margins. Theming is relatively low priority, and should not be in a first implementation - but it should be considered to make sure any first implementation could support it in the future. I think any good consistent and DRY solution would be able to, anyway

rubenwardy on 13 Oct 2017

👍17 ❤4

Hmm my statement there was too strong, even i like to have colourable boxes instead of only grey / black. Being able to design the background graphic is a good feature to have.

paramat on 13 Oct 2017

@weqqr That jQuery-like syntax is awful IMO.
@sfan5 What’s the problem of needing to register windows before showing them? That would be handy IMO, as long as the registration may be done at any time and not only on server load: you could setup the whole huge multi-tab window layout, all the callbacks, etc., and only set some content right before showing it.
Also, that way nodes could store window name in the meta instead of its full specification, thus easing updates.

numberZero on 13 Oct 2017

My thoughts (Some of these have been mentioned already in the thread):

There should only be a CSM GUI API (maybe with a compatibility layer for old-style formspecs). This would need to wait for servers being able to send client mods.
The API should have some way of drawing every frame so that mods can animate GUI components, making it possible for mods to create new interactive GUI elements, like selection wheels, without hardcoding engine code to handle them.
Layout should be automatic, i.e. you don't need to manually specify coordinates for everything if you don't want to.
The same API could be used for client-mod provided HUD elements.
(Optional) I like the idea of imgui's API. I've never actually used it though so it might not be as good as I imagine.

raymoo on 14 Oct 2017

👍2

MTs neutral grey boxes are visually fine and no problem.

Now your really pissing me off

Hmm my statement there was too strong,

Ooook then nevermind

But all jokes aside those "neutral grey boxes" look awful and they have to go they look unprofessional, they are ugly, they make minetest look like test software (the word test in minetest doesn't help with that) and they are NOT CUSTOMIZABLE WITH TEXTUREPACKS!!!!!!

tobyplowy on 14 Oct 2017

👍3

Using HTML/CSS, QT, or GTK would probably be too much for a game.

I'm in with QT and GTK but imagine how awesome it would be having HTML-based dialogs and menus and stuff. Of course not full HTML5 and CSS3, but a base set of HTML syntax (img, input, section, div, and some more) plus some Minetest-specific additions (for the inventories for example) would be friggin awesome!

ghost on 20 Oct 2017

👎1

@dsohler IMO some XML-based language would be way better: it is easier to parse and more flexible. E.g. mods could be able to define their own elements (using XML namespaces... oh yeah, it uses the same namespace:element naming convention as MT does...)

numberZero on 20 Oct 2017

@numberZero Using pure XML and then parse seems nice even if you limit the possibilities and make it harder to do something. If a dialog is nothing more than a viewport for a simple HTML parser to output the parsed HTML you basically could create websites within dialogs … which might not be uber useful within a game like Minetest but why limit modders? :smile:

Modders could even introduce own tags to the markup since HTML5 absolutely allows that and makes it very easy to do so.

ghost on 20 Oct 2017

@dsohler What do you consider that good in HTML? It’s old, bloated and messy; it is hard to lay out correctly, or even to lay out at all (remember how buggy all the web browsers were...). Also it doesn’t suit the GUI needs well.

numberZero on 20 Oct 2017

@numberZero I’m not talking about www-centric HTML 4 and earlier because yes, that sucks. I’m talking about a narrow subset of HTML5 (that would allow most coders and designers to create dialogs because they already know HTML). And yes: HTML is not a layout language at all. Back then it had design elements and those were overused and stupid bullshit. Fortunately nowadays HTML has nothing to do with that nonsense.

But yes: Instead of a HTML subset we could use XML and use a built-in XSLT file. Using a self-made XSLT processor it would even be possible to completely separate dialog definitions from core and have the API only talk through the XSLT processor. Mods could then implement their own syntax depending on whatever the mod author likes and add a XSLT file and send that to the processor and … :boom: But … oh well … that actually might be a little too much :smile:

But dialogs should absolutely be defined using some sort of semantic markup language instead an unnecessary complex and unflexible string that’s passed around.

ghost on 20 Oct 2017

@numberZero @dsohler So you imagine a Lua API to build a (dynamic) XML file which is sent to the players? Having HTML sounds nice but the libraries will be very heavy. Every library suggested here already offers more different elements than our formspecs do.
However, Minetest-special elements would still have to be added, like the inventory list.
nuklear (thanks to @weqqr for mentioning) looks very promising. A single header to include, with simple API functions. Even simpler are those from imgui, where the C++ examples almost look like Lua code. For the latter there's also an Irrlicht binding available, which embeds it perfectly into Irrlicht. But with the side effect of being a quite inactive project.

SmallJoker on 20 Oct 2017

👍2

@SmallJoker Yes. Whatever modern format (HTML5, XML, JSON, etc.) it will be is sent to the clients and parsed there using whatever parser is needed to generate the output. HTML would be awesome in combination with CSS and a proper HTML parser on client-side for maximum flexibility, XML would be nice because mod authors could create an own XSLT file and have the parser interpret whatever syntax the mod author prefers and JSON … well … the overhead is virtually nonexistent here.

ghost on 20 Oct 2017

@SmallJoker it's why i propose to use libxml2 or irrlicht XML parser (very low level) and create our own DSL

nerzhul on 22 Oct 2017

I agree with what @raymoo said about layout above

Layout should be automatic, i.e. you don't need to manually specify coordinates for everything if you don't want to.

With the current hard-coded layout, you can't handle different DPIs, different font/font sizes/translations being different length. This is the root cause of #6521 and #4607.

I don't see a way to fix those issues (not those specific cases, but the actual base problem) without using a GUI that can change based on the screen size available.

bvanrosendale on 25 Oct 2017

Please add the following items (from the modder's perspective):

Must have:

Complete and proof-read documentation, including, but not limited to:
- Available widgets and what they do
- Syntax
- What happens if user exists / hits Esc?
- Coordinates
- How to parse fields in on_receive_fields (for any given widget)
- Gotchas (if any)
Define which events trigger on_receive_fields and be consistent
Fix #4482

Should have:

Scrollable automatic-line breaking multi-line text: #3807
No gotchas, surprises, weird return values, etc.

Could have:

Clickable text (like hyperlinks)
Allow to check for supported widgets: #4834

General stuff:

I don't care much about the syntax
Please don't waste time on overly “clever” API designs, formpec syntaxes or other meta bullshit. That's not important; I just want something that works
I'm pretty much OK with the suggestions posted so far
I don't care if you use a GUI toolkit or do everything by hand if it works at the end
I agree that theming should be considered (not implemented!) as early as possible (otherwise it will be painful to add as an afterthought)
I'm not opposed to breaking changes if the result is a serious improvement over our current system. I guess the fucked-up coordinate system alone will already force you to break things anyway

Wuzzy2 on 27 Nov 2017

I'm not opposed to breaking changes if the result is a serious improvement over our current system. I guess the fucked-up coordinate system alone will already force you to break things anyway

It's possible to avoid breaking changes, however not doing so simplifies implementing a new system considerably.
We need to ask ourselves whether breaking every mod that uses formspecs is worth it.

sfan5 on 27 Nov 2017

I wonder how you can get around the fact that the coordinate system is wildly inconsistent and doesn't make any sense.
How are you going to fix this without breaking things? Adding a full-blown compability layer?

Wuzzy2 on 27 Nov 2017

Adding a compatibility layer is the only option, yes.

sfan5 on 27 Nov 2017

Imho, the best would be to be able to define formspecs without using any sort of string representation, neither formspec strings nor xml nor json, but directly through Lua in a similar way as to how it works in Tcl/Tk and other toolkits or frameworks with similar structure (JQuery UI and so).

The main problem with the coordinates in the current markup is that the properties are not named and you have to memorize the order and meaning... just having them properly named in a consistent manner (eg. name "pos" to the x,y position and add "width","height" as separate named fields) inside a table that is used as paramater to an appropriately named function for each widget would already be a significant improvement, imho.

Even if under the hood this translated to the current formspec format, it would already allow for consistency, stop caring about the parameter order and avoid any syntax errors in the string. You could easily make use of the lua syntax checker that you are meant to be using already.

I believe there was already an attempt to do something like this in the past, by abstracting the formspecs in Lua from a mod (was it this one? https://github.com/minetest-mods/smartfs ).

A way to bridge the old formspec with a new system like this would be to support, adding key names to the properties in guiFormSpecMenu.cpp. Instead of using a mix of split(element,';')/"split(element,',')" every time to get the list of arguments, use a reusable method to parse them and allow for optionally expressing the parameters in arbitrary order such as:

list[pos=0,3.5;size=8,4;inventory=current_player;name=main;]

This could be done in such a way that old strings that do not specify any named key would still be supported.

Then turning this into a more streamlined representation in the spirit of smartfs would be more straight forward.

local button1 = state:list({
  name = "main",
  pos = { 0, 3.5 },
  size = { 8, 4 },
  inventory = "current_player"
})

It would also work for an xml representation, if you do really want to go that way (but honestly I don't see why add yet another format, we are already using lua).

Ferk on 27 Nov 2017

It's true that a layouting GUI could be implemented on top of formspecs already. It might be interesting to work on something like that in modspace.

raymoo on 27 Nov 2017

I don't think it's a good idea to hack a new system into the existing string format just because that makes backwards compat "easier".
If backwards compat is desired, a converter would be a better approach.

sfan5 on 27 Nov 2017

👍7

Could have

Internationalization

numberZero on 27 Nov 2017

👍1

Should be possible with client-side translations. Mod authors just have to provide the translation in the needed format.

ghost on 27 Nov 2017

I think we don’t need any complex text layout system like HTML/CSS. Some flexibility may be needed to help localization (and to allow using larger fonts), but not that much, that’s GUI and not a web page; RTL may be handled by mirroring the whole form, in a sense. Several simple, well-defined layout patterns would be enough.

And HTML/etc. browser might be added as a special (optional) control, if necessary.

numberZero on 27 Nov 2017

Just my opinion, but I don't think there are team resources to handle yet another format. Specially if it has to deal with an entirely new additional parser and has to include an entire conversion and compatibility layer. HTML (both in its SGML and HTML5 variants) is not precisely a simple format, not to mention how crazy it is to render CSS correctly.

I'd say see what is wrong in the current format and improve it. My suggestion was actually meant as well as an improvement on the current format to fix what seems to be the issue. I assume the issue is not simply that it's not xml/json. If the problem is the syntax you can easily use a lua layer and get a syntax checking that's very fitting. If the problem is the inconsistent order and meaning of the parameters, name the parameters with consistent meaning. They actually do make sense, but the thing is that different widgets have different parameters and this makes the interface

Other things like the adapting containers, themes or scrollbars could be also included in the formspec, if they actually were supported by the underlying GUI. I don't believe that the formspec format is really the blocking reason why any of these features were not added before.

If you were to completelly reimplement the GUI in 0.5 with a different layouting system without caring about compatibility, I'd say go for it. But if you actually want to keep compatibility, why not do so improving the current formspec?

Ferk on 28 Nov 2017

The formpec format is really the least of our problems and I think it is even counter-productive to waste time by starting from basically 0.

There is nothing inherently wrong with the formspec strings. Quit talking about syntax all the time, this stuff really is not important! All you're going to achieve with this is syntactical sugar.

We haven't even talked about which coordinate system to choose for the new system.
Let's start here.
On first thought, I would go with inventory slots (for one simple reason: it's already used for most widgets), but use it for ALL widgets consistently and not with some weird exceptions and random offsets.

But I'm open to other suggestions.

Wuzzy2 on 28 Nov 2017

👍1

We haven't even talked about which coordinate system to choose for the new system.

Coordinate systems are bad and don't scale, containers would be better (linear, stack, table, etc)

Why do you all care about compatibility?

Because mods are our biggest asset?

If backwards compat is desired, a converter would be a better approach.

I think the best approach is to just have both simultaneously. They would both derive from GUIModelMenu, so the code wouldn't be too complex, just with some remaining functionality.

rubenwardy on 28 Nov 2017

👍1

I'd say see what is wrong in the current format and improve it.

I think it is even counter-productive to waste time by starting from basically 0.

Oh yes, why start from scratch and design a good format when you can just tack a few half-broken hacks onto the current format?

sfan5 on 28 Nov 2017

👍2

Wanna stop discussing and just start coding? ;) “Dear ImGui” seems to be the best choice for the backend. It’s lightweight, C++, doesn’t depend on other libraries. It’s not very customizable (e.g. it lacks theming) but should be hackable enough to fix that.

Disclaimer: I haven’t been using any GUI library for a while, so could miss something.

numberZero on 28 Nov 2017

Imgui is for prototyping, so isn't appropriate. A formspec replacement isn't a priority for 0.5.0 anyway

rubenwardy on 28 Nov 2017

why start from scratch and design a good format

Unless you have a lot of time and resources to maintain the compatibility layer, it'll be hard to start from scratch if you need to be backwards compatible and yet change the spec in fundamental ways that would truly not be suitable to support in the old formspec.

If the change required is suitable for being backwards-compatible, you might as well instead improve the old format to make it more extensible (formspecs are just elements with attributes after all).

If the change is truly not suitable and you are ok with the redundancy you can add it as a separate parallel implementation that doesn't need to be converted or related with the other in its interface, like Rubenwardy suggests. But I don't think it's sensible to expect a big fundamental change and yet design a layer of backwards compatibility that has to be maintained.

Ferk on 29 Nov 2017

Oh yes, why start from scratch and design a good format when you can just tack a few half-broken hacks onto the current format?

Because I don't think the current formspec syntax is broken. After years of modding, I never ever had a problem with the syntax. I don't get why some people claim there are problems.
Feel free to convince me that there is a problem.
But as I said several times earlier, I think the big problems are somewhere else.
So, if the syntax isn't broken, then adding a full-blown API layer with Lua tables and other fancy stuff is really just syntactical sugar.

I couldn't care less if I have to specify the formspecs via a string or via fancy function calls, but I think having a fancy syntax is really the least important thing to discuss about formspecs.

We haven't even talked about which coordinate system to choose for the new system.

Coordinate systems are bad and don't scale, containers would be better (linear, stack, table, etc)

I'm not sure what to think about this. Care to elaborate? I mean, how would I, as a modder, implement this? Let's say, how would the default Minetest Game be specified under your proposed container system?
Do you claim to have a system which does away with all coordinate systems altogether?

About using a GUI toolkit: I honestly don't understand why a GUI toolkit just hasn't been used right from the start. Implementing all widgets by hand is a time-consuming process. We are still lacking very basic widgets like scrollable multi-line text.
Whatever you do, I think the new system definitely needs to provide a healthy set of core widgets.

Wuzzy2 on 29 Nov 2017

Imgui is for prototyping, so isn't appropriate.

@rubenwardy explain please.

Feel free to convince me that there is a problem.

@Wuzzy2 yes there is a problem. There is a great problem. Not that the syntax itself is ugly, but that it forces to use ugly Lua code to generate that ugly formspec string and to regenerate it all the time. That it’s certainly not flexible enough to support e.g. multi-page containers (they are emulated currently using server’s capabilities). That it mixes controls and modifiers (although you might see the same in Borland Delphi/C++ Builder/etc., if you ever used that... Well even there, in a full-featured IDE, that wasn’t that handy).

By the way, can you write a formspec designer? Can that be done in a sane way at all?

numberZero on 29 Nov 2017

Not that the syntax itself is ugly, but that it forces to use ugly Lua code to generate that ugly formspec string and to regenerate it all the time.

That calls for a change of the system in which the formspec is used, rather than its syntax. You'll still have to do that even if it was in xml, json, lua tables or any other format. It has to do with the formspec being manipulated server side. Deeper changes would be needed if you don't want to regenerate the formspec every time the server wants to change something from it.

it’s certainly not flexible enough to support e.g. multi-page containers

There's already a container widget, defining a start and an end. Is the blocker for a multi-page one really the syntax? or the underlying GUI implementation?

That it mixes controls and modifiers

That's more of an issue of the implementation than of the format. This could happen as well with json or even with XML.

True that changing this could break compatibility (if you actively want to remove them, not just provide alternatives), but not a reason on itself to justify so much work and breakage, it's more of a thing of taste than pragmatism.

By the way, can you write a formspec designer? Can that be done in a sane way at all?

Most elements are just position and size. A designer wouldn't need to parse the formspec, just generate it, so I'd say the part that would be hardest is enforcing the coordinate limitations when placing controls and making sure it all looks like in Minetest. It certainly won't be easy, but under the same constrains a different format wouldn't make it any easier either, the difference would be in the generation of the output string.

I'm guessing you are assuming a new syntax is linked to different constrains. I think that's a bit like starting to build the house from the roof.

First determine what constrains are really wanted, then see if current formspecs can satisfy them in a reasonable way, and if not, then change the syntax.

Imho, there are 2 things that might be a problem in terms of syntax:

nesting
syntax checking

The nesting, depending on how the current container was done, might be already there, not sure (even if ugly for anyone used to xml / json / curly braces ...it wouldn't be the first language that defines blocks with start and end instructions).

The syntax checking, honestly to me the sanest would be to use not xml, nor json, but lua (since it's what mods are written on, so the syntax checking is for free). If lua is not gonna be used then I don't care too much about the syntax, because it's likely I won't be wanting to constantly copy-paste the strings to a syntax checker, even if they are XML, and I'm afraid if they are external files I can open in an IDE it might start becoming a hassle making changes to them on runtime. Not needing a separate file is also convenient when you have several simple formspecs as opposed to having just a few but complex ones. Thought I guess one could get used to it, I already had to get used to MyBatis at work after all.

Ferk on 30 Nov 2017

👍1

Pretty much what Ferk said. I'm still not convinced the syntax is inherently broken. I'm not opposed to a syntax change, I just think it's not worth the effort.

Wuzzy2 on 30 Nov 2017

The container tag is essentially implemented as a preprocessor. When the formspec is parsed, it adds the offset to all elements between the tags. The container element doesn't exist after then. It wouldn't be appropriate for a multipage widget.

rubenwardy on 30 Nov 2017

That calls for a change of the system in which the formspec is used, rather than its syntax.

Yes, mostly.

This could happen as well with json or even with XML.

Of course. There are many bad JSON-based formats.

I think that's a bit like starting to build the house from the roof.

I don’t mean we need to develop the syntax first. I just mean we don’t need, and should not (IMO) stick to the current syntax in any way while developing the new system.

Most elements are just position and size. A designer wouldn't need to parse the formspec, just generate it

Many controls are dynamic, so the designer would have to output formspec generator and not the formspec itself. Well, okay, it’s certainly possible to output Lua code that creates a formspec with given data. It actually resembles somehow the way many UI designers work. But in these other designers that is either for convenience (as in Delphi, where the generated code is fully optional and only GUI description is used to actually construct it) or because the language (C++) lacks something important (references to classes). Lua does support everything necessary. Only formspec API doesn’t.

numberZero on 30 Nov 2017

Let's use this as a tracker for all problems / bugs / shortcomings of the current formspec API as well. It would be best that if formspecs are being redone, everything should be fixed at once.

Formspec input resets after resize: https://github.com/minetest/minetest/issues/3732
Formspec names and node formspecs are insecure: https://forum.minetest.net/viewtopic.php?f=47&t=19129 (I haven't found an issue yet)
Please post more issues :-)

@rubenwardy, can you please update your 1st post (also consider previous posts for input)?

Wuzzy2 on 4 Jan 2018

What's the status on this? who will make a call on architecture for this and set milestones?
How can i contribute to moving this forward? Is someone already working on a solution based on this discussion?

QwertyCoolMT on 5 Feb 2018

❤1 👍1

I think, as usual for these kinds of big things, the best way to move it forward would be to PR a good foundation. I don't know if anyone is working on it, but I would guess probably not.

With such features, usually the person making the initial PR architects the rest of the system, subject to feedback from MT devs and other users.

raymoo on 5 Feb 2018

@numberZero

Dear ImGui is designed to enable fast iteration and empower programmers to create content creation tools and visualization/ debug tools (as opposed to UI for the average end-user)

from their read me

rubenwardy on 6 Feb 2018

The container tag is essentially implemented as a preprocessor. When the formspec is parsed, it adds the offset to all elements between the tags. The container element doesn't exist after then. It wouldn't be appropriate for a multipage widget.

Even if it's implemented as a preprocessor, it is still possible to expand on that to implement a paging system.

Example:

Before pre-processing:

container[pos=1,1;id=pets]
  page[]
     image[pos=0,0;size=1,1;texture=cats.png]
  page_end[]
  page[]
     image[pos=0,0;size=1,1;texture=dogs1.png]
     image[pos=0,1;size=2,2;texture=dogs2.png]
  page_end[]
  page[id=whatever]
     image[pos=0,0;size=1,1;texture=nyan1.png]
  page_end[]
container_end[]
pager_button[pos=0,0;size=1,1,label=cats;activate=pets,1]
pager_button[pos=1,0;size=1,1,label=dogs;activate=pets,2]
pager_button[pos=2,0;size=1,1,label=huh?;activate=pets,whatever]

After pre-processing:

image[pos=1,1;size=1,1;texture=cats.png;page=pets,1]
image[pos=1,1;size=1,1;texture=dogs1.png;page=pets,2]
image[pos=1,2;size=2,2;texture=dogs2.png;page=pets,2]
image[pos=1,1;size=1,1;texture=nyan1.png;page=pets,whatever]
pager_button[pos=0,0;size=1,1,label=cats;activate=pets,1]
pager_button[pos=1,0;size=1,1,label=dogs;activate=pets,2]
pager_button[pos=2,0;size=1,1,label=huh?;activate=pets,whatever]

Then, client side, hide any formspec element that has a "page" field different than the first found in each section of the same "page" set of values.

When the user clicks one of the "pager_button", check (client-side) the value of the "activate" field and make it be the current page, unhiding/showing all items marked with that page and hiding the rest. Just as a client-side visual effect, without really sending any packets to the server.

You can even chain this to have several layers of pagination.

I'm not saying this is the best way to implement it, just giving an example that shows it's not syntax what's blocking this.
If reimplementing the entire formspec system, adding new dependencies, stitching them to the engine and trying to maintain a compatibility layer is faster, easier and more maintainable than improving the current formspec system, then awesome, do that instead.

Ferk on 9 Feb 2018

I'm not sure if it is a good idea to require an image for every widget.

At least make it optional? Quick and easy colorization of widgets is also important. With baked-in images, that's not so simple.

Wuzzy2 on 14 Feb 2018

The easy way to make an image button act like a normal button is leave the image part of it blank or set it to blank.png. I think making duplicates would be unneccesary.

v-rob on 15 Feb 2018

It's better to go with vector-based stuff rather than bitmaps, since those support scaling and hiDPI out of the box.

Calinou on 15 Feb 2018

👍1

I was poking around and found a neat GUI library that is:

themeable
works with Irrlicht
should be fairly easy to implement into the modding API
can be scripted using Lua
cross platform

TMcSquared on 10 May 2018

i like cegui and i like the products listed on the site. It seems to be mature, but it's a new dep.

nerzhul on 10 May 2018

A new dep is perfectly fine for something as important as a GUI system

rubenwardy on 10 May 2018

While "can be scripted using Lua" is nice, it's not what we need.
We need a GUI library that can work "over" the network, since formspecs are shown on the client but the code that handles formspecs runs on the server.

Just a reminder, since this is not the first time someone has suggested a GUI library because it has Lua bindings.

sfan5 on 10 May 2018

@sfan5 i don't think GUI library over network exists somewhere. What i note here it's cegui permits to drop many shitty irrlicht old and crappy code on formspecs, and we can create a new formspec protocol for this new UI systems in parallel and drop formspecs at a point. (or keep the compat but the formspec format is... anoying)

nerzhul on 10 May 2018

A new GUI library can be used when writing a replacement, yes.
It's just doesn't solve the problem this issue is about: a replacement for Formspecs

(That doesn't mean it doesn't belong here, GUI libraries are still useful to know.)

sfan5 on 10 May 2018

The only useful part of the Lua binding is if it has the ability to make custom elements using it already. Otherwise it's not a benefit of the library for us

rubenwardy on 10 May 2018

formspecs are bound to irrlicht, we can make formspecs cegui aware, and it can be nice :)

nerzhul on 10 May 2018

No, formspecs need to be killed completely.

rubenwardy on 10 May 2018

👍9 👎1

The only useful part of the Lua binding is if it has the ability to make custom elements using it already. Otherwise it's not a benefit of the library for us

I believe the Lua might be only for animating the GUI, not sure on that though. But for the record, that is very cool nonetheless.

TMcSquared on 10 May 2018

formspecs are shown on the client but the code that handles formspecs
runs on the server.

That’s correct, currently. Although that may not be the best way. For
various reasons, it may be better to run GUI-managing code on the
client, and let the server communicate with it, rather than control
GUI on almost per-frame basis. That is, MT might expose the GUI library
to CSM, and have some builtin CSM to support server-supplied formspecs.
But other mods could have their own client-side parts using the GUI
library directly.

В Thu, 10 May 2018 07:11:09 -0700
sfan5 notifications@github.com пишет:

While "can be scripted using Lua" is nice, it's not what we need.
We need a GUI library that can work "over" the network, since
formspecs are shown on the client but the code that handles formspecs
runs on the server.

Just a reminder, since this is not the first time someone has
suggested a GUI library because it has Lua bindings.

numberZero on 12 May 2018

I would rather implement GUI in Lua. Also I think we should try to keep backwards compat.
Formspecs should probably work pretty much like HTML forms - you can do some preeval with lua, client side. In the end, the submission goes to the server.

For everything dynamic, requests have to be made explicitly. In general, MT needs such a request system. Probably worth another issue.

appgurueu on 9 Feb 2019

Do you think it would be possible to add IME (Input Method Editor) support in the features a GUI replacement should have? If you ever want to have support for Asian language inputs, you will need that at one point. Now that's a string of yarn that I understand most people don't want to pull, but that's why having IME support in the GUI framework (and making sure unicode works there but I think most recent UI framework wikk have good support) allows for some peace of mind.

yquemener on 10 Aug 2019

That feature is irrelevant to the GUI system used. It could already be added to our current GUI system, problem is that it needs to be added to irrlicht (apparently?)

rubenwardy on 10 Aug 2019

It can be added to any UI system (that supports unicode correctly) but it is a pain to do so. Some framework (I know Qt has) have done all the hardwork of taking into account the shortcuts, additional keys, unusual keyboard layouts and presenting a working input. That's definitely a "nice to have" feature.

yquemener on 11 Aug 2019

My proposal for a replacement for the formspec syntax is to use a Lua DSL:

gui.Window {
    width = 360,
    height = 230,

    gui.Button {
        name = "btn1",
        w = 4,
        h = 0.8
    },
}

This would be serialised to JSON when sending to the client.

This has the benefit of not requiring enough format to be added, and to be familiar to Lua modders. It's readable and clean.

There can also be Lua OOP equivalents for those that would prefer that, like in sfan5's example:

local root = gui.Window:new(window_name, 100 + 2*pad, 200 + 2*pad)

root:add(gui.Button:new("btn1", 4, 0.8)

The new GUI would support a formspec-like coordinate system using a Canvas element, but the user is encouraged to use containers like Stack, Frame, and Grid to position elements.

rubenwardy on 4 May 2020

TBH, I couldn't care less about the concrete syntax of formspecs. The formspec syntax is really the least of our worries. Yes, it's a bit unusual, but I can work with it.

Actual features of formspecs and coordinates that you can actually work productively with is much more important.

Besides, it seems that your syntax proposal is going to be nothing more than a wrapper anyway …

Wuzzy2 on 5 May 2020

TBH, I couldn't care less about the concrete syntax of formspecs

You're one of the only ones

Besides, it seems that your syntax proposal is going to be nothing more than a wrapper anyway …

With formspec containers and styling features, the only reason to replace the formspec format is to have a nicer format. Everything is already possible using the formspec format, it just requires lots of tags and verbose code.

There are multiple parts to this issue, the formspec format is one part.

Actual features of formspecs and coordinates that you can actually work productively with is much more important.

We have been working on more features, especially in UX polish. We have consistent coordinates. The next improvement in layouting is to introduce elements for stacks, panels, and grids, so that calculating spacing and other coordinates becomes largely unnecessary

rubenwardy on 5 May 2020

IMO a very nice feature (enabled by using Lua DSL) would be to be able to specify callbacks as part of the GUI definition, so that no manual tracking of the GUI is required (as is required in formspecs). This could include a callback for the whole GUI firing on any submission (similar to the current formspec handlers, but more convenient), and also callbacks attached to particular elements like buttons. @weqqr suggested something like this earlier in the thread, but it was part of a larger post and didn't get much comment.

raymoo on 8 May 2020

I personally care little for what the syntax will be, but a new one is important. Obviously, the current format is bad. One bad thing is its lack of extensibility. Sure, you can add extra arguments by increasing the formspec version, but this isn't ideal at all. And it's not like every argument is always going to be used. Named arguments are an important thing to have.

Another thing we have to consider is how we will handle callbacks.

v-rob on 12 May 2020

EDN could be an interesting choice for future formspec syntax. It provides all the same things that JSON does (arrays, maps, scalar values), but adds tagged elements, keywords, symbols and sets (and comments among other things).

Syntax wise it would be a short jump from the current syntax:

; comments start with semicolon

; arrays are inside square brackets []

; maps are inside curly brackets {}
; maps are always key value pairs, so they don't need special separators

; lists are non-indexable sequences that are inside round brackets ()

; syntax could be something like this:
(
  {:formspec_version 3} ; keywords start with a colon
  { 
  :size [2 2] ; commas are considered whitespace, so you can drop them
  :real_coordinates true
  :position [0, 0.5], ; it's still possible to use commas, wherever, if so wanted
  anchor [0 0.5]} ; symbols (without colon :) can also be used as keys
)

Or it could be written in a way even more similar to the current syntax, with only brackets being added:

({
  formspec_version[3]
  }
  { 
  size[2, 2]
  real_coordinates[true]
  position[0,0.5]
  anchor[0,0.5]
})

vs

formspec_version[3]
size[2,2]
real_coordinates[true]
position[0,0.5]
anchor[0,0.5]

The benefit (over current formspec notation) would be that EDN is an existing data format, meaning the syntax would be clear. It also allows deep data structures out of box.

EDN is a subset of clojure. That being said, EDN can be used in the same ways as one would use JSON (itself a subset of javascript). JSON has a better support, whereas EDN has more features.

More information:

EDN validator:

http://repo.tiye.me/mvc-works/edn-formatter/

edit: added secondary syntax that is more akin to the current formspec

mercorii on 19 May 2020

EDN has the same disadvantage of formspecs compared to a Lua EDSL that you would not be able to directly embed callback registration. Unlike formspecs though, EDN puts strings in double quotes, complicating escaping strings when constructing formspecs with the help of string literals (in a mod's source file).

raymoo on 20 May 2020

I think you've managed to find a format that looks even more horrific and terrifying than formspecs

rubenwardy on 20 May 2020

I think you've managed to find a format that looks even more horrific and terrifying than formspecs

It's just lisp. Whitespace is a bit weird in that example, in order to keep the syntax looking more like formspec, for demonstrative purposes.

How awful formspec's syntax even is in the end? It is mainly just key value pairs in a sequence.

EDN puts strings in double quotes, complicating escaping strings when constructing formspecs with the help of string literals (in a mod's source file).

It's a good point. (But can't you just use multiline strings with double brackets to mitigate the problem?) Although, the same would still apply to JSON (as it also uses double quotes), if it was left to be openly modifiable in a similar fashion.

You would probably want to use EDN mainly as a transmission format, while keeping it as a backup syntax available on the background, if someone truly wants to manually build the EDN strings, using a formspec like syntax.

EDN has the same disadvantage of formspecs compared to a Lua EDSL that you would not be able to directly embed callback registration

Then that same problem also applies to JSON, which is supposed to be used on the backend of the Lua EDSL, if I have understood correctly? EDN is just doing the thing that JSON would do.

Anyway, you want to have a declarative frontend, on top of your data transfer format, similar to rubenwardy's example, from a bit above.

gui.Window {
    width = 360,
    height = 230,

    gui.Button {
        name = "btn1",
        w = 4,
        h = 0.8
    },
}

There's a slight problem in the example with the element's parameters getting mixed with its children. I would try to keep them separate if possible.

When it comes to the underlying function calls, instead of using constructors for all the different elements, I'd probably go with the react.js way, and create a single function for creating all the different elements, for which you pass as a first parameter the element type, second the element parameters, and third its children.

There are some benefits from structuring the function this way:

there's only a single syntax for creating elements, and it cannot change for different elements just because someone at some point in the future decides it is a good idea that a specific object has a different constructor.
it easily allows using components in place of builtin elements, without needing to extend any objects.

local menu = createElement(gui.Window, {
    width = 360,
    height = 230
  }, {createElement(gui.Button {
    name = "btn1",
    w = 4,
    h = 0.8,
        onClick = handleButtonClickFn
    })
  })

render(menu)

You might want to take notes from react.js in general. They have shown how to declaratively define element hierarchies in a powerful way.

React is effectively a library for wrapping different render targets into a retained mode like library. It tracks the current state of the render target, and only does changes when they are required. It has been used initially for handling dom elements, but there are reconcilers from immediate mode targets like canvas and three.js (webgl), but also for retained mode targets like svg, and many others like curses (cli guis). It works with a wide array of different render targets. React also has a nice way for declaratively setting event handlers inside element declarations that could be used well with formspec's form submissions.

Roblox has a react.js implementation in lua called roact (https://roblox.github.io/roact/guide/hello-roact/). I'm not sure whether or not they are only superficially mimicing react's api, or if they are actually doing proper react style reconciliation below the surface. It's worth taking a look in either case.

mercorii on 20 May 2020

It's a good point. (But can't you just use multiline strings with double brackets to mitigate the problem?) Although, the same would still apply to JSON (as it also uses double quotes), if it was left to be openly modifiable in a similar fashion.

I don't support JSON as the interface exposed to the mod writer for constructing formspecs. The transmission format IMO doesn't matter that much as long as it's compact and easy to work with in the code. The data is made specifically to be produced/consumed by Minetest so it could even just be some custom binary format.

You're right about using brackets as string delimiters, that would avoid the issue with double quotes (if there are any issues with brackets in the string you could make the string delimiters long enough until they don't collide).

Then that same problem also applies to JSON, which is supposed to be used on the backend of the Lua EDSL, if I have understood correctly? EDN is just doing the thing that JSON would do.

Yes, that's right. I was only addressing the interface exposed to the modder, because in your original post bringing up EDN, it sounded like you were suggesting that modders should use it directly to specify the GUI, and didn't mention anything specifically about it being used as the transmission format. I don't really care about the transmission format, as long as it's easy to work with in code and compact, but I don't think EDN has much of an advantage over JSON in those respects (also EDN would require adding a new library dependency). A transmission format supporting schemas might be useful in checking Minetest-generated data against a handwritten schema. I don't think portability is much of a concern for transmitting GUI stuff, since its only possible purpose is to be produced/consumed by Minetest.

raymoo on 20 May 2020

@mercorii

It's just lisp.

Pretty sure you code in Emacs. But I prefer Vim. We Vimmers hate LISPers.

That’s a joke ofc. But seriously, it’s not the time to discuss transmission format, it’s just a technical detail. We only need to remember there will be one, so e.g. user-defined GUI elements are hardly possible, except of simple helpers defined in terms of standard elements (that can’t coexist peacefully with centralized element creation).

You might want to take notes from react.js in general.

And you might want to note that JS is a language designed for tiny cosmetic scripts that was being developed by two actively competing vendors so each was adding its own extensions, often incompatible with the competitor’s implementation. It was never intended for the programs it is used for currently.

I don’t want to say the library is bad, I just see nothing special. Except of maybe curses support, is it usable?

@paramat This issue became a forum thread effectively. What’s about splitting into something more specific?

numberZero on 20 May 2020

When it comes to the underlying function calls, instead of using constructors for all the different elements, I'd probably go with the react.js way, and create a single function for creating all the different elements, for which you pass as a first parameter the element type, second the element parameters, and third its children.

Why?

You can enforce the constructors for each element having the same format by convention and internal implementation utils, ie:

-- In the Minetest engine
gui.Window = make_gui({
    has_children = true,
    properties = {
        x = "number"
    }
})

Having a cleaner syntax makes it much more pleasant to write GUIs.

There's a slight problem in the example with the element's parameters getting mixed with its children. I would try to keep them separate if possible.

It makes it much nicer to write. It allows XML-like syntax without XML - properties on a node, and then children

It easily allows using components in place of builtin elements, without needing to extend any objects

This is not true, because you then have no control over how the tree is constructed with custom component elements

You might want to take notes from react.js in general. They have shown how to declaratively define element hierarchies in a powerful way

Taking notes from other frameworks would be a good idea, especially when it comes to the data and callback models

rubenwardy on 20 May 2020

@rubenwardy mercorii is probably more familiar with JS than Lua. In JS, arrays and mappings are very different things. But in Lua, they are both tables, just used a bit differently. In JS it would be highly problematic to iterate over children in such a mix. Here in Lua, it’s a matter of a single iparis call.

So, after 2.5 years, one question can be narrowed to: should we use a sort of Lua DSL (or maybe imperative Lua), or we really need an additional language to define GUIs?

numberZero on 20 May 2020

mercorii is probably more familiar with JS than Lua

Unfortunately very much this. Bear with me if I say stupid things. I mean no harm.

There's a slight problem in the example with the element's parameters getting mixed with its children. I would try to keep them separate if possible.

It makes it much nicer to write. It allows XML-like syntax without XML - properties on a node, and then children

My rusty lua skills failed me here. You can do the separation between properties and children with ipairs, as vitaliy explained. My bad.

:+1: from me for the proposed DSL style. It is declarative and easy to use.

Why I brought react into discussion? While we can argue about the js as a language and its warts all we want, react is in itself a great framework/library, regardless of the language that made it popular. And react is at least moderately popular even outside js world, as shows the fact that roblox, a non js project, has made a lua version of react for their own gui development.

React's strength lies in couple of features:

1) it handles elements declaratively as a tree structure, where
2) data pours only downwards in unidirectional way, making it very easy to see entanglement between different nodes (you just need to look up to see where the data came from). The only way to push data up in the tree is through callbacks or events. Nodes below in the tree don't know anything about their parents. (If you totally want to fight against the current, it is possible to write bidirectional code, and make your code messy, but fortunately it is usually not as easy with react.)
3) React allows creating complex tree structures by wrapping subtrees inside components (in js components are implemented as either functions or classes. the component's renderer returns the components child nodes). The components can have other components or built-in nodes as their children. But for the leaf nodes of the tree, only built-in elements are allowed. So the components just act as abstractions over subtrees.
4) As react is declarative, it must internally keep track of the render target's state. Because of this, it is very easy to understand what is happening, even if you don't understand exactly what the render target that react is controlling is doing. It is very powerful technique, one that can be used with many different render targets. Essentially react acts as a retained mode library, hovering over some other api.
5) As react keeps track of the internal state, it also keeps track of events. Users don't need to track events, react does that. The only thing users need to do is declaratively add events to nodes, when they are declaring the nodes. This is pretty much what every other framework is doing. The declarative spin here is that events are declared on the elements that they affect, as part of the tree declaration. This again reduces entanglement, as it is not possible to insert events into random places in the tree, from outside the tree.

_In essence, what react does, it provides a functional way for maintaining ui, declaratively, for people who don't want to do a deep dive into the functional world. It's sort of a soft-functional, a non-scary functional library. The only functional concept you need to know is that functions are first class citizens and can be passed as values (and you don't even need to know that functions as values is a core concept of functional programming). Luckily lua also treats functions as first class citizens._

The idea behind my earlier comments in this thread was something like the following:

It might be worth it to create three abstraction layers for the gui development:

Lowest layer: The part that is tied heavily to the gui system's implementation - whether this is the current formspec system, qt, or something else. This could be either formspec style json, or an imperative layer on top of the json. This layer needs to be powerful, as everything on top of it gets their strength from it.
Middle layer: React style declarative layer on top of it. Main idea here is to offer a layer that developers can use to powerfully control the lowest layer, and to be able to create complex guis with. It should remain stable, in case the layer below it is swapped to another at some point, so that user's would need to do minimal changes to their mods in order to take different gui implementation into use. It acts as a cushion on top of the implementation layer. It gives structure that will stay, even if all the nodes below it are swapped into different ones. It needs to be simple ("simple" doesn't mean "easy". simple is usually easy to use, but hard to make).
User facing, highest layer: Pretty layer on top of the middle layer. Basically the DSL layer that rubenwardy proposed. It makes using the middle layer effortless.

Why 3 separate layers? So that you can replace any layer with another if needed.

If the gui framework changes, it would need to be easy to get rid of the layer that is heavily tied to it.
If someone comes with a better middle layer framework. Let's say a compiler for compiling gui changes written in lua into even minimal lua code, it would then be able replace the second layer library with this compiler.
If someone comes up with a a better DSL, it would be easy to change the DSL with another.

Similarly, game developers and modders might want to swap out either of the two upper layers. As the lowest layer is heavily tied to the engine, it cannot be swapped by modders, unless they create a different gui system altogether on the modding level.

There's a good chance that there are large problems with this proposal, one's that I just haven't properly thought out. This proposal is just to bring different ideas into the mix. The thin red line going behind this proposal is that modular code is usually easier to maintain.

mercorii on 21 May 2020

So, after 2.5 years, one question can be narrowed to: should we use a sort of Lua DSL (or maybe imperative Lua), or we really need an additional language to define GUIs?

Would using an external browser be out of question as a secondary GUI system - one that would live beside the old/upcoming formspec? Browsers already offer a powerful GUI environment that a lot of developers are familiar with. The security of modern browsers is also pretty good, given the size and muscles of the organizations backing them.

The way I would think it to go:

the server could create a connection to a secondary GUI server that would talk with user's browser via websocket connection.
Clients could then be shown something like a QR code (for using additional phone for the web gui), or opening a link in a browser (when playing in a windowed mode), that would direct them to the correct GUI server and authorize them to interact with the server via browser. Perhaps, when playing in android, one could use the browserview that is available to apps, and it would be possible to slide it visible over the game view.
GUI created with full browser APIs could be shown to user.

The implementation of the GUI server could be done by anyone. It wouldn't be the responsibility of the minetest devs. It could be made by modders. It could be some stock GUI server made by the communtiy, or it could be custom made especially for a specific server.

This way, the browser GUI could contain all the client side logic that websites currently contain, and still in the average case, the data needed to be passed between the GUI server and the minetest server would be very limited, mostly just commands.

Example use case: a teleport booth. Transmitted data then being (from server to GUI server): request to open teleport GUI, and a list of possible jump points. To the other direction (from GUI server to server) the transmitted data would just be the selected teleport target. The interaction between GUI server and browser could be as js heavy (or maybe it wouldn't need js at all!) as the people implementing the GUI (and GUI server) would want it to be.

As far as I've understood, the naive approach would be for each mod on a server to create a connection of their own to the GUI server, by using the insecure environment. This is probably not wanted. One of the reasons being, that the server admins would probably want to keep track of all the outgoing internet calls (because security, privacy, and regulations like GDPR).

There would probably need to be only a single GUI server per server, one that the user would interact with. Because of this, and to keep track of outgoing data, the connection between server and GUI server would probably need to happen via engine. Basically, the server would give the full domain name of the GUI server to the engine (perhaps using external configuration files), and then the engine would establish this connection and keep it alive. Server could then pass information back and forth to the GUI server via functions provided by the engine. This way a regular mod could be given access to send messages to the GUI server, in a similar manner they now interact with formspec implementation. The GUI server could then filter what calls it wants to pass to the browser. On the client side, the engine would be responsible for opening the link to GUI server in the default browser or showing the relevant QR code.

Additionally, as the GUI server would talk with client over reqular web standards, it would allow wild things, such as creating a steam overlay like system for GUIs that could wrap the game, and show GUIs on top of the game's visuals, if people would want to do that.

There would still be use cases for the traditional formspec replacement, but a lot of the use cases could be offloaded to external tools.

The main benefit would be that this would drastically reduce the pressure to improve the engine's own GUI system.

Additional benefits would include:

reduction of GUI elements over the game world (some people consider it to be a source of ludonarrative dissonance or some such. (I read an article last week about it, and know fancy words now :D))
outsourcing the ugly parts to external services - in minetest, the game world is actually pretty, while GUI elements in most cases are not

mercorii on 30 May 2020

I'm going to be honest--unless I'm missing something big, this sounds like a terrible idea to me.

First of all, from a developer's perspective you now have multiple different GUI systems that you need to potentially learn and choose between. If you want to extend the GUI from other mods, you may need to even work with multiple systems in a single project! (And that assumes that these external server-based systems allowed any extension from other mods to begin with. It doesn't sound like it from your description)

From a user's perspective, this will likely result in even more disjointed and mismatched UIs. Right now, formspecs tend to look consistently awful because modders typically just use the default look for every element. Introducing multiple GUI systems will likely result in multiple incompatible default looks, making the problem much worse. It's also worth noting that formspecs can actually look pretty nice if you choose to flex some of the newer features--people just aren't doing that. There's also the UX nightmare that is having to switch to a separate application just to view an in-game GUI.

There would probably need to be only a single GUI server per server, one that the user would interact with.

So what happens to singleplayer folks? Do we just tell them to build and maintain their own GUI server?

The main benefit would be that this would drastically reduce the pressure to improve the engine's own GUI system.

In theory, but I suspect this would end up like mobs in practice--with modders clamoring for a better official in-engine API. Arguably this creates more pressure, since this your solution as proposed would exist side-by-side with formspecs.

reduction of GUI elements over the game world (some people consider it to be a source of ludonarrative dissonance or some such. (I read an article last week about it, and know fancy words now :D))

That's not what ludonarrative dissonance generally refers to.
Forcing the player to _leave the game and go somewhere else to interface with the game UI_ is 100x worse in terms of immersion. It's not in any way an improvement.
Even if that weren't the case, what you're describing doesn't require a reduction of GUI elements--diegetic UI design can also work.

Df458 on 30 May 2020

👍2

There would probably need to be only a single GUI server per server, one that the user would interact with.

You could also just have one mod (which would need to be set as trusted) that provides an API to other mods for interacting with the GUI server. If logging of connections is desired by server operators, then that could be part of that mod.

raymoo on 30 May 2020

Having the GUI be in a web browser separate to Minetest will be terrible for user experience, and will break when you put the game in full screen

rubenwardy on 30 May 2020

I totally understand the crushing reaction in the face of this novel idea about an external GUI. This is a brave idea from the depths that no ideas normally dare to enter, and it kind of has some of that sewer stench still there.

But seriously though, I'm just throwing things in the wall and trying to see what sticks. (I'm kind of trying to challenge others to think outside the box as well). I'm not sure whether this is a good idea or not, but I think there are some feature's or points of view here that are worth considering, if not worth implementing or adopting outright.

There is a kind of paradigm shift here.

The leading building block with this idea is that in today's world, more people have a smart phone than a computer. So if a person has a computer, they most likely also have a phone. (I have no idea how this would work, if person doesn't have a computer :grinning:). The idea is made from the perspective of someone playing on their computer, and having phone or tablet or stream deck at their hands as a sidecar on which they can do secondary actions.

Actually, this is not even that novel idea. Apple is pushing to this direction with their ipad pro as a sidecar for mac tech.

(If you think about it, this is also how a car functions. You have your steering wheel, your stick (unless you have automatic transmission), pedals and your view through the window. This is how you drive. In addition to this, you have your radios and air conditioning system controls, etc that are used for secondary utilities, possibly all even used through a central car computer.)

How would it work then? You would then have the secondary screen on your desk, and you could do all the secondary actions with it. You would play on your computer, and all the gui stuff, and chat, and what not, are held outside the games kingdom.

This would push for stricter separation between the actual game play events happening in the game's world, and the secondary game play controls used for simulating things that cannot be done in the game's world. You would want to avoid using GUI events in places where you can't take your hands from the controls, and at the same time, you could do more at the times you don't need to be glued to your keyboard. Again, the car analogy might or might not give some ideas for why this could work, or why it will not work.

There's also some new related development on the general direction of party games like You Don't Know Jack, where phones are used as controls when playing on a larger screen.

First of all, from a developer's perspective you now have multiple different GUI systems that you need to potentially learn and choose between. If you want to extend the GUI from other mods, you may need to even work with multiple systems in a single project! (And that assumes that these external server-based systems allowed any extension from other mods to begin with. It doesn't sound like it from your description)

This is a very real problem. It breaks from the formspec way of gui handling, meaning it cannot just be hidden behind some common middle layer. As I said, this is not a fully thought out proposal, but just something to challenge the old ideas.

On the positive side, it should be backwards compatible, as it is essentially a brand new render target.

You could also just have one mod (which would need to be set as trusted) that provides an API to other mods for interacting with the GUI server. If logging of connections is desired by server operators, then that could be part of that mod.

What raymoo said, is probably the best direction for POCing this.

That's not what ludonarrative dissonance generally refers to.

This goes a bit off-topic, but I tried to imply with the line that followed the mention of ludonarrative dissonance that I may not be an authority in ludology. People learning new "fancy words" from "articles" they read "last week" probably should not be taken as proper sources at the face value on said fancy words. But good that you pointed it out.

Okay, sure the term ludonarrative dissonance might have been wrong. There are other words describing UI elements that fit the game narrative, true.

I tried to raise the question about the conflict between the formspec gui and everything else happening at the screen. But I notice that in addition to using wrong terms, my English dropped some possibly crucial words in the short sentence that I wrote, which might not have carried the full intended information. :grin:

I get that minetest is an engine, but the narrative in a minecraft like game like mineclone 2 or minetest game, both that I've understood to be typical games made with the minetest engine, is that a block thingy in a blockverse does marvelous block things. While the narrative describes how rather rough tools are used for digging holes in the blocks, the actual gameplay happens through a weird fighter jet level hud system that the block thingies then control with, i don't know, their internal eye tracking system, straight out from the spring catalogue of stark enterprises.

That is some diegetic use of formspec GUIs, I guess. :grinning:

The engine's job possibly isn't to solve things like this, but what it can do, and what it does, whether the developers intend it to do, is make certain game design choices easier. The remark on ludonarrative dissonance was made with a tongue in the cheek, and regardless whether the term was used correctly, behind it is the question if there is a hidden, not-by-design bias, one that has just creeped-in somewhere in the development process, towards certain game development choices.

mercorii on 31 May 2020

Look, I get that you're trying to be creative. But creativity isn't enough, an idea has to be practical. What you're suggestion sounds more like a fairly specific gimmick, one that requires additional hardware to function, and is in effect a less convenient variation on dual-screen systems like the Wii U (which as you may recall, was a commercial failure). Minetest (and the vast majority of games in general) needs two hands to be played, and you're suggesting that we add a 3rd peripheral that forces users to drop what they're doing every time they want to use a UI.

While the narrative describes how rather rough tools are used for digging holes in the blocks, the actual gameplay happens through a weird fighter jet level hud system that the block thingies then control with, i don't know, their internal eye tracking system, straight out from the spring catalogue of stark enterprises.

Ignoring the fact that this is not the case in some Minetest games to begin with, this description is wrong on multiple levels. "weird fighter jet level hud system" is more or less a description of UI design and aesthetic choices, and has nothing to do with Formspecs themselves. Even in the specific cases where your description holds true, such UIs are a well-understood abstraction that has been honed over decades and is proven to work.

The engine's job possibly isn't to solve things like this, but what it can do, and what it does, whether the developers intend it to do, is make certain game design choices easier.

Indeed. So tell us then, what choices does your proposal simplify, and how does that simplification compare to what is already proposed?

behind it is the question if there is a hidden, not-by-design bias, one that has just creeped-in somewhere in the development process, towards certain game development choices

The bias is towards industry-standard tools and capabilities that both players and designers have come to expect and appreciate.

Your entire argument boils down to, "These GUIs are too old-fashioned. Let's do something new and exciting!" The obvious problem with this argument is that it completely violates one of the fundamental principles of UX: _Humans are most comfortable with familiarity._ This principle is the reason why users bitch and moan every time websites update their layout, and it's the reason why innovative interfaces are a hard sell that often get written off as gimmicks (consider the Microsoft Office ribbon, a pattern that was carefully designed to fix a real issue with the software but is still reviled to this day by older users). If we were Apple, Google, or Microsoft, we could spend millions on advertising and focus-testing to build the _Big New Thing_, but we're a small team of volunteers cobbling together a game engine. 'Innovative' is the last thing we want our UIs to be, because it cannot possibly end well for us if we wind up in that position.

...but let's step back, and pretend I'm wrong about the last paragraph (I'm not, this is just for the sake of argument). If that's the case, then we should still follow convention. Why? Because we're not the majority of end-users of this API. At the end of the day, we're building this for modders to use. And judging from most of the biggest, most popular mods... people want to make and play Minecraft-y stuff. There are alternatives in our ecosystem, but they're significantly more niche. As devs making a product, our duty is to cater to the users, not our cool experimental ideas.

If you truly believe in this idea, don't let me stop you. Build it with the existing APIs, and show it to folks. _If people truly want it, we'll hear that loud and clear._ But as long as an idea is just an unproven concept on a whiteboard, it has no value.

Df458 on 31 May 2020

👍3

Pleeeease, stoooop! I constantly get notifications about a futile suggestion that is going way beyond what this issue was meant to be. These experimental ideas are VERY off-topic here. At least move this experimental feature discussion into a new issue.

Thanks.

Wuzzy2 on 31 May 2020

Please don't make an issue for that! We want practical suggestions, not overly creative impractical suggestions. Perhaps experiment using a third-party mod

rubenwardy on 31 May 2020

👍4

That all is totally unrelated to formspecs. These are server-side GUI definitions only.

@mercorii Regarding web browsers, they are way too over-engineered to be usable. And a word on security...

numberZero on 1 Jun 2020

Something I've been thinking on recently is the problem of the event system and querying element states. In a normal GUI framework, you can do something like this (HTML + JavaScript):

var cbox = document.getElementById("checkbox");

cbox.onchange = function() {
    if (this.checked)
        console.log("Checked");
    else
        console.log("Unchecked");
}

Two things to notice here: The checkbox changing is a single function, not one giant function that receives all events at once (i.e. minetest.register_on_player_receive_fields). Secondly, this function can query the checkbox's current state, but in Minetest, you are forced to cache the value.

Minetest obviously has different stipulations than HTML + JavaScript, namely that cached values often have to be stored in some MetaDataRef to preserve them between games. So the question is, how can the new formspec system cache these values without any knowledge of where to store it? Even so, some formspec elements might not need caching (e.g. a formspec might not need key or mouse events, assuming #8679 is merged), and storing it would be a waste of space.

I don't really know how to solve this, but the formspec system really needs some event system with a caching of element states.

v-rob on 6 Aug 2020

@rubenwardy

we need to send commands over the network to the client

This is secure?

About a new frontend API I think we should take one of the lightweight suggestions:
tgui, sfgui, nuklear, clutter

BruceMustache on 10 Aug 2020

TGUI and sfgui are not production ready. Sfgui is unmaintained, and TGUI is still heavily WIP and largely undocumented

Those technologies would be for the backend / renderer, rather than the API

rubenwardy on 10 Aug 2020

What if use nuklear and use a protocol to send commands through network to the client?

BruceMustache on 10 Aug 2020

Why not just Irrlicht's GUI? We've already been working on it for quite a while, and it's capable enough. Adding another dependency sounds like a pain, and coming to a consensus even more so.

v-rob on 10 Aug 2020

👍1

@v-rob and others: are you aware of that there are at least 3 things discussed?

GUI backend replacement (patched IrrLicht is used currently but its input handling is too poor)
GUI specification format/API replacement (home-grown formspec format is used currently but it is awkward and inflexible)
GUI serialization replacement (the same formspec is used currently; awkwardness doesn’t matter much here fortunately)

Besides that, there are interesting problems like the fact formspec is often stored in the node meta in the current ecosystem, that’s weird by itself and may cause serious compatibility problems. It causes problems on mod updates already, although updating the form without updating the meta might cause problems too, but there are LBMs to deal with that. Of course they can be used to update the formspec too, but...

numberZero on 24 Aug 2020

Yes, I'm quite aware of all of these. My personal opinion for the GUI backend is to continue using Irrlicht since we already have it and patching it isn't hard at all. The GUI serialization for the future will be (I think this is a consensus) JSON; I'm working on a parsing refactor right now to make this possible in the future.

As for the specification, the opinion seems split between OOP (https://github.com/minetest/minetest/issues/6527#issuecomment-336449202 is an example) and Lua tables (similar to XML, https://github.com/minetest/minetest/issues/6527#issuecomment-631596025). I'm leaning towards Lua tables, but either seems fine to me.

I think node meta formspecs should be deprecated; they just seem like a big pain and can't migrate to a new format.

v-rob on 24 Aug 2020

You can use both Lua OOP and Lua Tables by having the Lua Tables be fancy constructors. Lua tables should be the recommended way, as it's syntactically nicer

rubenwardy on 24 Aug 2020

I've been experimenting lately in regards to this:

Define custom elements, with draw(). Draw is only called when required, which may be every draw step.

Right. Now, my thought was, is it possible to build the entire GUI system in pure Lua with this? Well, I made a prototypical attempt: https://github.com/v-rob/minetest/tree/gui

What is does is adds two callbacks to CSM: minetest.register_on_event(function(event) ... end) where event is an event definition similar to what is in #8679 and minetest.register_on_draw(function(dtime, drawer) ... end) where drawer is an interface to draw on the screen, with methods get_window_size, rect (draw filled rectangles), and image (draw images).

Optimally, after filling drawer out with more features, we could write the entire GUI system in Lua, ditching Irrlicht's GUI completely for formspecs, allowing custom elements, element callbacks, dynamic element editing, etc. as well as HUD and GUI merging. Development would be faster because of no compilation time. It would be awesome.

Unfortunately, while the event callback is perfectly feasible, the draw callback is much too slow. On my computer (nothing amazing, but runs Minetest at 60 fps), drawing 100 rectangles and 100 images slowed the framerate to ~57 fps, while 1000 of each slowed it to ~35 fps. If you do the equivalent tests with formspec box and image, 100 makes no noticeable dents with 1000 doing ~57 fps at the worst. Lua is almost certainly the bottleneck here. Now, while I could optimize the Lua callbacks some (rects have x, y, w, and h fields; could be changed to an array), I don't think it would help enough to bring the framerate up.

So, if custom drawing is to be achieved, we would have to do something like draw everything in one function call or cache what everything would draw and only change it from Lua when necessary. However, even these would be slow if people tried to make fancy animations by constantly updating the drawing.

So, I don't know. If this is possible, I would love to be able to do it because of all the benefits, but I can't think of many good ways to speed things up. Then again, there might be other problems that I haven't forseen. If nothing else, I've laid the groundwork for a CSM event callback and gained experience with Lua's C API. Anyhow, I would like feedback, if anyone has any to give, good or bad.

v-rob on 25 Sep 2020

Not redrawing the whole interface on every step, but rather what changed would work. This would either require individual objects engine side or the ability to only redraw certain regeons.

Alternatively just not redrawing when there are no state changes would be a good start.

benrob0329 on 25 Sep 2020

Not redrawing the whole interface on every step, but rather what changed would work.

@benrob0329 That would work in software. GPUs aren’t designed for that.

Lua is almost certainly the bottleneck here

@v-rob Or the immediate rendering you use. GPUs prefer large batches (IrrLicht too, I suppose). Drawing objects one by one leads to awful performance.

Also be very careful with draw2DImageFilterScaled. It scales in software, and caches the result, for each size it is called for IIUC.

numberZero on 25 Sep 2020

I'm thinking that the best way to speed things up would be to ditch the draw callback and instead have an internal engine object that stores everything it should draw with Lua having the ability to add, remove, and change things within it. That way, the engine draws everything itself every frame with Lua only changing what should be drawn every so often. That would have a more cumbersome API, but I don't think the callback can handle things fast enough.

Lua is almost certainly the bottleneck here

@v-rob Or the immediate rendering you use. GPUs prefer large batches (IrrLicht too, I suppose). Drawing objects one by one leads to awful performance.

I was saying that Lua -> C++ -> Lua -> C++ ... thousands of times per frame is what I thought the problem was. Formspecs draw every element separately too without a batch, so I don't think (?) the GPU's the problem, but I'm no expert on that stuff.

Also be _very_ careful with draw2DImageFilterScaled. It scales in software, and caches the result, for each size it is called for IIUC.

That's what formspecs use, so it's what I used too. Would plain driver->draw2DImage be better? I have no idea what feature draw2DImageFilterScaled has over it.

v-rob on 25 Sep 2020

You could batch the draw calls in Lua (ie: store them in a table), and send them to the engine all at once.

You could also reduce the number of times the draw callback is called by rendering to a render texture, and only rerendering when invalidated

Note that render textures are a bit hit and miss on Android though

rubenwardy on 25 Sep 2020

I tried out Rubenwardy's idea of storing eveything in a table and sending it all at once as well as making rectangles array-based. Unfortunately, the speed is not much increased, if indeed at all. So, any speed gained by the removal of many function calls is probably lost by accessing the array, which, I guess, is slower than accessing function arguments.

Seems to me like having an engine internal object is the only way to go from Lua, but even that would be just as slow if things are updated every frame, like fancy animations would do. Of course, it's unlikely that everything will be animated at once, so its more probable that performance would be better on average, only spiking low on screen resize.

I would envision the engine internal object as a linked list of drawing definitions with Lua userdata pointers to each definition for changing it and/or creating another definition after it. This would remove the minetest.register_on_draw since the engine would draw everything every frame itself with Lua only changing each thing when necessary. Window resizing would be exposed to the event callback.

Thoughts? I still want to pursue Lua-side if it's possible as I think it would open up more potential, but if this last idea doesn't work, though, I think the idea of pure Lua won't be possible.

v-rob on 28 Sep 2020

Have you tried profiling to see where time is being spent? Also accessing a table should be much faster than interop calls, especially if you keep track of the last index (but even if you don't).

raymoo on 28 Sep 2020

I have to ask, how does Love2d successfully overcome such bottlenecks with its Lua API? Rather than re-inventing the wheel from scratch, perhaps some effective solutions could be gleaned from that project.

sorcerykid on 30 Sep 2020

👍1

So, following a discussion on IRC, I decided to do some more performance testing. And now, I feel a bit ridiculous because the speed for a 1000 rectangles and images for the batch drawing is only ~3-5 fps different than a formspec with 1000 box[] and image[] elements (and this might be because of the different drawing techniques used for images for each). I don't know what was wrong with my earlier testing, but clearly, I did something horribly wrong.

I don't know whether I'm using LuaJIT or plain Lua (and I can't choose between the two anyway; I can't compile on Window and my Linux virtual machine is too slow for performance testing), but it probably doesn't matter a whole lot as it's just one function call per frame. I still want to test if I can get things to work right.

Anyway, I'm still doing testing, but it's looking more and more feasible. If all the testing is favorable, I'll work on adding more drawable things, like text, item images, etc, and then we'll see where it goes from there. Some of my worries are things like constantly querying the dimensions of text for text-heavy elements like hypertext and textarea. The constant function calls might bring the speed down.

v-rob on 4 Oct 2020

🎉2

Minetest: Formspec replacement

MuSCo (must have, should have, could have)

Actual GUI libraries / logic

Server API

CSM API

Other notes

Most helpful comment

All 117 comments

(The rest of this post is just throwing my ideas in here.)

This is secure?

Related issues