lwjgl3 🚀 - Add a GUI solution to LWJGL

Also I have a running version of Themable Widget Library, which works really awesome with LWJGL 3 and kappa's lwjglx library.
I would really love for someone to do the porting to native LWJGL3 without needing an adaptation layer.
The render code of TWL is really nicely decoupled from everything else and can be swapped once a new LWJGL3 implementation is available.

httpdigest on 8 Oct 2015

Do agree that TWL is a great option for rendering GUI on top of 3d content. It only supports LWJGL2 atm, though looking at the code base it doesn't seem all that much work to port to LWJGL3.

However for standalone ui's, SWT is probably the best and most mature option out there. It integrates nicely with the native platform and is small and lightweight. It also strips down pretty nicely (to a few KB's, the Avian example are good examples of how small, less than a meg for SWT support and the Avian JVM!).

So looking into getting LWJGL3 nicely integrated with SWT should be explored. IMO a much better investment then spending time on AWT or JavaFX.

Interestingly, l33tlabs (who host TWL), work on a really interesting project called Bling. They've somehow integrated SWT with TWL! I think they also use LWJGL2. They already have a full IDE running on the same, so pretty far along. There is a pretty long video here explaining how it works and theres a quick preview of what it looks like here.

I'm not sure what l33tlabs plans are but if they released the underlying SWT/TWL library or if something similar could be written, it could probably be the ultimate and complete solution for LWJGL3 i.e. a small lightweight API that can render ui inside OpenGL applications, create native platform widgets (like dialog's, system tray integration, native controls, etc) or even enable full blown native applications that can render opengl content inside it (e.g. particle editors, game engine editors, etc).

kappaOne on 8 Oct 2015

I wonder if it would be possible to somehow render HTML+CSS (and possible code the UI logic in javascript via nashorn maybe) for HUD?
I've researched quite a bit some time ago but couldn't find anything like that or even a reference.
It would also be nice to check how games such as Heroes of Storm and Starcraft 2 (which, IMO, have nice HUD) are doing their HUD.

Another thing that maybe could help is the project dukescript.

renannprado on 8 Oct 2015

I think using Nashorn and JavaScript would not be the problem; the library implementation and its API would be, as it sounds rather like the tasks of browser engines like WebKit or Blink and the domain of the Web platform in general. However it would sure be great. :)
IMHO when layouting with HTML/CSS, I would then switch to WebGL (like with three.js), Canvas "2d" based 3D frameworks such as phoria.js or CSS 3D Transformations, instead of developing a desktop application with Java/LWJGL.
dukescript however sounds fascinating! :)

httpdigest on 8 Oct 2015

Yeah, that's the thing. I didn't mean to _create_ something like WebKit, but if we could create a _binding_ for it and _use_ it to somehow render the HUD, that is what, IMO, would be awesome.
(I don't have any clue if it's possible, maybe I'm saying some non-sense here. I'm just trying to think from the library end user POV and, for me, HTML+CSS (and possible javascript) is the most flexible way to write UI today)

renannprado on 9 Oct 2015

This "awesomium" seems to be what you were looking for:

It also allows to render into an OpenGL surface (one can define its own surface factory).
That could be an option.

httpdigest on 9 Oct 2015

In my humble opinion its more of an issue if a GUI lib doesn't supports core OpenGL than not supporting LWJGL 3 (TWL is really nice but it's renderer is GL 1.1 exclusively).

NiftyGUI is there, has a core OpenGL backend, but doesn't supports LWJGL 3 (mainly Display/Keyboard/Mouse related changes and some function name changes).

Also another concern of mine is the state in which the GUI lib leaves OpenGL. This should be very well defined, ie, what state OpenGL needs to be before the GUI lib can do whatever it needs to do, and what state is OpenGL left with once it finishes. This is vital for efficient state tracking.

dustContributor on 10 Oct 2015

Unlike @dustContributor I'm not concerned with supporting "core OpenGL" standing as much as if it fits in the "lwjgl wheelhouse" of libraries and functionality, and if the general API fits thematically with other, existing lwjgl API calls. I don't have a specific recommendation on how to do that, so long as I would be able to pass in some UI component hierarchy, and just tell it to "show the gui".

What would be wrong with basically "copying" the AWT and making it render for lwjgl?

JustGregory-zz on 10 Oct 2015

Supporting OpenGL 1.1 and not >= 3.2 core is an issue if you yourself want to use a "modern" OpenGL context and functions therein, which would both not be possible in a single context. Either TWL would not function correctly because it used deprecated OpenGL functions or you would not be able to use "new" OpenGL functions. This is an issue with Mac OS X, which either supports OpenGL <= 2.1 if you don't ask for a new context specifically, or you would only get a "core" context if specifically asked for, but then TWL would not work.

One solution to this problem, and also to the problem of touched OpenGL state, could be multiple sharing contexts. One "deprecated" OpenGL context for TWL and another one for your own rendering.
TWL could be issued to render into a texture using a Framebuffer Object and then that texture can be shared between both contexts to simply blit the texture onto your own "new" context's window.

Sharing between an "old" deprecated context and a "new" core/forward compatible context does work at least under Windows with Nvidia.

However, only "requiring" OpenGL 1.1 is the smartest thing one can do for such a widespread library like TWL, in my opinion. Whenever you require anything higher than OpenGL 2.0 it will not work on some person's machine. Compatibility is a very big thing here!

httpdigest on 10 Oct 2015

That sounds both slow and bound to hit driver specific issues. No, I'd rather handle the state change myself.

At the very least it should have a renderer interface that doesn't assumes fixed-function pipeline working underneath, TWL has that issue in various places last time I checked, which makes it a pain in the butt to code a core renderer for.

I still propose NiftyGUI as a better alternative, right now it just needs a couple of classes using a LWJGL 3 backend. I opened an issue about it a while ago https://github.com/void256/nifty-gui/issues/304

There is literally no other GUI library that has a core renderer that I know of (for Java that is), thus why I think its an important matter. OpenGL 3.0 has been out there for 7 years, with hardware supporting it going as back as 2006 (GeForce 8xxx, Radeon HD2xxx), that was almost ten years ago.

dustContributor on 11 Oct 2015

However for standalone ui's, SWT is probably the best and most mature option out there. It integrates nicely with the native platform and is small and lightweight. It also strips down pretty nicely (to a few KB's, the Avian example are good examples of how small, less than a meg for SWT support and the Avian JVM!).

So looking into getting LWJGL3 nicely integrated with SWT should be explored. IMO a much better investment then spending time on AWT or JavaFX.

Do we even need anything special for SWT integration? Looks like @httpdigest already has a sample here.

Spasi on 18 Oct 2015

The only downside of SWT is that it still (even in the very latest nightly build) cannot produce a GL >=3.2 core context. I had a look at the sources and found that the WGL solution at least is devised but commented out... But this problem can be solved by letting LWJGL3 do the GL context creation. So we would need a cross-platform way to create GL contexts in LWJGL3 based on a window handle. This handle is easily API-accessible in SWT.
Another solution would of course be to hack SWT, but I heard that their OS X backend is really heavily outdated, so it would be quite some hacking.
I also dug out an old LWJGL forum entry about this: http://forum.lwjgl.org/index.php?topic=5531.0

httpdigest on 18 Oct 2015

I recall digging through the sources and finding no clear way to hijack the SWT's component so I could use it with my core context (although my experience with window management APIs is limited).

Both JavaFX and SWT (since 4.2 I believe) are theme-able with CSS, which is a pretty cool thing to have.

I wonder if we could nag SWT developers about this...

EDIT: Still no interest in NiftyGUI?

dustContributor on 19 Oct 2015

I am going to build an own GLCanvas, something like a org.lwjgl.opengl.swt.GLCanvas or org.lwjgl.swt.opengl.GLCanvas, based on the platform API exposed by LWJGL3. The native/JNI code of SWT is really horrible. They do like a thousand (*env)->GetField(...) invocations and other slow/nasty hacks.
Personally I think, an own custom SWT Canvas would be the best solution, and I don't think it will require much maintenance, since it is only about creating the GL context and that should be common knowledge how to do that.

httpdigest on 19 Oct 2015

So any update on lwjgl native gui rendering methods?
Any news on nanovg implementation?
Please keep us updated!

crisp00 on 23 Nov 2015

@httpdigest is making good progress with SWT integration, see his latest update.

I experimented a bit more with JavaFX and have a working OpenGL backend that runs on top of LWJGL. This may seem like a huge step forward, but it's not enough. It would be easy to implement context sharing with the JavaFX context and access the textures directly. The problem is, JavaFX is architected in a such way that the rendering layer has no idea which textures correspond to which GUI elements. I don't have a solution to this yet, I'll explore some more ideas when I have time.

I don't know if nanovg bindings will make it to the final LWJGL 3.0 release, but I'll try.

Spasi on 23 Nov 2015

The latest nightly build has jawt bindings, which in theory allows for the AWT integration in LWJGL 2 to be implemented in terms of LWJGL 3. Sample code can be found here (simplest possible implementation, Windows-only).

Spasi on 10 Jan 2016

I've been thinking. Now that NanoVG bindings are up, it _might_ be possible to use it as backend for other GUI libs that lack an OpenGL core context, like TWL for example.

I'm looking at TWL's sources, seems a lot of work though -_-

dustContributor on 11 Jan 2016

I completely forgot about libRocket: http://librocket.com/

Its C++ though.

dustContributor on 15 Jan 2016

This is kind of late but a while ago I did manage to make TWL run with LWJGL 3 though it wasn't good by any means and TWL does use old OpenGL functions.

I should also bring up that one of the main problems I was having was figuring out a good way to get the event data from GLFW and processes it. As there can only be one listener it would be easy to make a version that just splits that up however it gets more complicated for the end user and taking over isn't a very good option as its not very easy to detect if your listener has been removed and could lead to problems with the end users code.

ShadowLordAlpha on 7 Mar 2016

https://github.com/vurtun/nuklear

Spasi on 18 Apr 2016

👍11

Sweet, it even leaves rendering up to the user by just giving you commands to execute, perfect.

dustContributor on 20 Apr 2016

Hey, if at all interested, I'm making a library for lwjgl, but for a broader spectrum*, from scratch. It'll allow direct support for GLFW in LWJGL3, support for the latest OpenGL (4.5+), and a way to build shaders on the fly during runtime (in GLSL). The window component system will be made to support XHTML and XML based layouts for quick building of GUIs and easy management, so you could simply build it in a browser or other development environment.

(~~I don't have a repository yet, and I have a(n empty) repository~~I have a repository, but don't have internet. ~~ but I'll get one up as soon as possible. It's pretty bare bones right now anyways...)~~

*By which I mean it's for more than just lwjgl. It'll support GLFW, but it'll support other GUI systems as well: AWT, SWT, Swing, etc.

AdMaioremDeiGloriam on 5 May 2016

👍1

Hey! It would be great if you add bindings to this library NanoGUI
Can you please take a look on it? :)

SpinyOwl on 13 May 2016

👍1

NanoGUI is C++ and probably hard to wrap. But it could make sense to port NanoGUI to Java on top of lwjgl's NanoVG.
Edit: NanoGUI is quite decent, btw. IMHO would be a good choice.

arisona on 13 May 2016

👍1

SGL now has code in it

AdMaioremDeiGloriam on 24 May 2016

https://github.com/glegris/nuklear4j

Dafuq, never saw what O_o

EDIT: Oh, SWIG : /

dustContributor on 2 Jun 2016

LWJGL will very soon have Nuklear bindings, see the nuklear branch.

There's already support for everything, but I'm working on a Nuklear+GLFW+stb_truetype demo to understand and properly annotate the API (nullability, autosize parameters, etc).

Spasi on 2 Jun 2016

👍4

Awesome! I didnt see that branch. I better read up on Nuklear then :smile:

dustContributor on 2 Jun 2016

Very cool!

@Spasi: It seems to be pretty fast to add bindings. I'd be interested contributing by adding bindings (in particular Valvoe / OpenVR). I couldn't find much documentation how to get this started. What's the normal procedure to start with bindings for a new library? Cheers :)

arisona on 2 Jun 2016

@arisona

The best approach is cloning the repository, opening the IntelliJ project and exploring the Templates module. There are some important differences between bindings (loaded from a shared library vs baked into LWJGL, custom functionality like the capabilities classes, etc), but the general structure is the same:

The org.lwjgl.<library_package> package contains the type, struct and callback definitions. The type definitions provide an important level of type-safety and "refactorability".
org.lwjgl.<library_package>.templates package contains the actual binding templates, a file per generated class. These might look a bit scary, but it's mostly the documentation that makes it verbose. The most important thing is getting the "modifiers" right, but there is a ton of examples you can work with.

You should start with studying the GLFW bindings, I think that's the cleanest API. The LibOVR bindings should be fairly similar to OpenVR. You could also look for commit messages in the commit log that look like "Add <library name> bindings". These should give you a fairly good idea of everything required to add a new binding.

With that said, you don't have to do everything right. Simply contributing the types/structs/callbacks, templates and documentation will be a major help. I can do the rest of the work easily.

On OpenVR in particular: The C API has these FnTable structs that contain function pointers. They are loaded with the global VR_GetGenericInterface function (example here). No other library uses this scheme and LWJGL does not support it currently, so some customization will be required. Please open a new issue if you'd like to discuss this further.

Spasi on 2 Jun 2016

👍2

Will the nuklear bindings be moved into separate module that can be included optionally instead of being part of lwjgl3? I just feel that something like this, while valuable to the project and those who wish to use barebones lwjgl3, adds bloat and redundancy to other libraries that rely on lwjgl but provide their own gui solution, such as libgdx.

nhydock on 2 Jun 2016

@nhydock

Yes, see #100. The first 3.0.1 builds will have Nuklear built-in (adds about 200kb to the binaries) so that users can start working with it, but the plan is to split heavyweight bindings to separate artifacts.

Spasi on 2 Jun 2016

👍3

@spasi thanks re generator. this should get me started.

arisona on 2 Jun 2016

lwjgl_nuklear

Demo code here.

Spasi on 3 Jun 2016

👍1

i have a problem is the code of the demos,

import static org.lwjgl.nuklear.Nuklear.*;

in the version of lwjgl 3.0.0 build 90 no have the package

J-Ibarra on 4 Jun 2016

Like @Spasi said:

The first 3.0.1 builds will have Nuklear built-in

You are using the 3.0.0 Release, which does not have Nuklear included.
The sources of the Nuklear module, as well as the demo you are referring to, live in the nuklear branch, and you _could_ build LWJGL with Nuklear from that branch manually.

httpdigest on 4 Jun 2016

The first LWJGL 3.0.1 nightly build is now available and it includes the Nuklear bindings.

Spasi on 5 Jun 2016

❤2 👍2

Great job, thanks!

pierr3 on 5 Jun 2016

Awesome man, thanks a lot for your work!

dustContributor on 9 Jun 2016

Rendering GUI components inside 3D content (e.g. monitor/panel in 3D space displaying 2D content).

Is this something that can be done with nuklear?

jediminer543 on 21 Nov 2016

@jediminer543

You can render the GUI to a texture using a framebuffer and then use it to render the monitor panel triangles in the 3D space. I never worked with Nuklear, but this is how you will be doing normally.

sriharshachilakapati on 21 Nov 2016

Are there still plans to further explore JavaFX integration "natively" in lwjgl3, or is LWJGL-FX the current best alternative?

Most discussion in this ticket is about adding UI in a native GL display, what I have is an SWT based GUI that embeds lwjgl through an SWT GLCanvas to display a 3D representation that can be inspected using the SWT UI components around it, but we are looking at rewriting the GUI with JavaFX.

codemuse on 7 Dec 2016

Hey @codemuse,

Have you tried JavaFX Interoperability with SWT?

As for a pure JavaFX solution, I will probably try again when I have time, but failure is the most likely outcome.

Spasi on 8 Dec 2016

IMO any proper solution is really going to need help from the JavaFX team at Oracle, such as them adding an API for this sort of stuff or for them to expose certain internals of JavaFX which we can use to implement it properly.

Without the above, any solution is just going to be a hack, likely non-optimal and subject to breakage. JavaFX at present just isn't designed to allow this sort of integration.

kappaOne on 8 Dec 2016

Interestingly, planning for JavaFX.next (Java 10+) is currently taking place over at http://mail.openjdk.java.net/pipermail/openjfx-dev/2016-December/020025.html and Oracle are asking for suggestions.

Might be a good opportunity to chime in and request support for OpenGL/Vulkan via LWJGL.

kappaOne on 9 Dec 2016

It seems that has already been requested:

Also for JDK8:

http://mail.openjdk.java.net/pipermail/openjfx-dev/2013-May/007451.html

But I fear, because one of JavaFX's design goals being "abstract all and everything away" so that JavaFX is able to run on phones, on the Desktop, on the Browser platform and everywhere while being accelerated by either D3D, OpenGL or Vulkan (or what happens to be the hardware acceleration on that particular device), that Oracle will not break with this abstraction to actually expose an "OpenGLNode" (or even just a proposed "3D Canvas Node" telling the client that everything is currently being accelerated by D3D, OpenGL or Vulkan) on the API layer.

httpdigest on 9 Dec 2016

@Spasi, @kappaOne and @httpdigest: thanks for all your quick replies!

Your comments are similar to what I could find myself as I was investigating this. I'm looking for a "pure" JavaFX solution, i.e. embedding it in SWT wouldn't really make sense, since the idea is to move our application away from SWT.

@httpdigest thanks for the pointers to the OpenJDK mailinglist, I'll follow that to see what comes out of it.

In the meantime, I'll toy around what sort of hack would -- although suboptimal -- work for our specific situation.

codemuse on 12 Dec 2016

Now that we have JAWT bindings, can't we have it embedded into Swing and embed it into JavaFX using SwingNode?

sriharshachilakapati on 12 Dec 2016

I will second what sriharsha said above, except I fear that Swing/JavaFX/SwingNode adds altogether more of the abstraction-complexity to the mix. If it could be done easily, it would have been already; but maybe it will take a complex implementation to simplify it somewhat.

JustGregory-zz on 14 Dec 2016

Hi all, currently I working on GUI oop solution with api little bit similar to swing...
Here is my repository: legui

Here is result:

Radio button and text input	Widgets	TextArea

Button, togglebutton, checkbox, selectbox	Sliders	ScrollPanel

SpinyOwl on 22 Jan 2017

👍5 ❤1

Someone actually made JavaFX work with lwjgl3, ~~but it seems like it doesn't really use opengl core profile~~ ~~and it uses opengl 3.2 core profile~~: https://bitbucket.org/cuchaz/jfxgl

As it says in readme, it's a giant hack but it works and has good performance.

Barteks2x on 17 Mar 2017

@Barteks2x it looks like it's hard to make it work with Oracle java as standalone library, am I right?

SpinyOwl on 18 Mar 2017

@ShchAlexander I'm not the author so beyond what it says in the readme and what I see I can't tell you much. Based on my testing, it works as standalone library, but it uses implementation-specific hacks, including modifying the bytecode using asm library.

Obviously when using JFXGL you can't use normal JavaFX without lwjgl at the same time (and why would you?)

Barteks2x on 18 Mar 2017

Hi There. I'm the JFXGL author. It's still very much a work in progress, but it's getting better all the time. I've been talking to Spasi about it too, so it's already on the radar.

If you have any questions about JFXGL, ask away. Although maybe don't spam this issue thread?

cuchaz on 20 Mar 2017

Doing some research I got with this post!

https://engineering.riotgames.com/news/under-hood-league-client%E2%80%99s-hextech-ui

Here mention the use of javascript to create the client of this game, it would be interesting to be able to create GUI with HTML

OpenGL GUI using Chromium Embedded Framework (CEF)

J-Ibarra on 28 Mar 2017

SWT uses native OS widgets, why would one want to use that?

Did anyone evaluate whether it is possible to extend Graphics2D class/override repaint() methods and thereby redirect Java Swing UI painting to OpenGL/NanoVG?

Besides that, one could set JFrames as transparent and place them on top of the OpenGL window... :)

However, the best solution would be some generic Java UI lib that can be plugged into NanoVG, ie. a rewrite of Nuklear. CSSBOX provides html/css rendering, though I'm not sure how efficient animations would be. And interactions? In general, I'm not sure how efficient GUI animations would be with pre-written GUI libs. It would be an interesting task to think about the required structure of such a library to efficiently support all sorts of GUI animations.

BTW: because Vulkan support is now quite widespread, is NanoVG compatible with it?

jjYBdx4IL on 18 Jun 2017

Hi guys,

I'd like to propose also another possibility..

I'm doing a full jvm port of imgui here (nuklear is based on its concept), it's almost complete, text input/handling needs still to be addressed but all the rest is done

plus, it's also libgdx compatible :)

See the Screenshots Thread for some user creations. Also see the Mega screenshots for an idea of the available features.

Sample

elect86 on 21 Nov 2017

❤5

I was reading some Java articles and it popped up that Gluon published release/support plans for OpenJFX. For those who don't know, JavaFX is being removed from core JDK, and it will be released separately. So to use it you'd need it to import it like any other dependency (maven, raw jars, etc). And now it will be called OpenJFX.

https://gluonhq.com/javafx-11-release-and-support-plans/

They have a GitHub mirror of the OpenJFX repository and they say they pay attention to the issues there. https://github.com/javafxports/openjdk-jfx

Moreover, they say they have their bug tracking on the JDK thingy (although the site works like ass for me and I cant find it) https://bugs.openjdk.java.net/secure/Dashboard.jspa

Anyway, I'm wondering if that given the changes JFX is going through, if it would be possible to push for direct access to the GL internals from it? As a means to integration with other GL rendering.

All the stack is open source now so you got in that repo their "Prism" rendering engine and all, plus the HLSL/GLSL shader transpiler (Decora I think?) and some other nifty things.

In any case, they went a different route for it so instead of making bindings and "lower" the layer of native interop, they instead "raised" it by making the rendering part in C/C++ (doing all their multi platform context management on their own and all) and exposing higher level operations to Java. That doesn't sounds like it'd interact well with how LWJGL works but I wouldn't know.

It gets complicated because OpenJFX uses three backends, OpenGL ES 2, Direct3D and software. So what happens if you want to use OpenGL Core? Or Vulkan?

It's pretty complex, and I personally believe that The One True Way™ to do an UI lib for games is doing it like Nuklear does, handing out buffers of vertices/uvs/indices at rendering stage and have the user be responsible of displaying that to the screen, but I thought someone might be interested.

dustContributor on 21 Aug 2018

@dustContributor Yes, OpenJFX being decoupled from the JDK opens up opportunities, you can read some details in this JGO post.

(No work has been done yet, I grossly overestimated the free time I'd have this summer)

Spasi on 22 Aug 2018

I see you know already how to tackle it.

I inspected a bit OpenJFX code and it seems to have a rather direct rendering, in the sense that it has fairly hard codes operations in a bunch of classes that do specific things.

I'm guessing that achieving Nuklear-style of integration (ie, making it spit out vertices and texture bindings, API agnostic) would be pretty hard.

dustContributor on 26 Aug 2018

Hello everyone again.
I have several changes since last comment here.
Here is image of my gui library's current state:

preview

If anyone want to contribute, request any changes - you are welcome.
GitHub repo - https://github.com/SpinyOwl/legui/

SpinyOwl on 29 Aug 2018

If anyone want to contribute, request any changes - you are welcome.

I'll repeat what I said above (except longer!). I dont think having your own rendering backends is a good idea. You don't know how I, library user, do my rendering, you don't know what API I run, you don't know what version I run, and you really shouldn't.

Things that did their own drawing like Scaleform on big AAA games were phased out, because you have a complete subsystem trying its best to group rendering commands, group state and shader changes, and drawing all of that efficiently, and then you stuff your code with some other renderer that has some other API with some other concepts that just doesn't fits on how your engine does its drawing.

You have libraries like NanoVG, Nuklear and (now defunct) libRocket that are really really easy to integrate because they have a clear distinction between what is UI handling, what is rendering and what is input handling.

Without the library setup code, this is my rendering loop for Nuklear:

try ( MemoryStack mem = MemoryStack.stackPush() ) {
    // setup buffers to load vertices and elements
    final NkBuffer vbuf = NkBuffer.mallocStack( mem );
    final NkBuffer ebuf = NkBuffer.mallocStack( mem );
    nk_buffer_init_fixed( vbuf, tmpVertices/* , max_vertex_buffer */ );
    nk_buffer_init_fixed( ebuf, tmpIndices/* , max_element_buffer */ );
        // converts buffers to the layout specified in the config
    nk_convert( context, commands, vbuf, ebuf, bufferLayoutConfig );
}
// Upload data to the GPU
glAttribBuffer.bind().updateUnsafe( tmpVertices );
glIndicesBuffer.bind().updateUnsafe( tmpIndices );
// Process task commands.
int offset = 0;
// Compose render tasks out of nuklear's draw commands.
for ( NkDrawCommand cmd = nk__draw_begin( context, commands ); cmd != null; cmd = nk__draw_next( cmd, commands,
        context ) ) {
    final int elmCount = cmd.elem_count();
    if ( elmCount < 1 ) {
        continue;
    }
    // Submit task to the queue.
    // Matrices computed in shader.
    renderer.addToQueue( taskBuilder.reset()
            .type( RenderTask.Type.NUKLEAR )
            .meshId( glVao.id )
            .material( materialsById.get( cmd.texture().id() ) )
            .vertexCount( elmCount )
            .indicesOffset( offset )
            .build() );
    offset += elmCount * 2;
}

I just ask for Nuklear to fill vertex/index buffers, upload them to the GPU, then compose render tasks that use them and that's it.

I know that will best fit my rendering scheme because it is my rendering scheme. It will use the API I use, it will use the version I use, it will track state changes like I do, it will batch UBO updates like I do, and run without switching shaders unnecessarily like I do, and if I wanted I could make it use my instanced rendering by grouping by material first.

I dropped things like NiftyGUI because literally the first time I use them it turns out that there was some GL context interaction that they didn't account for so I had to spend time debugging what state my renderer was leaving GL with, what state the UI lib really needed (that may or may not be what the documentation says) and what state the GL context ended being in that made things not work correctly.

The GPU is a big fucking thing, wouldn't it be nice if you (lib author) didn't have to account for these things? Wouldn't it be a concern better suited for myself the library user since I know what API I target and how I want to use it?

I'd say: Write an UI lib, or a renderer, but not both in the same thing. They're separate concerns, and big enough to warrant their own separate libraries.

The same argument can be made for input handling but that'd make this too long. I have an input system that translates GLFW's stuff for my own scheme, and those events get pushed to Nuklear, Nuklear doesn't cares how do I map my keys, devices or anything, it just cares for being fed input events in a format it understands.

At least that's my POV after trying out integrating JavaFX, Swing, TWL and NiftyGUI multiple times over time. Nuklear, being the pain in the ass of a library it is to use, was way, way easier to integrate. And since the API is so clean cut on that regard, it was easier to debug. I know when it renders because I render it. I know when it processes input because I tell it to.

That said, I know UI is hard, that's why I'm always in the lookout for a lib that does it for me. I understand making such a simple integration point isn't easy, and may require rethinking a lot of abstractions. For example, it's really hard to move TWL as-is to such scheme, or even to using a core GL context (nevermind Vulkan) with it because not only believes it should draw itself, it believes it should draw itself like they did with OpenGL 1.1 in 1998, which for any kind of recent project is a problem. The right abstractions just aren't there.

Moreover, I will argue that moving to such scheme not only opens up the integration possibilities, it has no drawbacks. Nothing prevents you from implementing a GL 1.1 renderer for Nuklear, or a Vulkan one, or a software renderer, nothing prevents you from supplying separately your own backends to draw it if you really really want to code an UI renderer. Hell, it wouldn't surprise me if data-driving it that way made the whole thing faster via handling flat buffers of draw commands and vertices. It just makes the whole ordeal way easier for everyone involved.

I'm not writing this for you specifically anyway, so dont take it as a big ass critique of your code, I've only skimmed through your library and it probably does a lot of things right that aren't even on my mind right now. I'm just writing it for anyone who is interested in doing/using an UI library in Java for videogame (or any real-time graphics) purposes.

dustContributor on 8 Sep 2018

I understand what you mean, @dustContributor, and of course big projects often use their own decisions that fit better to the project and work faster.

I'm not saying that my decision is good, but I tried to make it as much extensible as it could be.
Renderers could be implemented using OpenGL, Vulkan or and any other technology.

Any way every solution is good for it's purposes. If you need best performance - you implement something in your own way, if you need fast development without pain - you can choose existing solution. 👌

SpinyOwl on 9 Sep 2018

https://tomsondev.bestsolution.at/2018/12/04/announcing-efxclipse-driftfx-integrating-native-rendering-pipelines-into-javafx/

RUSshy on 4 Dec 2018

👍2

Interesting new feature for JavaFX https://twitter.com/johanvos/status/1128627206295629824

kappaOne on 15 May 2019

Sounds like it still uses texture copies though. =( See: https://twitter.com/tomsontom/status/1128698978520977411 DriftFX still looks much better. Just wish it were more portable.

cuchaz on 15 May 2019

Looks like some momentum is starting to build for integrating DriftFX into LWJGL.

See: https://github.com/eclipse-efx/efxclipse-drift/issues/9

cuchaz on 23 May 2019

would there be a way to surround a 3d scene with a gui i made using a flat quad?

hYdos on 1 Jun 2019

Now that https://github.com/eclipse-efx/efxclipse-drift/issues/9 has been closed, what's next for the integration branch (https://github.com/LWJGL/lwjgl3/tree/driftfx)? Is it available for testing as a snapshot build?

zugaldia on 16 Nov 2019

The latest snapshot (LWJGL 3.2.4 build 4) includes DriftFX. It's a build of the wip_performance branch.

Since the implementation is unstable, performance is suboptimal and only Java 8 is supported, lwjgl-driftfx is considered experimental. Like lwjgl-bullet, it will be available in snapshots only until the situation improves.

Spasi on 27 Dec 2019

Is there a way I can skip building this or get it to build with java11-openjfx installed? It can't seem to locate classes related to openjfx.

gudenau on 8 Jul 2020

Hey @gudenau,

As of 92c88d461df60e1017322ea3f6372c00ded3892a there is support for JavaFX 11. The build should download the necessary OpenJFX jar files automatically via Maven. If it doesn't work for you, please open a new issue and paste the error you're getting.

You can also skip the DriftFX bindings by setting binding.driftx to false in config/build-bindings.xml.

Spasi on 8 Jul 2020

I've been working on a 3D RuneScape map editor using JFXGL and threekt:

RuneScape map editor

The performance is not bad at all. It does have some UI issues but there are workarounds for that :)

However, I have yet to figure out how I'm going to render popups and child windows and such.

Displee on 25 Feb 2021

🎉3 👍1

Holy cow, that's awesome!

cuchaz on 25 Feb 2021

❤1

Lwjgl3: Add a GUI solution to LWJGL

Most helpful comment

All 75 comments

Related issues