Imgui: Re-render only on user interaction

[Person-93]

_(you may also go to Demo>About Window, and click "Config/Build Information" to obtain a bunch of detailed information that you can paste here)_

Version/Branch of Dear ImGui:

Version: 1.66
Branch: master

Back-end/Renderer/Compiler/OS

Back-ends: opengl3
Compiler: gcc, mingw, clang
Operating System: linux/windows
My Issue/Question:
Is there a way to tell if the user has interacted with the window since the last rendering?

I want to skip rendering most of the window if the user hasn't interacted. (i.e. clicked, typed, moved mouse).

Also, this might not be directly related to this library, but is there a way to tell if the window is completely covered by other windows? In that case, I would skip rendering entirely.

[ocornut]

Is there a way to tell if the user has interacted with the window since the last rendering?
I want to skip rendering most of the window if the user hasn't interacted. (i.e. clicked, typed, moved mouse).

Effectively you want to check for "clicked,typed,moved mouse" and you have this information already because your back-end is passing it to dear imgui and you are looking into using a custom back-end if you want to go idle. You need to setup a counter, and reset it to 2 or 3 whenever an input came (so a single button click will "wake up" rendering for 2 or 3 frames).

Also see #71 and #1206.

Also, this might not be directly related to this library, but is there a way to tell if the window is completely covered by other windows? In that case, I would skip rendering entirely.

This is a different question worthy of a different topic.
It's not possible currently and it's not something I put much energy on because the default styles have transparent window background. You may also want to measure the cost that you are trying to avoid.

[ocornut]

Addendum: Your subject line says "re-render only on user interaction" but the point of the technique highlighted is in my first paragraph is mainly to avoid running the ImGui:: functions and touching your source data. The actual rendering cost is generally lesser than the building of it.

[djdeath]

For a Gtk+/OpenGL backend I have a redraw on user interaction heuristic :
Render 2 frames on input events 1 and on text entry focused, redraw at the blinking speed of the cursor 2
For interactive graphs, I added a schedule frame method (a bit like requestAnimationFrame in Javascript) 3

[jpcy]

You can skip rendering a frame if the contents of ImDrawData hasn't changed since the last rendered frame.

[ocornut]

Thank you @djdeath for those links. I would like to eventually make it a more obvious first-party citizen.
We should probably expose various counters and timers carrying specific semantic (e.g. cursor blink timer which the user may choose to use or ignore or adapt).

For the blinking cursor I would like to implement this idea where by storing a persisting reference to the vertices used by the cursor, we could have a fast pass patching the alpha of the 4 vertices. So some ImGui:: function could be independently handling cursor patching, and when the timer elapse we only do patching + request GPU refresh.

[tomasiser]

Just wanted to put a comment here as I had to deal with this problem last year.

Also, this might not be directly related to this library, but is there a way to tell if the window is completely covered by other windows? In that case, I would skip rendering entirely.

According to my previous research, this is incredibly difficult to do on Windows, or at least I did not find a 100% working solution*. If someone knows, please let us know! Now I have no idea about other platforms, but let me explain how I managed to at least partially solve this problem on Windows.

So first of all, in Windows API, there is an IsIconic function available (provided you know the HWND of your main window, which should be easy to get from most rendering backends). The tricky part of this function is that it's almost _useless_ as it only returns true if the window is actually minimized using the minimize button in the top right corner. It does _not_ even return true when you press Win+D. So this is not going to help us.

Then there is an IsWindowVisible, but don't get misled by its name. It has nothing to do with your window being covered by other windows. It only returns whether the show/hide flag of the window is set to visible. So again, nothing which can help us.

Then we have GetForegroundWindow, GetActiveWindow, and GetTopWindow. These are also useless because you probably want to render animated stuff even if your window is not on top of everything else, but still visible on the screen. So again, nothing which can help us.

Finally, I decided to implement a solution based on manually finding windows from coordinates. The idea is fairly "simple". You just need to choose a few coordinates on your window and ask Windows which HWND is visible at that coordinates. In my implementation, I decided to use 3 points: the top-left corner, the window center, and the bottom-right corner. If _all these 3 points_ are obscured by another window, I conclude that I do not need to render anything as my window is probably not visible at all.

Note that this solution also works if the window is "behind" desktop (i.e., when you press Win+D).

Example code copy pasted from my implementation:

// do not forget this:
#include "windows.h"

bool is_window_covered(HWND wnd) {
    if(IsIconic(wnd)) {
        return true;  // early return, window is minimized (iconic)
    }

    RECT windowRect;
    if(GetWindowRect(wnd, &windowRect)) {
        // check if window is obscured by another window at 3 diagonal points (top left, center, bottom right):
        bool isObscuredAtDiagonal = true;
        POINT checkpoint;
        // check window top left:
        checkpoint.x = windowRect.left;
        checkpoint.y = windowRect.top;
        auto wndAtCheckpoint = WindowFromPoint(checkpoint);
        isObscuredAtDiagonal &= (wndAtCheckpoint != wnd);
        // check window center:
        checkpoint.x = windowRect.left + (windowRect.right - windowRect.left) / 2;
        checkpoint.y = windowRect.top + (windowRect.bottom - windowRect.top) / 2;
        wndAtCheckpoint = WindowFromPoint(checkpoint);
        isObscuredAtDiagonal &= (wndAtCheckpoint != wnd);
        // check window bottom right:
        checkpoint.x = windowRect.right - 1;
        checkpoint.y = windowRect.bottom - 1;
        wndAtCheckpoint = WindowFromPoint(checkpoint);
        isObscuredAtDiagonal &= (wndAtCheckpoint != wnd);
        if(isObscuredAtDiagonal) {
            return true;
        }
    }

    return false;
}

---

* Note that there is another solution explained at the bottom of this page. It is based on clipping. You may want to check it out. I have no idea if it works as I have not tested that one.

[malamanteau]

In my current prototype I hash the current drawlist, and compare this against the previous hash. If the hash is the same I don't redraw. I force a full redraw every second to make sure that it get redrawn in certain edge cases (waking up from stand-by etc). I keep a separate hash for every ImGuiViewport.
XXHash64 works really well!

I also enable and disable vsync based on input events to reduce latency.

I limit the framerate to four times the monitor refresh rate (when vsync is off), and with all these changes CPU usage is very low, and only spikes a bit during heavy interaction.

static void ImGui_ImplDX11_RenderWindow(ImGuiViewport* viewport, void*)
{
    uint64 lastHValue = 0;
    if (viewport->UserData)
    {
        lastHValue = ((XXHash64*)viewport->UserData)->hash();

        delete (XXHash64*)viewport->UserData;
        viewport->UserData = nullptr;
    }

    viewport->UserData = new XXHash64(1105121757519812621);
    XXHash64 * h = (XXHash64 *)viewport->UserData;

    ImGuiIO & io = ImGui::GetIO();

    const float width_points  = io.DisplaySize.x;
    const float height_points = io.DisplaySize.y;

    ImDrawData * draw_data = viewport->DrawData;

    const int  width_pixels = (int)(draw_data->DisplaySize.x * io.DisplayFramebufferScale.x);
    const int height_pixels = (int)(draw_data->DisplaySize.y * io.DisplayFramebufferScale.y);

    h->add(&width_pixels,               4_u64);
    h->add(&height_pixels,              4_u64);
    h->add(&(draw_data->CmdListsCount), 4_u64);

    for (int i = 0; i < draw_data->CmdListsCount; i++)
    {
        h->add(&(draw_data->CmdLists[i]->VtxBuffer[0]), draw_data->CmdLists[i]->VtxBuffer.size() * sizeof(ImDrawVert));
        h->add(&(draw_data->CmdLists[i]->IdxBuffer[0]), draw_data->CmdLists[i]->IdxBuffer.size() * sizeof(ImDrawIdx));
        h->add(&(draw_data->CmdLists[i]->CmdBuffer[0]), draw_data->CmdLists[i]->CmdBuffer.size() * sizeof(ImDrawCmd));
    }

    uint64   nuHValue = h->hash();

    if (lastHValue == nuHValue)
    {
        viewport->NeedSwap = false;
        return;
    }
    else 
        viewport->NeedSwap = true;
....

and also:

static void ImGui_ImplDX11_SwapBuffers(ImGuiViewport* viewport, void*)
{
    if (!viewport->NeedSwap)
        return;
....

I had to add two items to ImGuiViewport to keep track of these things:

    bool                NeedSwap = false;
    void*               UserData = nullptr; 

Here is my function for detecting user input (I use this to temporarily disable vsync, to make the app more responsive):

inline bool AutoShouldVSync()
{
    ImGuiIO & io = ImGui::GetIO();

    static bool mouseDown0Prev = io.MouseDown[0];
    static bool mouseDown1Prev = io.MouseDown[1];
    static bool mouseDown2Prev = io.MouseDown[2];
    static bool mouseDown3Prev = io.MouseDown[3];
    static bool mouseDown4Prev = io.MouseDown[4];
    static Math::Vector2 mousePosPrev = io.MousePos;

    bool mouseDownChanged =
        (io.MouseDown[0] != mouseDown0Prev) ||
        (io.MouseDown[1] != mouseDown1Prev) ||
        (io.MouseDown[2] != mouseDown2Prev) ||
        (io.MouseDown[3] != mouseDown3Prev) ||
        (io.MouseDown[4] != mouseDown4Prev);
    bool mouseMoved = io.MousePos != mousePosPrev;

    bool mouseIsDown =
        io.MouseDown[0] ||
        io.MouseDown[1] ||
        io.MouseDown[2] ||
        io.MouseDown[3] ||
        io.MouseDown[4];

    mousePosPrev   = io.MousePos;
    mouseDown0Prev = io.MouseDown[0];
    mouseDown1Prev = io.MouseDown[1];
    mouseDown2Prev = io.MouseDown[2];
    mouseDown3Prev = io.MouseDown[3];
    mouseDown4Prev = io.MouseDown[4];

    bool mouseWheelChanged = io.MouseWheel != 0.0f || io.MouseWheelH != 0.0f;

    return !(mouseIsDown || mouseDownChanged || mouseMoved || mouseWheelChanged);
}

@ocornut is this a functionality you might want to merge into the official project?

[Lectem]

I think this is a very interesting topic, especially when deploying imgui to the web !
Maybe a hash could be computed directly by imgui each time a new draw call is added, and then expose this hash ?

[ocornut]

I think this is a very interesting topic, especially when deploying imgui to the web !
Maybe a hash could be computed directly by imgui each time a new draw call is added, and then expose this hash ?

It would be too costly for imgui to do this by default, so best left to the user.
We could easily optimize the best case (e.g. first hash and compare ImDrawCmd data only), but we have to deal with the worst case which is that nothing changed and would need hashing of the whole vertex data to confirm it.

An alternative would be to allow variable-frequency refresh of individual windows (e.g. non-focused windows could be configured to refresh at lower frequency), and the ImDrawList can carry a timestamp.

Note that all of this is different from the original post which was to render on user interaction.