Kitty: Kitty freezes

[blueForestIcarus]

I frequently get an issue where rendering in kitty freezes until I either click on the window (which doesn't always work) or change the window size (for example, by going fullscreen).

I have not noticed anything in particular which triggers the freezes, they happen both when scrolling and typing, although they are more frequent when scrolling.

I am running arch linux with i3wm, Xorg, mesa 18.1.4.
My laptop has Intel HD Graphics 520 and I have the xf86-video-intel package installed for the i915 driver.

This has been a persistent issue for me across a few versions and I would like to help track it down if I can. Unfortunately, I have no idea where to start.

[maximbaz]

I actually experience some freezes as well, but for me kitty unfreezes in a second by itself. So, not sure if I should intervene in your issue or create a new one.

I can reproduce this pretty easily, maybe every few minutes. Best way to notice this is just hold a key and wait until the next freeze happens.

I usually use zsh, but just to make sure the problem is with kitty, I commented out my ~/.config/kitty/kitty.conf and started a fresh kitty instance running /bin/dash (which has no custom config whatsoever).

Notice how kitty froze for a moment while I was holding Backspace removing bs.

[kovidgoyal]

@maximbaz The first thing to do is set

GLFW_IM_MODULE=none

which will prevent kitty from using ibus which is a frequent cause of freezes/poor performance when handling key presses. Run kitty like this:

GLFW_IM_MODULE=none kitty

@blueForestIcarus I doubt your issue has anything to do with the IM module since you say it has been present across multiple releases. When you say kitty freezes what does that mean exactly? It stops showing updated output from the child process? It does not respond to key presses when pressing shortcuts for example to open new windows/tabs? Does switching to another application and back unfreeze it?

[blueForestIcarus]

When kitty freezes it stops updating the window. Switching application focus, clicking the window, resizing / fullscreening and sometimes typing work to unfreeze the window (but only fullscreening/resizing works every time). When the output unfreezes the rendering updates to what I would expect (implying that the child process is still receiving keystrokes). For example, if the output freezes in vim, I can continue typing and when I click to unfreeze the terminal it is evident that vim received the keystrokes. Likewise, when output freezes during scrolling, it unfreezes scrolled to the right position.

I have this issue most often in vim, or when scrolling through the output buffer, though it does happen occasionally when running commands.

[kovidgoyal]

Does pressing shortcuts to open new window work? Run kitty as

GLFW_IM_MODULE=none kitty --debug-keyboard --debug-gl > /tmp/log 2> /tmp/log

and see if anything is written to the log when your freeze happens -- you should see every key event being logged. Does the logging continue during the freeze?

From your description it sounds like glfw is hanging waiting for events from the X server. You can test that by putting some debug prints into the main_loop function in child-monitor.c Most likely it is hanging in wait_for_events()

[maximbaz]

I'm running kitty 0.11.3 which doesn't have ibus implementation yet, so GLFM_IM_MODULE=none is probably not affecting me (correct assumption?). By the way, why not treat the absence of that variable as "none", if it known to improve performance? I don't think every kitty's user needs ibus anyway, it can be an opt-in feature.

When I said "I can reproduce this pretty easily", I swear I could reproduce this every few minutes, I even recorded a few gifs in order to choose the best one, but right now I can't reproduce this anymore. I will get back to you if I manage to gather some additional info, e.g. using the --debug switches.

[kovidgoyal]

Yes, 0.11.3 does not have ibus, That env var is an override. IBUS is only turned on if some ibus env var is set and the ibus daemon is running. By setting GLFW_IM_MODULE=none you can ensure ibus is not used even if the above conditions are met.

[kovidgoyal]

@maximbaz if you are having difficult reproducing, then it might be dependent on external conditions, such as system load, keep an eye on htop or similar when it happens.

[blueForestIcarus]

Well this is odd.
When I run that command you gave me and tail -F /tmp/log, the log only updates what seems to be every 20 keystrokes (it seems to be 20 pretty consistently). After 20 keystrokes tail gets all twenty events all at once. What is even stranger is that if I get the terminal to freeze by scrolling and then type twenty keystrokes, then the output of kitty unfreezes at exactly the same time as the log updates.

Needless to say, I tailed the file in a differn't terminal than kitty to make sure it was the log and not kitty that wasn't updating. I tried this in two terminals to double check: same results.

Also yes, shortcuts to open a new window will unfreeze kitty.

[kovidgoyal]

That's just an artifact of using tail. Instead of redirecting the output to /tmp/log just look at it in a terminal and it should update immediately.

[blueForestIcarus]

Right you are.

Nothing is printed to the log when the terminal freezes. However, if the logging window is visible when kitty freezes then typing unfreezes it. I think this might be the output in the logging window forcing the screen to be redrawn.

[kovidgoyal]

You are using kitty itself as the output window? Use a non-kitty terminal or a separate instance of kitty. Or are you saying that you are already doing that and changes in any non-kitty window are enough to unfreeze it?

[kovidgoyal]

It does not sound like this bug is in kitty, if external window activity wakes it up. It sounds like for some reason X events are not being delivered to kitty byut he window manager during a freeze. You can confirm that by adding some prints to child-monitor.c as I described earlier.

[Synray]

Since a few days ago, I started getting the same freezes that @maximbaz showed. I agree with @kovidgoyal, the freezes could be a driver issue, or else maybe caused by an xorg.conf option.

I run Arch on an optimus laptop, with i3-wm, using the kitty-git package.
I was testing Vulkan, and during the process I uninstalled bumblebee. After that I had to change my xorg.conf to use the intel driver instead of nvidia. Once that was done, the freezes started, not just in kitty, but also on youtube videos, and even pastebin.com, (tested on qutebrowser and firefox).

Once I reinstalled bumblebee, and reverted my xorg.conf, the freezes stopped. It's a really strange bug, I'm curious how many people are also on optimus laptops? The freezes could be caused by the intel driver, but that might just be specific to my case.

[maximbaz]

I use Intel driver as well (on Dell Precision 5520, I have additional Nvidia GPU, but not using it). However I don't see this happening in youtube or any other apps.

@kovidgoyal here's a gif showcasing the issue side by side with htop. Although CPU is used, I have 8 cores and there's plenty of free resources available. Also, I tried a different terminal (sakura) and couldn't reproduce the issue.

I think the problem is either in kitty, or in the Intel driver, but I have no idea how to find this out.

kitty --debug-gl prints nothing except this one line, even when freezes occur:

GL version string: '4.5 (Core Profile) Mesa 18.1.5' Detected version: 4.5

As for --debug-keyboard... Now this has given me some new information. It started with the following seemingly weird observation: every time I run kitty --debug-keyboard from a terminal (doesn't matter from which), in the newly spawned kitty that is open side by side with the parent terminal, I cannot reproduce the bug. As soon as I remove the flag and just run kitty, I can reproduce it. So I return the --debug-keyboard flag. Now if I redirect the output to /dev/null or a real file, again I can reproduce the issue. If I put two terminals on different i3 workspaces, I can reproduce the issue.

This gave me an idea: as long as there is something else refreshing the screen, kitty has no lag. So my next test was to put kitty side by side with a youtube video. And guess what: I play the video, kitty has no issues, I pause the video and the lag returns.

Also notice how consistent the lag is with htop redraws in the gif above.

As funny as this finding is, I am still not sure whether this is a bug in driver or in kitty. @kovidgoyal I would be very curious to hear your thoughts on this matter 🙂

[kovidgoyal]

@maximbaz: @blueForestIcarus already established that any unrelated screen activity prevents the freeze from happening, see my last post in this thread. As far as I can tell, this is definitely a driver issue, or at best a combination of glfw+driver issue. I suspect that what happens is the x server stops delivering messages to kitty when the freeze happens, so kitty does not have any events to process and simply sits waiting. You can confirm that by adding debug prints to wait_for_events() as I described in this thread a week ago. I have no idea why the X server does this, but it definitely seems to be driver related/Xorg.conf related.

[maximbaz]

Huh, sorry I didn't connect the dots yesterday 🙂

I just performed the test, doesn't look like kitty stops getting events.

Here's the patch I applied. The idea is to print the number of times the loop is executed per second:

Click to see

diff --git a/kitty/child-monitor.c b/kitty/child-monitor.c
index 606fe396..2eb5ae60 100644
--- a/kitty/child-monitor.c
+++ b/kitty/child-monitor.c
@@ -18,6 +18,7 @@
 #include <signal.h>
 #include <sys/socket.h>
 extern PyTypeObject Screen_Type;
+#include <time.h>

 #define EXTRA_FDS 2
 #ifndef MSG_NOSIGNAL
@@ -755,7 +756,18 @@ main_loop(ChildMonitor *self, PyObject *a UNUSED) {
 #define main_loop_doc "The main thread loop"
     bool has_open_windows = true;

+    int count = 0;
+    char prev[100];
     while (has_open_windows) {
+        count++;
+        time_t ltime = time(NULL);
+        char new[100];
+        strcpy(new, asctime(localtime(&ltime)));
+        if (strcmp(prev, new) != 0) {
+            printf("loop %d - %s", count, new);
+            count = 0;
+            strcpy(prev, new);
+        }
         double now = monotonic();
         if (global_state.has_pending_resizes) process_pending_resizes(now);
         render(now);

Now to reproducing the bug. I disabled my i3 bar and moved kitty on a new workspace, this allows me to see kitty hanging for a few seconds.

My expectation was that I would see missing timestamps if kitty hangs in wait_for_events or somewhere else.

Here's how it went:

❯ python .
loop 1 - Mon Aug  6 12:13:07 2018
loop 39 - Mon Aug  6 12:13:08 2018
loop 41 - Mon Aug  6 12:13:09 2018
loop 4 - Mon Aug  6 12:13:10 2018
loop 169 - Mon Aug  6 12:13:11 2018         <--- begin holding the key and typing "aaaaa...."
loop 294 - Mon Aug  6 12:13:12 2018
loop 291 - Mon Aug  6 12:13:13 2018
loop 294 - Mon Aug  6 12:13:14 2018
loop 294 - Mon Aug  6 12:13:15 2018
loop 293 - Mon Aug  6 12:13:16 2018
loop 290 - Mon Aug  6 12:13:17 2018
loop 289 - Mon Aug  6 12:13:18 2018         <--- stop holding the key, this time bug didn't reproduce
loop 38 - Mon Aug  6 12:13:19 2018
loop 121 - Mon Aug  6 12:13:20 2018
loop 294 - Mon Aug  6 12:13:21 2018         <--- begin holding the key and typing "aaaaa...." again
loop 307 - Mon Aug  6 12:13:22 2018
loop 305 - Mon Aug  6 12:13:23 2018
loop 304 - Mon Aug  6 12:13:24 2018
loop 290 - Mon Aug  6 12:13:25 2018
loop 313 - Mon Aug  6 12:13:26 2018
loop 304 - Mon Aug  6 12:13:27 2018         <--- around this time kitty stops refreshing, I still hold the key, but no new "aaaa" appear on the screen
loop 307 - Mon Aug  6 12:13:28 2018
loop 303 - Mon Aug  6 12:13:29 2018
loop 304 - Mon Aug  6 12:13:30 2018         <--- .... I still hold the key, still no new letters appear on screen....
loop 296 - Mon Aug  6 12:13:31 2018
loop 307 - Mon Aug  6 12:13:32 2018
loop 291 - Mon Aug  6 12:13:33 2018         <--- stop holding the key, kitty still hangs, I press Ctrl+C and this refreshed the kitty
loop 41 - Mon Aug  6 12:13:34 2018
loop 30 - Mon Aug  6 12:13:35 2018
loop 303 - Mon Aug  6 12:13:36 2018
loop 170 - Mon Aug  6 12:13:37 2018
loop 43 - Mon Aug  6 12:13:38 2018
loop 6 - Mon Aug  6 12:13:39 2018
loop 3 - Mon Aug  6 12:13:40 2018
loop 4 - Mon Aug  6 12:13:41 2018
loop 5 - Mon Aug  6 12:13:42 2018
loop 16 - Mon Aug  6 12:13:43 2018
loop 1 - Mon Aug  6 12:13:45 2018
loop 4 - Mon Aug  6 12:13:46 2018

[kovidgoyal]

You should also turn off cursor blink otherwise the loop will tick every cursor_blink_interval even if no events are received. Also print out maximum_time in your print statement to be sure that kitty is waiting for events and not a timeout such as for cursor blink.

[maximbaz]

Here it is. What is maximum_time, is it maximum_wait?

Patch

diff --git a/kitty/child-monitor.c b/kitty/child-monitor.c
index 606fe396..67ad5d8f 100644
--- a/kitty/child-monitor.c
+++ b/kitty/child-monitor.c
@@ -18,6 +18,7 @@
 #include <signal.h>
 #include <sys/socket.h>
 extern PyTypeObject Screen_Type;
+#include <time.h>

 #define EXTRA_FDS 2
 #ifndef MSG_NOSIGNAL
@@ -755,7 +756,18 @@ main_loop(ChildMonitor *self, PyObject *a UNUSED) {
 #define main_loop_doc "The main thread loop"
     bool has_open_windows = true;

+    int count = 0;
+    char prev[100];
     while (has_open_windows) {
+        count++;
+        time_t ltime = time(NULL);
+        char new[100];
+        strcpy(new, asctime(localtime(&ltime)));
+        if (strcmp(prev, new) != 0) {
+            printf("maximum_wait: %.2f, loop: %d - %s", maximum_wait, count, new);
+            count = 0;
+            strcpy(prev, new);
+        }
         double now = monotonic();
         if (global_state.has_pending_resizes) process_pending_resizes(now);
         render(now);

Test run:

maximum_wait: -1.00, loop: 1 - Mon Aug  6 12:45:28 2018
maximum_wait: -1.00, loop: 64 - Mon Aug  6 12:45:29 2018         <--- begin holding the key and typing "aaaaa...."
maximum_wait: -1.00, loop: 42 - Mon Aug  6 12:45:30 2018
maximum_wait: -1.00, loop: 181 - Mon Aug  6 12:45:31 2018
maximum_wait: -1.00, loop: 304 - Mon Aug  6 12:45:32 2018
maximum_wait: 0.00, loop: 295 - Mon Aug  6 12:45:33 2018
maximum_wait: 0.00, loop: 297 - Mon Aug  6 12:45:34 2018
maximum_wait: 0.00, loop: 306 - Mon Aug  6 12:45:35 2018
maximum_wait: -1.00, loop: 309 - Mon Aug  6 12:45:36 2018
maximum_wait: -1.00, loop: 309 - Mon Aug  6 12:45:37 2018         <--- somewhere around here kitty stopped refreshing
maximum_wait: -1.00, loop: 307 - Mon Aug  6 12:45:38 2018
maximum_wait: -1.00, loop: 300 - Mon Aug  6 12:45:39 2018
maximum_wait: -1.00, loop: 113 - Mon Aug  6 12:45:40 2018         <--- stop holding the key, pressed the key again a few times, nothing happened, pressed Ctrl+C and kitty refreshed
maximum_wait: -1.00, loop: 43 - Mon Aug  6 12:45:41 2018
maximum_wait: -1.00, loop: 40 - Mon Aug  6 12:45:43 2018
maximum_wait: -1.00, loop: 7 - Mon Aug  6 12:45:45 2018

[maximbaz]

Also, Ctrl+C is not guaranteed to make kitty start refreshing again, but it feels like it helps sometimes, that's why I press it in the end of the test 🙂

[kovidgoyal]

That output doesn't make sense, why would it be refreshing hundreds of times a second? With cursor blink turned off the loop should basically hang in wait_for_events() indefinitely until you press a key. For example, with the following one line patch

 diff --git a/kitty/child-monitor.c b/kitty/child-monitor.c
index 606fe396..beee82c1 100644
--- a/kitty/child-monitor.c
+++ b/kitty/child-monitor.c
@@ -760,6 +760,7 @@ main_loop(ChildMonitor *self, PyObject *a UNUSED) {
         if (global_state.has_pending_resizes) process_pending_resizes(now);
         render(now);
         wait_for_events();
+        printf("tick at %f\n", monotonic());
 #ifdef __APPLE__
         if (cocoa_pending_actions) {
             if (cocoa_pending_actions & PREFERENCES_WINDOW) { call_boss(edit_config_file, NULL); }

running kitty as
kitty -o cursor_blink_interval=0

I get a few initial ticks and then no output until I press a key which gives a few more ticks and so on.

Is that loop running continuously for you? If so something else is seriously wrong.

[maximbaz]

It runs hundreds of times per second because I press and hold "a" key, just like on my gifs above. This is just the way to easily see when kitty hangs. If I don't press anything, indeed no prints occur.

What the log shows is that the loop runs hundreds of seconds even after kitty hangs, i.e. I still hold the key, kitty still runs that loop, but not draws anything new.

[maximbaz]

With this test I expected that if kitty doesn't render anything on the screen while I keep holding a keyboard button, it would mean that it is stuck somewhere and the main loop doesn't run, and so I would see missing timestamps. But the loop runs, hundreds of times per second, so kitty receives my key presses, just doesn't render them sometimes.

[kovidgoyal]

I dont see how that could be, according to xset -q the keyboard repeat rate is 25 by default which would mean 25 repeats per second, not hundreds of them.

A render will happen when some output is received from the child process (or the kernel echoing the key press to stdout). You can track if that is happening in by putting a print at line 603.

[maximbaz]

I use xset r rate 250 45, so for me the repeat rate is 45, but yeah, still not hundreds. Do you get the expected counts with my patch, around 25?

xset -q

❯ xset -q                         
Keyboard Control:
  auto repeat:  on    key click percent:  0    LED mask:  00000002
  XKB indicators:
    00: Caps Lock:   off    01: Num Lock:    on     02: Scroll Lock: off
    03: Compose:     off    04: Kana:        off    05: Sleep:       off
    06: Suspend:     off    07: Mute:        off    08: Misc:        off
    09: Mail:        off    10: Charging:    off    11: Shift Lock:  off
    12: Group 2:     off    13: Mouse Keys:  off
  auto repeat delay:  250    repeat rate:  45
  auto repeating keys:  00ffffffdffffbbf
                        fadfffeffffdffff
                        9fffffffffffffff
                        fff7ffffffffffff
  bell percent:  50    bell pitch:  400    bell duration:  100
Pointer Control:
  acceleration:  2/1    threshold:  4
Screen Saver:
  prefer blanking:  yes    allow exposures:  yes
  timeout:  0    cycle:  600
Colors:
  default colormap:  0x50    BlackPixel:  0x0    WhitePixel:  0xffffff
Font Path:
  /usr/share/fonts/misc,/usr/share/fonts/TTF,/usr/share/fonts/OTF,built-ins
DPMS (Energy Star):
  Standby: 600    Suspend: 600    Off: 600
  DPMS is Disabled

---

Added the printf to line 603, it is also getting executed, around 45 times a second as per your prediction 🙂 And I see the prints also when kitty is not drawing my key presses.

[kovidgoyal]

If line 603 is getting executed, then kitty is drawing the changes as expected, which means that for some reason the changes are not being updated by the X server/graphics driver. Which brings us back to driver issues. Sadly, I am no expert on the X server, so I haven't a clue where to start debugging that.

I'll test your patch for the repeat rates a little later, as I am currently away from my dev box. It could be that there are some additional events waking up the loop (there will be some, for example data from the child process, but that I would expect to cause a 2-4x change not a 8x change in loop ticks).

[kovidgoyal]

Here are my results with your patch (repeat rate of 25)

loop 1 - Mon Aug  6 17:17:09 2018
loop 25 - Mon Aug  6 17:17:11 2018
loop 5 - Mon Aug  6 17:17:12 2018
loop 58 - Mon Aug  6 17:17:13 2018
loop 157 - Mon Aug  6 17:17:14 2018
loop 155 - Mon Aug  6 17:17:15 2018
loop 153 - Mon Aug  6 17:17:16 2018
loop 154 - Mon Aug  6 17:17:17 2018
loop 157 - Mon Aug  6 17:17:18 2018
loop 152 - Mon Aug  6 17:17:19 2018
loop 160 - Mon Aug  6 17:17:20 2018
loop 146 - Mon Aug  6 17:17:21 2018

This is a factor of 6x between the repeat rate and the tick rate. A bit higher than my expected 4x so there are probably a couple more event types generated that I did not think of. But, in any case, this has no bearing on the freeze issue.

[kovidgoyal]

Oh and there is one more check you can make, in kitty code to be sure. In the screen_draw() function in screen.c you can print out what character is being drawn. If that prints out the key you are pressing, then we know that the key event has been processed as expected and the updated buffer has been rendered as expected (at line 603)

[maximbaz]

It does print the key I'm pressing.

Well, I think we knew that it will be a driver issue, since you cannot reproduce this.

I also have a theory on why Ctrl+C almost always "wakes up" kitty: in my particular test (typing aaaaa in shell prompt), one key press corresponds to one character rendered, but Ctrl+C is special, it causes the entire prompt to be re-rendered, which is a lot of characters. My theory is that such kind of burst wakes up the rendering.

[kovidgoyal]

I am closing this, since as best as I can tell, this is not an issue in kitty code. If anybody has evidence to the contrary feel free to post and I will re-open.

[Ealhad]

I am running arch linux with i3wm, Xorg, mesa 18.1.7
My laptop has HD Graphics 630 and I have the xf86-video-intel package installed for the i915 driver.

I encounter the same issue, Kitty regularly stops updating window (most of the time when scrolling), even though it receives input (when I change the window size, I can see Kitty processed all the input, be it from mouse or keyboard).

This doesn't happen with other terminal emulators (i.e. Termite)

[yrizk]

same, I have i3 and this issue is happening.