Alacritty: Significant startup time

[occivink]

Linux, X11. Using eb231b3e70b87875df4bdd1974d5e94704024d70, release build (the AUR one).

Starting up alacritty takes a significant amount of time. Simply doing time alacritty -e false reports about 390ms. Other terminal emulators, such as urxvt or termite start much faster, at about 160ms for both.
It's also not just a matter of this simple benchmark, in normal usage it also feels slower to start.

[lord-re]

I just tried with an older build and after a git pull and got more or less the same duration ~120ms.
What's your gpu and driver ?

[occivink]

It's an intel HD520, and I don't have the DDX driver installed (so modesetting).
I don't think it's opengl issue, other programs that require opengl contexts (e.g. mpv) are able to start faster.
I'll try to profile the startup time

[alexozer]

I also wish alacritty's startup time was near instant, it's a surprisingly large component of what I look for in my preferred terminal (I would probably switch to alacritty from gnome-terminal if it just started faster). Latency matters.

I tried changing some font / dpi settings and that seemed to affect the startup time too, so who knows which factors cause the most latency. Has a server-client mode for alacritty ever been considered?

[jwilm]

There's some pretty low hanging fruit in this area still before we consider something like a client-server model. For instance, we don't even start the shell (or whatever program) until after the renderer is fully initialized. We have plans (#673) to put the renderer on a secondary thread which will help both startup time and input latency.

[alexozer]

Not really sure what happened but startup performance seems to have, as of now, significantly improved! I'm assuming putting the renderer on a separate thread would still be helpful as a takes a brief bit for the window to draw after the frame opens, but it still seems much faster overall.

[zacps]

I get the following results on a GTX970 with NVIDIA 387.12 drivers and Alacritty @geb231b3:

 Performance counter stats for 'alacritty -e false' (10 runs):
        132.778607      task-clock (msec)         #    0.482 CPUs utilized            ( +-  0.64% )
               187      context-switches          #    0.001 M/sec                    ( +-  5.24% )
                 0      cpu-migrations            #    0.003 K/sec                    ( +- 55.28% )
            20,744      page-faults               #    0.156 M/sec                    ( +-  0.03% )
       472,537,165      cycles                    #    3.559 GHz                      ( +-  0.93% )
       557,793,536      instructions              #    1.18  insn per cycle           ( +-  1.10% )
       115,965,180      branches                  #  873.372 M/sec                    ( +-  1.06% )
         1,361,352      branch-misses             #    1.17% of all branches          ( +-  0.22% )
       141,775,463      L1-dcache-loads           # 1067.758 M/sec                    ( +-  1.39% )
         4,855,266      L1-dcache-load-misses     #    3.42% of all L1-dcache hits    ( +-  0.14% )
         1,337,835      LLC-loads                 #   10.076 M/sec                    ( +-  0.30% )
           601,179      LLC-load-misses           #   44.94% of all LL-cache hits     ( +-  0.53% )
       0.275591435 seconds time elapsed                                          ( +-  2.48% )

Run with sudo perf stat -r 10 -d alacritty -e false

[occivink]

I took a flamegraph out of curiosity, and a surprisingly long time seems spent in libfontconfig and libfreetype.

[torpak]

I just started using the 0.2.0 version because of the scrollback feature.
I noticed that larger history values increase startup time siginficantly:
About a second on my system for 100000 and about 10 seconds for 1000000.
Runtime performance seems unchanged and better than anything else.

[sanga]

I've seen this too. And that's why I dropped the scrollback history to only 1000. Anecdotally (it was quite a while ago when I made that change and I didn't measure it at the time, so take with a grain of salt) that more that halved the startup-time from when scrollback history was at 10,000

[mstoeckl]

With current (release build) master, 80% of callgrind time for alacritty -e false is spent in fontconfig or libc functions called by fontconfig.

For sake of comparison, time alacritty -e false gives 0.890±0.030s real time, while xterm -e false takes 0.216±0.005s; the default st uses 0.193±0.003s, and a heavily patched st hits 0.060±0.005s.

If startup latency is really important to you, a workaround that I've used for some of my programs is to set up a ~50MB bitmap file containing images corresponding to all ~137,000 allocated unicode code points, prerendered in grayscale at my usual font size. Then font loading reduces to calling mmap once and copying memory as needed. The obvious downside is that combining characters, color emoji, oversize characters, and other unicode features are unavailable. (Of course, for some tasks, this "downside" is a feature.)

[chrisduerr]

There has been some further discussion and profiling on this issue in #1603.

I'd be interested to see how Linux and MacOS are different here. So far it seems like most reports are about fontconfig. However if there are caching issues on linux, it's not unlikely they are present on MacOS too.

[jonhoo]

So, specifically the insights from #1603 is that the fontconfig is spent in FcFontMatch, so it's not about rendering specific glyphs as far as I can tell, but about looking up the fonts in the first place. In particular, FcFontMatch is called 14 times during startup, and each call is quite costly, even though I think in theory we should only have to call it a handful of times. Especially if the user is only using a single font. Here's a flamegraph of master that includes the fontconfig symbols as well:

[zacps]

Using some manual tracing (callgraph generator when) I found the following path to FcFontMatch, which I suspect is the problematic one:

* render_cells (for cell in cells)
* GlyphCache::get
* get_glyph
* get_rendered_glyph
* face_for_glyph
* load_face_with_glyph
* FcFontMatch

I'll see if I can eliminate this.

[chrisduerr]

@zacps Thanks for investigating this. Also thanks to @jonhoo for providing so many useful flamegraphs.

It looks like these metrics should allow us to get rid of a significant portion of the startup time. If you need any help, please let me know.

[jwilm]

I'm kind of thinking that I want to try pulling in the font-kit crate rather than investing more time in our own font abstraction. I did some research yesterday on how Pango is doing fallbacks with fontconfig, and I've come to the conclusion that we are pretty far off the mark with our impl.

[zacps]

That sounds like a good idea, do you think it makes sense to start of by writing implementing our current font API for font-kit or adapting the current renderer to font-kit's API?

[kimsnj]

In one of my linux laptops, this is issue is particularly severe: 2s for alacritty -e false vs 140ms for gnome-terminal.

I've tried to generate a flamegraph from master (uploaded along the setup and times) in this gist: https://gist.github.com/kimsnj/6ee1a1992c856fb6a68ee94bee4780d1
(It is however missing some symbols and the rest is mangled).

Update: The issue on my laptop does not seem linked to fonts, but to winit… which takes 2seconds to create the window. I updated the gist with logs that clearly show this.

[jonhoo]

@kimsnj yeah, that's where I ended up too. It's been filed as https://github.com/tomaka/winit/issues/682.

[sersorrel]

The startup time for me (measured via time alacritty -e false) significantly increases if I have a compositor running, as well as if I have a second monitor attached:

              | 1 monitor | 2 monitors |
--------------+-----------+------------+
no compositor | 150ms     | 200ms      |
compositor    | 200ms     | 270ms      |

[jonhoo]

@anowlcalledjosh can you give https://github.com/tomaka/winit/issues/682#issuecomment-466767319 a try?

[sersorrel]

@jonhoo With that version of winit, the results are much more consistent regardless of the number of monitors! With a compositor, startup takes between 150-200ms; without a compositor, it's pretty consistently about 100ms.

[yaymukund]

Just wanted to attach my benchmarks on wayland, in case it's at all helpful:

# Dell XPS 15" 9560 (just the laptop screen, no external monitors, no touchscreen)
# sway version 1.0-rc1-148-g0df76ed9 (wayland, wlroots)
# alacritty 0.2.9 (`alacritty-git` AUR package, autogenerated alacritty.yml)
$ time alacritty -e bash -c sleep 0
==> alacritty -e bash -c sleep 0  0.10s user 0.04s system 29% cpu 0.475 total

$ time termite -e false
==> termite -e false  0.08s user 0.02s system 101% cpu 0.101 total

# If I disable wayland, it's fast:
$ export WAYLAND_DISPLAY="" ; time alacritty -e false
==> alacritty -e false  0.06s user 0.02s system 82% cpu 0.099 total

Note: I did -e bash -c sleep 0 because -e false would hang indefinitely, for some reason. But ~0.45s seems to match my experience of opening a new alacritty window.

Please let me know if there's anything I can provide that might be helpful. I'm really excited to see alacritty take off :rocket:

[yaymukund]

Figured it out! My issue was related to having nvidia-utils installed. Once I uninstalled that, everything was fast again. nvidia-utils depends on egl-wayland and eglexternalplatform, which may also have something to do with it. Much thanks to UndeadLeech for walking me through debugging this and generating flamegraphs.

[jeremyong]

Uninstalling nvidia-utils is definitely not a sustainable fix as that package contains both opengl and vulkan drivers (among other things). I can confirm that alacritty has at least double the startup time as urxvt.

Moving to a client-server model so you can share GPU resources instead of needing to build fonts every time definitely seems like good improvement...

[aude]

I can confirm Alacritty as of today (0.3.2) is faster.

In the start of 2019, alacritty -e false took a long time, 500-1000ms maybe.
Today, alacritty -e false takes around 150ms.

Thanks @jonhoo, @ZeGentzy and you others for tracking down the root cause, you da mans!

[drewboardman]

Mine takes about 2-3 seconds to open a new terminal.

[chrisduerr]

@drewboardman Which version is that?

[chrisduerr]

Usually startup should be ~0.1-0.3 seconds.

[kchibisov]

I'd bet it's on some nvidia system with libglvnd or something like that, since libglvnd is traversing configs, etc.

[drewboardman]

I do have an nvidia card, and nvidia-utils installed.

My version is 0.4.1 @chrisduerr

[Thovthe]

I have a somewhat exotic arm setup. I'm attracted to Alacritty mostly because of the well documented and expansive config file but I would love it to start in less than a second on this setup, I could make it my day to day then.

RK3399 SOC

Here are some hyperfine benchmarks for startup times.

Alacritty:

hyperfine --warmup 3 --export-markdown alactests.md 'alacritty -e false'
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| alacritty -e false | 1.488 ± 0.064 | 1.420 | 1.632 | 1.00 |

Run from Alacritty (Expand)

| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| alacritty -e false | 1.446 ± 0.045 | 1.388 | 1.530 | 1.00 |

Kitty:

hyperfine --warmup 3 --export-markdown kittests.md 'kitty -e false'
| Command | Mean [s] | Min [s] | Max [s] | Relative |
|:---|---:|---:|---:|---:|
| kitty -e false | 1.691 ± 0.272 | 1.160 | 1.850 | 1.00 |

Run from Kitty with --single-instance

I think this is cheating because it took a while for these to close after the test and AFAICT they weren't taking input.
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|:---|---:|---:|---:|---:|
| kitty --single-instance -e false | 401.0 ± 9.5 | 386.3 | 417.8 | 1.00 |

XFCE Term:

hyperfine --warmup 3 --export-markdown xftests.md 'xfce4-terminal -e false'
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|:---|---:|---:|---:|---:|
| xfce4-terminal -e false | 852.1 ± 59.4 | 723.3 | 902.2 | 1.00 |

Run from xfce4-term (A bit unfair, but legitimate unlike Kitty.)

| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|:---|---:|---:|---:|---:|
| xfce4-terminal -e false | 170.0 ± 7.9 | 159.0 | 185.4 | 1.00 |

[chrisduerr]

@Thovthe That seems like extremely high startup times, even for xfce4-terminal which isn't rendering purely in OpenGL.

Do you have an extremely slow drive (like SD card) or something like that?

I'd assume that the difference between xfce4-terminal and Alacritty is mostly the additional driver startup time which I don't think we can do much about. But on a "normal" system that should mean that the startup time is < 0.2-0.3s, so it seems like your setup is rather unfortunately slow.

[baudlord]

I found out in Arch Linux, with an Intel UHD 620 (KBL-R) laptop, Alacritty was incredibly slow (~2s startup time) when using any of the patched fontconfig packages available at AUR.

Returning to the official repository packages made a huge diffference, Alacritty now takes ~100ms to start up on my system.

Hope this is helpful to anyone that ends up here after googling. :)

[ddevault]

I just ran into this. The built-in fontconfig is really bad, why can't this just be a build error?

[kchibisov]

I just ran into this. The built-in fontconfig is really bad, why can't this just be a build error?

This is something decided by servo. I can ping them to provide an option to disable (but there're other crates that are doing so) that or just try to find all the libraries in alacritty and error out if something is missing/can't find. In theory the second approach could work just fine, since I also hate that sometimes it's building some fontconfig on its own.

@chrisduerr are you fine with second approach in general, since IIRC a lot of crates are using such 'fallback'?

[chrisduerr]

I don't quite understand what you mean by "try to find all the libraries in alacritty and error out if something is missing/can't find".

[kchibisov]

"try to find all the libraries in alacritty and error out if something is missing/can't find".

I mean that we can run pkg-config and try to find libraries we dynamically link to in build.rs script and error out on failure, knowing that some crates we're using will build their own copies of those libs.

[chrisduerr]

I find it strange to crash on a setup we should be able to support without any problems.

If anything, it seems like we should check if we could improve the fontconfig bindings to speed things up, instead of just outright crashing.

I get the idea of notifying the user about missing dependencies, but that's one of the reasons why packaging is important. There should be packages or at least build scripts available so the user doesn't have to worry about it.

[ddevault]

I was building a patched version of alacritty with @cyclopsian's gles3 work added, for what it's worth. Not all users are coming from packages, and it would be helpful to avoid saddling them with a broken build.

[kchibisov]

Yeah, but on the other side of things, if you won't have fontconfig at all you won't have a broken build actually, also it's stated a lot of times that fontconfig is required in INSTALL.md. However it's indeed possible to build without it.

Not all users are coming from packages

This file is purely for them https://github.com/alacritty/alacritty/blob/master/INSTALL.md , I guess they can take a look on it, but I guess most of the time you just build -> fix -> repeat, so you don't mind reading what you actually could need for your build.

I saw some reports from slow startup from builtin fontconfig, so I kind of agree on a point to throw or at least warn during build that some dependencies were not used from the system. But there's a huge chance that such warning will be missed by a user.

IIRC servo has this fallback to builtin fontconfig due to android primarily as well as some fallbacks for freetype-sys. I'd try to propose them to do what freetype-sys is doing honestly, since there's more pain from it than gain.

[ddevault]

30 second startup time to a crappy looking terminal doesn't seem like a working build to me, unless you believe this to be the kind of behavior which alacritty is judged by as a project.

[kchibisov]

30 second startup time to a crappy looking terminal doesn't seem like a working build to me, unless you believe this to be the kind of behavior which alacritty is judged by as a project.

Convince @chrisduerr , not me :- ) , I'm +1 (to error out) for that in general, since I hate debug such things from the users.

[kchibisov]

30 second startup time to a crappy looking terminal doesn't seem like a working build to me, unless you believe this to be the kind of behavior which alacritty is judged by as a project.

Note, @cyclopsian's gles3 and gles2 branches are blending text with the window behind it IIRC, so it depends from where it came to you tbqh.

[chrisduerr]

The thing is, if the startup time is 30 seconds, that is what should be fixed. Just aborting the build completely isn't really a fix for that, it just removes support for everyone who might actually try to rely on it.

[kchibisov]

Just aborting the build completely isn't really a fix for that, it just removes support for everyone who might actually try to rely on it.

If they really rely on that they could patch build script, I guess.

[cyclopsian]

My patch is not intended to be a real long-term solution, it's a hack for personal use. I've noticed that issue with the alpha blending but never bothered to fix it because it doesn't affect me. (I only really use a terminal emulator maximized against a solid color background)

[chrisduerr]

If they really rely on that they could patch build script, I guess.

At which point it's not supported anymore though. A bit like stating that Alacritty supports having videos as the background, you just need to write the code to patch it in. That's exactly what not supporting it is.

[kchibisov]

@ddevault your particular issue wrt system's Fontconfig was addressed here 0210a43196611c7c9216e923e3e11c6fe1b98876

[ddevault]

Nice.