Neovide: Unicode glyphs from the technical range won't render again

[SirJson]

It's kinda sad to see that issue #153 is back again after I fixed it.

I also know why following the Gitter chat. Basically the bug fix of another guy broke mine because there is no standard in Linux. Can we make the symbol fonts a config variable? A few terminals on Linux do that, and for good reason.

[yngman]

I also know why following the Gitter chat. Basically the bug fix of another guy broke mine because there is no standard in Linux.

May I ask for clarification? I can confirm the issue on Ubuntu 20.04, where my build is missing e.g. the U+2630 (☰) glyph, but have not participated in the chat mentioned; I am naïve of the history of this code base.

Is it possible to implement a strict font coverage sanity check as a feature with embed-fonts? I am not interested in continuing the build if the font lookup fails unexpectedly; I'd rather fix that. I'm rather surprised by this issue since font fallback has worked fantastically for me on Linux when coordinated by the system at runtime (but I get that statically is different).

As a final question; would it be infeasible to configure a default, cross-platform repository pull of an OFL font (e.g. DejaVu Sans Mono)? May the issue of the size bloat be skirted by a feature option and a git submodule (thus not requiring an init if unused)?

[SirJson]

In my original patch I tried to address Windows (where I had the problem first), Linux and OSX but could only get it to work for the first two.

For Windows it's easy because there is a defined Symbol Font available. But on Linux there is no such thing as a definition in user space most of the time. That's why I chose as default value GNU Unifont for three reasons, one it is tried and tested, two it became a GNU Project in 2013 thus is still maintained and three has one of the best coverage of all Unicode Symbols from the Technical Unicode Range. That's the range that people use for nice dashes and dots (Visual Studio Code would be an example)

The problem with Unifont was that we couldn't just embed this font because it would mean a licence change for Neovide and that's not my decision.

Status quo was: People on Linux please install Unifont it's so old you will have it in your package manager and everything will be fine.

That was fine until someone had rendering issues mixing Unifont with his font which led to this pull request which broke my rendering and yours maybe too.

The kicker is tho: Neovim-Qt can render everything just fine. It is not impossible it is just a ton of work and I think the most efficient solution for this project would be to do it just like Konsole is doing it. Having a default but letting the user override it if he got the better / correct glyphs.

I would implement it but at the moment for me it is unclear how the rendering will evolve which already held me back and made me more observe what will happen. It would be a shame to solve a problem for like 2 weeks and then it gets replaces by something completely different.

[jamespotz]

Maybe implementing something like $HOME/.config/neovide.toml or .neoviderc for overriding default configs.

[j4qfrost]

I believe @Kethku is currently working on something along these lines. He's just been a bit busy.

[Kethku]

The real issue here is a lack of os level font fallback. Neovim-qt can handle this correctly because it can use operating system apis to build a font fallback list on the fly based on the installed fonts. We don't have that option because cross platform font fallback libraries don't exist for rust yet.

I apologize for the churn @SirJson. It is very difficult for me to keep in mind all the different constraints while working on the project in spare my spare time. I beg some amount of leniency and understanding

[Kethku]

As for the render rewrite issue, I wouldn't worry too much. I have doubts that such a rewrite is going to happen. It's a huge effort, and I won't merge it until such an effort is well and truly baked. At the moment nobody has contributed an attempt outside somewhere basic proof of concept expirements I made a month or two ago.

I have to walk a fine line between encouraging contribution from a wide variety of people, and gating those contributions to make sure we make good decisions for the project. If I say no all the time, nobody will want to help out. So in general I encourage working on stuff.

[Kethku]

As mentioned by @j4qfrost (thanks so much again for helping manage these issues) I have merged a modification of guifont. You can now specify multiple fonts by separating the font name by commas. I am also going to add a separate neovide specific variable which allows the user to specify the fallback list without being incompatible with other guis

[jamespotz]

I also would like to contribute with this project, But I am still very new to rust.

[j4qfrost]

@jamespotz https://github.com/Kethku/neovide/issues/230 This is a good place to start. I think the best place is in the animation_utils.rs file, since those functions are fairly simple. Refer to the settings folder if you need help or ping me on gitter.

[jamespotz]

Thanks @j4qfrost

[SirJson]

@Kethku I know, this project grew extremley fast and you really try to support everyone which is something big companies strugle with.

But it's good to hear that there might not be the big change in rendering technology because I was thinking about getting this to run on something like a Raspberry Pi. That would open up the option of using OpenGL ES as Fallback with Angle on Windows which translates straight to DirectX. Basically how Skia is used by Google right now if you don't enable experiments.

And if that works I want to have a look if I can take advatage of the EGL backend in SDL2 because that would mean you could in theory boot straight to neovide. I have no idea if this is even a good idea, but something I always wanted to try 😀

[SirJson]

Ok back to the topic, I just saw this on a fresh build of the master branch on Windows

I think there is more to this than just: "Someone changed the default font on Linux"... (I'm using the same font here as I did in #153)

EDIT: I investigated the problem on Windows and can't reproduce it here on my personal workstation. I think this could be a local configuration problem on the system I made the screenshot from because it is owned and administered by the company I work for. That means I don't always have full control over the configuration. For example 30 minutes before took that screenshot I had to ask our administrator to unblock SDL so there might be more lock downs in place like on fonts.

After seen this I'm certain Windows is not affected by this issue.

[Kethku]

Just to add, font fallback is hard in rust not because theres a bunch of work to be done, but because the cross platform libraries we use don't have a solution. Raph Levien has done a really great writeup on the complexity here: https://raphlinus.github.io/rust/skribo/text/2019/04/04/font-fallback.html and made a follow up issue here: https://github.com/servo/font-kit/issues/37

Until these resolve, we are always going to have limited success because we don't have the big hammer that browsers, and qt based uis have. System font fallback is the correct solution for this problem, but we don't have access to it.

The short term solution is manual fallback specification. As mentioned above, we already have manual listing of font names in the guifont. So it can be solved today if you know what fonts to list. It is possible that font-kit also has some capabilities for font fallback, but more research is needed.

I have also highlighted the need for a separate fallback list variable so that we are actually compatible with the ad-hoc guifont spec, but I haven't actually had somebody complain about the incompatibility so its probably not urgent. That said, @jamespotz if you're looking for something to do, this task would be low hanging fruit like @j4qfrost said.

[SirJson]

Mh okay this idea comes from a different me that isn't programmer and has no clue what a qt is: (I have no better way to describe this but we have word for this 😂)

Can't we just draw our own technical glyph font? Like how hard can it be to draw dots and lines or use svgs that we are allowed to use for more complex figures. It's work yes but this would solve it on every platform. The question would be which unicode range is the most used by coders to paint dots and lines... The powerline glyphs should be no problem...

Now that I think about it the nerd font base font could be a good starting point. It's MIT licenced so we only have to extend the stuff that is missing and boring and hoping it isn't something we need an artist for.

EDIT: I mean this is really all I'm missing, dots and lines and those are not included in nerd font for some reason.

[Kethku]

I think thats the idea behind these https://github.com/Kethku/neovide/tree/master/assets/fonts @exoticus can elaborate on how he made them. Maybe they just need extended?

[SirJson]

I wonder why I never asked my self what those extra fonts are 😃 That's good to hear tho!

I had a look at my old bug report where I found the glyphs that failed to render and had a look with FontForge if those could be missing, and the only one missing is u+2b1d aka "BLACK VERY SMALL SQUARE". I use this glyph for spaces which explains why it is super visible to me.

I never created a font, creating a black very small square is easy but adding it to a font without breaking it? I help where I can but maybe the creator of that font can help here again? Otherwise I see what I can do.

[Kethku]

closing in favor of https://github.com/Kethku/neovide/issues/444