Nixpkgs: fontconfig: Enable color emoji as a fallback

[wmertens]

Describe the bug
Without this config, many emoji are rendered as black-and-white. This makes me sad.

Enter https://github.com/stove-panini/fontconfig-emoji - see the readme for a full breakdown.

To make it work best, the DejaVu font needs to be disabled, because it has B&W emoji embedded. Vera doesn't, and it is an ancestor of DejaVu, so that could be used as a replacement for the default system font.

So this issue is about if this config should be included by default and if DejaVu should be replaced as a default font.

IMHO, NixOS should default to configurations that most people would pick, and I'd argue that the most common way to see emoji nowadays is in full color.

@jtojnar @matthewbauer

Maintainer information:

module: fonts

[worldofpeace]

We do install a color emoji default, are you not using fonts.fontconfig.enable and fonts.enableDefaultFonts?

[jtojnar]

What emoji font and what app are you using? At least fonts with embedded PNG (noto-fonts-emoji, twitter-color-emoji and joypixels (non-free)) should work with Pango (GTK) based apps:

Firefox supports SVG in OT type fonts (twemoji-color-font, emojione (unmaintained))).

More about font formats: https://www.colorfonts.wtf/#section4

[wmertens]

All I did was use gnome3 as my desktop environment, and I had to change the fontconfig before Google Chrome would use color emoji. Does the link above render color emoji for you?

[matthewbauer]

Is that link actually using color emoji? This link works for me in chromium & firefox: https://guoyunhe.me/demo/noto-color-emoji/index.html (fonts.enableDefaultFonts = true; fonts.enableFontDir = true;).

[wmertens]

Yes the link i gave renders color emoji for me on Chrome and on my android phone. If you have B&W emoji, you now know which problem I'd like to fix ;)

Your example link specifically uses an emoji font, which is not the case most of the web.

[emilazy]

I agree that the defaults could use improvement here. My fonts configuration is very bespoke, but these excerpts might be relevant to getting colour emoji used more often:

          <!-- fonts of last resort -->
          <match>
            <edit mode="append" name="family"><string>Twitter Color Emoji</string></edit>
            <edit mode="append" name="family"><string>Noto Sans</string></edit>
            <edit mode="append" name="family"><string>Source Han Sans</string></edit>
            <!-- TODO: why does http://www.fileformat.info/info/unicode/block/sutton_signwriting/utf8test.htm use Filling? -->
            <edit mode="append" name="family"><string>SignWriting 2010</string></edit>
          </match>

          <!-- ... -->

          <!-- multilingual defaults -->

          <alias binding="same">
            <family>sans-serif</family>
            <prefer>
              <family>Noto Sans</family>
              <family>Twitter Color Emoji</family>
              <family>Source Han Sans</family>
            </prefer>
          </alias>

          <alias binding="same">
            <family>serif</family>
            <prefer>
              <family>Noto Serif</family>
              <family>Twitter Color Emoji</family>
              <family>Source Han Serif</family>
            </prefer>
          </alias>

          <alias binding="same">
            <family>monospace</family>
            <prefer>
              <family>Cascadia Code</family>
              <family>Twitter Color Emoji</family>
              <family>Source Han Mono</family>
              <family>Noto Sans Mono</family>
            </prefer>
          </alias>

(I empty defaultFonts.* and set defaults manually to have language-specific default fonts, but this should transfer over to the standard options reasonably.)

[benley]

Using config snippets from https://github.com/stove-panini/fontconfig-emoji as @wmertens suggested seems to produce excellent results for me. I've put the following in fonts.fontconfig.localConf:

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
  <!-- Priority:
       1. The generic family OR specific family
       2. The emoji font family (defined in 60-generic.conf)
       3. All the rest
  -->
  <alias binding="weak">
    <family>monospace</family>
    <prefer>
      <family>emoji</family>
    </prefer>
  </alias>
  <alias binding="weak">
    <family>sans-serif</family>
    <prefer>
      <family>emoji</family>
    </prefer>
  </alias>

  <alias binding="weak">
    <family>serif</family>
    <prefer>
      <family>emoji</family>
    </prefer>
  </alias>

  <selectfont>
    <rejectfont>
      <!-- Reject DejaVu fonts, they interfere with color emoji. -->
      <pattern>
        <patelt name="family">
          <string>DejaVu Sans</string>
        </patelt>
      </pattern>
      <pattern>
        <patelt name="family">
          <string>DejaVu Serif</string>
        </patelt>
      </pattern>
      <pattern>
        <patelt name="family">
          <string>DejaVu Sans Mono</string>
        </patelt>
      </pattern>

      <!-- Also reject EmojiOne Mozilla and Twemoji Mozilla; I want Noto Color Emoji -->
      <pattern>
        <patelt name="family">
          <string>EmojiOne Mozilla</string>
        </patelt>
      </pattern>
      <pattern>
        <patelt name="family">
          <string>Twemoji Mozilla</string>
        </patelt>
      </pattern>
    </rejectfont>
  </selectfont>
</fontconfig>

...and the result is that glyphs like ⌛️ and 🤔 are rendered correctly using my desired emoji font, and glyphs that should _not_ be covered by emoji (numbers, regular text arrows, etc) are not. Seems good so far.

[cole-h]

@benley That is fantastic. Works a treat for me. I monologued my testing on #nixos-chat -- the short of it being that placing the above snippet in ~/.config/fontconfig/fonts.conf does not work (or rather, starts replacing numbers with their emoji counterparts), while the suggested fonts.fontconfig.localConf NixOS option works a treat. After the recent fontconfig 2.13 bump, I now have color emoji again, with no downsides (or at least, none that I've found in the past 10 minutes)!

[hpfr]

As of recently my emoji fontconfig broke. Every space character was replaced with large gaps, numbers were replaced with emoji numbers, and alacritty went a bit insane (left window launched pre-rebuild):

I use 69-emoji.conf from the fontconfig-emoji repo alongside some logic from another emoji override file I found at some point. I'm guessing some of it is redundant, but fontconfig is hard to parse. Anyway, it doesn't work anymore. I had to disable the weakly-bound aliases for serif, sans-serif, and monospace to prefer emoji, and I also had to disable some match append rules.

Commenting out those rules fixed the issue, but I'm left with numbers replaced occasionally with emoji as @cole-h mentions.

Using the system-wide option instead of home-manager works as you describe @cole-h, but it's a bit strange, no? Why should this have to be system-wide? In any case, big thanks for following up with the solution. Fontconfig is a strange beast.

[cole-h]

Fontconfig configurations broke recently due to fontconfig being bumped from 2.12 to 2.13. Can't tell you exactly why, but I know that's the root cause.

I'd imagine this cannot be used in h-m config so simply because the system-wide option is affected by the rest of /etc/fonts/conf.d -- the local.conf that the aforementioned NixOS option creates is loaded by 51-local.conf. I don't know exactly how fontconfig uses conf.d and the file prefixes in there, but I imagine it's an ordering of sorts. Maybe if you were to copy /etc/fonts/conf.d to ~/.config/fontconfig/conf.d, as well as both fonts.conf and local.conf, the result will be the same (properly rendered emoji)? I don't have the time to test this (and I'm fine with configuring this system-wide) so it's up to any interested parties to go from there.

[hpfr]

I determined that only the rules rejecting fallback fonts like DejaVu were necessary in the system-level local.conf option. The other rules can be at the user level.

I've learned enough about the config format, but I'm having a new issue. Using benley's config I get pretty much perfect emoji support, except in rofi and dunst, which render emoji extremely small, for some reason. Maybe they share some rendering method incompatible with the latest fontconfig, but fontconfig doesn't appear to have changed on nixos-unstable recently, so I'm not sure what happened. If someone else could test benley's config with rofi I'd appreciate it.

The two yellow dots at the end are emoji.

[andresilva]

@hpfr I am also using benley's config, I had been using it globally from the beginning and everything was working great. It seems to have broken after https://github.com/NixOS/nixpkgs/pull/93562 and now I get the same issue you have in rofi (it was working fine previously). It works fine everywhere else (terminal, editor, browser), I have spent some time trying to debug it but I have no clue what the issue is.

[isaacwhanson]

I seem to be having the same issues with broken fonts and it seems /etc/fonts/local.conf is no longer being linked, and my localConf no longer seems to be used by fontconfig. oh also seems 51-local.conf reads (note the include path)

<fontconfig>
  <description>Load local customization file</description>
    <!-- Load local system customization file -->
    <include ignore_missing="yes">/etc/fonts/2.11//etc/fonts/2.11/local.conf</include>
</fontconfig>

[sternenseemann]

Can confirm that the issue @hpfr describes persists on master. I can reproduce it with bemenu and sway. My previously working config is:

{
    fonts = {
     # …

      enableFontDir = true;
      enableGhostscriptFonts = true;
      enableDefaultFonts = true;

      fontconfig = {
        enable = true;
        antialias = true;
        hinting.enable = true;
        defaultFonts = {
          monospace = [ "DejaVu Sans Mono" "Noto Mono" ];
          serif = [ "Vollkorn" "Noto Serif" "Times New Roman" ];
          sansSerif = [ "Open Sans" "Noto Sans" ];
          emoji = [ "Noto Color Emoji" "Twitter Color Emoji" "JoyPixels" "Unifont" "Unifont Upper" ];
        };
      };
    };
}

I never had a local.conf which suggests there is another issue.

[ciil]

@sternenseemann can you confirm that ~/.config/fontconfig/fonts.conf does not exist for you either? I honestly have no idea how that's generated, but deleting it just fixed my issue on current master (f14522334077f0dcc98bb9c80353f3cb1b6f1e48).

[sternenseemann]

@ciil Issue persists for me, I've rebuild with the given commit and deleted ~/.config/fontconfig

[jtojnar]

Quick script for copying the configuration to a directory for easy tweaking:

FONTCONFIG_PATH=$PWD/etc-fonts
cp --dereference -r /etc/fonts/2.11/ "$FONTCONFIG_PATH"
chmod -R +w "$FONTCONFIG_PATH"
sed -i "s#>/etc/fonts/2.11#>$FONTCONFIG_PATH#g" $(rg -l /etc/fonts "$FONTCONFIG_PATH")

Then you can run env FONTCONFIG_FILE=$PWD/etc-fonts/fonts.conf FC_DEBUG=1024 pango-view --text="Příliš 😂" --font='"Noto Color Emoji" 20' and see how much scaled the emoji is.

[jtojnar]

Commenting out https://gitlab.freedesktop.org/fontconfig/fontconfig/-/blob/e735abcfe139de91fc2e2d323cb4bf16345e1419/conf.d/10-scale-bitmap-fonts.conf#L66-74 seems to fix the scaling issue but the line has been there for ages.

Changing the file to

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
<match target="font">
  <test name="outline" compare="eq">
    <bool>false</bool>
  </test>
  <edit name="test-pattern-ps" mode="assign">
      <name target="pattern">pixelsize</name>
  </edit>
  <edit name="test-font-ps" mode="assign">
      <name target="font"   >pixelsize</name>
  </edit>
</match>
</fontconfig>

I got the following metrics:

$ env FONTCONFIG_FILE=$PWD/etc-fonts/fonts.conf fc-match -v 'Noto Color Emoji' | rg test
    test-pattern-ps: 12.5(f)(w)
    test-font-ps: 109(f)(w)

Of course, while I was building fc 2.12.6, I realized the scaling code is not idempotent and is being run twice thanks to

https://github.com/NixOS/nixpkgs/blob/742c7340d2b998065c5e03355e657ee338057658/pkgs/development/libraries/fontconfig/make-fonts-conf.xsl#L35

We will need to revert edf2541f02c6b24ea791710d5cadeae36f9b1a3a. https://github.com/NixOS/nixpkgs/pull/95358

[andresilva]

The issue with small emojis is fixed for me on current nixos-unstable (a31736120c5de6e632f5a0ba1ed34e53fc1c1b00).

[sternenseemann]

Persists for me.

[cole-h]

Fixed for me (with benley's config from here: https://github.com/NixOS/nixpkgs/issues/86601#issuecomment-670158155).

[sternenseemann]

Fixed also for me (without a local.conf), turns out I had a typo in my NIX_PATH override.

@jtojnar seems like the versioned config directory was indeed the issue.

[petabyteboy]

The config posted by @benley worked for me as well, but it was important to set the defaultFonts and omit the rejectfont blocks instead, because otherwise it would not allow those fonts to be used at all (which would make it unnecessary to put them in the fonts list in the first place).
My config ended up looking like this:

  fonts = {
    fontconfig = {
      localConf   = lib.fileContents ./fontconfig.xml;
      defaultFonts = {
        emoji = ["Noto Color Emoji"];
        serif = ["Bitstream Vera Serif"];
        sansSerif = ["Bitstream Vera Sans"];
        monospace = ["Bitstream Vera Sans Mono"];
      };
    };
    fonts = with pkgs; [
      noto-fonts-emoji
      ttf_bitstream_vera
      font-awesome_4
      unifont
    ];
  };

With fontconfig.xml containing:

<?xml version="1.0"?>
<!DOCTYPE fontconfig SYSTEM "fonts.dtd">
<fontconfig>
  <alias binding="weak">
    <family>monospace</family>
    <prefer>
      <family>emoji</family>
    </prefer>
  </alias>
  <alias binding="weak">
    <family>sans-serif</family>
    <prefer>
      <family>emoji</family>
    </prefer>
  </alias>
  <alias binding="weak">
    <family>serif</family>
    <prefer>
      <family>emoji</family>
    </prefer>
  </alias>
</fontconfig>

I think this fontconfig file should be included by default in NixOS, or are there any downsides I haven't discovered yet?

[jtojnar]

According to https://gitlab.freedesktop.org/fontconfig/fontconfig/-/issues/136#note_82092 it should be up to applications.

I asked about the downsides https://gitlab.freedesktop.org/fontconfig/fontconfig/-/issues/111#note_630720.

[colemickens]

(try number two on this reply)

I have tried to follow your example @petabyteboy, but it's not working for me? https://i.imgur.com/PXtWYw1.png. Most of the "numbers" on my desktop now either just don't render (or are too tiny to see), or are rendering as emoji. (This is basically the same result I'd gotten with @benley's config as well.)

I assume it's because I have a different mix of fonts installed (or some other change somewhere I'm not accounting for).

FWIW, I tried to be careful to stop sway, clear ~/.cache and ~/.config/fontconfig*, then re-activate, then start sway.

[jtojnar]

That is precisely what the config is supposed to do – prioritize an emoji font over the font that would be selected normally.

You will either need to find an emoji font that does not include those characters, remove them manually using fontforge (someone must have made a script for that already), or leave it up for applications to choose emoji. The last option should already work by default in most applications and this config will not help applications that do not support emoji.

Really, the only reason to use the config above would be if you want to render old Unicode characters like ⏳ or ⚠ that were retrospectively made emoji to render as emoji.

Edit: Looks like twitter-color-emoji might not contain numbers so it might be safer.

[colemickens]

Thank you so much for the explanation @jtojnar; that helps a lot.

[hpfr]

As others have mentioned, jtojnar's regression fix resolved my issue and a system-wide config is no longer necessary. I use the same weak binding config benley listed above in my user fontconfig. @colemickens I don't have your issue.

That is precisely what the config is supposed to do – prioritize an emoji font over the font that would be selected normally.

@jtojnar is this the case with binding="weak"? I thought it would come after the font that would be selected normally. For me, numbers render normally.

The config posted by @benley worked for me as well, but it was important to set the defaultFonts and omit the rejectfont blocks instead, because otherwise it would not allow those fonts to be used at all (which would make it unnecessary to put them in the fonts list in the first place).

@petabyteboy note that the Twemoji Mozilla reject block is necessary if you want to avoid displaying that font in Firefox when sites like https://emojipedia.org specifically request it before emoji, since it is included with the browser. Furthermore, even when fonts.enableDefaultFonts is false, DejaVu Sans is still installed, hence the second reject block. Some people don't want these fonts to be used at all, since if sites specify them explicitly, they'll override system emoji fonts.

[jtojnar]

That is precisely what the config is supposed to do – prioritize an emoji font over the font that would be selected normally.

@jtojnar is this the case with binding="weak"? I thought it would come after the font that would be selected normally. For me, numbers render normally.

Hmm, that might be the case. I would just expect the weak binding to be ignored if there is a strong one that matches. But I think that it depends on how the app using fontconfig is implemented.

[hpfr]

Yeah, I looked at the docs and it looks like binding has something to do with language. I removed that and it still seemed to work, so I don't even know if that affects anything.