Nixpkgs: fonts: provide opentype files for X11 bitmap fonts

[rnhmjoj]

Since 23edc9cf14d3a7c427abfb69fe8b542ed5bf0365 bitmap fonts in PCF and BDF format stopped working in applications using pango as a backend (gtk apps for example).

Bitmap fonts packaged in nixpkgs should be updated to provide OTB (OpenType Bitmap) fonts. If unsupported upstream the xorg.fonttofsnt tool may be used to generate them.

Possibly affected packages:

• [x] clearlyU (#79638)
• [x] creep (#79641)
• [x] dina (#79664)
• [x] dina-pcf (#79664)
• [x] envypn-font (#79666)
• [x] gohufont (#75163)
• [x] ~powerline-fonts~ (already provides OTF and TTF)
• [x] profont (#79756)
• [ ] proggyfonts (no upstream support or good results with ff or fonttosfnt)
• [x] siji (#80220)
• [x] spleen (#79671)
• [x] tamsyn (#80228)
• [x] terminus_font (#80261)
• [x] tewi (#80270)
• [x] ucs-fonts (#80366)
• [x] unifont (#80322)
• [x] uni-vga (#80326)
• [x] ~unscii (already provides OTF and TTF)~
• [x] uw-ttyp0 (#79633)

Maintainer information:

# a list of nixpkgs attributes affected by the problem
attribute:
  - 'clearlyU'
  - 'creep'
  - 'dina'
  - 'dina-pcf'
  - 'envypn-font'
  - 'gohufont'
  - 'powerline-fonts'
  - 'profont'
  - 'proggyfonts'
  - 'siji'
  - 'spleen'
  - 'tamsyn'
  - 'terminus_font'
  - 'tewi'
  - 'ucs-fonts'
  - 'unifont'
  - 'uni-vga'
  - 'unscii'
  - 'uw-ttyp0'

# a list of nixos modules affected by the problem
module:

[lheckemann]

There's also the fonts in the xorg attrset such as fontmiscmisc.

[nixos-discourse]

This issue has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/what-are-your-goals-for-20-03/4773/13

[multun]

Should we keep the former PCF / BDF fonts in?

[rnhmjoj]

Should we keep the former PCF / BDF fonts in?

Definitely: most software still use PCF/BDF fonts, especially older one.

[multun]

@rnhmjoj given the following powerline-fonts package and this i3 font config: font pango:Terminess Powerline, 10, i3 fails to load properly the font when the pcf / bdf fonts are present. It succeeds, however, when the two rm -rf commands are uncommented. Any idea why?

[rnhmjoj]

@multun: I'm not sure. It could be pango is loading the pcf first and failing. Does that happen with other fonts too?

[mtreca]

@multun I think that the fonttofsnt package messes up foundry names when converting bdf/pcf fonts. You could try exactly matching the font name using fc-match but in my experience it is better to keep a single version of your font on your system.

[mtreca]

@rnhmjoj I tried the font myself with awesomewm and I have the same issue. I indeed think that Pango loads the pcf/bdf version first.

I tried a disabling type1 bitmap fonts and/or bitmaps as a whole in fontconfig using

  fonts.fontconfig = {
    allowBitmaps = false;
    allowType1 = false;
  };

but it seems that otb fonts are still considered as bitmap fonts by fontconfig, so this doesn't work.

[rnhmjoj]

Void decided to downgrade pango to 1.43. In facts the 1.44 doesn't really bring any new feature (the package is in "maintenance mode") but only breakages: besides bitmap also lots of vector fonts now have spacing issues. We may consider this.

[jtojnar]

Many programs (e.g. Nautilus) already depend on Pango 1.44.

[mtreca]

Another solution that is found on the AUR is to provide two different packages, for instance gohufont and gohufont-otb. It duplicates a fair amount of code but at least there is no clashing between bitmapped and non-bitmapped fonts.

[rnhmjoj]

@vxid: maybe we could use multiple outputs to avoid duplicating the packages.

[mtreca]

@rnhmjoj I did not know this was possible. That would be much cleaner indeed. I am not sure on how to do it though. I could change all the problematics don't packages if I had an blueprint on how to do so. I did the same thing with bitmap fonts on the AUR, by copying existing PKGBUILDs.

[rnhmjoj]

Before doing this on a scale, I'd like to solve the problem that fonttosfnt is not deterministic, or rather, it produces slightly different outputs on diferent architectures. Currently this breaks all fonts being fixed-output derivations.

[jtojnar]

Things generated by fonttosfnt probably should not be FODs anyway.

[rnhmjoj]

@jtojnar Why not? Fonts are just data: they shouldn't depend on the platform generating them and should be fixed outputs.

[rnhmjoj]

Good news: I sorted out the trouble with fonttosfnt in #75163.
I also reproduced the issue @multun and @vxid described and I can say splitting the outputs seems to work:

• Installing the otb output works (apparently but I haven't tested extensively) for both X11 (using Xft) and GTK applications.

• Installing the out output works for X11 only (and console, optionally).

• Installing both still confuses pango resulting into broken glyphs. I opened an issue at pango about this.

The bad news is that fixed output and multiple outputs are incompatible so we'll have to choose one. I don't know if there is any workaround.

Another option I tried is renaming the font family to something like nameOT to avoid the confusion. It works but can't be done with fonttosfnt (yet, I could try to patch it); if the fonts are provided as BDFs it's pretty trivial to change the name with, say, sed.

[mtreca]

@rnhmjoj That's very good news. If the otb output works as intended for both GTK and X11 applications, I think your last solution involving renaming old bdf files would be the best. I am ready to test-drive some fonts to make sure they all display correctly after renaming them.

[rnhmjoj]

By the way, @jtojnar made me realized fixed output for a package that isn't strictly unpacking some source is a cheat: because an update to any dependency could theoretically change the result but nix won't rebuild the package. So it's actually better to remove the fixed output constraint and split the outputs.

I think your last solution involving renaming old bdf files would be the best.

I now think the best approach is to split the outputs. Renaming can't be done in general, because many fonts are provided as pcf and they can't be modified so easily (it requires writing a fontforge script). Also renaming the font will require people to change their configuration files.

[rnhmjoj]

You can check out my PR for gohufont.

[jtojnar]

Should not have been auto-closed.

[rnhmjoj]

Oh, yes: thank you.

[mtreca]

@rnhmjoj Thanks a lot for your work. I will wait for your merge to hit nixpkgs-unstable to try it out.

[dtzWill]

What's the expect usage for these multiple-output fonts, by the way?

Is there a good way for NixOS (for example) to facilitate this, maybe options that pull in one format or another based on module options (which perhaps would be configured by NixOS modules, like if gnome is being used/etc)?

Hopefully it'll be fixed re:issues when multiple formats are available, so such configuration is safe...

[rnhmjoj]

What's the expect usage for these multiple-output fonts, by the way?

The idea is that $out output will continue to work the same way, so users that don't have or care about pango/harfbuzz based applications won't have to change their configuration.
On the other hand the $otb output will allow to use the fonts everywhere but requires to switch from XLFD to an Xft-based string in X11 applications.

Is there a good way for NixOS (for example) to facilitate this, maybe options that pull in one format or another based on module options (which perhaps would be configured by NixOS modules, like if gnome is being used/etc)?

There could be an option (maybe defaults to gnome3.enable) that does map (x: x.otb) fonts.fonts if the otb output is available. I'm not sure yet.

Also If the pango issue will ever be fixed the pcf,otb outputs could even be merged back together.

[rnhmjoj]

The next phase is what @lheckemann noticed: there are at least 37 more fonts in the xorg attrset that needs to be taken care of. Hopefully their structure is strict enough to create a hook or something to generate the otb automatically.

[mtreca]

I did a fair share of reading on this issue today.

According to the Fedora bugtracker, converting bitmap fonts automatically using fonttofsnt can introduce errors and spacing issues (I did witness some when converting fonts like misc fixed when I was on Arch). They however came up with a python script that seems to yield better results (even though it still calls fonttofsnt under the hood).

This issue also mentions that pango is never likely to be "fixed" since there is a common willingness to drop support for type 1 bitmap fonts in order to support a newer standard. I guess that means that type 1 bitmap support will eventually have to be dropped.

[rnhmjoj]

This issue also mentions that pango is never likely to be "fixed" since there is a common willingness to drop support for type 1 bitmap fonts in order to support a newer standard. I guess that means that type 1 bitmap support will eventually have to be dropped.

I know it's very unlikely that type 1 will ever be supported in harfbuzz: I meant the issue that Pango still loads the wrong fonts if both type 1 and open type are present.

They however came up with a python script that seems to yield better results (even though it still calls fonttofsnt under the hood).

For starters, If the fonts are not iso10646 fonttosfnt doesn't ever run. In that cases fontforge may work but sometimes it doesn't: for example proggyfonts. I even tried bdf2sfd (after @dtzWill packaged it) but the results are just terrible on proggyfonts.

I think some fonts will never be converted.

[wucke13]

I have the issue that awesome wm displays only garbage even though my bitmap font of choice has .otb files. Could this be related to the thing you describe here, @rnhmjoj :

I meant the issue that Pango still loads the wrong fonts if both type 1 and open type are present.

If yes, what is a workaround/fix for that? Breaking all old bitmap fonts is _bad_, breaking new ones though such a thing is __worse__.

[rnhmjoj]

Could this be related to the thing you describe here, @rnhmjoj :

You should check whether the package contains both pcf/bdf and otb fonts with the same font-family name. If it does then Pango will still load the type 1 fonts and produce garbage.

If yes, what is a workaround/fix for that? Breaking all old bitmap fonts is bad, breaking new ones though such a thing is worse.

The workaround is splitting the outputs of the package in out (for pcf, bdf...) and otb and only intall the otb output in your system.

[mtreca]

@wucke13 This is what I used ( I use awesome as well):

    fonts.fonts = with pkgs; [
      gohufont.otb
      uw-ttyp0.otb
    ];

[wucke13]

You should check whether the package contains both pcf/bdf and otb fonts with the same font-family name. If it does then Pango will still load the type 1 fonts and produce garbage.

It does. Is upstream pango recognizing this as a bug? If so, do you happen to have a link to the issue? The typical search engines are not to useful for anything pango & bitmap related...

Is the splitting of font packages a well recognized effort for upstreaming in NixOS? Other than that, wouldn't it be nice to write a hook for font packages, that automatically checks for the file formats of the font files and renames the font family names, appending something alike (OTB) or (BDF) and so on, according to the font file's ending?

@vxid Thank you! Also, that is a sweet choice for fonts :)

[rnhmjoj]

It does. Is upstream pango recognizing this as a bug? If so, do you happen to have a link to the issue? The typical search engines are not to useful for anything pango & bitmap related...

I opened an issue but it did not receive much attention. You can find it here: https://gitlab.gnome.org/GNOME/pango/issues/457

Is the splitting of font packages a well recognized effort for upstreaming in NixOS? Other than that, wouldn't it be nice to write a hook for font packages, that automatically checks for the file formats of the font files and renames the font family names, appending something alike (OTB) or (BDF) and so on, according to the font file's ending?

It's the best solution @jtojnar and I came up with and it's what other distro are doing: providing multiple packages like font, font-ttf, font-otb etc., instead we use multiple outputs .

About renaming the font families, yes, it would work but in general requires using fontforge, it tried keeping down its use because it's not deterministic, unlike fonttosfnt.

[wucke13]

When will all the fixes be ~merged to~ available in the 20.03 channel?

[rnhmjoj]

When will all the fixes be merged to available in the 20.03 channel?

I don't think it's a good idea to bring these changes to 20.03. It's not complete yet and I introduced a few breaking changes in the structure of the packages.

[wucke13]

Letting people wait another 6 months for their beloved fonts to work with pango doesn't sound like a good Idea as well.

[rnhmjoj]

You can install the unstable fonts easily:

$ nix-channel --add https://nixos.org/channels/nixos-unstable nixos-unstable

and in configuration.nix

{
    fonts.fonts = 
      let unstable = import <nixos-unstable> { };
      in [ unstable.uw-ttyp0.otb ];
}

[wucke13]

You can install the unstable fonts easily:

That is a good point.

I don't think it's a good idea to bring these changes to 20.03

I kindly disagree. As is well known, the whole pango issue already caused a lot of headache for many people, having it fixed in NixOS once and for all ASAP would be my personal preference.

[rnhmjoj]

I kindly disagree. As is well known, the whole pango issue already caused a lot of headache for many people, having it fixed in NixOS once and for all ASAP would be my personal preference.

Backporting these changes as is going to to very little: since the fonts are split in different outputs and OpenType fonts are not in the default one, GTK applications will still be broken by default.
Users must be made aware of the otb output and choose to use it: this is only possibile if the outputs are document in the manual or mentioned in the release changes.

Moverover, I was using the same bitmap font in GTK and X11 applications so I had to change my .Xresources to use the OpenType variant because the two can't be installed together.

This situation could change if Pango were to fix the font loading issue: I would just merge the outputs and backport the changes along with a pango patch.

[rnhmjoj]

I'm taking a look at the xorg fonts, now. The packages containing type 1 bitmap fonts apparently are:

  "fontadobe100dpi"
  "fontadobe75dpi"
  "fontadobeutopia100dpi"
  "fontadobeutopia75dpi"
  "fontadobeutopiatype1"
  "fontarabicmisc"
  "fontbh100dpi"
  "fontbh75dpi"
  "fontbhlucidatypewriter100dpi"
  "fontbhlucidatypewriter75dpi"
  "fontbhtype1"
  "fontbitstream100dpi"
  "fontbitstream75dpi"
  "fontbitstreamtype1"
  "fontcronyxcyrillic"
  "fontcursormisc"
  "fontdaewoomisc"
  "fontdecmisc"
  "fontibmtype1"
  "fontisasmisc"
  "fontjismisc"
  "fontmicromisc"
  "fontmisccyrillic"
  "fontmiscethiopic"
  "fontmiscmeltho"
  "fontmiscmisc"
  "fontmuttmisc"
  "fontschumachermisc"
  "fontscreencyrillic"
  "fontsonymisc"
  "fontsunmisc"
  "fontwinitzkicyrillic"
  "fontxfree86type1"

These are all collections of fonts and there are pcf, pfa and afm, which I'm not sure they are supported by fonttosfnt. Reading their fonts.dir they contain complexively 3439 fonts, ugh.

[rnhmjoj]

Having removed the vector fonts and run fonttosfnt we are left with:

    "fontadobeutopiatype1"
    "fontbhtype1"
    "fontbitstreamtype1"
    "fontibmtype1"
    "fontxfree86type1"

    "fontmicromisc"
    "fontsunmisc"
    "fontcursormisc"
    "fontdecmisc"

The first "type1" half simply can't be converted with this tool, the second half should but it's failing.
I'll have to try with fontforge again. Of course, all of these should be checked to see if the output is usable.

[rnhmjoj]

So, with #91777 and #95907 applications using pango should be able to load bitmap fonts without more user intervations.

The only problem left is the convertion of the xorg fonts but I give up on it: the current tools (fontforge, fonttosfnt) can't reliably handle most fonts and proportional bitmap fonts at all. I'll soon open a PR to convert the type1 vector fonts (xorg.font*type1) and post the expression I used to test, in case anyone else wants to try it.

[rnhmjoj]

Here's the expression: https://gist.github.com/rnhmjoj/bfa1af90c7f0f99b730eb8d4a7f6bd32

[mtreca]

@rnhmjoj Thanks a lot for your on bitmap fonts, I have been using the otb output you provided for the past six months without any issues.

[rnhmjoj]

Note that on unstable the otb output will disappear in the next channel update. So, if you're tracking unstable you're ok because pango will be updated at the same time with the fix needed; on the other hard, if you're on NixOS 20.03 you may have to pin the unstable channel, because the pango update won't be backported.

[schmittlauch]

So the only thing preventing this from getting closed seem to be proggyfonts because of its bad conversion result.

How do we want to proceed with this?

• keep this open until proggyfonts provides some OTFs upstream
• close the issue regardless of proggyfonts and maybe add a comment in its Nix expression
• close this but drop proggyfonts from nixpkgs

[rnhmjoj]

Well, yes but there are also hundreds of fonts in pkgs.xorg that I couldn't convert.
Removing the packages doesn't make sense because the fonts work perfectly fine in Freetype-based programs.
If the fonts are still actively supported I would create an upstream issue, in the meantime I think this can be kept open for visibility.