Libertinus: Unicode support overview

[Nescio0]

It would be nice if there was a readme table to keep track of Unicode coverage (cf. https://en.wikipedia.org/wiki/DejaVu_fonts#Unicode_coverage ), e.g.

U+0000 to U+007F Basic Latin                             : a / 128
U+0800 to U+00FF Latin-1                                 : b / 128
U+0100 to U+017F Latin Extended-A                        : c / 128
U+0180 to U+024F Latin Extended-B                        : d / 208
U+0250 to U+02AF IPA Extensions                          : e /  96
U+02B0 to U+02FF Spacing Modifier Letters                : f /  80
U+0300 to U+036F Combining Diacritical Marks             : g / 112
U+0370 to U+03FF Greek and Coptic                        : h / 135
U+0400 to U+04FF Cyrillic                                : i / 256
U+0500 to U+052F Cyrillic Supplement                     : j /  48
…
U+1AB0 to U+1AFF Combining Diacritical Marks Extended    : k /  15
…
U+1C80 to U+1C8F Cyrillic Extended-C                     : l /   9
…
U+1D00 to U+1D7F Phonetic Extensions                     : m / 128
U+1D80 to U+1DBF Phonetic Extensions Supplement          : n /  64
U+1DC0 to U+1DFF Combining Diacritical Marks Supplement  : o /  63
U+1E00 to U+1EFF Latin Extended Additional               : p / 256
U+1F00 to U+1FFF Greek Extended                          : q / 233
…
U+2070 to U+209F Superscripts and Subscripts             : r /  42
U+20A0 to U+20CF Currency Symbols                        : s /  32
U+20D0 to U+20FF Combining Diacritical Marks for Symbols : t /  33
…
U+2C60 to U+2C7F Latin Extended-C                        : u /  32
U+2C80 to U+2CFF Coptic                                  : v / 123
…
U+2DE0 to U+2DFF Cyrillic Extended-A                     : w /  32
…
U+A640 to U+A69F Cyrillic Extended-B                     : x /  96
…
U+A700 to U+A71F Modifier Tone Letters                   : y /  32
U+A720 to U+A7FF Latin Extended-D                        : z / 174
…
U+AB30 to U+AB6F Latin Extended-E                        : α /  56
…
U+FE20 to U+FE2F Combining Half Marks                    : β /  16
…

[khaledhosny]

There are several tools and websites that can give such reports, maintaining this manually gets often outdated. I used to run a tool at build time that generated Unicode charts-like PDFs, but it is too much a requirements for such a common task.

[alerque]

@khaledhosny Would you accept a PR that added a Makefile target to generate a plain text and/or PDF chart like this? It wouldn't necessarily need to be run as part of the release process to update the artifact (although it could), but could just be something the repository could handle generating if people happened to have the dependencies anyway and wanted such a thing. If so feel free to re-open and assign this issue to me.

@Nescio0 @waldyrious You may be interested in fontproof. You could feed it a font file from this project and get such a PDF chart back.

[khaledhosny]

If it can be done with no extra dependencies (or something that can be easily installed with pip), then I’m fine with it. I don’t like keeping out-of-sync stuff around, so it would need to be run each time a font is modified, or at release time if it takes too long.

I remember now one extra reason I stopped doing this, the coverage is inconsistent across fonts, so we need separate reports for each font file.

[alerque]

Okay please do assign this to me and I'll look into what can be done with those qualifications. I've needed something like this before and wished more fonts had something like it to review when consider them for a project.

[alerque]

@Nescio0 Where did you get that ASCI chart format? I'm looking into possible ways of displaying this and having a hard time coming up with something clear and readable that would cover multiple fonts well. Also if there is anything standardized enough that it would be useful for comparison with other projects it would be nice to match formats.

[alerque]

@khaledhosny A quick question: is the Makefile supposed to be BSD/other compatible or is assuming GNU Make okay? Are there any external tools I can count on being available besides the python font tools used and some basic shell stuff?`

[khaledhosny]

GNU Make, I never use any other make. FontTools will always be used, standard GNU utils are OK too, Python packages available on PyPI are fine as well (as long as they are Python 3-compatible).

[alerque]

Music to my ears, BSD-Make has the devil's red horns! :smiling_imp:

Another quickie: is pandoc in the cards or is that reaching for the stars?

So far I've surfaced two ways¹ to tackle this, using a tool already supplied with Python fonttools to dump a character map as an XML table (ttx -t cmap -o - <file>.otf) and parsing that into something useful using shell tools –or– adding a dependency on Python fontaine which has some spiffy output formats baked in. I ask about Pandoc because that would normally be my go-to tool to cleanup and normalize output formats, not because what I want to do can't be done with awk, sed, etc.

¹ _{A third, not quite as promising for this project but that I'm cross linking just in case others find it useful, is nototools which has a ./coverage.py script that also does roughly the same job.}

[khaledhosny]

Pandoc is fine, but it feels like an overkill for such a simple task (I’m not happy for using XeTeX to generate the sample either).

[Nescio0]

@Nescio0 Where did you get that ASCI chart format? I'm looking into possible ways of displaying this and having a hard time coming up with something clear and readable that would cover multiple fonts well. Also if there is anything standardized enough that it would be useful for comparison with other projects it would be nice to match formats.

Basically it's [Unicode block range] [Unicode block name] : [glyphs in font] / [assigned characters in Unicode block]. It was inspired by https://en.wikipedia.org/wiki/Template:Unicode_blocks
https://www.unicode.org/Public/12.1.0/charts/CodeCharts.pdf (108 MB) lists all Unicode glyphs and blocks in the newest standard.
An example of a more detailed yet simple format would be e.g. https://software.sil.org/gentium/support/character-set-support/
Even better would be https://dejavu-fonts.github.io/Samples.html

[iandoug]

Symbola version: https://dn-works.com/wp-content/uploads/2020/UFAS-Docs/Symbola.pdf

I recently switched to Libertinus from Symbola. No good at drawing glyphs, but made a comparison chart between various fonts in the same market, which may or may not be useful, as a way of giving something back:

https://www.keyboard-design.com/academic-font-coverage-comparison.html

Suggestions welcome of course.

Cheers, Ian