Asciidoctor-pdf: Many Unicode characters disappear

[DanielSWolf]

I have an AsciiDoc file in UTF-8 encoding. When I run Asciidoctor PDF, many non-ASCII characters just disappear. I'm not specifying a custom font.

Consider the following file, unicode-test.adoc.

= Unicode test

* Greek letters
** α U+03B1 GREEK SMALL LETTER ALPHA
** β U+03B2 GREEK SMALL LETTER BETA
** γ U+03B3 GREEK SMALL LETTER GAMMA

* Emoticons
** 😀 U+1F600 grinning face
** 😂 U+1F602 face with tears of joy
** 🤠 U+1F920 cowboy hat face

* Bent arrows
** ⤶ U+2936 ARROW POINTING DOWNWARDS THEN CURVING LEFTWARDS
** ↵ U+21B5 DOWNWARDS ARROW WITH CORNER LEFTWARDS
** ⏎ U+23CE RETURN SYMBOL
** ↲ U+21B2 DOWNWARDS ARROW WITH TIP LEFTWARDS
** ↩ U+21A9 LEFTWARDS ARROW WITH HOOK

* Half-circles
** ◐ U+25D0 CIRCLE WITH LEFT HALF BLACK
** ◑ U+25D1 CIRCLE WITH RIGHT HALF BLACK
** ◒ U+25D2 CIRCLE WITH LOWER HALF BLACK
** ◓ U+25D3 CIRCLE WITH UPPER HALF BLACK

When I run asciidoctor-pdf unicode-test.adoc, the resulting PDF is this: unicode-test.pdf

The Greek letters work, but all the other symbols are just omitted. There are no warnings.

System information:

Asciidoctor PDF 1.5.0.alpha.18 using Asciidoctor 1.5.6.1 [http://asciidoctor.org]\
Runtime Environment (ruby 2.3.1p112 (2016-04-26 revision 54768) [x86_64-darwin14]) (lc:UTF-8 fs:UTF-8 in:- ex:UTF-8)

[mojavelinux]

This is just part of dealing with fonts in PDF. No one font (that looks decent) has all the characters in Unicode. And for the ones that do, they are absolutely huge. So we ask you to bring your own font if you need more glyphs. This is explained here: https://github.com/asciidoctor/asciidoctor-pdf/blob/master/docs/theming-guide.adoc#fonts

[DanielSWolf]

Thank you for your explanation. I see that font handling in PDF files is complicated. Still, I would appreciate it if special characters could work out of the box.

By the way: According to the document you linked, there should have been a warning for each unsupported character. While not ideal, this might have helped me track the problem down. But there was no warning.

[mojavelinux]

I would appreciate it if special characters could work out of the box.

Special characters do work out of the box. Just not all of them. And yes, in a perfect world, there would be a font that looks good and has all the characters. But there just isn't.

You can specify a different fallback font that has a broader range of characters. One of those is VLGothic. It's not the most elegant font, but it has just about everything. I'll document below how you can use that.

there should have been a warning for each unsupported character.

Not when using a TrueType font. This is just a limitation of Prawn that is out of our control.

[mojavelinux]

Here's how you can use VLGothic as your fallback font so that you get more special characters.

1. Create a theme file named more-glyphs-theme.yml.

2. Populate it with the following:

extends: default
font:
  catalog:
    Noto Serif:
      normal: notoserif-regular-subset.ttf
      bold: notoserif-bold-subset.ttf
      italic: notoserif-italic-subset.ttf
      bold_italic: notoserif-bold_italic-subset.ttf
    M+ 1mn:
      normal: mplus1mn-regular-ascii-conums.ttf
      bold: mplus1mn-bold-ascii.ttf
      italic: mplus1mn-italic-ascii.ttf
      bold_italic: mplus1mn-bold_italic-ascii.ttf
    VLGothic:
      normal: /usr/share/fonts/vlgothic/VL-Gothic-Regular.ttf
      bold: /usr/share/fonts/vlgothic/VL-Gothic-Regular.ttf
      italic: /usr/share/fonts/vlgothic/VL-Gothic-Regular.ttf
      bold_italic: /usr/share/fonts/vlgothic/VL-Gothic-Regular.ttf
  fallbacks:
  - VLGothic

1. Load the theme when invoking the Asciidoctor PF command:

asciidoctor-pdf -a pdf-style=./more-glyphs-theme.yml unicode-test.adoc

[DanielSWolf]

At our company, we're increasingly using AsciiDoc for technical specifications of all kinds. Typically, such an AsciiDoc document won't be part of an elaborate build system, but is just a standalone .adoc file. So I'm limited to the features AsciiDoctor offers out of the box, without any setup or configuration outside the .adoc file itself.

Currently, I'm trying to limit myself to ANSI characters, but often, some special characters would improve readability a lot.

[mojavelinux]

If you're creating PDF and are limited to using the themes provided out of the box (and not willing to create a custom theme and use custom fonts), then yes, you'd be limited in which characters you can choose. However, it's a much broader range of characters supported than just ASCII (not ANSI, btw). The default theme with the fallback font (default-with-fallback-font) supports just about every major language except for CJK languages (for which there is a separate gem) and emoji.

So what characters are you missing? If we can identify characters that are important, I'm happy to include them in the fallback font. Sadly, that won't include emoji (though we could consider bundling an emoji font as a fallback).

Keep in mind, Asciidoctor PDF does support Font Awesome, so that's a good alternative to emoji. For example: icon:smile-wink[].

[mojavelinux]

When I run asciidoctor-pdf unicode-test.adoc

You need to use:

asciidoctor-pdf -a pdf-theme=default-with-fallback-font unicode-test.adoc

Btw, the test suite now includes a Unicode test:

• https://github.com/asciidoctor/asciidoctor-pdf/blob/master/spec/reference/font-i18n-default-with-fallback.pdf
• https://github.com/asciidoctor/asciidoctor-pdf/blob/master/spec/fixtures/i18n-font-test.adoc

[mojavelinux]

but all the other symbols are just omitted. There are no warnings.

Unfortunately, Prawn doesn't issue a warning for missing glyphs in a TTF font like it does for an AFM font. However, we could monkeypatch the Prawn::Text::Formatted::Box#find_font_for_this_glyph to warn when it fails to resolve the glyph in any TTF font. However, this would only work when the fallback font is in use since that's the only time Prawn checks for the glyph to be present in the font. (There's one other way to do it that would work without the fallback font in use, but it's less elegant).

[mojavelinux]

To be consistent with the other built-in fonts, we're going to use the Noto Emoji (colorless) font. It will be enabed when the default-with-fallback-font theme is in use. We can only support emoji which the font provides, though it does support most of the well-known emoji.

If you do want to use your own emoji font, it's much simpler now:

extends: default
font:
  catalog:
    merge: true
    Symbola: /path/to/symbola.ttf
  fallbacks: [ Symbola ]

As far as I can tell, Prawn doesn't support color fonts, so you won't get any color in your emoji (unless you change the font color).

[mojavelinux]

I've added the emoji font as the fallback font. If it's missing emoji chars you'd like to see added, and they exist in either M+ 1p or NotoEmoji font, please request those characters in a separate issue. In the interest of gem size, I'd like to only include characters we absolutely need. This shouldn't be an academic exercise in Unicode support.