asciidoctor-pdf 🚀 - Japanese doesn't show

By default, Asciidoctor PDF only bundles two themes. One theme references a subsetted font (NotoSerif) which only supports Western languages (English and most European languages). It also comes with a theme that references a subsetted font (M+ 1p) that includes a broad set of CJK characters, but not all. So I would start with trying the later theme:

asciidoctor-pdf -a pdf-theme=default-with-fallback-font document.adoc

If that doesn't work, what I recommend is making your own theme that references your own font. It's not difficult to do that anymore. I don't think the asciidoctor-pdf-cjk-kai_gen_gothic is really necessary. See:

mojavelinux on 7 Jan 2020

👍1

Thank you very much for the reply.
I will try your advice and tell you the result. :)

dusicong on 7 Jan 2020

Making my own theme seems to take a lot of time.
There is a time-saving solution.
downgrade to version 1.5.0.alpha.18, then problem solved.

Still don't understand why the new version can not work with Japanese (or CJK).

dusicong on 8 Jan 2020

Still don't understand why the new version can not work with Japanese (or CJK).

This is not a true statement. The current version works the same way with Unicode as the previous version did. Likely there is an incompatibility in the configuration.

Please provide me with a sample document (in text format), an example which gems you are using, and a copy of your theme so that I can see what you might be doing wrong.

mojavelinux on 8 Jan 2020

I'd say it looks pretty great with the default-with-fallback-font theme.

Here's sample input:

Anypoint Security は、多層的アプローチでアプリケーションネットワークを保護します。これらのレイヤは連携し、API へのアクセスを制御したり、ポリシーを適用したり、すべてのインバウンドトラフィックまたはアウトバウンドトラフィックをプロキシしたりして、アプリケーションネットワークとネットワークの個々のノードの両方を保護します。これにより、外部の脅威や攻撃が軽減されます。Anypoint Security には、ネットワーク実装に高い負荷をかけずに攻撃を検出してトラフィックを検証する専用のエンドポイントが用意されています。

== ポリシー

Anypoint Security の特徴として、Runtime Fabric にデプロイされているすべてのノードにポリシーを適用する、パフォーマンス、信頼性、拡張性の高いサービスが挙げられます。その後、Anypoint Security ポリシーは、すべてのトラフィックが通過するデフォルトのファイアウォール/ルータ機能として動作します。

== シークレットマネージャ

Anypoint Security には、デプロイメントで使用される TLS 証明書とキーストアを保存するセキュアコンテナが用意されています。シークレットマネージャを使用すれば、Runtime Fabric へのインバウンドトラフィックを暗号化する TLS コンテキストをセットアップするために必要なシークレットを保存できます。

== トークナイゼーションサービス

Anypoint Security のトークナイゼーションサービスでは、クレジットカード番号などのキー値をトークンに置き換えて機密データの不必要な漏洩を保護します。

Here's how you'd run it:

asciidoctor-pdf -a scripts=cjk -a pdf-theme=default-with-fallback-font jp.adoc

The -a script=cjk is important because it tells the converter to put semantic breaks in lines to accommodate mixing of English and Japanese.

Are you seeing different results?

mojavelinux on 8 Jan 2020

I also used the KaiGenGothicJP theme from the asciidoctor-pdf-cjk-kai_gen_gothic gem, and it worked decently:

asciidoctor-pdf -r asciidoctor-pdf-cjk-kai_gen_gothic -a pdf-theme=KaiGenGothicJP jp.adoc

Note the absence of the -a scripts=cjk in that case. That's because that theme already adds the semantic breaks.

mojavelinux on 8 Jan 2020

If you want to install the asciidoctor-pdf-cjk-kai_gen_gothic gem to get the fonts, but then use your own theme. Here's how you'd do it:

Install the gem:

$ gem install asciidoctor-pdf-cjk-kai_gen_gothic
Get the fonts:

$ asciidoctor-pdf-cjk-kai_gen_gothic-install

Create a theme that extends default:

extends: default
font:
 catalog:
   KaiGen Gothic JP:
     normal: KaiGenGothicJP-Regular.ttf
     bold: KaiGenGothicJP-Bold.ttf
     italic: KaiGenGothicJP-Regular-Italic.ttf
     bold_italic: KaiGenGothicJP-Bold-Italic.ttf
   M+ 1mn:
     normal: GEM_FONTS_DIR/mplus1mn-regular-subset.ttf
     bold: GEM_FONTS_DIR/mplus1mn-bold-subset.ttf
     italic: GEM_FONTS_DIR/mplus1mn-italic-subset.ttf
     bold_italic: GEM_FONTS_DIR/mplus1mn-bold_italic-subset.ttf
 fallbacks:
 - KaiGen Gothic JP
base:
 font-family: KaiGen Gothic JP
heading:
 font-family: $base-font-family
abstract:
 title:
   font-family: $heading-font-family
sidebar:
 title:
   font-family: $heading-font-family

Load your theme when running Asciidoctor PDF:

$ asciidoctor-pdf -a scripts=cjk -a pdf-theme=./jp-theme.yml -a pdf-fontsdir=$(dirname $(gem which asciidoctor-pdf-cjk-kai_gen_gothic))/../data/fonts jp.adoc

or

$ asciidoctor-pdf -a scripts=cjk -a pdf-theme=./jp-theme.yml -a pdf-fontsdir=$(ruby -r asciidoctor-pdf-cjk-kai_gen_gothic -e "puts File.expand_path '../fonts', (Gem.datadir 'asciidoctor-pdf-cjk-kai_gen_gothic')") jp.adoc

I'd be happy to add this info the theming guide.

mojavelinux on 8 Jan 2020

Thank you very much for your concern.
Well, I really want to know what is going on with my situation.
Since I changed my versions some times to solve the problem, I would like to tell you the situation from beginning.

First, I tried the command as you recommend:
asciidoctor-pdf -a pdf-theme=default-with-fallback-font document.adoc
But that does not work. PDF is produced, but Japanese doesn't show as usual.
At that time, ruby2.7 with asciidoctor-1.5.0.rc.1

Then I set another computer with ruby2.5 with asciidoctor-1.5.0.alpha.18, and that works.

I suppose it is a version problem.
To check it, I set a 3rd environment as ruby2.7 and asciidoctor-1.5.0.alpha.18. I get Japanese too, but with little problem. There is a weird space between number and Japanese character 「の」.

dusicong on 8 Jan 2020

I'm confident that if you install asciidoctor-pdf cleanly on Ruby 2.7, the default-with-fallback-font theme will work. Keep in mind that the document does have to be encoded in UTF-8.

You can fix the large spaces using -a scripts=cjk.

mojavelinux on 8 Jan 2020

I will try again later.
Thanks !

dusicong on 8 Jan 2020

I've added all this information to the README. https://github.com/asciidoctor/asciidoctor-pdf#support-for-non-latin-languages

mojavelinux on 8 Jan 2020

I tried to install the asciidoctor-pdf-cjk-kai_gen_gothic gem to get the fonts.
When I did step 2, there comes an error:

[1/20] Downloading KaiGenGothicCN-Bold-Italic.ttf
Traceback (most recent call last):
        25: from C:/Ruby27-x64/bin/asciidoctor-pdf-cjk-kai_gen_gothic-install:23:in `<main>'
        24: from C:/Ruby27-x64/bin/asciidoctor-pdf-cjk-kai_gen_gothic-install:23:in `load'
        23: from C:/Ruby27-x64/lib/ruby/gems/2.7.0/gems/asciidoctor-pdf-cjk-kai_gen_gothic-0.1.1/exe/asciidoctor-pdf-cjk-kai_gen_gothic-install:30:in `<top (required)>'
        22: from C:/Ruby27-x64/lib/ruby/gems/2.7.0/gems/asciidoctor-pdf-cjk-kai_gen_gothic-0.1.1/exe/asciidoctor-pdf-cjk-kai_gen_gothic-install:30:in `chdir'
        21: from C:/Ruby27-x64/lib/ruby/gems/2.7.0/gems/asciidoctor-pdf-cjk-kai_gen_gothic-0.1.1/exe/asciidoctor-pdf-cjk-kai_gen_gothic-install:31:in `block in <top (required)>'
        20: from C:/Ruby27-x64/lib/ruby/gems/2.7.0/gems/asciidoctor-pdf-cjk-kai_gen_gothic-0.1.1/exe/asciidoctor-pdf-cjk-kai_gen_gothic-install:31:in `each_with_index'
        19: from C:/Ruby27-x64/lib/ruby/gems/2.7.0/gems/asciidoctor-pdf-cjk-kai_gen_gothic-0.1.1/exe/asciidoctor-pdf-cjk-kai_gen_gothic-install:31:in `each'
        18: from C:/Ruby27-x64/lib/ruby/gems/2.7.0/gems/asciidoctor-pdf-cjk-kai_gen_gothic-0.1.1/exe/asciidoctor-pdf-cjk-kai_gen_gothic-install:33:in `block (2 levels) in <top (required)>'
        17: from C:/Ruby27-x64/lib/ruby/gems/2.7.0/gems/asciidoctor-pdf-cjk-kai_gen_gothic-0.1.1/exe/asciidoctor-pdf-cjk-kai_gen_gothic-install:33:in `open'
        16: from C:/Ruby27-x64/lib/ruby/gems/2.7.0/gems/asciidoctor-pdf-cjk-kai_gen_gothic-0.1.1/exe/asciidoctor-pdf-cjk-kai_gen_gothic-install:34:in `block (3 levels) in <top (required)>'
        15: from C:/Ruby27-x64/lib/ruby/2.7.0/open-uri.rb:50:in `open'
        14: from C:/Ruby27-x64/lib/ruby/2.7.0/open-uri.rb:744:in `open'
        13: from C:/Ruby27-x64/lib/ruby/2.7.0/open-uri.rb:174:in `open_uri'
        12: from C:/Ruby27-x64/lib/ruby/2.7.0/open-uri.rb:233:in `open_loop'
        11: from C:/Ruby27-x64/lib/ruby/2.7.0/open-uri.rb:233:in `catch'
        10: from C:/Ruby27-x64/lib/ruby/2.7.0/open-uri.rb:235:in `block in open_loop'
         9: from C:/Ruby27-x64/lib/ruby/2.7.0/open-uri.rb:764:in `buffer_open'
         8: from C:/Ruby27-x64/lib/ruby/2.7.0/open-uri.rb:346:in `open_http'
         7: from C:/Ruby27-x64/lib/ruby/2.7.0/net/http.rb:932:in `start'
         6: from C:/Ruby27-x64/lib/ruby/2.7.0/net/http.rb:943:in `do_start'
         5: from C:/Ruby27-x64/lib/ruby/2.7.0/net/http.rb:958:in `connect'
         4: from C:/Ruby27-x64/lib/ruby/2.7.0/timeout.rb:105:in `timeout'
         3: from C:/Ruby27-x64/lib/ruby/2.7.0/timeout.rb:95:in `block in timeout'
         2: from C:/Ruby27-x64/lib/ruby/2.7.0/net/http.rb:960:in `block in connect'
         1: from C:/Ruby27-x64/lib/ruby/2.7.0/net/http.rb:960:in `open'
C:/Ruby27-x64/lib/ruby/2.7.0/net/http.rb:960:in `initialize': A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond. - connect(2) for "github-production-release-asset-2e65be.s3.amazonaws.com" port 443 (Errno::ETIMEDOUT)

Is it a ruby problem?

dusicong on 8 Jan 2020

It appears you must be behind some sort of proxy or other network filter that's blocking the download. Or maybe GitHub is throttling the download. I'm not sure.

You can download the fonts directly from here: https://github.com/chloerei/asciidoctor-pdf-cjk-kai_gen_gothic/releases

mojavelinux on 8 Jan 2020

I still don't think you actually need that theme. The default-with-fallback-font covers most of the common glyphs needed for Japanese. You have yet to provide me with a document that I can test so that I can see if you have characters which are not covered.

mojavelinux on 8 Jan 2020

Thank you for your reply.
Yes, proxy is the problem.
Just now I set a path environment variable and successfully installed the fonts. ;)

dusicong on 8 Jan 2020

Yeah!

mojavelinux on 8 Jan 2020

😄1

I still don't think you actually need that theme. The default-with-fallback-font covers most of the common glyphs needed for Japanese. You have yet to provide me with a document that I can test so that I can see if you have characters which are not covered.

Understand.
Here is a short version of my document.
[file deleted]

dusicong on 8 Jan 2020

I tried it with Asciidoctor PDF 1.5.0.rc.1 and the output looks flawless to me.

$ asciidoctor-pdf -a scripts=cjk -a pdf-theme=default-with-fallback-font jp.adoc

[file deleted]

mojavelinux on 8 Jan 2020

Hi, I would like to try again.
I upgrade my asciidoctor-pdf to 1.5.0.rc.1. And as usually Japanese doesn't show.
I would like to finish "install the asciidoctor-pdf-cjk-kai_gen_gothic gem to get the fonts".
Sorry for taking you such much time.
Could you tell me in step 4 what kind of address I should use after '-a' ?

dusicong on 9 Jan 2020

If you use the kai_gen_gothic theme, then you'd enter:

asciidoctor-pdf -r asciidoctor-pdf-cjk-kai_gen_gothic -a pdf-theme=KaiGenGothicJP document.adoc

But I can tell you right now that if the default-with-fallback-font doesn't work, the problem is not the font. The problem is with encodings.

Can you run the following commands and report the output:

asciidoctor -v
asciidoctor-pdf -v

mojavelinux on 9 Jan 2020

You might need to set either your terminal or your Windows to use UTF-8 encoding by default. Windows likes to use non-standard encodings by default, which messes up everything.

mojavelinux on 9 Jan 2020

If you use the kai_gen_gothic theme, then you'd enter:
asciidoctor-pdf -r asciidoctor-pdf-cjk-kai_gen_gothic -a pdf-theme=KaiGenGothicJP document.adoc
But I can tell you right now that if the default-with-fallback-font doesn't work, the problem is not the font. The problem is with encodings.

Can you run the following commands and report the output:
asciidoctor -v
asciidoctor-pdf -v

Asciidoctor 2.0.10 [https://asciidoctor.org]
Runtime Environment (ruby 2.7.0p0 (2019-12-25 revision 647ee6f091) [x64-mingw32]) (lc:Windows-31J fs:Windows-31J in:UTF-8 ex:UTF-8)

and

Asciidoctor PDF 1.5.0.rc.1 using Asciidoctor 2.0.10 [https://asciidoctor.org]
Runtime Environment (ruby 2.7.0p0 (2019-12-25 revision 647ee6f091) [x64-mingw32]) (lc:Windows-31J fs:Windows-31J in:UTF-8 ex:UTF-8)

I already set my Windows to use UTF-8 encoding.
RUBYOPT=-Eutf-8

dusicong on 9 Jan 2020

for the command

asciidoctor-pdf -r asciidoctor-pdf-cjk-kai_gen_gothic -a pdf-theme=KaiGenGothicJP document.adoc

I got

undefined method `/' for nil:NilClass
  Use --trace for backtrace

dusicong on 9 Jan 2020

Can you add the trace flag as it suggests?

mojavelinux on 9 Jan 2020

Setting RUBYOPT is not the same as setting the system encoding in Windows.
In fact, it could complicate things even further.

Perhaps try without setting RUBYOPT.

Beyond that, I'm really not sure what else to suggest. It's a prerequisite
to be able to work in UTF-8. The strings are being corrupted due to how you
have things configured, and that's largely due to the mixing of the JP
encoding somewhere along the way.

I think what's happening is that the input is identifying as not UTF-8
encoded, and Asciidoctor reencodes to UTF-8, which is than corrupting the
input. So the system is telling Asciidoctor one thing, but in reality it's
another.

This is definitely not a font issue.

mojavelinux on 9 Jan 2020

The reason for the change from alpha.18 is that the version of Asciidoctor core changed (so, again, nothing to do with Asciidoctor PDF). In Asciidoctor 1.5.x, the processor would force encode the input to UTF-8 (not actually reencode it). In Asciidoctor 2.0.x, the input encoding (at least what it claims to be) is honored and the processor reencodes to UTF-8. So if Ruby is configured with the wrong input encoding, you'll get a corrupt document. I'll be sure to add this information to the documentation.

mojavelinux on 9 Jan 2020

I think we finally find the true reason.
Because all of these start with the error:

incompatible character encodings: UTF-8 and Windows-31J

Right now, my ruby is 2.7, asciidoctor 2.0.x (which is the true reason), asciidoctor 1.5.0.rc.1,
I deleted RUBYOPT, and I got:

[image deleted]

I would like to try the set the system encoding in Windows.
Sorry for directly asking you how, I searched but many pages say it is impossible to change system encoding. (they even say it is impossible to check the encoding right now)

dusicong on 10 Jan 2020

incompatible character encodings: UTF-8 and Windows-31J

Yep, that's the trouble.

I've stated this before. I do not understand why Microsoft insists on using encodings other than UTF-8 by default for Windows. Every other operating system uses UTF-8 by default. And that's why Windows users are always struggling with encodings.

Follow these steps to enable UTF-8 globally:

Open Control Panel (the old school looking one)
Click "Change date, time, or number formats"
Click the "Administrative" tab
Click "Change system locale..."
Check the box "Beta: Use Unicode UTF-8 for worldwide language support"
Click OK

If that doesn't work, I'm not sure how to do it.

mojavelinux on 10 Jan 2020

Thank you for the your guide.
I tried, Japanese still doesn't show. And worse:

[image deleted]

Let me change the subject,
in the situation of ruby 2.5, asciidoctor 2.0.x, asciidoctor 1.5.0.alpha.18 in windows10 version1709
I also got the error:

incompatible character encodings: UTF-8 and Windows-31J

but when I use RUBYOPT=-Eutf-8, I get JP-showed pdf file.

Meanwhile, the situation of ruby 2.7, asciidoctor 2.0.x, asciidoctor 1.5.0.rc.1 in windows10 1909
even when I add RUBYOPT, I cannot get JP-showed pdf file.

If there is nothing wrong with the version of windows and it is only the problem of asciidoctor, then two situation above should have the same results, I suppose.

dusicong on 10 Jan 2020

I have to be honest, I just don't know. I have tried to understand Windows encodings, and even when I try it myself on a Windows machine, I just can't reproduce it. Asciidoctor follows all the rules it's supposed to follow to get encoding correct, and is thoroughly tested for those scenarios. As far as I'm concerned, it's a misconfiguration of Ruby. But I can't provide any more detail than that.

mojavelinux on 10 Jan 2020

I understand.
Thank you for all your help.
I at least have one right configuration and can produce Japanese pdf file now, so it is OK already.
Maybe some day when I learn more about ruby, I will find out the problem.

I always feel like that you will yell at me, because of your profile. haha
Thanks again.

dusicong on 10 Jan 2020

I always feel like that you will yell at me, because of your profile. haha

:rofl:

Nope, just trying to empower you with the best information I can give you.

mojavelinux on 10 Jan 2020

Another approach you might try is to use the provided Docker image via Docker for Windows. See:

That way, you take your Windows problems out of the picture. Linux handles encodings perfectly and you'll never run into these types of problems. (Another route is cygwin, but I'm less familiar with that environment to recommend it).

mojavelinux on 10 Jan 2020

Thank you very much !!

dusicong on 14 Jan 2020

❤1

FYI: https://qiita.com/kuboaki/items/67774c5ebd41467b83e2

kuboaki on 25 Jun 2020

😄1

Thanks for sharing. There are few outdated things about that article. First, it uses the deprecated pdf-style attribute instead of pdf-theme. It also doesn't use the extend feature for the theme with font catalog merging, which makes it a lot simpler to define and use custom fonts. See https://github.com/asciidoctor/asciidoctor-pdf/blob/master/docs/theming-guide.adoc#keys-extends

mojavelinux on 25 Jun 2020

👀1 👍1

Mr. mojavelinux
Thank you for your help 10 months ago.
Now I already solve Japanese Language problem, of course. ;)

I edited some of my commits recently for personal information.
Hope it doesn't bother you. :)

And I would like to ask you a favor.
Could you please delete the generated pdf file for me.
Thank you in advance.

dusicong on 9 Nov 2020

I have removed the file at your request.

mojavelinux on 9 Nov 2020

❤1

Asciidoctor-pdf: Japanese doesn't show

All 38 comments

Related issues