Asciidoctor-pdf: Add text hyphenation

[brodo]

Hi there,
it would be great if asciidoctor-pfd could insert hyphens at the appropriate places in german texts. I had to manually set this up in asciidoctor-fopub (which was a hassle). Is there a way to archive this in asciidoctor-pdf? I've set :lang: de in the document header.

Thanks

[mojavelinux]

Asciidoctor PDF does not yet support any form of hyphenation (nor does it do anything different when you change the language, atm). These features are certainly planned, and in such a way that it's not a pain to setup or use...but we aren't there yet.

One way to implement this today is by using a Treeprocessor extension for Asciidoctor that uses a library such as text-hyphen to insert hyphenation opportunities in the content. These will then be interpreted by Prawn (the underlying PDF library) and result in hyphenations in the text when a word needs to be broken.

See https://github.com/asciidoctor/asciidoctor-extensions-lab/issues/3

[mojavelinux]

To clarify the title of the issue, there is no hyphenation in any texts at the moment, regardless of language.

[mojavelinux]

I gave this a try using the text-hyphen gem and the gem does indeed work.

require 'text/hyphen'

Hyphenator = Text::Hyphen.new

def layout_prose string, opts = {}
  ...
  string = Hyphenator.visualize string, SoftHyphen
  ...
end

We'll need to work around XML tags like we do with the text_transform operation. See https://github.com/asciidoctor/asciidoctor-pdf/blob/master/lib/asciidoctor-pdf/sanitizer.rb#L28-L34.

However, either Prawn isn't doing the right thing with the soft hyphen or our text formatter is breaking how they are handled in the Prawn text arranger. Prawn is leaving behind hyphens in the middle of a line. I'll need to look into it more.

[mojavelinux]

The method would look something like:

def hyphenate_pcdata string
  if BuiltInEntityCharOrTagRx =~ string
    string.gsub(SegmentPcdataRx) { $2 ? Hyphenator.visualize($2, SoftHyphen) : $1 } 
  else
    string
  end
end

We might even want to be smarter and not hyphenate literal text (text inside <code>).

[mojavelinux]

...and yes, we can build on the lang attribute when instantiating the Text::Hyphen object.

[lit-uriy]

• russian, ukrainian, belarusian texts also use hyphenation

[ahus1]

I've had a looked into this and found a tree processor quite difficult. I didn't find a way to process the content or text and write it back to the node.

Then I came up with the idea of prepending some hyphenation. Adding the shy hyphens worked for both HTML and PDF. I don't see any hyphens in the middle of the line as @mojavelinux described in July 2015. I'm using asciidoctor 1.5.4 and asciidoctor-pdf 1.5.0.alpha.14 with prawn 2.1.0 on jruby 9.0.4.0.

It might be possible to reduce the code even further. But for me it now works by adding hyphenation to text blocks, lists and tables. I load it like any other extension using -r

This is my first time hacking ruby. Comments are welcome. Let me know what you think and I'll add a PR to close asciidoctor/asciidoctor-extensions-lab#3

require 'asciidoctor/extensions'
require 'text/hyphen'

include ::Asciidoctor

HyphenatorGerman = Text::Hyphen.new(:language => "de")

# ignore entities and things in pointy brackets
SegmentPcdataRx = /(?:(&[a-z]+;|<[^>]+>)|([^&<]+))/

def hyphenate string
  words = string.split(/[^[[:word:]]]+/).uniq
  if (words) then
    words.each do |word|
      hyphenatedWord = HyphenatorGerman.visualize word, '&#173;'
      string = string.gsub word, hyphenatedWord
    end
  end
  return string
end

module HyphenationExtension
  def content
    content = super
    case @content_model
      when :simple
        content.gsub(SegmentPcdataRx) { $2 ? (hyphenate $2) : $1 }
      else
        content
    end
  end

  def text
    content = super
    content.gsub(SegmentPcdataRx) { $2 ? (hyphenate $2) : $1 }
  end
end

class Asciidoctor::Block
  prepend HyphenationExtension
end
class Asciidoctor::ListItem
  prepend HyphenationExtension
end
class Asciidoctor::Table::Cell
  prepend HyphenationExtension
end

[mojavelinux]

Using a treeprocessor is definitely the right solution. However, it doesn't mean you have to start there. Sometimes, it's good to start with a crude solution just to explore how it's going to work. But eventually, we want to migrate the solution to a treeprocessor.

[jstasiak]

I also need hyphenation and I'm happy to report that @ahus1's extension (from the comment above) seems to be working with with the newest asciidoctor and asciidoctor-pdf.

[znerd]

@jstasiak What versions of Asciidoctor and Asciidoctor PDF would that be?

[jstasiak]

The latest rubygems releases on the 22nd of September - asciidoctor-pdf 1.5.0.alpha.16 and asciidoctor 1.5.7.1 if I remember correctly. I'm working with ascciidoctor-pdf that's based on b47f53cf2f7a right now and it still works.

[odrotbohm]

I'd love to see that integrated into Asciidoctor PDF. In the meantime. Is there any place I can read up on how to integrate an extension that's presented as sources into my existing APDF installation?

[ahus1]

@odrotbohm - please have a look at this repository for a sample setup:

Ruby only: https://github.com/ahus1/asciidoctor-deepdive/tree/master/manual-ruby

Maven running Ruby: https://github.com/ahus1/asciidoctor-deepdive/tree/master/manual

[mojavelinux]

I'm open to accepting a PR for this (esp now that we have a test suite). I still wonder, though, whether this is something that should go into Prawn directly, or whether it's better suited for Asciidoctor PDF. Until I got my hands in there, I'm not sure I'd know which place is more appropriate.

[weberja]

I stated to write a treeprocessor on the groundworks of @ahus1 ad i encounter the same problem as he had. It's quit difficulty to write the hyphened text back to a block because it is already converted. If i did this wouldn't i destroy inline makros? The only functions to write back the text are text or lines an in my understanding they need to be asciidoc text. Am i wrong?

I begin to have a felling that this should live in prawn (or near the end of the processing of an .adoc) because it is a thing we want do later so we don't break other things that doesn't ignore shy hyphen.

What are your opinions?

[mojavelinux]

I agree Prawn is ultimately the right place for this. The trouble is, text handling in Prawn is like brain melting complicated. I have been in that code in the past, and I find myself quickly getting lost. That's not to say you shouldn't try. I'm just sharing the experience I've had. The abstraction model really isn't great, so you have to work at quite a low level. Of course, we will only know what's possible if we try ;)

[mojavelinux]

Fascinating article on the nuances of hyphenation: http://clagnut.com/blog/2395

[weberja]

Yah I had stated to take a look in to prawn but immediately back out of it in the hope to find another spot to bring this in. Maybe I find the time to look in to it :'D

[mojavelinux]

It's possible we could monkeypatch Prawn. In fact, we do that a lot already. So it doesn't necessarily need to be accepted upstream for us to go that route. But we do need to know what code to monkeypatch and how to do it. There's always a risk of introducing incompatibility between Prawn versions, but there are ways to avoid that.

[mojavelinux]

(And by monkeypatch, I mean we can use Module.prepend).

[mojavelinux]

I've submitted a proposal (see #1380). Here's how it works:

• You must install the text-hyphen gem separately
• You must set the hyphens attribute in the document header or via the API/CLI
• The value of the hyphens attribute is used as the language; otherwise it defaults to the value of the lang attribute, finally falling back to en_us
• All text is hyphenated except for headings (section and sidebar titles), the title page, description list terms, and the auto-generated index

The one limitation is that if a word is split by different formatting, it will not be hyphenated. In this case, the hyphenator cannot see the whole word. This would not be easy to solve. But in these cases, you can insert a shy hyphen manually, which always works.

We can improve on this behavior as we get feedback from real world usage.

[mojavelinux]

@ahus1 My mistake is that I was applying the hyphenation to the entire paragraph at once instead of individual words. Thanks for the tip!

[ahus1]

@mojavelinux - I looked at the tests and the example PDF and the tests; look great. Will try it once the next beta or RC is available. I've opened a pull request for asciidoctorj-pdf to include text-hyphen by default.

[robertpanzer]

The CI build seems to fail because we're still running on an old build image version, however the build succeeded locally for me. (I'll rather migrate to GH actions before investing more time in travis)

If I understood the related ticket correctly it currently requires a custom extension to profit from this PR.
Therefore I am a bit undecided if I should merge this. If this is becoming _the_ solution for hyphenation, then I am fine with it.
@mojavelinux What do you think?

[ahus1]

@robertpanzer - once the changes from Dan are merged (and this ticket is closed), no extension is needed any more. The author just sets the hyphens attribute.

I'd ask you to keep the PR for asciidoctorj-pdf open until this in a new asciidoctor-pdf release, but merge it before publishing the next asciidoctorj-pdf release.

[robertpanzer]

Sorry for commenting here, definitely the wrong place 😞
I'll make sure to check the PR again once hyphenation has made its way into asciidoctor-pdf.

[mojavelinux]

@ahus1 is correct, no extensions are necessary. The functionality is provided by a direct integration with the text-hyphen gem.

You need two things to enable hyphenation:

1. The text-hyphen gem on the load path
2. The hyphens attribute on the document

AsciidoctorJ PDF should not enable the hyphens attribute by default since hyphenation adds significant overhead (plus the language isn't known). Instead, this should be something the author turns on.

[mojavelinux]

I'm going to go ahead and proceed with this implementation. I like how it ended up fitting in. We're applying the modification at the same place where the text transform (uppercase, lowercase, etc) is applied. And that just makes sense.

[odrotbohm]

Just wanted to throw in a big thank you for solving this. Really closes one of the features I hear most people asking about, especially coming from legacy publication systems. 👍 🙇

[mojavelinux]

Thanks @odrotbohm!

[mojavelinux]

There's one important caveat to note. If a line gets advanced to the next page because there isn't enough room for it on the current page, the hyphen opportunities in that line end up being ignored. This is caused by a bug in Prawn. For some reason, it strips out the hyphen characters when advancing the line. I'll file an issue upstream to get that sorted out.

[mojavelinux]

I figured it out. See #1383.

[mojavelinux]

I think the attribute should be renamed from hyphens to hyphenate. Thoughts?

[ahus1]

@mojavelinux: yes, hyphens is probably quite short. Looking at other attributes they are usually nouns, see https://asciidoctor.org/docs/user-manual/#builtin-attributes

When making it a noun, it would be hyphenation. I vote for the noun and not the verb.

[mojavelinux]

It's not perfectly consistent, but it does happen to be the noun more times than the verb. hyphens is definitely out because it's too ambiguous. I'm stuck between hyphenate and hyphenation. But if there's no objection, I'll proceed with hyphenation.

[mojavelinux]

Now I remember why I initially selected hyphens. It's because that's the name of the CSS property that controls hyphenation. See https://developer.mozilla.org/en-US/docs/Web/CSS/hyphens But as we've established, using that as a document-wide attribute may make it too ambiguous.

[mojavelinux]

I've thought about the proposed change a lot over the last few days and I'm now of the opinion that we should stick with the name hyphens. Here's my reasoning:

1. It's a noun, making it consistent with other attributes like toc and sectnums.
2. It's short, like toc and sectnums. In contrast, hyphentation really looks out of place when put next to these other attributes
3. It's the name used by CSS, and the styling in Asciidoctor PDF is heavily inspired by CSS

So let's go with hyphens and see what feedback we get. It's not a big deal to change it if we found out we made the wrong choice.