Pandoc: Syntax highlighting broken in Apple Books v2.4 (1923)

[vlad-goshko]

Hi, recently I'm recompiled my markdown ebook and found that is all blocks of code are invisible in Apple Books.
I don’t know which update broke this OSX or Pandoc itself, but digging deeper I found a simple example that breaks it.

Info:

OS: MacOS Catalina 10.15.4 (19E266)
Apple Books v2.4 (1923)
pandoc 2.9.2.1
Compiled with pandoc-types 1.20, texmath 0.12.0.1, skylighting 0.8.3.2

File:

test_case.md

Runs:

pandoc -t epub3 -o test_case.epub test_case.md - second block of code is invisible

pandoc -t epub3 -o test_case.epub test_case.md - test_case.md without 16 line, everything is fine

pandoc --no-highlight -t epub3 -o test_case.epub test_case.md - original test_case.md with 16 line, everythins is fine

[jgm]

Can't reproduce this with iBooks 2.2. (Though I do note that the end of the comment disappears -- this is actually due to the size of the enclosing box, and if you highlight the text you can see that it's really there and will scroll. Are you sure that isn't the situation in your case?)

[jgm]

Also try unzipping the epub (use unzip) and examining the HTML produced by pandoc to see if anything looks amiss.

[vlad-goshko]

Yes, I checked unzipped archive and there everything is ok.
As for the text highlighting - yes I can highlight it but it seems to be invisible

And if I copy the text - I see the code block in buffer

[vlad-goshko]

Also I checked produced epub in Calibre and it is ok, so this is most likely a problem of Apple Books or something, but I really need to view my epubs on iOS devices

[jgm]

Sounds like a problem on Apple's end. Hm. You can experiment with the epub and see if you can fix things by twiddling HTML or CSS -- maybe this would point to a workaround.

[jgm]

You might report this to apple. The epub, xhtml, and css pass validation tests. I don't see a problem on pandoc's side.

[nateberkopec]

I just noticed this as well on my own epub. Disabling syntax highlighting means at least the code shows up, though of course without highlighting!

No other reader exhibits this issue - tested in Calibre and in various web browsers.

I _suspect_ it has something to do with the inline style block used by Pandoc and Apple Book's renderer (which, IDK, what does it use? Safari?). This is not necessarily a Pandoc _issue_ but it's possible it could be mitigated on Pandocs side.

[nateberkopec]

OK, I have no idea _why_, but removing the follow CSS rule fixes the issue in Apple Books 2.4/1923:

@media screen {
div.sourceCode { overflow: auto; }
}

This doesn't really make any sense.

I confirmed this by editing the generated CSS in the EPUB archive and just kind of binary searching through the CSS until I removed the rule that caused the problem.

[nateberkopec]

So, the workaround is to add the following in your EPUB CSS:

/* Apple Books 2.4+ doesn't like overflow:auto on syntax highlighting generated by skylighting */
@media screen {
  div.sourceCode { overflow: visible !important; }
}

[vlad-goshko]

Yes, I confirm, this fixes the issue, thank you
Also if you have some overflow: hidden in other blocks it may also break the highlighting, so resulting block would be:

/* Apple Books 2.4+ doesn't like overflow:auto on syntax highlighting generated by skylighting */
@media screen {
  div.sourceCode { overflow: visible !important; }
  p, span, pre, code, blockquote {
    overflow: visible !important; 
  }
}

Or in more general way:

/* Apple Books 2.4+ doesn't like overflow:auto on syntax highlighting generated by skylighting */
@media screen {
  .sourceCode { overflow: visible !important; }
}

Add this to the end of your EPUB CSS file

[mb21]

nice find! so should just add this to https://github.com/jgm/pandoc/blob/master/data/epub.css ?

[jgm]

Adding this means that code will overflow the box in all EPUBs, correct?
Maybe worth doing this if iBooks can't handle the 'auto' setting, but I can imagine many others may prefer the current behavior.

[nateberkopec]

Hmmm, maybe I can try overflow:hidden later.

[jgm]

Have you tried reporting this as a bug to Apple, by the way? It does seem a problem on their end, and it seems a shame to eliminate scrollbars from overflowing code blocks on ALL epubs just because of a problem with a specific version of iBooks. But that might be the thing to do until they fix the issue...

[nateberkopec]

Overflow hidden does _not_ work. I'll report to Apple.

[vlad-goshko]

I already created a ticket to them 3 weeks ago, but it just disappeared after 3 weeks, maybe such a policy, I don’t know

[jgm]

I now have Books 2.4 on my mac. So I tried this. It works fine, as far as I can see, although there are no scroll bars, so in lines that don't fit the content is simply cut off. (But that wasn't the problem I took you to be reporting.) Screenshot:

[jgm]

OK, I do see some funny behavior. With the default font size, everything fine. If I do Command-+ to increase the font size, then still fine -- the second code block continues on p. 2:

But if I increase size again, I get something similar to what you reported:

[jamtur01]

I'm also seeing this - I've worked around it by disabling syntax highlighting with --no-highlighting. I've logged a ticket with the Apple books team but my track record on responses from them is pretty bad.

[ageitgey]

I also ran into this issue in the wild in my previously-published book. I can confirm that adding the following CSS (suggested by @vlad-goshko) resolved the issue:

/* Apple Books 2.4+ doesn't like overflow:auto on syntax highlighting generated by skylighting */
@media screen {
  .sourceCode { overflow: visible !important; }
}

[jamtur01]

@ageitgey I get code overflowing my listing boxes using that CSS so it doesn't resolve it for me I am afraid. :(

[rauschma]

I ended up doing the following:

@media screen {
  .sourceCode {
    overflow: visible !important;
    white-space: pre-wrap !important;
  }
}

Wrapping was necessary because Books.app would cut off text otherwise.

[jgm]

Should we change the default CSS to the one @rauschma used? That would change behavior in all readers, not just ibooks. In general I prefer the scrollbar to wrapping, but until ibooks is fixed it might make sense to do something that works for everything.

(Or: is there a way to make the CSS rule conditional so it only affects ibooks?)

[bitfield]

This issue is still happening with the updated CSS; I found it was necessary to remove this line to get code blocks to be visible in iBooks:

    white-space: pre-wrap !important;

So the CSS that works for me is:

css @media screen { /* Workaround for iBooks issue; see #6242 */ .sourceCode { overflow: visible !important; } }

[jgm]

@bitfield, it should work fine out of the box (without custom css) in the latest pandoc release.
(I just tested with the sample above.) If you think the issue is still there, we'll need full instructions on how to reproduce. Pandoc version, exact command line and inputs used.

[bitfield]

With pandoc 2.10.1, and the following command line:

pandoc -o "dist/For The Love of Go - Fundamentals.epub" metadata.yaml src/*.md -t epub3 --css css/styles.css --highlight-style=haddock --epub-embed-font=fonts/SourceSerifPro-Regular.ttf --epub-embed-font=fonts/RobotoSlab-VariableFont_wght.ttf --table-of-contents

and the following CSS:

/* This defines styles and classes used in the book */

body {
  margin: 5%;
  text-align: justify;
  font-size: medium;
  font-family: SourceSerifPro;
  src: url("fonts/SourceSerifPro-Regular.ttf");
}

code {
  font-family: monospace;
}

h1 {
  text-align: left;
  font-family: RobotoSlab;
  src: url("fonts/RobotoSlab-VariableFont_wght.ttf");
}

h2 {
  text-align: left;
}

h3 {
  text-align: left;
}

h4 {
  text-align: left;
}

h5 {
  text-align: left;
}

h6 {
  text-align: left;
}

/* For title, author, and date on the cover page */

h1.title {}

p.author {}

p.date {}

nav#toc ol, nav#landmarks ol {
  padding: 0;
  margin-left: 1em;
}

nav#toc ol li, nav#landmarks ol li {
  list-style-type: none;
  margin: 0;
  padding: 0;
}

a.footnote-ref {
  vertical-align: super;
}

em, em em em, em em em em em {
  font-style: italic;
}

em em, em em em em {
  font-style: normal;
}

code {
  white-space: pre-wrap;
}

span.smallcaps {
  font-variant: small-caps;
}

span.underline {
  text-decoration: underline;
}

q {
  quotes: "“" "”" "‘" "’";
}

div.column {
  display: inline-block;
  vertical-align: top;
  width: 50%;
}

div.hanging-indent {
  margin-left: 1.5em;
  text-indent: -1.5em;
}

the code blocks are not visible in iBooks:

With the same pandoc version and command line, if I add the following block of custom CSS:

@media screen {
  /* Workaround for iBooks issue; see #6242 */
  .sourceCode {
    overflow: visible !important;
    white-space: pre-wrap !important;
  }
}

the code samples are visible:

Here's the relevant Markdown source:

First, the test sets up a variable `want` to establish what it _wants_ to receive from the function:

```go
var want float64 = 2
```

Then it calls the function with the values `4, 2`, and stores the result into another variable `got`:

```go
got := calculator.Subtract(4, 2)
```

The idea is to compare `want` with `got` and see if they are different. If they are, the `Subtract()` function is not working as expected, so the test fails:

```go
if want != got {
  t.Errorf("want %f, got %f", want, got)
}
```

On the other hand, if `want` and `got` are equal, then everything's fine, so the test function ends here. (You don't need to explicitly make a test pass in Go; if it doesn't explicitly fail, that's considered a pass.)

So let's look at the code for the `Subtract()` function (in `calculator.go`). Here it is:

```go
func Subtract(a, b float64) float64 {
  return b - a
}
```

If you spot a problem, try altering the code to fix it. Run `go test` again to check that you got it right.

When the test passes, you can move on!

**STRETCH GOAL:** Find out what other ways there are to fail a test using the `testing.T`. What other methods does it have? Do any of them look potentially useful?

[jgm]

Here's what I see in iBooks after doing with your sample files

pandoc --css 6242.css 6242.md -o 6242.epub

This looks fine. iBooks 2.4, 1923.1.

[bitfield]

This looks fine.

I'm delighted to hear it! But you said that you thought the issue _should_ be fixed with the latest pandoc, and I have confirmed that it isn't. You asked for the exact command line, input, and style files I'm using. I've supplied them. If I can supply any further information to help you diagnose and fix the issue, please let me know.

[jgm]

I can't fix the issue if I can't reproduce it.
Can you reproduce it using the exact commands I use?
Save the markdown text as 6242.md:
``` First, the test sets up a variablewant` to establish what it _wants_ to receive from the function:

var want float64 = 2

Then it calls the function with the values 4, 2, and stores the result into another variable got:

got := calculator.Subtract(4, 2)

The idea is to compare want with got and see if they are different. If they are, the Subtract() function is not working as expected, so the test fails:

if want != got {
  t.Errorf("want %f, got %f", want, got)
}

On the other hand, if want and got are equal, then everything's fine, so the test function ends here. (You don't need to explicitly make a test pass in Go; if it doesn't explicitly fail, that's considered a pass.)

So let's look at the code for the Subtract() function (in calculator.go). Here it is:

func Subtract(a, b float64) float64 {
  return b - a
}

If you spot a problem, try altering the code to fix it. Run go test again to check that you got it right.

When the test passes, you can move on!

STRETCH GOAL: Find out what other ways there are to fail a test using the testing.T. What other methods does it have? Do any of them look potentially useful?
````

Save the CSS as 6242.css:
````
/* This defines styles and classes used in the book */

body {
margin: 5%;
text-align: justify;
font-size: medium;
font-family: SourceSerifPro;
src: url("fonts/SourceSerifPro-Regular.ttf");
}

code {
font-family: monospace;
}

h1 {
text-align: left;
font-family: RobotoSlab;
src: url("fonts/RobotoSlab-VariableFont_wght.ttf");
}

h2 {
text-align: left;
}

h3 {
text-align: left;
}

h4 {
text-align: left;
}

h5 {
text-align: left;
}

h6 {
text-align: left;
}

/* For title, author, and date on the cover page */

h1.title {}

p.author {}

p.date {}

nav#toc ol, nav#landmarks ol {
padding: 0;
margin-left: 1em;
}

nav#toc ol li, nav#landmarks ol li {
list-style-type: none;
margin: 0;
padding: 0;
}

a.footnote-ref {
vertical-align: super;
}

em, em em em, em em em em em {
font-style: italic;
}

em em, em em em em {
font-style: normal;
}

code {
white-space: pre-wrap;
}

span.smallcaps {
font-variant: small-caps;
}

span.underline {
text-decoration: underline;
}

q {
quotes: "“" "”" "‘" "’";
}

div.column {
display: inline-block;
vertical-align: top;
width: 50%;
}

div.hanging-indent {
margin-left: 1.5em;
text-indent: -1.5em;
}
````

Then

pandoc --css 6242.css 6242.md -o 6242.epub

and open 6242.epub in ibooks.

If you get the same result I did, that would be helpful to know.

[jgm]

The idea is to prune things down so you have a minimal case -- just enough to show the problem, without anything extraneous.

[bitfield]

No, your 6242.md file does not reproduce the problem for me. Here is one that does:

This test failure is telling you exactly where the problem occurred:

```
calculator_test.go:20
```

It also tells you what the problem was:

```
want 2.000000, got -2.000000
```

So what exactly is the test doing? Let's take a closer look. Don't worry about the `t.Parallel()` part for now, as it merely relates to how Go runs the tests.

First, the test sets up a variable `want` to establish what it _wants_ to receive from the function:

```go
var want float64 = 2
```

Then it calls the function with the values `4, 2`, and stores the result into another variable `got`:

```go
got := calculator.Subtract(4, 2)
```

The idea is to compare `want` with `got` and see if they are different. If they are, the `Subtract()` function is not working as expected, so the test fails:

```go
if want != got {
  t.Errorf("want %f, got %f", want, got)
}
```

On the other hand, if `want` and `got` are equal, then everything's fine, so the test function ends here. (You don't need to explicitly make a test pass in Go; if it doesn't explicitly fail, that's considered a pass.)

So let's look at the code for the `Subtract()` function (in `calculator.go`). Here it is:

```go
func Subtract(a, b float64) float64 {
  return b - a
}
```

If you spot a problem, try altering the code to fix it. Run `go test` again to check that you got it right.

When the test passes, you can move on!

**STRETCH GOAL:** Find out what other ways there are to fail a test using the `testing.T`. What other methods does it have? Do any of them look potentially useful?

Here's the result:

You can see that the first two code samples are now showing, but the subsequent ones are not. I feel like this is related to the (non-Go) backtick-delimited blocks; if I remove the first one, so that the test document starts at "It also tells you what the problem was...", the first missing code sample becomes visible, but the second one is still invisible.

I hope this helps!

[jgm]

Good. I can reproduce the issue with this input.

[jgm]

But adding the extra CSS noted above to the end of the CSS didn't seem to help.

@media screen {
  /* Workaround for iBooks issue; see #6242 */
  .sourceCode {
    overflow: visible !important;
    white-space: pre-wrap !important;
  }
}