Sumatrapdf: Regression : large PDF cannot be opened since v3.2

[Lucas-C]

Hi.

I am working on a PDF video game, made with and targeted for Sumatra PDF:
https://github.com/Lucas-C/undying-dusk

The game takes the form of a 150MB PDF file.

Sadly, thanks to some playtesters, I found out that since v3.2 of Sumatra PDF this reader cannot load the file anymore (it freezes, taking forever to load).
I used v3.1.2 during the development of the game, and it worked all fine.

Do you agree that it is some sort of regression?
Do you think it could be fixed?

I'm willing to help. I'm a developper but I have no experience developping on Sumatra PDF, so any pointers on how to debug this would be appreciated 😊

(linked issue on the game repo, in French)

[SumatraPeter]

@kjk: Perhaps an update to MuPDF 1.18.0 will help, since on my PC that could open the PDF just as easily as Sumatra 3.1.2 could (while both 3.2 and the latest prerel-13163 - based on MuPDF 1.17.0 I believe - choked).

@Lucas-C: BTW the "Compatible PDF readers" table on your GitHub page seems to be wrong - MuPDF _can_ scale graphics to fit the screen using the 'Z' keyboard shortcut. It also uses 'G' IIRC for going back to the prev. page.

[Lucas-C]

Thank you for your answer @SumatraPeter.

I did not know about MuPDF shortcuts. I'd be really happy to consider it compatible with the game.
However I have just tested the shortcuts you mentioned on mupdf.exe on Windows (version 1.17.0) and 'z' does not have any effect and g only brings back to the first page of the document :(

If you wish to dig deeper into this, could you please open an issue on Lucas-C/undying-dusk in order keep the discussion separate ?

[SumatraPeter]

@Lucas-C: If I have something more substantial to add about your game in particular I'll certainly open an issue for it. However for now while we wait for kjk to chime in all I want to add about MuPDF is that its KB shortcuts are case-sensitive. So try 'Z' and 'G' instead of 'z' and 'g'. 🙂

[Lucas-C]

'Z' works fine ! Thank you ! However 'G' only bring to the end of the document...
I'm going to update the table though, and maybe suggest the 'ALT + g' shortcut to the MuPDF project.

Edit: done here

[Lucas-C]

Actually, after checking the source code, I found out that MuPDF shortcut to "go back" is 't' !

[GitHubRulesOK]

@Lucas-C
Whilst running MuPDF-GL the shortcuts are listed via F1 no need for users to read source code of search the web.
A similar feature I would like to have seen in SumatraPDF see https://github.com/sumatrapdfreader/sumatrapdf/issues/1467

[kjk]

Yes, we do a little bit more work per page in 3.2 and usually that's not a problem but this PDF has over 100k pages and this additional work adds up.

Based on profiling, the time is spent in this loop: https://github.com/sumatrapdfreader/sumatrapdf/blob/master/src/EnginePdf.cpp#L806

It should be possible to optimize this. We call pdf_lookup_page_obj for each page which re-does a linear search in the same dict object for a given page number.

Reading mupdf code it seems like it would help if the dict of pages info in the PDF was sorted. Then mupdf code would do binary search instead of linear. So if you can tweak PDF generation code to do that, it would help.

But ultimately we should be able to just grab that dictionary and get at this information without doing a search for each page number.

@Lucas-C that's something you should be able to figure out just by tracing the code in pdf_lookup_page_obj so you can give it a try. If not, I'll look into it when I process a backlog of issues.

[kjk]

Should be fixed in latest daily build https://www.sumatrapdfreader.org/dailybuilds.html

For me it opens under 3 seconds.

[Lucas-C]

Thank you @kjk for your detailed explanation!
And for the fast fix!

Is your comment about sorting the dict of pages info in the PDF still relevant after your fix?
Right now I don't see how to do it, but I can look into it.
Otherwise maybe pdf_sort_dict should be called by Sumatra?

On the fix, I have good news and bad news:
the opening time issue is definitively solved,
but on the other hand Sumatra from the daily build has rendering issues...
I'm going to open another ticket about that.

[kjk]

Sorting wont matter for Sumatra after this fix.

In general, if you're not already doing that, you can optimize final PDF with e.g. mutool clean https://mupdf.com/docs/manual-mutool-clean.html

Reading the docs mutool clean -g -g -g -l -z input.pdf output.pdf.

Not sure if it'll make a big difference, but it might make some.

There are other tools that optimize PDF file, not just mutool.

[Lucas-C]

I tried it, but after more than 1h running...

mutool clean -g -g -g -l -z undying-dusk-v0.9.4.pdf optimized-undying-dusk-v0.9.4.pdf
error: exception stack overflow!

Thanks for the tip anyway!

[GitHubRulesOK]

after more than 1h running...🤣 either third g or -l will take ages for that many objects
using just
mutool clean -g -g undying-dusk-v0.9.4.pdf optimized-undying-dusk-v0.9.4.pdf
is quick and reduces file size a little so may help to reorder objects enough but I have not enough insight to confirm difference is as desired

Is it not possible to sort pages during generation rather than re-shuffling billions of bytes afterwards ?