Vulkan-docs: Provide working & chunked version of the spec, including extensions

[ghost]

It appears these are the only officially hosted HTML pages of the spec:

https://www.khronos.org/registry/vulkan/specs/1.0/html/vkspec.html
https://www.khronos.org/registry/vulkan/specs/1.0-wsi_extensions/html/vkspec.html
https://www.khronos.org/registry/vulkan/specs/1.0-extensions/html/vkspec.html
https://www.khronos.org/registry/vulkan/specs/1.0/man/html/

The first three basically include everything on a single page. This is very unwieldy: it takes long to load, grinds some browsers to a halt, and searching for keywords is made harder. It also won't scale. You can't append every Vulkan spec ever made to vkspec.html.

The last link is chunked, but doesn't include extensions. It also appears to have dead links or links into the one-page spec. E.g. inspect the links "depth bias clamping" and "recording state" links on this page:
https://www.khronos.org/registry/vulkan/specs/1.0/man/html/vkCmdSetDepthBias.html

I also didn't find the index for those pages (other than the raw file index).

Would it be possible for Khronos to officially host the complete spec in a nicer way?

[haasn]

To give another example, http://vulkan-spec-chunked.ahcox.com/index.html this has a really nice interface but it's horrifically out-of-date

[oddhack]

asciidoctor, which is our toolchain now, does not support chunked HTML output generation unless you go through the fairly horrid Docbook intermediate path. This isn't something we've wanted to put time into supporting given a lack of demand from inside Khronos itself, but if you want to figure out how to make that work, we'd certainly evaluate the PR. Whether we would want to commit to supporting all those generated spec artifacts indefinitely is another question that would have to be asked of the working group. Note that the ref pages are not a complete specification, nor are intended to be.

It does take a while to load KaTeX and process all the equations in the spec. I'm curious what browsers are grinding to a halt, though. Chrome/Chromium and Firefox seem OK.

Our current belief is that we can, in fact, include every extension, although it's probably more useful for vendors to generate specs that contain only and exactly the sets of extensions their devices expose.

Loading the PDF spec should be faster in general.

I don't dispute that there are drawbacks, as well as benefits to the single-file specs. It's an issue of time and demand.

[krOoze]

This is very unwieldy: it takes long to load, grinds some browsers to a halt, and searching for keywords is made harder.

Dam kids! In the olden days of Vulkan it took like 2 minutes for the spec to load! :trollface:

Beside using the PDF, disabling JavaScript (perhaps through AdBlock or something) might help if you are fine with math formulas not being translated.

It does take a while to load KaTeX and process all the equations in the spec. I'm curious what browsers are grinding to a halt, though. Chrome/Chromium and Firefox seem OK.

Potentially some KaTeX elements can still be translated to the Asciidoc elements. That might help.

Anyway I do get this even with Firefox 56 and Chrome 61. Basically both browsers become unresponsive for a moment. Chrome is bit more well behaved (and allows user to switch to another tab) but loads the page longer.

unless you go through the fairly horrid Docbook intermediate path.

Tried that; not PR ready though. It is mostly readable. The formatting is bit ugly (i.e. plain). Some formatting does not work, and KaTeX is not injected so math does not work. Links seem to work though, which is nice.

Anyway, here's a starting point if someone wants to try it or bring it to usable state:
https://github.com/krOoze/Vulkan-Docs/tree/add_chunked

[oddhack]

@krOoze it's definitely posssible to use KaTeX via the Docbook->HTML path - the previous asciidoc toolchain did this. Probably that could be resurrected fairly straightforwardly.

[haasn]

Beside using the PDF, disabling JavaScript (perhaps through AdBlock or something) might help if you are fine with math formulas not being translated.

I personally have both javascript and CSS disabled, which helps a lot (although it does break the scrollable table of contents); but it's still nowhere near what I'd call realtime / comfortable.

[ghost]

There's an in-progress (but mostly abandoned) extension to asciidoctor to generate chunked output here: https://github.com/asciidoctor/asciidoctor-extensions-lab/blob/master/lib/multipage-html5-converter.rb

I played with it once, mostly works, but crucially it doesn't handle table of contents or cross-page links properly. It wouldn't be too hard to hook all that up to make it work; though asciidoctor's extension interface doesn't make it obvious how you'd do it...

[oddhack]

We'll monitor asciidoctor in the hope of them supporting chunked output properly, but this is likely to be pending for quite a while.

[oddhack]

N.b. I've seen some claims that KaTeX's load time is (a) mostly due to the large font package it includes and (b) those fonts aren't needed on modern browsers. If I can substantiate that and figure out how to strip out the fonts, that may help (or, if someone else wants to take a look at that - I'm pretty oversubscribed at present).

[ghost]

@oddhack

N.b. I've seen some claims that KaTeX's load time is (a) mostly due to the large font package it includes and (b) those fonts aren't needed on modern browsers.

@Khronoswebmaster and I looked at this a few weeks ago, and couldn't find a hugely compelling way to improve this situation unfortunately. We had some success with allowing people to download and install the fonts offline and then use their local copies, but that's only a win for a few people at best, so I haven't pursued that option.

[jeremyong]

I'm looking at a javascript profile of just loading the page now. It takes over 10 seconds to process a single frame of the initial load and the entire page isn't interactable for a full 20 seconds. Over 4 of those seconds are spent in garbage collection alone.

The load time is certainly not just fonts. The time spent is largely the huge size of the documentation and the amount of layout churn that occurs when individual DOM nodes are removed, recreated, and reinserted.

[krOoze]

@jeremyong It is not, in of itself the fonts.

Anyway, anyone who is in this thread for the non-chunked spec load time, I refer you to PR #702 (and others linked therein). With those changes it loads within units of seconds (which is IMNO still too much in the 21st century, but nevertheless several times improvement.).

[Zingam]

The apispec.html doc also needs to have a chunked version.
Any low power machine chokes on loading, even resizing the docs in Chrome.

[Zingam]

Is this fixed now?

[oddhack]

Mostly. There are a few enhancements to be added like a search box, and it hasn't received much testing yet, so we haven't linked it into the registry index page, but the chunked specs are present in the repo. We might change the URLs too.

[Zingam]

Is this the final chunk size: https://renderdoc.org/vkspec_chunked/chap40.html#extensions
It feels too big still.

[oddhack]

AFAICT from the asciidoctor-chunker project, it just splits at the chapter level. All we could do is split longer chapters, then. Maybe someone who has time to look into this more deeply will have a better suggestion.

[oddhack]

@Zingam we are not supplying apispec.html any more. The "chunked" version of it is the individual reference pages, because literally all apispec.html is, is an index followed by all the refpages.

[krOoze]

They have added search box JS on top of the official build that adds 2.3 s to the load time, so...
Their server sends some content without compression (which adds 200 kB). Also not sure why their favicon is 300 kB.

Math is still being converted (of which there are none in this chapter, I think). That adds about 1 s to the page load. This can be helped by #733 or #704.

PS: I guess it could be split to three chapters (from the original subchapters), though for 1.0 build most of the content would be in the first chapter anyway.
PPS: It might also make sense to exclude the old extensions from the build of the Flagship Specification.

[Zingam]

Wouldn't splitting the chapters at a subtitle level such as "31.7. Sparse Resource API" be preferable (if possible) or even at "31.7.1. Physical Device Features" as the reader of the specification won't have to scroll as much ad will be easier to find/not to lose the location he wants tot read?

[krOoze]

@Zingam Possibly, but as the asciidoctor-chunker does not seem to support that at the moment, the suggestion is moot.

[procedural]

Vulkan 1.0: https://www.khronos.org/registry/vulkan/specs/1.0/html/
Vulkan 1.0 with wsi extensions: https://www.khronos.org/registry/vulkan/specs/1.0-wsi_extensions/html/
Vulkan 1.0 with all extensions: https://www.khronos.org/registry/vulkan/specs/1.0-extensions/html/

Vulkan 1.1: https://www.khronos.org/registry/vulkan/specs/1.1/html/
Vulkan 1.1 with khr extensions: https://www.khronos.org/registry/vulkan/specs/1.1-khr-extensions/html/
Vulkan 1.1 with all extensions: https://www.khronos.org/registry/vulkan/specs/1.1-extensions/html/

[1ace]

Vulkan 1.1 with wsi extension: https://www.khronos.org/registry/vulkan/specs/1.1-wsi_extensions/html/

That's a 404 right now.

(All the other links work, thanks!)

[krOoze]

@1ace https://www.khronos.org/registry/vulkan/specs/1.1-khr-extensions/html/index.html
or simply https://www.khronos.org/registry/vulkan/ for navigation

[oddhack]

The 1.1.106 spec update includes a searchbox in the chunked specifications and a search index (unfortunately serving to bloat the already enormous registry repository even more) backing the searchbox. Closing this.

[Zingam]

@krOoze

@Zingam Possibly, but as the asciidoctor-chunker does not seem to support that at the moment, the suggestion is moot.

https://github.com/wshito/asciidoctor-chunker/issues/5#issuecomment-782168645

Well, maybe our wish for finer chunks was granted.