Docs.rs: Downloadable docs

[Kapeli]

I'd like to integrate docs.rs inside Dash.

To achieve this, I need a way to download the docs for a package as HTML files. Please consider supporting this.

[njskalski]

How about just sharing the database dump, like a backup.torrent or whatever. Crates.io for sure have a backup, this way some users will co-host it for free.

[ketsuban]

That's not an acceptable solution - note that @Kapeli isn't just asking for tossing the database over the wall, they specific want to integrate docs.rs into a offline documentation viewer so that users can download the documentation for specific crates as needed. The docset format isn't exclusive to Dash, either - other projects such as Zeal also make use of it.

[Kapeli]

One note: I'm not asking for docs.rs to generate Dash docsets. I'm asking for it to provide downloadable docs.

[lilyball]

I'd really love to see this done. I use Dash for everything else and it's really annoying dealing with third-party Rust crate docs since I can't view them anywhere outside a web browser.

[reeze]

I am really looking forward this integration.

[dmilith]

Can't wait for both integration and Rust syntax coloring support for Dash snippets :)

[libratiger]

is there any progress or any new plan?

[nhynes]

It'd also be nice to get the nightly rustc docs without having to build the rustc. Recursive wget is a(n inefficient) workaround, I suppose.

[brinsche]

https://github.com/Robzz/cargo-docset was recently released, maybe that's enough for some people here or maybe the some of the code can be reused and integrated into docs.rs somehow

[lerencao]

Any update on this?

[pietroalbini]

It's not too practical to generate a downloadable archive of a crate's documentation, as each file is stored individually on S3.

We'd need to fetch all the files individually and generate an archive of that on the fly, which is not practical for large crates. Preparing an archive at build time and storing it separately would increase our storage costs, and due to the unbounded nature of docs.rs we should try avoiding that.

If y'all have better implementation ideas I'd love to read them.

[lilyball]

Could we prepare an archive at build time but only for crates that are opted in to this using some notion of "important to the community"? For example I'd love to see docs.rs provide a docset for tokei that Dash can keep automatically up-to-date. I don't know who'd provide that curation though. There could be some way to nominate crates and leave it up to the docs.rs maintainers to approve it, or maybe it could be based on traffic to a particular crate's documentation.

[Kapeli]

Could the archive be generated only when requested and have a fixed-size cache where older archives get removed?

Would the archives really be that big though? Docs are generally just text, which compresses very well. You could have a separate archive for the common resources (CSS, images, fonts and so on) and then the docs archives would just be compressed HTML files.

[pietroalbini]

Could the archive be generated only when requested and have a fixed-size cache where older archives get removed?

That's not really feasible, as some crates (like stm32f0) have ~200k HTML files in them. They're all stored on S3, and just listing them took awscli 2 minutes and 43 seconds from the docs.rs server.

Would the archives really be that big though? Docs are generally just text, which compresses very well. You could have a separate archive for the common resources (CSS, images, fonts and so on) and then the docs archives would just be compressed HTML files.

Resources are already deduplicated on S3, and all files will be compressed soon^tm. Once we do that storing the prebuilt archives will double our storage requirements. Today we can afford that, but thinking long term we'll want to avoid using too much storage.

[Kapeli]

just _listing_ them took awscli 2 minutes and 43 seconds from the docs.rs server

Taking a long time is fine. For API access you can return a message saying the docs archive isn't ready yet and to try again later, for users trying to download the docs from their browser, show a page saying the same thing, maybe a bit nicer with automatic refresh and so on. With a big enough cache size, you could optimise both CPU and disk space needs.

[jyn514]

It'd also be nice to get the nightly rustc docs without having to build the rustc

@nhynes this is out of scope for docs.rs, we only build user documentation. I'm not sure the right place to open a new issue, maybe https://github.com/rust-lang/www.rust-lang.org/issues ?

[jyn514]

We discussed this internally and this probably won't see action at least until Rust All Hands in March.

Personally, I would like to see #379 implemented and #532 merged before we make any decisions, which would let us see how much storage we'll be using in the future.

[teohhanhui]

Not directly related, but it'd be great to turn docs.rs into an offline-first PWA (Progressive Web App). So the user would still be able to browse the docs they have already visited before even when offline, without having to use a separate website or app.

The same could be done for doc.rust-lang.org

[Kixiron]

That can be achieved with cargo doc to build local crates and rustup doc for the book, std, and everything else on doc.rustlang.org

[teohhanhui]

Unfortunately, that's not the best UX. As a user, I want to be able to keep using the familiar links I've already visited before from my browser history, and have them still work while offline. Since, let's be honest, we usually discover things using the search engine, not start by browsing through / searching inside the docs.

Another alternative is a browser extension to redirect online version -> offline version, similar to what the IPFS Companion extension does.

For example:
https://doc.rust-lang.org/std/sync/struct.RwLock.html -> file:///home/teohhanhui/.rustup/toolchains/stable-x86_64-unknown-linux-gnu/share/doc/rust/html/std/sync/struct.RwLock.html

[jyn514]

I opened https://github.com/rust-lang/docs.rs/issues/845 since this is not related to downloadable docs.

[scouten]

As a long-time Dash user, I would like to reawaken this thread. I've recently come to working in Rust after three years of working in Elixir. The Elixir community managed to solve exactly this problem. The integration with Dash and third-party libraries is IMHO part of the delight of working in that language environment.

I recently completed the Rust 2021 survey and this was my #1 complaint about transitioning into Rust work. I hope the Rust community will take this more seriously.

[jyn514]

Don't worry, we haven't forgotten about it. This is currently blocked on https://github.com/rust-lang/docs.rs/issues/1004; if you're interested in helping out, that would be a good place to start.