Docs.rs: Configurable index location

[Nemo157]

To support a staging server and other users running their own docs.rs instance against their own registry the index repository used should be configurable.

[Nemo157]

Of course this is a bit more involved than I thought at first.

As well as changing the index this will need to:

1. Change the docbuilder to pull the crate to be built from the alternative registry.
2. Use the api key from the index to determine which API to hit for additional details, and deal with registries that do not support this API.

[jyn514]

Relevant code:

• https://github.com/rust-lang/docs.rs/blob/8fb139b/src/docbuilder/rustwide_builder.rs#L203
• https://docs.rs/rustwide/0.7.1/rustwide/struct.Crate.html#method.crates_io
• https://docs.rs/crate/rustwide/0.7.1/source/src/crates/cratesio.rs

This will require changes to rustwide.

[Nemo157]

I've got ~everything except the rustwide changes done in https://github.com/rust-lang/docs.rs/compare/master...Nemo157:alternative-registry, though I'll likely split off some cleanup I've done there to make a small PR beforehand.

[repi]

This would be great to have!

We currently use a private crate registry on cloudsmith.io for crates that are private or not fully ready yet to be open sourced. https://github.com/EmbarkStudios/rust-ecosystem/issues/47

[jyn514]

This would be great to have!

We currently use a private crate registry on cloudsmith.io for crates that are private or not fully ready yet to be open sourced. EmbarkStudios/rust-ecosystem#47

Note that this would still not allow using crates from a private registry on docs.rs. Rather, this allows you to host your own docs.rs instance which uses a private registry.

[repi]

Note that this would still not allow using crates from a private registry on docs.rs. Rather, this allows you to host your own docs.rs instance which uses a private registry.

Yes that makes sense 😀 and was my expectation & use case as well.

[Nemo157]

Another point to deal with here: authentication. Probably just needs documenting that the user account that docs.rs is running under must have a git credential helper setup correctly (I assume this works with libgit2 correctly? In my own alternative registry experiments I've had to use the net.git-fetch-with-cli option because libgit2 doesn't support the "dumb" http transport, but cloudsmith don't mention having to enable this for basic auth).

---

@repi from looking through your linked issues I see that cloudsmith support at least the subset of the crates.io API required for publishing. Do you know if there's any documentation on what other parts of the API they support? I can only find documentation on their own API, not on the package manager specific APIs. (We pull some other details from the crates.io API for each build).

---

Another-nother point, we link to crates.io in the templates. It is impossible to derive a correct url to link to for an arbitrary registry, so we probably want a separate config item for this (or 2, one to the homepage, one to create a url for each crate). For crates.io we'd get away with just a prefix we append the crate name to, though maybe it'd be better to take a template string that we insert the crate name into.

[repi]

@Nemo157 haven't seen any details of what part of the Cargo Registry API cloudsmith.io supports.

But @lskillen at cloudsmith should know and hopefully can help

[Kixiron]

There's some various decoupling that I need to do in templates, namely not making the https://docs.rs domain hard-coded, I can add in the registry changes as well, I can expose both via a Tera function

[lskillen]

@Nemo157 I can certainly find out for you what parts we support, although I suspect that (beyond publishing) our implementation is more focused on the git/fetch handling than anything crates.io specific. Is there a good reference somewhere on the calls that docs.rs makes so we can check compatibility? (just realising this is the inverse of your question, ha!)

There is absolutely room for us implementing something if needed though. :) Also, if there's anything that we could do, to make this easier, we're more than happy to help out. Sounds like a fantastic idea!

[Nemo157]

@lskillen The current calls we use are /api/v1/crates/{}/versions and /api/v1/crates/{}/owners, the latter at least is documented in https://doc.rust-lang.org/stable/cargo/reference/registries.html#web-api. I'm planning to make it so that we don't _need_ these to be supported, but was wondering if they might already be supported by other registries.

[apoclyps]

Hi @Nemo157 , thanks for outlining the calls and docs. I've taken a look into the existing implementation and currently, the calls mentioned are not yet supported. As Lee said, the Cloudsmith implementation is focused primarily around publishing and git/fetch however if support is required, we are more than happy to look into providing the support required; is this something you need? If so, I can raise it with the team to be discussed during the next planning session on Monday (8th). Alternatively, we can provide data in other ways, if needed by the new approach.

[jyn514]

@apoclyps We instead decided to gracefully fallback to loading dummy information if it's not available from the registry (https://github.com/rust-lang/docs.rs/pull/813/commits/2cbf25226d6b383992c7f74bea9ae4c83dc714e4). So your docs would still show up even if you didn't support those endpoints, they just wouldn't have information about

• owners
• authors
• release time
• whether the crate was yanked
• number of downloads

If you don't need that information for your use case, you don't have to add it to your registry.

[repi]

Cool! That info is nice, but less important for our use case of generating docs.rs docs on a custom instance for our private registry (cloudsmith)

[jyn514]

Status update: this is still waiting on the rustwide changes. Currently we use https://docs.rs/rustwide/0.10.0/rustwide/struct.Crate.html#method.crates_io, which doesn't allow customizing the registry. For us to make more progress, rustwide needs a new API that allows using custom registries for builds.

[Nemo157]

Oh yeah, I implemented that a while ago, and had tested using it inside docs.rs, but never got round to figuring out how to setup testing of it.

[l4l]

Hmm, then it seems I did it wrong then. Custom registries looks much more complicated than crates.io : Do you plan to continue working on that issue, @Nemo157?

[Nemo157]

Yeah, custom registries have a custom URL template, so you can't just construct the download URL like the crates.io implementation does; you have to download the index and read it's config.

I'm not planning on working on it again soon, so if you feel like taking what I've done and get it actually merged feel free (if not I will probably get back to it sometime).

[l4l]

Nice, thanks for clarifying things out! I try to finish that out asap.

[jyn514]

1140 was reverted in https://github.com/rust-lang/docs.rs/pull/1147.