Rust: Host compiler documentation

[brson]

The API documentation for the compiler is useful, and only available online via @Manishearth. It should be hosted by rust-lang.

Need to decide where on the doc server they are hosted - probably separately from the end-user docs; ensure the makefiles can produce them in a separate location from the user docs; add the build rule to the dist builders; make buildbot upload them to the right place.

cc @edunham

UPDATE: Mentoring instructions available here.

[steveklabnik]

This is a dup of https://github.com/rust-lang/rust/issues/28309, but gets to the heart of the issue, so I'm going to close it in favor of this one.

I've been talking with @edunham about this a few times over the last few weeks, I think the plan we discussed was a /internals path.

[edunham]

@Manishearth how are you currently generating the contents of https://github.com/Manishearth/rust-internals-docs ?

[Manishearth]

./configure --enable-compiler-docs; make docs. I was considering at some point to make it turn on private docs as well (using rustdoc --passes)

[steveklabnik]

This probably belongs on the forge. /cc @nikomatsakis

[nikomatsakis]

cc @mgattozzi, who is working on this.

[nikomatsakis]

cc @Manishearth, who has been hosting docs as a generous service of late. =)

oh, they were already cc'd =), sorry

[mgattozzi]

I mentioned in #42359 what I was doing and the status of things

[SimonSapin]

The "compiler plugins developers" part of the audience for this is shrinking with procedural macros 1.1 / 2.0, but this can still be useful for people working on the compiler or on custom lint plugins.

In my case, I’d use these docs when one of Servo’s lint plugins breaks in a compiler update. So to be useful they need to stay up to date with master or Nightly, and aren’t updated only when someone remembers to do it

[mgattozzi]

Since it's a Travis build we can set it as a cron job to reupdate everything daily

[nikomatsakis]

It would be really great to see some action on this in the near term. Any idea how much work it is to make this happen?

[mgattozzi]

Looks like someone opened up an issue on rust-forge and I pointed them towards my previous work. I've been a bit busy and haven't been able to do much work on it. My branch I point to there just needs to be able to build rustc and it should work once that happens. The problem was getting Travis to have the right deps to build it in the first place.

[Michael-F-Bryan]

I'm currently using GitHub pages to host the docs ([repo]/[rendered]) and have a cron job set up to run when I'm asleep and when I'm at work.

This afternoon I rewrote the old bash script I'm using into a more legit Rust program... Would the program I've made be useful for this?

[Manishearth]

I think the ideal solution would be one where the nightly build does it. If this is not part of the nightly build, the doc build will break, and these days it has been breaking _all the time_.

[Michael-F-Bryan]

@Manishearth have you got any idea what I'd need to do to integrate internal docs generation into the nightly build?

As it is, my rustc-internal-docs program should almost work out of the box for our needs. You'd just need to add in some alternate mechanism for stashing away the generated docs.

[Manishearth]

I don't know how nightly dist works, so no.

[Mark-Simulacrum]

Nightly dist just pushes existing artifacts which are built on every PR. I think that we can enable the compiler docs on the aux builder to make sure that they work before merging each PR; I hope that the only thing that needs to be done for that to happen is passing something like --enable-compiler-docs to the aux configure run (look into src/ci/docker/aux...) but @alexcrichton may have more comments on any problems that could cause.

[Manishearth]

IIRC the current system is such that a nightly dist build is equivalent to a regular CI build, yes? We should probably keep that property since it avoids broken nightlies (and forces nightly breakers to get fixed before landing)

[alexcrichton]

I don't think that this is trivially fixable but that being said I also don't think that this will be too too hard to fix. The steps to fix this would look like:

• [ ] Add support to rustbuild for a "rustc docs" component tarball. For comparison you can see how the standard library docs component is created. When doing this we'll have to make sure, however, that rustc docs go into a different directory than the standard library docs. This may also want to mirror how the standard library docs are generated
• [ ] Configure the dist-x86_64-linux builder to produce compiler documentation. When you run this builder it show now also produce a rustc-docs-....tar.gz in addition to rust-docs....tar.gz.
• [ ] Make a PR with these changes. Now every single PR will produce and upload compiler documentation for the x86_64-unknown-linux-gnu target.
• [ ] Next rust-central-station will be updated to deploy this documentation to S3. You can see how the standard library works today. Here we'll probably just want to change the upload_dir option to something like nightly-rustc or something like that. I think we'll also ony want to publish this documentation for the nightly channel.

After all that'd done and deployed we should have (a) gated builds on compiler documentation and (b) continuous deployments of https://doc.rust-lang.org/nightly-rustc

[kennytm]

Configure the dist-x86_64-linux builder to produce compiler documentation. When you run this builder it show now also produce a rustc-docs-....tar.gz in addition to rust-docs....tar.gz.

According to #45366, building the compiler docs now require a full-bootstrap compiler, meaning we can only do it on the x86_64-gnu-full-bootstrap builder. (Alternatively we could improve the copy-stage1-to-stage2 procedure so that a non-full-bootstrap builder could work?)

[Mark-Simulacrum]

No, you just need a stage2 compiler, no need for full bootstrap. The requirement is that target libraries and sysroot libraries are identical due to how proc macros work today.

[davidtwco]

Re-opening as this still depends on this rust-lang/rust-central-station#40 to land.

[SimonSapin]

https://doc.rust-lang.org/nightly/nightly-rustc/rustc/ is up. Should it use RUSTDOCFLAGS="--document-private-items"?

[davidtwco]

@alexcrichton @SimonSapin I agree that this would be very useful, I've submitted a PR that should handle this: #49654.

[kennytm]

If we are documenting private items, I hope rustdoc can clearly indicate that an item is private (or pub(crate) or whatever restriction). For instance, if I'm working in the rustc_mir crate, I would not want to know any private detail about the syntax_pos crate.

Also, I hope there's a -Z flag to tell rustdoc to shut up about the rustc_private feature, because we all know the compiler API is unstable 😄.

cc @rust-lang/docs

[Manishearth]

I've long wanted --document-private-items to produce a "Private" checkbox that lets you toggle private items as well. Maybe I'll work on that if @QuietMisdreavus thinks its a good idea

[mark-i-m]

Also, I hope there's a -Z flag to tell rustdoc to shut up about the rustc_private feature, because we all know the compiler API is unstable smile.

I would appreciate that, since those little yellow boxes take up a lot of space...

EDIT: and all of the [Experimental] notes everywhere...