Rust: Tracking issue for rustbuild, the alternate build system for rustc

[alexcrichton]

I'm opening this up as a tracking issue for all work that needs to happen to get rustbuild, our new build system, turned on by default. The end goal here is to completely jettison our thousands of lines of makefiles and configure scripts for tons and tons of Rust code! Initial support has landed in #31123, but we've still got quite aways to go!

Here's a list of items left that need to be implemented, and help is always appreciated when tackling these! If you want any clarifications or any help on any of these, feel free to reach out to me (acrichto) on IRC.

Documentation

work in progress

• [x] nomicon - #31713
• [x] rustbook - #31713
• [x] error index - #32133
• [x] crate docs - #32133
• [x] compiler docs - #32133
• [x] standalone docs - #31713
• [x] style guide - #31713

Testing

• [x] crate tests - #33282
• [x] crate doctests - #33282
• [x] rustbook/nomicon tests - #33092
• [x] standalone documentation tests - #33092
• [x] error-index tests - #33092
• [x] run-pass - #32755
• [x] run-fail - #32755
• [x] compile-fail - #32755
• [x] parse-fail - #32755
• [x] codegen - #32755
• [x] debuginfo - #32755
• [x] codegen-units - #32755
• [x] pretty - #32755
• [x] run-pass-fulldeps - #32755
• [x] compile-fail-fulldeps - #32755
• [x] run-pass-valgrind - #32755
• [x] rustdoc - #32755
• [x] run-make -- move maketest.py to rust as well - #33093
• [x] Android testing - ship things to the emulator - #34567
• [x] multi-platform testing, handle multiple hosts and multiple targets - #34567
  

  Porting

These should all be verified to work and then ideally we'd add a builder verifying they continue to work!

• [x] Android - #34567
• [x] iOS
• [x] various cross compiled linuxes
• [x] musl-x86_64

configure script

• [ ] Migrate compiler version detection to Rust (maybe optional?)

• [ ] Ensure all tool detection is moved into Rust

  Misc

• [x] Add build automation to pass --enable-rustbuild, but not run tests - https://github.com/rust-lang/rust-buildbot/pull/61

• [x] Verify that multi-host builds work - #31884
• [x] Verify sha256 of downloaded nightly rustc and nightly cargo - #32926
• [x] Migrate tidy to rust - #32590
  
  
  
  • binary check WIP
  
  
• [x] Migrate licenseck to rust - #32590
  
  
  
  • work in progress
  
  
• [x] Migrate featureck to rust - #32590
  
  
  
  • work in progress
  
  
• [x] Migrate errock to rust - #32590
  
  
  
  • work in progress
  
  
• [x] make clean - https://github.com/rust-lang/rust/pull/31591
  

  make install + make dist

These are pretty huge portions of the make files, they may take _quite_ some time to migrate over, and it's probably fine to keep them around for now. @brson's plans with multirust-rs and reusable library components may become relevant here as well.

This may be subdivided into a bunch of smaller tasks over time!

• [x] make dist - https://github.com/rust-lang/rust/pull/32237

Far future (not required)

• [x] Support src/bootstrap/bootstrap.py being called directly (e.g. on MSVC where make isn't available)
• [ ] verify gpg signatures of downloaded nightlies - #34598
• [x] Support building rustc for architectures that can themselves not be run. For example a x86_64 host should be able to build an arm compiler (e.g. a full installation tarball) - #31884
• [ ] Move building LLVM out of rustbuild (not actually sure if this is possible)
• [ ] Move building compiler-rt out of rustbuild (eases cross-compiling libstd) - #34400
• [ ] migrate htmldockck to Rust

[alexcrichton]

Also as the most bike-sheddy thing, I'm not a huge fan of the term "rustbuild", so I'd also be fine calling this something cooler like "rust laser build"

[steveklabnik]

This is a very simple thing, but I hit control-C while it was downloading stage0 and because the file wasn't valid, it just bailed out and I had to remove it manually. I'm not sure if deleting it and retrying is the right thing, or if this is just going to be a papercut generally.

[alexcrichton]

Sounds like a bug! (added a checkbox)

[nodakai]

By going from Makefile to build.rs, are we not throwing away the notion of dependency graph?

Please enable resuming on curl if possible, for people with narrowband.

[alexcrichton]

No, I would expect any build system to take dependencies into account. It's an implicit assumption of mine that any sort of dependency tracking done by the current build system _must_ be done by rustbuild as well.

[steveklabnik]

Regarding the documentation features... maybe we should try to land https://github.com/rust-lang/cargo/pull/2256 and then they could just be their own crates?

[alexcrichton]

I suspect that may not end up helping too much unfortunately, most of our docs are rustbook-based rather than rustdoc-based.

[steveklabnik]

Oh duh. I should, uh, know that :sweat_smile:

The new book is going to use mdBook, and I _think_ it should just be a drop-in replacement. Maybe it's time to kill rustbook for real?

[alexcrichton]

It probably doesn't matter either way in terms of dependency management and instrumentation of the compiler building mdbook/rustbook, so I suspect switching over wouldn't change things too much really

[steveklabnik]

@azerupi is okay with mdbook becoming a dependency https://github.com/azerupi/mdBook/issues/105#issuecomment-183518235

I will poke at that docs branch sometime next week.

[japaric]

I'm going to look into building compiler-rt as a crate using cargo instead of having the bootstrap binary build it. This should get us closer to being able to (cross) compile the std crate with cargo, i.e. just cd src/libstd && cargo build.

[alexcrichton]

(added some bullet points to that effect)

[shepmaster]

While trying to help out the AVR-Rust fork, I started using rustbuild. I've been able to compile LLVM, but then realized that I had to change the configuration of LLVM. I edited src/bootstrap/build/native.rs to add a new value to the LLVM_TARGETS_TO_BUILD definition. When I ran make again, LLVM was not rebuilt.

[shepmaster]

As a work-around, I've deleted all the llvm-config executables in my build directory. This seems to trigger a _complete_ rebuild of LLVM. I have a feeling that this will make me a sad :panda_face: when it comes time to make single bits of change to LLVM and incorporate them into Rust. Perhaps this is what the custom LLVM option is made for?

[alexcrichton]

@shepmaster ah yeah right now the build system doesn't track changes to itself. I personally found that to be the most annoying part of the makefiles and I was glad to _not_ reimplement that, but I suspect we'll have to add that eventually.

[alexcrichton]

Also yeah, the handling of LLVM specifically may not be as great as it needs to be. Currently it unconditionally always configures via cmake before building with cmake. There's currently no entry point for "run a build in this directory but don't reconfigure". I'd probably recommend using an external LLVM for now, and that should be much easier through the config.toml configuration rather than ./configure itself

[Zoxc]

libc and alloc_system should be removed from liballoc's dependencies. With that change I can cross-compile a freestanding sysroot by depending on libcollections, https://github.com/AveryOS/avery/blob/master/sysroot/Cargo.toml

[bstrie]

Does this imply that any of the zillion crates under src are going to be split out onto Crates.io?

[hanna-kruppe]

@bstrie Not in the near future, since the old buikdsystem will still be around for a while. Once the Makefiles are gone, I expect that at least the crates that already have an official (owned by rust-lang or rust-lang-*) crates.io equivalent will be phased out in favor of of depending on the crates.io versions.

[alexcrichton]

Ok, almost all work is now done on this metabug, and I believe that rustbuild is now at feature parity with respect to the set of the makefiles that we run on all our automation (dist bots, auto bots, etc). As a result I'm going to close this out as it's all "basically done" (yay!).

There's a number of issues still tagged A-rustbuild, and we can use that label for tracking issues with rustbuild moving forward.