Cargo: Support producing a "build plan" without executing anything

[alexcrichton]

This is an issue extracted from the discussion over at https://github.com/rust-lang/rust-roadmap/issues/12. The general high-level idea is that Cargo should be able to produce a build plan for the explicit purpose of consumption by another tool. This step would not iteslf execute any work but will likely assume that all the source code is available for consumption.

Once Cargo is able to support this it should be explored to see what this integration would then look like into external build system, such as Buck or Bazel. This may involve writing generators which translates from Cargo's build plan to a Buck/Bazel configuration file, or it may involve more dynamism at build time.

The goal here is to leverage Cargo as much as possible for drawing dependencies between projects and learning how the compiler is executed. This should give us quite a bit of flexibility to continue to implement new features in Cargo (such as the recently stabilized proc-macro) with little-to-no changes in external build tools.

[P-E-Meunier]

I love the idea, and I actually wrote such an external tool relying on Nix, see https://nest.pijul.com/pmeunier/nix-rust

Just parsing Cargo.lock was enough to get reproducible builds on NixOS, without recompiling the world every time. However, just two things still feel hackish:

• I had to guess what the main file of a crate was, and whether the crate was a binary or a library. For some crates (such as "untrusted"), the Cargo.toml simply says lib.name = "untrusted", and there's a single file in the package, also called "untrusted".

• Features. Nix computes a hash of everything used to produce a package, so it can track feature changes easily.

Including these two things in the lock file would solve everything!

[alexcrichton]

Some more notes from various discussions:

• This output should include all inputs necessary to produce such output. Basically this should imply running --emit dep-info on the compiler to get a list of input files
• The output here should be a full dependency graph for everything on the Cargo side of things, with edges between crates everywhere.
• The output should also also list all the outputs of each step, rlibs/dylibs/etc.

[bennofs]

Clang/LLVM has something similar: build_commands.json

[glandium]

FWIW, the compilation database for clang made a big mistake where one needs to parse the command on their own. It should have been a list instead of a string. Please don't repeat that mistake.

[da-x]

This feature is akin to gcc's -M family of flags. For the purity of the solution I suggest to add the following:

• Support for non-existing inputs, i.e. input .rs files that are supposed to be generated prior to the execution of the actual rustc invocation. E.g. outputs of larlpop. Perhaps these external tools could be integrated in the output build-plan so that it covers a greater scope?
• Build plan should not depend on any prior built state (i.e. the target/ directory need not exist and is not read for the build plan to be generated).

[budziq]

It would be ideal if such functionality could be some how triggered/available while in build.rs.
Then relatively hacky testing crates such rust-skeptic could stop parsing unstable cargo .fingerprint internals while building documentation testcases :wink:
https://github.com/brson/rust-skeptic/issues/37

[mshal]

I made an attempt at implementing this as a way of learning Rust. So far it just prints out the info from DependencyQueue and the rustc command-line invocations as Python data structures. With this I'm able to import the data structures into a Python program that writes out Tupfiles, and build a small dummy Rust program (a hello-world with two local library dependencies) with tup.

There are a number of things missing so far, so this isn't ready to land. In particular:

1) build.rs support is not yet implemented. Confusingly, it will output a build step for a build.rs file with the same name as the rustc invocation.
2) The final output of a program is not copied. It will create a rule for target/debug/deps/hello_world-b6a758bc870b5332, but not target/debug/hello_world, for example.
3) Running cargo with --build-plan still creates some directories under target/. This should probably be left to the external build system.

[aturon]

Just a quick note -- the Cargo team is actively hashing out high-level design questions around this and other aspects of build system integration. Hope to have some writeups soon!

[Xanewok]

@mshal awesome that you did some work on this!
If possible, I'd like to help with the implementation as it'd be very convenient to have that in the RLS, since right now we're going to use an approximation to perform our own builds using rustc.

[alexcrichton]

There's an RFC for build system integration now up at https://github.com/rust-lang/rfcs/pull/2136

[mshal]

Sorry for the delay - I've been distracted by other things. I have the latest attempt based off of cargo-0.22 here: https://github.com/mshal/cargo/tree/build-plan-0.22.0

I used 0.22 as the base because that's what we currently use in mozilla-central. I'm able to run cargo with --build-plan against the toolkit/library/rust/Cargo.toml file to generate a plan.json which contains (almost) all the information needed to invoke the rustc commands and build scripts with an external tool. The only missing thing I am aware of is the outputs of a build script, but those can be supplemented externally as a workaround.

It almost works when I run it through tup, but that has other problems with rustc's weird file usage patterns that still need to be resolved. I believe this is a reasonably complete implementation in the cargo side of things however.

I'd appreciate any feedback and improvements! If it would be easier, I could submit it as a pull request.

[mshal]

I forgot to mention, but I did convert the json file to a shell script that just invokes the rustc commands in the right order, and that shell script does complete successfully. So I think that shows it can at least function as a way of exporting commands to an external "build system"

[alexcrichton]

Awesome thanks for the update @mshal! A PR would probably be the easiest thing for review, yeah, so feel free!