Cargo: Cargo needs a way to run an executable out of a build-dependency

[erickt]

As best as I can tell, cargo only builds the libraries from a dependency. This has the unfortunate side effect of preventing build scripts from being able to execute a binary. This is unfortunately preventing @dwrensha's capnproto-rust from being used by a cargo build script to generate files.

The reason why is due to the design of the Cap'n Proto project. According to doc, the way Cap'n Proto does code generation for languages is that there is a driver executable, capnp that parses the .capnp file and generates a schema, which is then passed off to a plugin executable to generate the language specific files. There are 3 options to do this right now:

• build capnproto-rust outside of the project, generate the code and check it into the project.
• have build.rs clone capnproto-rust, run cargo build in there and execute the binary to generate the code.
• modify the upstream project to turn capnp into a library, and use that in a build.rs script. I'm not familiar with that codebase, but it does seem pretty big, so this would probably be a lot of work.

Instead, I think things would be a lot more simple if a cargo dependency could be built to include the executables. Then it would be pretty trivial to do this code generation.

[alexcrichton]

cc #748, a similar issue. I would personally prefer to use libraries for this whenever possible as binaries have the problem of not being able to disambiguate one another. If I have two build dependencies that both provide an executable named foo, then there's no way to differentiate between them.

[dwrensha]

For what it's worth, my current approach for projects that need to do Cap'n Proto code generation is to require that both executables, capnp and capnpc-rust, already be installed on the system. Then I have a build.rs that invokes capnp compile -orust and puts the generated code in $OUT_DIR. And, of course, I add a Cargo dependency on capnproto-rust, which brings in the runtime library required by the generated code. capnp-rpc-rust is an example that follows this pattern.

[dwrensha]

Here's one way I could eliminate the need to install capnpc-rust. I could split the code generation library into its own repo / Cargo package. The new package would still have the binary target capnpc-rust, but it would also have a library target that exposes a function that could easily be called from a build.rs file. That function would execute capnp compile -o/bin/cat and forward the output to the capnpc_rust::codegen module.

So you would still need to separately install capnp, but once you do that you'd be able to stay entirely within Cargo.

[erickt]

@dwrensha: that would be nifty!

I think this would be useful for other things than capnproto of course. We are eventually going to grow projects as big as Ruby on Rails or Django, which will come along with their own administration tools. I like that cargo provides for project level isolation, So it would be a shame if we force people to have to install something in order to use it.

@wycats: I think you've dealt with this in bundler, how do you handle it?

[alexcrichton]

I'm going to close this as working as intended, and I'm also not sure that this has come up recently as a desire for various build scripts.

[comex]

I'm trying to do the same thing. I don't see why there is no story for depending on or installing executables other than manually checking out and building... unless I'm missing something?

[alexcrichton]

Does your build dependency not offer its functionality through the interface of a library? (that is the recommended method)

[comex]

The dependency is bindgen, so yes, but this seems like a waste of time to me... I already have a (hacky) Python script to do some extra processing on the output, so I can't just use the plugin version. I could redo the processing in Rust in order to use the bindgen library (rather than popen), but what is the point of requiring me to do that when the executable is right there in the repo and just needs a way to be compiled?

[alexcrichton]

what is the point of requiring me to do that when the executable is right there in the repo and just needs a way to be compiled?

Calling a library has a few benefits:

1. It is _far_ faster than calling a whole separate proces
2. Avoids conversions between languages through stdout/stderr/args/env vars. You can use native Rust types everywhere (and get compilation warnings, etc)
3. Cargo can manage everything from the upstream dependencies to what architecture it's built for.
4. Cargo can precisely tell you where bindgen is. Otherwise a bindgen executable elsewhere in PATH may cause bugs.
5. Cargo can allow a project to end up running two versions of bindgen if necessary (by saying which links to which)

[torarnv]

@alexcrichton: Not all build dependencies can be made into libraries I think. Eg, https://github.com/stepancheg/rust-protobuf, where the binary is a plugin for protoc. Please consider re-opening.

At the moment Cargo effectively builds all deps with --lib, is that what we want? What if I as a package author know that my package consists of both a lib and a binary, wouldn't it make sense for me to be able to override it in my manifest? Or as a user of package, where I know the package provides more than just a lib, wouldn't it make sense be able to declare in the dependency section of my manifest that the dependency should include more than just the lib target?

[per-gron]

+1 for reopening this ticket. I am also trying to use protobuf (gRPC in my case), and I really don't like that the way people seem to do it is to ask their users to install grpc_rust_plugin and a bunch of other things globally. I would like my crate to be able to have versioned dependencies to these things.

Performance of the build is a pretty poor argument if the consequence is unversioned dependencies.

[simonbuchan]

My use case is embedding the dependency binary in the dependant binary. Am I missing some obvious alternative?