Rust: Tracking issue for RFC 1977: public & private dependencies

[withoutboats]

This is the tracking issue for rust-lang/rfcs#1977 - public and private dependencies in cargo.

Blocking stabilization:

• [ ] Determining the impact & migration path
• [ ] Implementation
  
  
  
  • [ ] [Command to update Cargo.lock to minimal versions](https://github.com/rust-lang/cargo/issues/4100)
  
  
  • [ ] Make cargo publish use the minimal versions allowed by Cargo.toml
  
  
• [ ] Documentation

[Boscop]

Maybe it's useful to look at how this problem is approached in Haskell.
Here are some relevant slides: https://wiki.haskell.org/wikiupload/b/b4/HIW2011-Talk-Loeh.pdf

Here is an example of a subtle issue that occurs with private dependencies:

Btw, I found it through this post: http://harry.garrood.me/blog/purescript-why-bower/

[Eh2406]

Would Rust2018 be a good opportunity to make the braking changes to Cargo.toml?

[SimonSapin]

Maybe. Though such changes should be designed and proposed soon, have an easy migration path, and preferably only affect a minority of existing crates.

[Nemo157]

Thought I just had related to ecosystems like futures that provide a "core" stable crate along with less stable utilities re-exported from a "facade": items from a public dependency re-exported through a private dependency should be treated as public. (This is likely how the implementation would behave anyway, but having a testcase for this would be nice). Basically how I see using a tightly coupled ecosystem like that work is that you would declare both the "facade" crate and "core" crate as dependencies, but only the "core" crate as public:

[dependencies]
futures = { version = "10.17", public = false }
futures-core = { version = "1.2", public = true }

But then, in code you would never import items from the "core" crate, instead you would simply have public APIs that return types which have been re-exported through the "facade" from the "core" crate and rely on the compiler to warn you if you accidentally use something that didn't originate in the "core" crate.

[Eh2406]

I don't quite understand something about your example. Can you elaborate on what the Toml and the code in the futures crate look like?

[Nemo157]

@Eh2406 the setup's a little annoying to just explain, so here's a full compiling set of example crates: https://github.com/Nemo157/rust-44663-facade-example

In this example futures-core exports a single trait Future, futures re-exports this trait from futures-core along with a "utility" Foo.

Now mylib wants to use futures, including returning some Futures from its public API. But futures itself intends to have breaking changes to update utilities, users that care about interoperability should only use types from futures-core in their public APIs. Even as new major versions of futures are released they will all link against the same version of futures-core so the underlying traits will be interoperable.

So, to have the compiler enforce this restriction the developers of mylib want to use the public/private dependencies feature, they have both futures and futures-core as dependency but only futures-core marked as public. But for ergonomics it's nicer to just use the re-exports directly from futures, instead of having to remember which parts of the API need to be imported from futures and which from futures-core.

[Aaron1011]

I'd like to work on this.

[Eh2406]

Thanks for offering to work on this! @alexcrichton, do you have any mentoring advice?

I can at any time get a branch ready for the resolver part of cargo, it will be good enough to start experimentation, although not good enough to be ready to stabilize.

[alexcrichton]

Sure! @Aaron1011 so this is split roughly into two features, one being rustc knowing what crates are in the "public interface" and a second being the support in Cargo's resolver to make various decisions differently. Are you interested in one of those in particular?

[Aaron1011]

@alexcrichton: I've started working on the rustc part in a branch: https://github.com/Aaron1011/rust/commits/feature/pub-priv-dep, and I plan to submit a (WIP) PR soon. I might also be interested in working on the cargo part as well.

[Eh2406]

@Aaron1011 please cc me when you make that PR. I know nothing about the internals of Rust, so will be of no help, but I do want to follow the discussion.

I will redouble my efforts to get the Cargo's resolver part mergeable. I have been enjoying ( and hating ) it because it is the hardest algorithmic problem I have ever worked on. I would be happy for some help, perhaps a new perspective will get me un-stuck.

[alexcrichton]

@Aaron1011 ok and it seems like you're off to a great start! I'd be up for reviewing that PR, and if you've got any questions along the way just let me know

[Nemo157]

There was another usecase for this mentioned on i.rl.o, if cargo doc had a form of "local development" mode to build the documentation you care about when working on a crate, this would allow that mode to know exactly which dependencies it needs to produce the documentation for (all direct dependencies + their public dependencies).

Similarly, if you're self-hosting documentation for a crate somewhere you may want to be able to generate a documentation bundle including the crate and all its public dependencies so that it is fully self-contained.

[petrochenkov]

The current command line syntax for passing private and public extern crates to rustc is

--extern-private name=PATH
--extern name=PATH

(this differs from what was written in the RFC).

I think the syntax would benefit from being slightly more general

--extern:modifier name=PATH

with private and public extern crates being written like this

--extern:priv name=PATH
--extern:pub name=PATH
--extern name=PATH // OK, `pub` is the default

(We may also want other modifiers, or multiple modifiers (--extern:m1:m2) in the future.)

[Aaron1011]

@petrochenkov Do you want me to modify my PR to use that syntax instead?

[petrochenkov]

@Aaron1011
A separate PR would probably be better, this is a novel syntax in general, it may require an FCP etc.

[Aaron1011]

The Cargo frontend of the implementation has been merged

Things still left to do:

• Decide on how public/privates dependencies and dependency naming will interact.
• Merge https://github.com/rust-lang/crates.io/pull/1685 (once the above issue is resolved)
• Decide on a plan to rollout the change to cargo public (picking the minimal versions from Cargo.toml)

[Eh2406]

The discussion of the first point is continuing at https://github.com/rust-lang/cargo/issues/6129#issuecomment-464845052

minimal versions is being tracked at https://github.com/rust-lang/cargo/issues/5657. It is implemented, but is not wide enough used to be stabilized as a cli flag. I doubt we can add it to cargo publish, without a magere rethink.

[ehuss]

This is now available on the latest nightly (2019-05-07) if anyone wants to experiment with it (docs). Just add cargo-features = ["public-dependency"] to the top of Cargo.toml of your project to get things started. You can then look at the warnings produced by rustc to understand which dependencies are exposed in your public API.

[dekellum]

I like this feature! The produced warning messages were very clear. In 1 out of 5 crates trialed, I decided based on the results to make 2 dependencies private that were needlessly public. This makes me think there is value and wonder if there is an opportunity to stabilize just the information collection part of this sooner (public dependency attribute, rustc warnings, uploads to the crates.io index) and the dependency resolution changes (warnings or any resolution errors) later? In particular, wouldn't that allow progress on information collection and utility without serializing on rust-lang/cargo#5657? Or perhaps the tracking details just need to be updated here?

Some additional minor feedback below:

% rustc --version
rustc 1.36.0-nightly (a784a8022 2019-05-09)
% cargo --version
cargo 1.36.0-nightly (759b6161a 2019-05-06)

• If you label something public=true that isn't actually publicly exposed, you currently don't get a warning. Would a warning or lint for that case be an appropriate future addition?

• With the cargo feature enabled, cargo doc currently fails, due to rustdoc not understanding the --extern-private flags as passed. Workaround is to temporarily remove the feature _and_ all public attributes from deps, or replace --extern-private with --extern and run rustdoc manually.

cargo doc --no-deps --open
 Documenting barc v1.2.0 (/home/david/src/body-image/barc)
error: Unrecognized option: 'extern-private'

error: Could not document `barc`.

Caused by:
  process didn't exit successfully: `rustdoc --edition=2018 --crate-name barc barc/src/lib.rs --color always -o /home/david/src/body-image/target/doc --cfg 'feature="body-image"' --cfg 'feature="brotli"' --cfg 'feature="default"' --cfg 'feature="memmap"' --cfg 'feature="mmap"' --cfg 'feature="olio"' -L dependency=/home/david/src/body-image/target/debug/deps --extern body_image=/home/david/src/body-image/target/debug/deps/libbody_image-f6307513eafde4da.rmeta --extern-private brotli=/home/david/src/body-image/target/debug/deps/libbrotli-914be93b87d00a74.rmeta --extern-private bytes=/home/david/src/body-image/target/debug/deps/libbytes-60c7a2c9551c05c4.rmeta --extern-private flate2=/home/david/src/body-image/target/debug/deps/libflate2-3410ad175205616a.rmeta --extern http=/home/david/src/body-image/target/debug/deps/libhttp-1bcf97f938c6fdd7.rmeta --extern-private httparse=/home/david/src/body-image/target/debug/deps/libhttparse-b04e539fbadba356.rmeta --extern-private memmap=/home/david/src/body-image/target/debug/deps/libmemmap-b29e83fdba55bd43.rmeta --extern-private mime=/home/david/src/body-image/target/debug/deps/libmime-df20e72307836353.rmeta --extern-private olio=/home/david/src/body-image/target/debug/deps/libolio-c7a6754af0be8b56.rmeta --extern-private tao_log=/home/david/src/body-image/target/debug/deps/libtao_log-7da4b359386cbcf7.rmeta --extern-private tempfile=/home/david/src/body-image/target/debug/deps/libtempfile-5c5a144b50e3000b.rmeta -Z unstable-options --cfg barc_std_try_from` (exit code: 1)

I'm happy to elaborate here or elsewhere if its useful.

[Aaron1011]

If you label something public=true that isn't actually publicly exposed, you currently don't get a warning. Would a warning or lint for that case be an appropriate future addition?

That sounds like a good idea to me!

Implementing that will be a little tricky. Currently, rustc isn't explicitly aware of the underlying public-dependency feature - it simply treats all dependencies as public by default. We would want to avoid showing 'unecessary public dependency' warnings to users who don't have the public-dependency feature enabled, as these warnings would be both useless and unfixable.

We would probably need to add a -Z public-dependencies flag to rustc to ensure that we only enable the lint when the cargo feature is also enabled.

Is it necessary to make another RFC for something like this?

[dekellum]

Or the alternative "--extern(_:pub_|:_private_)? name=PATH" syntax proposed by @petrochenkov could also get rustc that detail, if it chose to preserve the distinction for this? It sounds like an implementation detail to me, but what do I know! ☺

[dekellum]

I just tried enabling this feature on another lib crate (also for the purpose of collecting information on public deps) and noticed that examples/*.rs are another interesting edge case. If the examples define any pub items (here really just for purpose of example) that reference the lib crate types, these will result in warnings of the pattern:

warning: type `Foo` from private dependency 'libcrate' in public interface

This is because each example is compiled, including as part of cargo test, with --extern-private libcrate=... Note this dependency isn't written in the Cargo.toml, its just implied for examples. Shouldn't examples be compiled with --extern:pub libcrate=... instead (using newer proposed syntax just for clarity)?

Cc: @Eh2406 this last case (and possibly the one above about cargo doc) might be more specific to cargo?

[petrochenkov]

The feature's implementation doesn't currently work correctly with transitive dependencies.
If we load a crate named foo as a transitive dependency, we still mark it private (or not) by looking at the --extern foo option, even if refers to an unrelated crate and --extern options in general are entirely irrelevant to transitive dependencies.

The dependency private-ness should be written into metadata and read from there for transitively loaded crates.

This brings a question - what if the same crate is private direct dependency and a public transitive dependency (or vice versa)? Should it be treated as private or public?

(I'm currently refactoring code in that area and may fix this issue as well.)

[petrochenkov]

@Aaron1011
By the way, do you still have plans to implement the --extern:modifier syntax from https://github.com/rust-lang/rust/issues/44663#issuecomment-475853173?

[ehuss]

@petrochenkov I like the idea of the modifier syntax, but I'm curious how that would work with getopts. If the intent would be to support multiple modifiers, would every permutation need to be registered as separate flags?

[petrochenkov]

how that would work with getopts.

No idea :)
I didn't look into implementation details, only suggested a syntax that's more generic than --extern-private.

[ehuss]

I have posted a proposal for a new --extern flag syntax at #67074.

[ehuss]

I'm a bit confused about how re-exports are being treated. From the RFC: "Q: Can I export a type from a private dependency as my own?" says NO. However, it looks like various ways of re-exporting do not generate a warning:

• pub extern crate private_crate; or pub use private_crate;
• pub use private_crate::some_item;

Is that intentional?