Cargo: Publish lockfile for binary crates

[matklad]

Implementation PR: #5093

Summary:

For binary crate, add

cargo-features = ["publish-lockfile"]`
[project]
publish-lockfile = true

to Cargo.toml. cargo install will then use the lockfile of binary crates.

Steps:

Feature ready for stabilization

Stabilization TODO:

• move docs from https://github.com/rust-lang/cargo/blob/5845464cf92a348f95dc6b43133c017c37fcb684/src/doc/src/reference/unstable.md#publish-lockfile to appropriate location.

• remove feature gate

[polyzen]

Any ETA on this, or anything I could do to help?

[dwijnand]

Continuing from https://github.com/rust-lang/cargo/issues/6556#issuecomment-455209954:

I think it would be good to start looking at fixing any known issues and moving towards stabilizing that. Although if it is optional, I'm not sure if most people will use it, so it may not help here. I wonder if it was considered if it should always be published, and let the installer decide whether or not to use it?

Why _shouldn't_ it be published? Why would an installer _not_ want to use it?

[Nemo157]

One reason would be if there were security issues with the any of the versions pinned in the lock file, even if those crates were yanked and semver compatible releases made the lock file would still cause the old versions to be used until the binary crate released a new patch version with an updated lock file.

[ehuss]

Why would an installer _not_ want to use it?

My only thought is that using the latest versions will bring in the latest and greatest fixes. But maybe the frequency of "cargo install fails" outweighs that? Maybe it could be the default to always use Cargo.lock if available, and have an option to ignore it? 🤔

This bit me recently where mdbook would fail to install (https://github.com/rust-lang-nursery/mdBook/issues/852).

[dwijnand]

I guess we could dual cargo build --locked and have cargo install --unlocked 😁

[dwijnand]

Btw, there's more detail in the chat in #5093.

[Nemo157]

I should say I'm for having this feature working, I have encountered the same failure in a binary crate I manage as well.

I think having cargo install --unlocked along with a prominent warning when cargo install with a lockfile brings in a yanked package could be enough to mitigate the security issue. If that's not enough, maybe even require cargo install --allow-yanked or something so that the user must affirmatively allow it.

[polyzen]

On January 17, 2019 10:27:49 AM EST, Dale Wijnand [e-mail] wrote:

Continuing from
https://github.com/rust-lang/cargo/issues/6556#issuecomment-455209954:

I think it would be good to start looking at fixing any known issues
and moving towards stabilizing that. Although if it is optional, I'm
not sure if most people will use it, so it may not help here. I wonder
if it was considered if it should always be published, and let the
installer decide whether or not to use it?

Why _shouldn't_ it be published? Why would an installer _not_ want to
use it?

https://doc.rust-lang.org/cargo/faq.html#why-do-binaries-have-cargolock-in-version-control-but-not-libraries

--
Best,
polyzen

[Nemo157]

@polyzen that's related to libraries, this issue is only about installing binaries (a crate can contain both a library and a binary, but my expectation is that even if it contains a lockfile that will be ignored when used as a library).

[polyzen]

I was responding to @dwijnand's original question, but (on FastHub) it doesn't seem like his message got quoted.

[dekellum]

For some reason before finding this issue, I thought including the lock file was now default (and therefore stabilized) for all bin crates publishing. Its awkward that this is nightly only, as enabling it would normally interfere with CI (using rust versions other than nightly). However I'm already publishing some crates via a dedicated, publishing-specific git branch, so I can disable CI (or only CI on nightly) in that branch and still use it. Yah!

@matklad's top _Summary_ has a minor typo:

cargo-features = ["publish-lockfile"]`
- [project]
+ [package]
publish-lockfile = true

I do agree with @Nemo157, that minimally, cargo install --allow-yanked should be implemented, and ideally _also_ the cargo install --unlocked flag, including an initial warning that the lock file was downloaded/found and used in the first place. Only the former would seem necessary for the security concern and stabilizing this.

Thanks!

[dwijnand]

I was responding to @dwijnand's original question, but (on FastHub) it doesn't seem like his message got quoted.

Thanks, but that link's about version control, though, while this issue is about publishing crates.

[dwijnand]

Its awkward that this is nightly only, as enabling it would normally interfere with CI (using rust versions other than nightly).

Agreed, it's a shame that one cannot opt-into cargo unstable features on stable.

[polyzen]

Thanks, but that link's about version control, though, while this issue is about publishing crates.

Crates used to publish all files checked into version control, including lockfiles. Due to the issue with libraries, people would exclude their libraries' lockfiles from their repos.

Eventually lockfiles were filtered out from crates so people wouldn't have to do that. Now we have publish-lockfiles so people can use --locked for binary crates.

[dekellum]

Actually, if crates.io security advisories were implemented, along the lines of:

https://internals.rust-lang.org/t/pre-rfc-reviving-security-advisories-in-crates-io-rfc-pr-1752/9017

..then this would probably cover the security concern as well, and in a more general and complete way.

cc: @tarcieri

[tarcieri]

This seems related/relevant: including Cargo.lock in binary artifacts (cc @Shnatsel)

https://github.com/rust-secure-code/wg/issues/14

One of the goals of the above is definitely to allow RustSec audibility of compiled Rust artifacts, and it seems like that might be useful for solving this particular problem as well.

[dekellum]

In setting up to try and use this feature (above referencing commit), I note that:

cargo-features = ["publish-lockfile"]

...is retained in (re-generated) Cargo.toml that is included in the package. In testing a local install at least, I get the following error:

% cargo +1.31.0 install --path ./barc-cli
error: `/home/david/src/body-image/barc-cli` is not a crate root; specify a crate to install from crates.io, or use --path or --git to specify an alternate source

Caused by:
  failed to parse manifest at `/home/david/src/body-image/barc-cli/Cargo.toml`

Caused by:
  the cargo feature `publish-lockfile` requires a nightly version of Cargo, but this is the `stable` channel

The first error line is confusing or perhaps a different issue, but I think I understand the "Caused by" part. Before I go through the trouble of publishing a potentially broken patch release of barc-cli, could anyone confirm/deny if the same error will occur when attempting to install such a published crate on a stable-channel rust via crates.io? If this is the case, was it intentional that the feature is limited to publishing crates (using a nightly), _that can only be consumed by a nightly rust_? I find that surprisingly narrow of a target. It would be helpful to clarify that, here, ether way.

[ehuss]

that can only be consumed by a nightly rust

I believe this is correct. It's a general problem of testing any nightly feature.

The error message does look odd, I'll take a look and see if it can be easily fixed.

[dekellum]

Regarding the first error line, note: my root is a cargo workspace with 4 crates, only one of them, this bin crate.

And thanks for confirming—I can't workaround the nightly-only "cargo publish" aspect, just by configuring and publishing with a different git branch. Seems like cargo could really use the ability to make _some_ non-stable features, like this one, still work on stable cargo (with the opt-in feature flag). Lacking this, and if stabilization is going to be further delayed, it would make this usable if cargo package stripped this package/publish only feature out of the re-generated Cargo.toml (but left the Cargo.lock in place.)

[dwijnand]

Seems like cargo could really use the ability to make _some_ non-stable features, like this one, still work on stable cargo (with the opt-in feature flag).

I think the risk is that once it's accessible in stable it'll be relied upon.

Lacking this, and if stabilization is going to be further delayed, it would make this usable if cargo package stripped this package/publish only feature out of the re-generated Cargo.toml (but left the Cargo.lock in place.)

That would require the guarantee that crates published by nightly cargo can be consumed by stable cargo. I _think_ that's ok, but I don't know if it is.

But that initial cargo install --path error message is definitely wonky.

[dekellum]

Well, if its _not ok_ to publish via nighty cargo a crate with MSRV to some stable version, then that seems like a much bigger problem than (lack of) this feature! Isn't that how many or most crates are published? The tendency to develop on nightly, CI and publish for nightly+stable+MSRV?

[dwijnand]

Yeah, I think we do this by making cargo forward-compatible first, let it ride the trains to stable, then make the change to master/nightly cargo. For example what we're doing in #6180.

[nrc]

We discussed this today in the Cargo meeting. We do think it is good to stabilise the core concept soon, we want to change a few aspects of the UX:

• by default, publishing a binary would include the Cargo.lock file. This could be overridden in the Cargo.toml.
• by default, installing a binary does not use the Cargo.lock file, but this could be overridden with a command line argument
• Cargo.lock files should be verified as part of the pre-publication verification step.

[Shnatsel]

I would strongly oppose this because this makes propagating security updates to the ecosystem effectively impossible, at least until a first-class mechanism for security updates is in place.

Right now the user is not notified about security updates, but at least cargo install pulls in latest versions with presumably latest fixes. If binary crates ship with a specific Cargo.lock, propagating a security update to a library would require action not only from the library author, but also from maintainers of _every single_ binary crate that depends on the library, including indirect dependencies.

[dwijnand]

@Shnatsel as per @nrc's comment, cargo install will, by default, continue to pull in the latest versions of dependencies (ignoring the lock file). The change is that a lock file _will_ be published (after being verified) and that lock file _may_ be used, if a user chooses to.

[dekellum]

_Right now_ and without this feature, some users (myself included) are opting to use a specific/controlled Cargo.lock, by simply committing and deploying a project's Cargo.lock with the source, building the application/service on the target (or an intermediary) host/container using that lock file, then running it. This makes good sense for repeatable deployment of non-public applications, which we must assume are the vast majority. I think rust-lang/docs.rs is one prominent, public example of this strategy, though. There are likely others.

It seems to me that any security concerns should put that in perspective, and any security efforts would be much better spent on doing the thing of most obvious utility: Any cargo command that downloads crates, and ideally any subsequent build or run, should emit WARNINGs for any crates with known security issues, or at the very least, crates that have been yanked (for any reason).

Said another way: Security and stabilizing this feature isn't an either/or situation, particularly if actually using the Cargo.lock is stabilized here as a non-default option.

[dekellum]

@nrc:

• Cargo.lock files should be verified as part of the pre-publication verification step.

What does that mean in this context? Would cargo, for example, refuse publication if a lock dependency has been yanked (already, at that time)? Security wise I still think the above described warnings are both _sufficient_ and more broadly useful. Users should hopefully see such warnings sooner in there process. But it would make sense to fully re-run such checks at publication time.

[sanmai-NL]

Fully agree @dekellum! There are plenty of vulnerability scanning and auditing tools to deal with vulnerable dependencies (e.g., container layers, Linux distro packages, malware patterns within executables, and even open source components). @Shnatsel, see https://github.com/RustSec/cargo-audit.

[ehuss]

This is a proposal to stabilize the publish-lockfile feature (in a limited sense).

@rfcbot fcp merge

Stabilization Target: 1.37 — Release date August 15th 2019

What is being stabilized

Cargo will be changed to always include Cargo.lock in the .crate file if the package has binaries or examples.

This has no immediate impact on users. cargo install ignores the lock file by default. A user must use cargo install --locked to use the packaged lock file.

NOTE: The publish-lockfile key in Cargo.toml is specifically not being stabilized, and will be removed at some date in the future if we decide it is ultimately not useful. At this time there does not seem to be compelling use cases to forcefully prevent Cargo.lock from being included.

Stabilization will also include documentation updates to make note of this change, and calling out the use of cargo install --locked.

The primary motivation for this feature is to allow cargo install to use the same set of dependencies that was tested when the crate is published. This can be useful for automated environments to always install the same thing. It is also useful when a semver-incompatible change is accidentally published, providing a way for users to fall back to a compatible set.

More details

• Cargo.lock is regenerated when the .crate file is created, guided by any existing Cargo.lock. The general consequence is that the Cargo.lock may be different from the one in the current workspace. The published version will not include unrelated workspace members or [patch] entries, and some things like path dependencies may resolve differently.
• Warnings have previously been added for:
  
  
  
  • Creating a .crate file with yanked dependencies.
  
  
  • cargo install --locked where an entry in the Cargo.lock is yanked.
  
  

[rfcbot]

Team member @ehuss has proposed to merge this. The next step is review by the rest of the tagged team members:

• [x] @Eh2406
• [x] @alexcrichton
• [x] @ehuss
• [ ] @joshtriplett
• [x] @nrc
• [ ] @withoutboats

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[Nemo157]

Cargo.lock is regenerated when the .crate file is created, guided by any existing Cargo.lock. The general consequence is that the Cargo.lock may be different from the one in the current workspace.

Can --frozen/locked be used to ensure it is identical; or does that only apply to updating the in-workspace lockfile, not the newly generated one?

[ehuss]

Can --frozen/locked be used to ensure it is identical; or does that only apply to updating the in-workspace lockfile, not the newly generated one?

No. IIRC, that applies to the verify-step only.

In the cases where the lock file changes, using the original would either be broken, or just contain extra entries that are ignored.

[rfcbot]

:bell: This is now entering its final comment period, as per the review above. :bell: