cargo publish multiplate packages at once

[alexcrichton]

It would be nice to have a flag to cargo publish which publishes all local packages in a DAG fashion.

Notes (edit ehuss):

• It would be nice if it supported atomic publishing.
• --dry-run should work correctly (will need to pretend that the previous crates have been published, maybe via patch?) See also https://github.com/rust-lang/crates.io/issues/1515.
• See https://github.com/rust-lang/crates-io-cargo-teams/issues/82 for other considerations for a new publish API to better accommodate this.

[wycats]

This would be awesome ;)

[huonw]

I suspect we may want to have a story with https://github.com/rust-lang/cargo/issues/883 if/when we implement this: I could easily imagine unintentionally publishing crates by not realising that I'm depending on them.

[botev]

Is there any progress on this? I think this would a very good thing to have. One of the nice things about crates is that they are modular, thus reduce the compile time of larger projects. However, if you are developing a large library compile times can get pretty slow (when it hits >2min I start to compile after 50% of the fixing, so it can compile while I'm fixing the other 50% of the issues). In this case you can split the library into sub-crates, where each sub-crate is in some sense standalone, or a build-up over "core" structure. However, each one of them would not make any sense on its own. Publishing each individually does not make any sense also and not packaging it together. If we can have this kind of publish --all this would give as an option to reduce compile times without any drawback from the user and publishing side of the library.

[matthiasbeyer]

Any updates whether this will be implemented and when?

[wycats]

@matthiasbeyer I think this fell through the cracks when we did workspaces. This issue was supposed to track wrapping --all up for publish but I guess it got lost since it was originally opened earlier?

@alexcrichton what's the status?

[alexcrichton]

@wycats AFAIK this has always been in the 'nice to have' category and hasn't progressed to the 'someone has put time into designing this' category.

[matthiasbeyer]

One thing I just thought about: When doing a workspace release, cargo should build everything and after everything is fine, publish all crates at once... if that is even possible. Not like "build & publish the first crate and continue for each crate" but rather "build everything, then publish everything".

[kpcyrd]

It seems that in a foo-rs, foo-rs/foo-sys project layout, you can't cargo package in foo-rs before foo-rs/foo-sys has been published.

[ufoscout]

I created a simple PR to deal with this annoying issue; it permits to execute cargo publish from a workspace.

One thing I just thought about: When doing a workspace release, cargo should build everything and after everything is fine, publish all crates at once... if that is even possible. Not like "build & publish the first crate and continue for each crate" but rather "build everything, then publish everything".

Unluckily I did not find a way to implement this logic because cargo requires that all the dependencies of a package are in the repository or the package phase (when the tarball is created) fails; consequently, before publishing a package with a "path" dependency, that dependency must be in the repository.

[Nemo157]

Something else I would want that I don’t think that can handle is only publishing updated packages, if one of the packages already exists at its current version number it should be downloaded and verified that the new package is identical.

[softprops]

being new to workspaces but not to cargo this feels very much like a paper cut. My finger just got nipped when I tried to publish a new workspace project by following the docs

...then realized that order matters. The validation of packages will fail of one of the workspace packages depends on another in the same release but which may not have been published first. In my case it's a very simple ordering but for those new to cargo, something like cargo publish --all could get the ordering right without putting a burden on me at all. Something like that would be a much nicer new user experience, also a much nicer convenience to those that have learned the ordering semantics for the more more manual publishing approach.

[matthiasbeyer]

It would definitively help me with imag where I publish over 50 crates in one release!

[softprops]

https://github.com/Byron/google-apis-rs and https://github.com/rusoto/rusoto would also like benefit greatly from this

[torkleyy]

This sounds useful, but I'm wondering what the exact behaviour should be. What if you have a path dependency and there's no version specified, should Cargo modify the Cargo.toml for that?

[mitchmindtree]

@torkleyy interesting point. It probably should not modify the Cargo.toml at all, but it would be worth considering whether or not cargo should allow for path dependencies but only for crates within the same workspace.

E.g. If building foo and a dependency on bar is specified via a path, perhaps cargo could implicitly publish the new version of foo with the version of bar that was published during the same cargo publish --all? This would also allow for cargo to easily build everything first locally before publishing any of the packages.

Simplified Steps

I'm imagining cargo publish --all should do something like the following:

1. Check manifests and version validity of all packages before starting to build any.
2. Create DAG of workspace packages and determine the build order.
3. Build packages. If any failures occur, bail out of the whole process.
4. If all manifests and valid and packages are built, publish all packages in the order in which they were built.

It would be worth considerinng if step 4 should be a special "atomic step" recognised by crates.io so that if for some reason the net drops out or there's a crash the user doesn't end up with only half of their packages published.

[Nemo157]

@mitchmindtree I would like to also see an extra step between 3 and 4 doing full workspace package validation to replace the current pre-publish validation. This does the normal per-package validation steps with 2 changes:

1. crates.io is queried for an exact version match of any of the crates. If there is an exact version match then the contents are compared to make sure the crate hasn't changed, if they match the crate is removed from the set to publish. (Basically making publishing a crate idempotent so that you can republish a workspace where you have only updated some of the crates.)

2. during this validation crates can depend on crates that are either available on crates.io or are part of the current set to publish.

[torkleyy]

I started working on this here: https://gitlab.com/torkleyy/cargo-publish-all

[sehz]

https://gitlab.com/torkleyy/cargo-publish-all works well.

[softprops]

Thanks for sharing. This looks like a good stop gap for not having this as a built in. The main reason I'd still champion this as a built-in is for consistency as other built-in commands support the --all flag.

[KSXGitHub]

What's the status of this issue?

[pksunkara]

Sorry for the drive by, but in case anyone is wondering, I got inspired by lerna and made https://github.com/pksunkara/cargo-workspaces. rust-analyzer and chalk are using it to auto publish weekly.