Cargo: Cargo fails to run when an *optional* dependency is missing

[marcbowes]

#!/usr/bin/env bash

set -e

cd /tmp
[ -d cargo-optional-deps ] && rm -rf cargo-optional-deps
cargo new cargo-optional-deps
cd cargo-optional-deps

cargo build
cat >> Cargo.toml <<EOF
optional_dep_does_not_exist = { version = "1.0", optional = true }
EOF
cargo build

Fails with

error: no matching package named optional_dep_does_not_exist found (required by cargo-optional-deps)

I did not expect this because the dependency is not required as per http://doc.crates.io/manifest.html#the-features-section:

optional dependencies, which enhance a package, but are not required; and

I'm running into this behavior while integrating Cargo into a private build system that wants control over dependencies. As such, I need to elide optional dependencies by default in this system's dependency list.

I'm opening this issue to get some input on:

1. Would ignoring missing optional dependencies for inactive features be a problem for Cargo?
2. I tried to look through existing issues - has something like this been discussed before?
3. An idea about resourcing/priority of the feature and how it fits into the roadmap for Cargo's integration with existing systems.

[alexcrichton]

Thanks for the report! This is currently expected behavior, however, as Cargo will asset that all dependencies exist (optional, platform-specific, or not) to generate a lock file.

Can you describe your use case a bit for depending on optional dependencies that don't exist? There may be another way to tackle the same problem!

[marcbowes]

The build system at my company is dependency aware. It uses this knowledge to trigger rebuilds. For example, if A depends on B and then B is built, so too will A. In this manner, a commit to a package will land up running a build (e.g. run tests, static analysis) for everything that is using it. The dependencies are listed in a package config file and are broken down by "build", "normal", "test" and "runtime" dependencies.

The "third party" import process for Rust crates involves a step where we use the crates.io API to map the dependencies onto internal packages. Thus, the dependencies in Cargo.toml land up being in the package config (normal) dependencies section while dev-dependencies land up in build/test dependencies. When we build a package, our build system processes the dependencies and builds a Cargo registry (the "soon to be deprecated" variant) that has precisely the crates we need.

The impedance mismatch is that our build system doesn't have a concept of optional dependencies. Thus we're stuck between two extremes: "all optional dependencies go in the package config" and "no optional dependencies go in the package config". After an internal discussion, we decided the latter was much better. The story in this world is that if you (package A) want to consume a crate (package B) and turn on an optional dependency (package C), then you need to depend on A + C and enable the feature in your Cargo.toml.

If we went for the "put all optional dependencies in the package config" we'd have a bunch of issues such as triggering unnecessary rebuilds, potentially running into licensing issues and so forth. Less is more.

Just to add some more color:

Our build system is pervasive, old and opinionated. When I read https://github.com/rust-lang/rfcs/pull/2136 I assumed we'd need to turn every knob and dial you provided. However, despite being old and opinionated, the integration we have now is actually pretty neat. We have an import process that worries about turning Cargo.toml dependencies into our dependency manifest format and we have a custom registry build process. That's it. Cargo does everything else and that means that our developers get to use everything they find on stackoverflow without any weirdness.

Also note that there is no way we're going to use crates.io. Not only for security/policy reasons, but because we have internal build system concepts that allow developers to "have their own universe" - this makes any "central" or "single" source nonsensical. Each developer can have their own crates.io for their own universe. Another interesting fact is that our build system handles all types of dependencies, so we don't have any issues with native code, e.g. ensuring that the right version of openssl is present.

[marcbowes]

It's likely that "platform specific" will also cause similar issues, but I haven't thought through that too deeply.

[alexcrichton]

Ok thanks for the explanation @marcbowes! I wonder if there may be a good balance to strike here perhaps.

So the reason that Cargo needs optional dependencies is that when generating a lock file for a crate it'll ensure that the lock file doesn't change depending on what you do with the crate (test, build with features, etc). This means that even though some compiles don't use all the dependencies it instead requires all deps to be present at all times.

This sounds like a case, though, where there's definitely an impedance mismatch between your internal build system and "idiomatic Cargo". In that sense it may be best to follow the "build system integration" threads and features to see how we can better accommodate your build system. When you say "there is no way we're going to use crates.io" my guess is you don't mean that you won't use crates from crates.io but rather you won't be directly downloading from crates.io (which is totally ok!). We basically want to make sure that we're empowering and enabling anyone who just wants to use code from crates.io!

One thought I had when writing this is that maybe some "shim crates" could work for now? You're right that pulling in unused crates could have license problems and such like that, but you may also be able to have a set of crates that are "stubbed out" which are empty and contain no dependencies and fail to compile. That way if you accidentally pull it in you know that something needs to be updated to really import the crate.

[marcbowes]

So the reason that Cargo needs optional dependencies is that when generating a lock file

I think you've hit the nail on the head here. I'm not sure "lock files" are necessary in our system which has its ways of ensuring reproducible builds. Our package template for Rust projects has a .gitignore for Cargo.lock.

but rather you won't be directly downloading from crates.io

Yup, exactly. Sorry for the ambiguity. We have an "import" process to pull an external dependency into our build system. That process is aware of crates.io.

One thought I had when writing this is that maybe some "shim crates" could work for now?

I'm open to the idea but also skeptical. Right now I don't care about 32 bit, Windows, etc. There are trees of dependencies I don't need to worry about if I just remove them. That works great because I can just scrub them from the Cargo.toml and nothing breaks. Optionals are harder because enabling the feature becomes impossible without "unscrubbing" the manifest on demand.

In that sense it may be best to follow the "build system integration" threads and features to see how we can better accommodate your build system

Yeah, I am (trying to). If something in particular is worth investigation, I'd appreciate if you could ping me.

[cramertj]

Has any further thought been given to what a solution here might look like? I'd be happy to help with implementation if there's a decision about what should be done. Currently, we're checking in a huge number of vendored crates into the fuchsia tree because they're pulled in as optional dependencies of something or other, and I'd like to make it possible to omit them.

[alexcrichton]

@cramertj that may or may not be a bug in Cargo and/or the Cargo integration, optional dependencies are only pulled in for members of a workspace, which typically isn't true for crates from crates.io (just those you yourself are writing). Is that the behavior you're seeing?

[bryteise]

So I likely have a similar issue as @marcbowes in that I'm trying to avoid packaging optional dependencies as it just creates more work for myself (as then I need to package all of the optional dependency packages and their dependencies).

I also noticed something unexpected based on what I understand from lock file generation. I have a package shipping a lock file so I go to package up the content in the lock file (the lock file content should contain the full dependency resolution if I understand correctly). As part of this process I notice one of the dependencies itself has an optional dependency but that is not part of the primary package's lock file. Is that expected?

(side story, in the libc repo, it ships a Cargo.lock file which I slighly remember buzz around libc breaking folks and maybe that's why the library is shipping a lock file but I don't see any of the dependencies listed in the Cargo.toml so I'm just left with adding all of dependencies I see from running cargo vendor for libc is that the expected workflow?)

[cramertj]

@alexcrichton I'm not sure yet-- I've made a note to myself to investigate further. I started pushing after this because the Fuchsia repo currently includes 57 megs worth of winapi dependencies.

[alexcrichton]

@cramertj heh I've thought we'd hit a problem like this at some point.

A random solution which may help solve this though is to perhaps list the valid targets for a project in a workspace root Cargo.toml. That way Cargo could be smarter and just vendor dependencies for those targets (and alter resolution so it won't include winapi in the lock file)

[cramertj]

@alexcrichton That would be perfect! I'll message you on IRC and we can chat about the details.

[siscia]

Hi Folks,

I hit the same problem with an use case that I believe is quite reasonable.

My project redisql has an open source part and a close source part.

For my own sanity, I was keeping everything in the same cargo folder and using a git submodule to a private repo for the close source part. Like this: https://github.com/RedBeardLab/rediSQL

Now users correctly report that they cannot build it: https://github.com/RedBeardLab/rediSQL/issues/29#issue-314814341

Is there any way around it?

Cheers,

Simone

[alexcrichton]

@siscia unfortunately there's no workaround at this time, although https://github.com/rust-lang/cargo/issues/5133 may be workable if the dependency is placed behind a Cargo feature

[siscia]

I see, thank you :)

[acfoltzer]

I've just hit the exact same situation as @siscia, where I have an open source project with an optional proprietary component that adds a few extra features. I'm currently working around it by checking in an empty package to the open source project, and then instructing users to overwrite that with the proprietary code if they need it, but this is pretty bad ergonomically.

[joshtriplett]

@acfoltzer Cargo does avoid requiring optional dependencies when you use cargo install. So if you create a "directory registry" with the one crate in it, and then cargo install the crate by name from that registry, cargo will build and install the crate without requiring optional dependencies.

I'd like to have a general solution to the problem, though.

[Michel-Haber]

Has a decision been made for this issue?
I have encountered a case where it is really needed.
In fact, one of my dependencies is optional, and needs a crate that is generated locally. I want to be able to build my project without this dependency when its feature isn't used, and get a compile error if I try to build with this feature and the generated code is missing.

Is there no possible workaround for now?

[keszybz]

I'd like to add another "me too" case. I'm working on packaging crates in Fedora. We cannot use crates.io directly, and all crates that are needed during the package build process must be packaged as RPMs. Not all crates are packaged (for various reasons, sometimes lack of resources, sometimes we don't want to package something), and we want to use cargo build with some optional features disabled and without those crates installed. This is very close to the situation described in the original report for this bug, except that this is a public distro so everything is in the open.

Right now the only workable solution seems to be to edit Cargo.toml files to remove the optional dependencies we don't want to enable, but that is a terrible hack and makes it much harder to automate the whole process.

[emilio]

Somehow Firefox manages to build without optional dependencies in the tree... But I'm not sure how that even works :).

Edit: Note that only optional _path_ dependencies.

[emilio]

Here's why the Firefox thing works if anyone's curious: https://mozilla.logbot.info/servo/20180917#c15322773

[Qeenon]

Here is my solution hack/workaround https://github.com/Qeenon/Amadeus/commit/56a1892fc0b5e6630caf9ade67345f5259f08761