Cargo: Activating features for tests/benchmarks

[ctz]

Given a library crate, I want to have optional features which are available to integration-level tests and examples so that 'cargo test' tests them.

Here's what I have:

[package]
name = "rustls"
version = "0.1.0"
(etc)

[features]
default = []
clientauth = []

[dependencies]
untrusted = "0.2.0"
(etc)

[dev-dependencies]
env_logger = "0.3.3"
(etc)

Here's what I've tried so far. Attempt one: add the crate into dev-dependencies with the feature, ie:

[dev-dependencies.rustls]
path = "."
version = "0.1.0"
features = ["clientauth"]

I kind of expected that to work, but it panics:

jbp@debian:~/rustls$ RUST_BACKTRACE=1 cargo test
thread '<main>' panicked at 'assertion failed: my_dependencies.insert(dep.clone())', src/cargo/util/dependency_queue.rs:86
stack backtrace:
   1:     0x558d17c3a2a0 - std::sys::backtrace::tracing::imp::write::h9fb600083204ae7f
   2:     0x558d17c4383b - std::panicking::default_hook::_$u7b$$u7b$closure$u7d$$u7d$::hca543c34f11229ac
   3:     0x558d17c434c3 - std::panicking::default_hook::hc2c969e7453d080c
   4:     0x558d17c28048 - std::panicking::rust_panic_with_hook::hfe203e3083c2b544
   5:     0x558d17701fef - std::panicking::begin_panic::h4ebf9fe884b2415f
   6:     0x558d17802d27 - cargo::ops::cargo_rustc::job_queue::JobQueue::enqueue::h98b20a33b8744ebe
   7:     0x558d177f4faf - cargo::ops::cargo_rustc::compile::h93336dd21aba29d0
   8:     0x558d1777a513 - cargo::ops::cargo_rustc::compile_targets::h7223dffae01d6b9d
   9:     0x558d1776eec9 - cargo::ops::cargo_compile::compile_pkg::h48656ffa9b1541f3
  10:     0x558d1776bd09 - cargo::ops::cargo_compile::compile::h1b43b20047c53d10
  11:     0x558d1785fb0d - cargo::ops::cargo_test::compile_tests::h028a22d3131e0d65
  12:     0x558d1785f379 - cargo::ops::cargo_test::run_tests::h0ad98c54a6f40c9d
  13:     0x558d176de256 - cargo::call_main_without_stdin::hd484f08b17c419d1
  14:     0x558d1768afe7 - cargo::execute::hab7accd6bf9b5c64
  15:     0x558d17683f8b - cargo::call_main_without_stdin::hf099bd36acb849a3
  16:     0x558d17680da1 - cargo::main::h2b219a79f378c1ef
  17:     0x558d17c430d8 - std::panicking::try::call::hc5e1f5b484ec7f0e
  18:     0x558d17c4e0eb - __rust_try
  19:     0x558d17c4e08e - __rust_maybe_catch_panic
  20:     0x558d17c42b03 - std::rt::lang_start::h61f4934e780b4dfc
  21:     0x7f79ea9bf86f - __libc_start_main
  22:     0x558d17680728 - <unknown>

(version: "cargo 0.11.0-nightly (259324c 2016-05-20)")

Attempt two: see if features.* works like profile.*:

[features.test]
default = ["clientauth"]

alas

$ cargo test
error: failed to parse manifest at `/home/jbp/rustls/Cargo.toml`

Caused by:
  expected a value of type `array`, but found a value of type `table` for the key `features.test`

Is this possible?

[alexcrichton]

Unfortunately, no, it's not possible for cargo test to automatically enable features currently. The panic though definitely seems bad!

[jjpe]

I just ran into this issue because I'm currently using a feature to decide whether or not to do some rather verbose tracing. I'd like to be able to set this during testing so that when tests fail I can look at the trace.
The reason I made it a feature is that the code itself is in a library, but at the same time it seems more appropriate to have the consuming crate decide whether or not to perform the trace.

To be clear, an actual consuming crate has no problem setting the feature and getting corresponding results. It is just for testing the library crate itself that it doesn't seem possible at the moment.

Update: There is a workaround I just found:
If the feature is called foo, then this will test with the feature activated:
cargo test --features "foo"

The only thing is that this is activated manually, and being able to specify it in the library's Cargo.toml would activate the feature by default, which is desirable in my use case.

[sanmai-NL]

@alexcrichton:

As of yet, the Cargo panic is gone.

  cargo --version
cargo 0.19.0-nightly (fa7584c14 2017-04-26)

Excerpt from Cargo.toml

[features]
default = [] # "clippy"

[profile.test]
default = ["clippy"]

The output is now as follows:

cargo test --color=always --package my_app --bin my_app test_1 -- --nocapture

warning: unused manifest key: profile.test.default
    Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
     Running target/debug/deps/structure_web_server-3dc5f86e9b4fb259

running 1 test
test tests::test_1 ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured

You wrote this isn't possible. It appears there are no immediate objections to adding such a feature. Could you explain how fundamental the limitation is? And, if possible, sketch a solution? Just so a PR becomes more likely to happen.

[alexcrichton]

Ah I believe this was fixed in https://github.com/rust-lang/cargo/pull/3921 as https://github.com/rust-lang/cargo/issues/3902 was a dupe of this issue. I guess I was wrong before!

[sanmai-NL]

@alexcrichton: the crashes have been resolved yes, but the original feature request still stands AFAIK. Please reopen.

[alexcrichton]

Oh sorry, missed that aspect.

[sunjay]

Is there any way to add features per profile now? That is, if I want to enable certain features during tests, can I do that yet?

[alexcrichton]

@sunjay unfortunately no, you'll still need to pass --features on the command line

[sunjay]

Is this change something that would require an RFC or could it just be implemented? It sounds like being able to specify features in the profile configuration is something that could benefit a lot of people.

[alexcrichton]

@sunjay I think due to the subtle possible ramifications of auto-enabling features I'd personally want to see an RFC, but it may be worth canvassing others from @rust-lang/cargo first as well

[sunjay]

I would love to work on an RFC for this. Very interested to hear what the cargo team thinks.

Right now, I'm having to constantly remember to pass features in when I run cargo test. I would really prefer to not have to do that. It's an extra point of friction for anyone who uses my code.

Though it would be great to even just be able to configure features in tests, I would really like to see cargo support a features/default features option in each of the [profile.*] sections. This would make it so you could configure default features for your release, debug and benchmarking profiles too. My RFC would be proposing something like that.

[matklad]

I think we have an RFC by @withoutboats for this already: https://github.com/rust-lang/rfcs/pull/1956, which is postponed because of profiles.

[sunjay]

@matklad Ah yup that's exactly what I would have proposed. Thanks for looking that up! I read the discussion and it seems like that's pending on some work with how multiple profiles are applied together (e.g. test --release). It seems that the current behavior is more complicated than what it looks like at first glance.

Is there any way to do this in a backwards compatible way? It would be great if there was some way to just add another configuration option that is in effect the same as the command line argument while keeping everything the same as it is now. Would that make it too hard to change to something better (like profiles) later?

The main issue I'd like to solve is that right now, introducing features into your codebase makes it so that you have to pass features in as command line arguments every time you run a certain commands. If these aren't default features that you need for all builds, it's not possible to simply configure the ones you want in Cargo.toml. This makes it harder for people new to your codebase or to cargo set things up. It is very repetitive and easy to forget. Is there an easier way to just address that issue?

[brson]

I've also run into the need to have features automatically enabled in test builds. In my case I want the python bindings "extension-module" feature on for production builds, but off for test builds so that I can link directly to python during testing.

[Object905]

Also it's very useful for #![no_std] ergonomics. Since you need to test with libstd linked, otherwise you get
error: no #[default_lib_allocator] found but one is required; is libstd not linked?. So you constantly running cargo test --features std

[ebfull]

Note that you can do this with required-features if you specify your test targets explicitly, aka:

[[test]]
name = "whatever"
path = "tests/whatever.rs"
required-features = ["mycoolfeature"]

[RReverser]

@ebfull That only skips building/running tests if feature is not enabled instead of force-enabling the feature during build.

[synek317]

Some ugly workaround, at least for some cases, is to extend all your #[cfg(feature = ...)] with test (e.g. #[cfg(any(test, feature = "..."))] and copy all optional dependencies in [dev-dependencies] and remove optional = true.

It works nice for me but for sure there are cases where it is not enough.

[nixpulvis]

Yea I feel like an RFC here would make sense if one doesn't already exist. I'm using rust again, and finding myself really wanting something like cargo test --feature-matrix for example.

[Boiethios]

Some ugly workaround, at least for some cases, is to extend all your #[cfg(feature = ...)] with test (e.g. #[cfg(any(test, feature = "..."))] and copy all optional dependencies in [dev-dependencies] and remove optional = true.

It works nice for me but for sure there are cases where it is not enough.

I like your trick, but unfortunately, it does not work for the documentation tests (not sure why, though).

[isavegas]

Having something akin to #[features("chrono", "alloc")] and so on to set a specific feature set for a test module (with #[features()] to use no features) would be awesome! Of course, having this functionality available elsewhere is probably not a great idea.

Of course, this might not even be feasible without rewriting the core of the testing framework, but perhaps it could be a nice quality of life feature to look at down the line.

[cormacrelf]

If you have a workspace going, a better workaround is to create a test_utils crate. You can make the feature activate automatically for an integration test.

# test_utils crate
[dependencies]
main = { path = "../main", features = ["test"] }

pub use main as main_test;
// or, may as well use this crate to compile the integration testing infrastructure
pub fn exported(file: &str) { main::whatever(); assert_eq!(1, 1); }

# main crate
[features]
test = []

[dev-dependencies]
datatest = "..."
test_utils = { path = "../test_utils" }

// lib.rs
cfg_if::cfg_if! {
    if #[cfg(feature = "test")] {
        fn whatever() { do_something_else(); }
    } else {
        fn whatever() {}
    }
}

// tests/integration.rs
use test_utils::main_test as main;
use test_utils::exported;

#[datatest::files( ... )]
fn integration(file: &str) {
   exported(file);
}

Lotta moving parts but I think I wrote it all right.

[Diggsey]

What's the current state on this, has there been another RFC?

[Nashenas88]

@cormacrelf unfortunately that doesn't work for my case, where I have multiple features I'd like to test. My current workaround is to use a makefile to keep track of different test crates, and each of those depends on the main crate with specific features turned on. With workspaces, you can't mix features. Would really love to see some progress here, and I'd be willing to help with the RFC if help is needed.

[ChaseElectr]

We are working on a really big project and use a lot of protobuf structs. Now we want enable Serialize and Deserialize on those struct only under test situations, but found control through features may be our only choice:
First we have a lot of structs define like this

// crate a, which dependens on crate b
#[cfg_attr(feature = "with-serde", Serialize, Deserialize)]
struct Foo {
    bar: Bar
}

// crate b
#[cfg_attr(feature = "with-serde", Serialize, Deserialize)]
struct Bar {
    b: i32
}

Then we want write tests for crate a

#[cfg(test)]
mod tests {
    #[test]
    fn my_test() {
         let foo: Foo = serde_json::from_str("xxxxx");
         balabala.....
    }
}

We have to add #[cfg(feature = "with-serde")] for my_test or for mod tests, or the clippy will complain about not implement Serialize/Deserialize for Foo.

We alse have tried with using #[cfg_attr(test, Serialize, Deserialize)] instead of #[cfg_attr(feature = "with-serde", Serialize, Deserialize)], but found that when we run test for crate a, crate b won't have cfg_attr(test) to be true, which will lead Serialize/Deserialize not implement for Bar.
Another choice is use other config attribute instead of test, such as #[cfg_attr(with_serde, Serialize, Deserialize)], but this needs additional effort for both writing tests and running tests.

If we can have default features for test, it seems is the simplest way to solve this problem😋

[Fishrock123]

Would folks here be open to an RFC which proposes special-casing a Cargo.toml features feature named test (similar to default) which is used when compiling for tests?

[DesmondWillowbrook]

There is currently a work-around for this. (Source)

[dev-dependencies]
self-package-name= { path = ".", features = ["desired_feature"] }

where self-package-name is the name of the package in the Cargo.toml file.

Sample repo for the same