Rust: cfg(test) is not set during doctests

[sunjay]

_From @sunjay on October 27, 2017 21:23_ (Moved from rust-lang/cargo)

It might be that I'm just not understanding how something works, so please let me know if that is the case!

Since doctests are tests, I expect that #[cfg(test)] would work and cfg!(test) would return true. This works during unit tests, but during doctests, both assumptions are wrong.

The reason I need this is because I have some code (not in doctests) that manages windows (like GUI windows) that I need to not run during any tests. I think this is a cargo problem because the attributes work consistently, I believe it's just cargo that doesn't set the attribute. If there is a workaround I can use, I would really appreciate it. :) (EDIT: My workaround is at the bottom.)

Here's some code that reproduces the problem: (Run with cargo test)

#[test]
fn cfg_test() {
    // This will pass, as it should
    assert!(cfg!(test))
}

/// ```rust
/// # fn main() {
/// // This will fail, even though we *are* in a test
/// assert!(cfg!(test), "cfg!(test) was false");
/// # }
/// ```
pub fn cfg_test2() {}

/// ```rust
/// # fn main() {
/// #[cfg(test)]
/// fn foo() {}
/// #[cfg(not(test))]
/// fn foo() {panic!("cfg(test) was not set")}
/// foo();
/// # }
/// ```
pub fn cfg_test3() {}

This produces the following output:

$ cargo test
    Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
     Running target/debug/deps/foo-3890a4e1c4133892

running 1 test
test cfg_test ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

   Doc-tests foo

running 2 tests
test src/lib.rs - cfg_test3 (line 15) ... FAILED
test src/lib.rs - cfg_test2 (line 7) ... FAILED

failures:

---- src/lib.rs - cfg_test3 (line 15) stdout ----
    thread 'rustc' panicked at 'test executable failed:

thread 'main' panicked at 'cfg(test) was not set', src/lib.rs:6:10
note: Run with `RUST_BACKTRACE=1` for a backtrace.

', /checkout/src/librustdoc/test.rs:315:16
note: Run with `RUST_BACKTRACE=1` for a backtrace.

---- src/lib.rs - cfg_test2 (line 7) stdout ----
    thread 'rustc' panicked at 'test executable failed:

thread 'main' panicked at 'cfg!(test) was false', src/lib.rs:3:0
note: Run with `RUST_BACKTRACE=1` for a backtrace.

', /checkout/src/librustdoc/test.rs:315:16


failures:
    src/lib.rs - cfg_test2 (line 7)
    src/lib.rs - cfg_test3 (line 15)

test result: FAILED. 0 passed; 2 failed; 0 ignored; 0 measured; 0 filtered out

error: test failed, to rerun pass '--doc'

In both cases, since cfg(test) is not set, the test fails.

---

This is a temporary workaround you can use if you are in need of this functionality:

In Cargo.toml, add the following:

[features]
# The reason we do this is because doctests don't get cfg(test)
# See: https://github.com/rust-lang/cargo/issues/4669
test = []

Then run with cargo test --features "test".

This adds a feature called "test" to tests when they run. This works for both unit tests and doctests.

Then, to test for it, use the following:

#[cfg(any(feature = "test", test))]

You could also just go with the following, since the feature is set for all kinds of tests.

#[cfg(feature = "test")]

_Copied from original issue: rust-lang/cargo#4669_

[jonhoo]

I may be mistaken here, but I believe doctests are treated as integration tests (i.e., not unit tests), and consider the crate under test external. This is supported by the fact that you must manually add extern crate <cratename>; if you extern any other crates in the doctest. Doctests (like integration tests) also cannot access private functions, or otherwise reach into the internals of the crate. The test attribute is only set for the compilation unit that holds the test, which in this case is not the crate under test, but rather the test harness itself.

Now, I do believe there is a bug here, and that is that test should technically be set inside the doctest code itself, since the doctest is running in test code. However, the crate under test should not be compiled with test, as the original post above requests.

That all said, I have also wanted the ability to have the crate under test behave differently in some way when it is run in "test mode", even when used as an extern crate, so I'm sympathetic with @sunjay's complaint. It's pretty common to work around this by adding a special test configuration parameter or feature to the crate under test, but this feels like a hacky workaround, as this toggle can then also be used by other consumers of the library. It is more okay for integration tests and doctests to reach into the crate under test than it is for some random other crate.

[sunjay]

Maybe the answer is to have a specific doctest attribute so that we can choose what to include in either. My current hacky solution (adding a "test" feature), is only inconvenient because you have to manually pass it in when you run tests. Otherwise it works great and accomplishes exactly what I want. If there was a cfg flag I could use specifically to check for doctests, there would be no problem.

With that, unit tests could remain unit tests, and we could do more specialized behavior in doctests when we need to. Adding an attribute like that isn't a breaking change either so that's great too.

Note: if we add this, we should probably have something for tests that aren't unit tests or doctests. The tests in the separate tests directory could probably benefit from their own cfg flag.

[fschutt]

It would be great if a #[cfg(doctest)] was added or #[cfg(test)] would work for doc tests. The workaround is simply annoying. I stumbled on this because I wanted to create a mock rendering function that is only available in doc-tests.

[dvdplm]

+1 for a #[cfg(doctest)] – the workarounds are very unsatisfying.

[D4nte]

I am facing similar situation for the tests folder. I have code that provide me with helpers to the tests folder. I would like this code to only compile under #[cfg(test)] as it's only meant for test.

[briansmith]

Doctests are integration tests so any solution for this problem should work for integration tests too.

I agree that we need a way to enable features that are available (only) to integration tests.

[paulvt]

Agreed. I am writing a crate that manages some devices via libusb. I would prefer if it wasn't necessary for me to connect a device to be able to perform doctests; the same actually also holds for the integration tests.

(If somebody known solutions for these kinds of issues and common ways to mitigate them, please let me know in private. Thanks in advance!)

[asomers]

My problem is quite similar. I maintain https://github.com/asomers/mockall , which is normally only used as a dev-dependency. So some of my examples won't really do anything unless cfg(test) is set. Ideally I'd be able to run the doc tests both ways: both with and without cfg(test).

[ggriffiniii]

I would also point out that this causes the most prominent use of the doc-comment crate to seemingly not work for users. Each of the top 5 most popular crates (lazy_static, regex, aho-corasick, byteorder, num_cpus) are all using doc-comment behind a #[cfg(test)] conditional. Meaning that they intend to check their README.md files, but in fact are not. These crates are maintained by some of the best members of the rust community indicating me me that the current behavior is not what even experienced rust engineers expect.

I only discovered this issue after trying to use doc_comment to test my README.md file, trying to figure out why it wasn't working. Comparing my code to the examples in those crates above, failing to see what I was doing wrong. Eventually compiling those crates only to discover they also are not testing their README.md files, bisecting older versions of the rust compiler and doc_comment crate to discovery that this seemingly never worked.

[sunjay]

I should note that my usecase will be pretty much addressed by the stabilization of #[cfg(doctest)]. This issue can probably be closed once that is released on stable Rust. (see next comment)

[sunjay]

Ah actually it turns out I misunderstood what was added. The cfg(doctest) is only used when collecting doctests. It isn't used when compiling the code (:cry:).

This does not resolve this issue then. We would need cfg(doctest) to be added when compiling doctests for me to fully remove the --features "test" hack I've been using so far.

[GuillaumeGomez]

@sunjay I think we actually should set it when compiling as well. I opened https://github.com/rust-lang/rust/issues/67295 to keep track of it.

[sunjay]

Thank you! I agree. I was initially very confused about the behaviour.

[erayerdin]

I just would like to point out this thread from official user forum since it's settled that this is an issue affecting integration tests as well. Just in case.