cargo test documentation

[lspitzner]

I am currently fighting with an instance of "cargo test does stuff and i don't even know what and why" and I find the documentation around this thoroughly disappointing. Apparently it is calling rustdoc --test which somehow interprets certain comments as testcases, and those fail to compile (of course, I don't think any of those comments are meant to be tests. They are autogenerated by bindgen though, so I don't even know for sure.)

In my opinion having doctests included in cargo test by default without appropriate documentation is bad and a disservice.

It means that users (like me) can run into situations where cargo test reports "failures: foo::bar::some_identifier_0" and where:
1) some_identifier_0 is not even in the sourcecode anywhere;
1) the only hint where this test is coming from is the headline "Doc-tests $cratename/libname" which still does not give me any clear indication of what this test does;
1) cargo test --help does not explain what the command even does, to any sufficient degree;
1) http://doc.crates.io/ does not explain things either, it does not even have a section around testing;
1) cargo test --verbose reveals that it runs rustdoc --test. What does that do? I have looked into rustdocs --help, its manpage and looked for a userguide, and have still have found only:
1) https://doc.rust-lang.org/book/documentation.html explains some basics. But the source where the failing tests (probably?) originate from contains neither /// nor three backticks.

This is rather annoying. Not because the tests don't pass, or because rustdoc --test has no good documentation yet, but because it happens by default without me asking for it.

3559 seems to be close, although its description is more specific.

(I will be using cargo test --lib which does not do whatever the doctests do. So I am not asking about help for my particular case.)

(thanks for the all maintainers and contributors for their work on this project.)

[lspitzner]

Oh, and to be clear: bindgen is creating #[test]s as well; I am not talking about those (at least I am pretty sure because I have commented those out and they vanished from the test output. But the other failing tests still remained.)

[alexcrichton]

Ah yeah so I can at least help clarify what's happening here. By default when you run cargo test Cargo attempts to run as many tests as possible, inferring lots of locations and such to avoid forced configuration. One such test is "doc tests" which is where we'll run rustdoc --test over a library (if one is available). This will attempt to compile code snippets in documentation as Rust programs and report errors, allowing you to verify that doc examples are indeed correct!

An example of this is:

/// ```
/// foo::foo();
/// ```
pub fn foo() {}

I believe bindgen generates doc blocks with documentation from C header files and it isn't currently tagging the code snippets as c or ignore (e.g. "not rust").

You can disable doctests locally via:

[lib]
doctest = false

And I definitely agree the docs can be improved around this!

[lspitzner]

In this particular comment can you spot what parts of the syntax make it qualify as a testcase? /** is significant, but not all such doc comments lead to testcases.

Thanks for the doctest = false hint.

[alexcrichton]

For that it may be the <PRE> block if the translation covers that, otherwise it may be lines like these which are indented over and markdown could erroneously consider those code instead of part of the list above.