Rust: Doctests don't work in bin targets, non-public items

[kornelski]

It's surprising that doctests sometimes don't run at all, and there's no warning about this.

cargo new foo --bin

/// ```rust
/// assert!(false);
/// ```
fn main() {
    println!("Hello, world!");
}

cargo test --all

I'd expect the above to fail, but it looks like the doc-comment code is never ran. Even if I explicitly enable doctests for the binary:

[[bin]]
name = "foo"
path = "src/main.rs"
doctest = true

they still aren't run.

(moved from https://github.com/rust-lang/cargo/issues/5477)

[GuillaumeGomez]

It sounds weird to have documentation examples in a bin target, don't you think? There is no documentation to generate so why would there be documentation's code to run?

[kornelski]

This bug is based on feedback from a user.

Cargo makes new users create bin targets by default, so the first time a new Rust user tries doc tests, it's probably in a bin.

There's a mismatch between what rustdoc does (tests publicly-visible documentation) and expectations (that tests in doccomments, in general, are run).

[GuillaumeGomez]

Maybe we give bad information ahead? I wonder if users know the difference between comments and doc comments... For now I don't see this as a rustdoc bug but as a misunderstanding that should be resolved in tutorials/books.

cc @rust-lang/docs

[vi]

Maybe doctests should just run unconditionally?

The code for a non-public bin-crate item may end up in a public, lib-crate item later, so what's wrong having documentation and tests a bit earlier?

[QuietMisdreavus]

The problem with running doctests on bin targets lies with how doctests work. The way rustdoc creates doctests is by building the code sample as a bin target of its own, and linking it against your crate. If "your crate" is really an executable, there's nothing to link against.

If you run cargo test --doc --verbose, you can see that Cargo needs to build your crate ahead of time, and pass the rlib to rustdoc:

[misdreavus@tonberry asdf]$ cargo test --doc --verbose
   Compiling asdf v0.1.0 (file:///home/misdreavus/git/asdf)
     Running `rustc --crate-name asdf src/lib.rs --crate-type lib --emit=dep-info,link -C debuginfo=2 -C metadata=31e1a62d36de9a99 -C extra-filename=-31e1a62d36de9a99 --out-dir /home/misdreavus/git/asdf/target/debug/deps -C incremental=/home/misdreavus/git/asdf/target/debug/incremental -L dependency=/home/misdreavus/git/asdf/target/debug/deps`
    Finished dev [unoptimized + debuginfo] target(s) in 0.79 secs
   Doc-tests asdf
     Running `rustdoc --test /home/misdreavus/git/asdf/src/lib.rs --crate-name asdf -L dependency=/home/misdreavus/git/asdf/target/debug/deps -L dependency=/home/misdreavus/git/asdf/target/debug/deps --extern asdf=/home/misdreavus/git/asdf/target/debug/deps/libasdf-31e1a62d36de9a99.rlib`

running 1 test
test src/lib.rs - SomeStruct (line 3) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

Semantically, i find nothing wrong with allowing doctests in bin crates. However, to pull this off, we would need to tell Cargo to compile bin crates as libs, then hand that rlib to rustdoc for testing. There's nothing stopping us from doing that right now:

[misdreavus@tonberry asdf]$ rustc --crate-type lib --crate-name asdf src/main.rs
[misdreavus@tonberry asdf]$ rustdoc --test --crate-name asdf src/main.rs --extern asdf=libasdf.rlib

running 1 test
test src/main.rs - SomeStruct (line 3) ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out

cc @rust-lang/cargo

[kornelski]

Wow, that's a nice surprise that it works. I was afraid that duplicate main would be a problem, but I guess rlib namespaces it!

[QuietMisdreavus]

Yeah, the main in your bin goes inside the lib crate, so it's just a regular function at that point. I did have to tag it with #[allow(dead_code)] to silence a warning, but otherwise it's all good.

[PhilipDaniels]

I just hit this today. I came here after finding https://github.com/rust-lang/cargo/issues/2009 in a Google search, which mentions that it is 'addressed in the documentation'. I suspect it may have been at one point but it doesn't appear to be now. I checked

https://doc.rust-lang.org/book/ch11-01-writing-tests.html

and from there the link

https://doc.rust-lang.org/book/ch14-02-publishing-to-crates-io.html#documentation-comments-as-tests

(Of course it's possible its mentioned somewhere and I missed it. A very careful reading of the surrounding text implies it, because it's all talking about libs, but nowhere I see is it mentioned explicitly.)

[joshtriplett]

I ran into this today, and found it confusing. I would love to be able to write doctests in the modules of my binary crate.

What would it take to make that work out-of-the-box?

[ehuss]

I can think of a few hurdles that someone would need to address:

• How to resolve symbols in a binary? Doctests behave as-if they are a separate crate linking in the library. But a binary is normally not used a library, so it has no name to refer to. For example:

/// ```
/// do_something();
/// ```
fn do_something() {}

fn main() {}

Here, do_something(); would fail because it is not in scope. You could try to scope it with the name of the binary (mything::do_something();), but then you have two new problems: what if you have a lib target of the same name? And, deal with the fact that you normally don't export any public items in a binary.

• How to deal with lack of public exports? Building a binary as a library will result in a lot of unused warnings. And since rustdoc imports the crate, usually nothing will be publicly exported, so you won't be able to access anything.

One (rough) idea to address this would be to fundamentally change how rustdoc --test works. Instead of creating a synthetic test that links to a library, embed the tests directly in the module. So it would rewrite the above example to:

fn do_something() {}

#[test]
fn do_something_doctest() {
    do_something();
}

So all local (private) symbols would be in scope.

That's a pretty radical difference, so I'm not sure if it would fly (would probably be confusing if there are two styles of doctests).

Also, I think it's worth examining if this is a good idea from the start. Documenting binaries still seems like a strange concept to me. Why can't you just move all the code to a library?

[joshtriplett]

On Sat, Mar 28, 2020 at 02:23:19PM -0700, Eric Huss wrote:

I can think of a few hurdles that someone would need to address:

• How to resolve symbols in a binary? Doctests behave as-if they are a separate crate linking in the library. But a binary is normally not used a library, so it has no name to refer to. For example:

/// ```
/// do_something();
/// ```
fn do_something() {}

fn main() {}

Here, do_something(); would fail because it is not in scope. You could try to scope it with the name of the binary (mything::do_something();), but then you have two new problems: what if you have a lib target of the same name? And, deal with the fact that you normally don't export any public items in a binary.

• How to deal with lack of public exports? Building a binary as a library will result in a lot of unused warnings. And since rustdoc imports the crate, usually nothing will be publicly exported, so you won't be able to access anything.

One (rough) idea to address this would be to fundamentally change how rustdoc --test works. Instead of creating a synthetic test that links to a library, embed the tests directly in the module. So it would rewrite the above example to:

fn do_something() {}

#[test]
fn do_something_doctest() {
    do_something();
}

So all local (private) symbols would be in scope.

I would love to see that as the normal approach to doctests,
personally. It would also unify the different kinds of tests. I do agree
that that would be a more invasive change, though.

Also, I think it's worth examining if this is a good idea from the start. Documenting binaries still seems like a strange concept to me. Why can't you just move all the code to a library?

Why should the concept of documentation and testing be limited to
libraries, and not supported in binary crates? I use documentation in
binary crates to document the code for later understanding. I'd like to
use tests to verify individual modules of functionality; that doesn't
necessarily mean I want to split that functionality into a separate
library crate.

[dkasak]

Just hit this today. I also frequently document internal modules of binary crates and intuitively expected that doctests would still work. Effectively, I use internal modules as non-public libraries. The reason I do not make them into separate library crates is that they are often not useful in a general context, but I do sometimes end up turning them into libraries. This kind of flexibility seems like a win.

[ayroblu]

Hey, so I just learnt about this while learning about the language. It's disappointing, and I'm not sure this has been prioritised so it doesn't sound like it's going to be addressed in the near future.

Anyways, I guess the only way then is to do the plebeian way and write a test for it

[GuillaumeGomez]

You can split your crate into two parts: the binary part (containing only the main function and the args parsing eventually) which calls all other functions and the "lib" part. Then you can document all functions inside the lib part. I did that for a few crates of mine and I think it's more than enough and is a better match for how doc should be handled. What do you think?

[ayroblu]

@GuillaumeGomez Do you have a good example or minimum POC with like cargo build? Sorry still learning here

[GuillaumeGomez]

Sure, here you go: https://github.com/gtk-rs/gir/blob/master/Cargo.toml

[slightlytwisted]

It sounds weird to have documentation examples in a bin target, don't you think? There is no documentation to generate so why would there be documentation's code to run?

I think mostly for consistency. If someone is used to writing docs with rustdoc and doctests on their library functions, it makes sense that they would want apply the same documentation style to their bin functions. On a small project, having to separate out helper functions to a separate library is an extra hoop to jump through, especially if it is just for the sake of getting tests to run.

I think underneath this is the assumption that people don't need to generate documentation for binaries. But in a team environment, its a good idea to document as much as you can, including examples. Having to switch documentation and test styles between libraries and binaries seems like an arbitrary constraint.

[dkasak]

But in a team environment, its a good idea to document as much as you can, including examples.

Yes, I even document extensively for myself! And you can never be sure whether someone will join you on a project in the future.

[steveklabnik]

I think underneath this is the assumption that people don't need to generate documentation for binaries.

It's not quite this, exactly. Rustdoc is a tool for documenting a Rust API. A binary does not have a Rust API. So rustdoc is not really meant for documenting binaries.

Where the friction really comes is when you want to produce internal documentation. This is a concern for libraries too, and that's why --document-private-items exists. But the way that rustdoc goes about generating this information ends up being tough for binaries, because it's oriented around lbiraries, by nature.

[kornelski]

Doccomments unfortunately couple two things, but I think in this case the concern of running (doc)tests could be separate from concern of documenting things. Some tests like compile_fail just have to live in a doccomment, regardless whether one wants to generate documentation or not.

To look at it from a different perspective: tests are run by cargo test --all, not cargo doc. The fact that rustdoc is involved in testing is an implementation detail. If you ignore the mechanics of how the tests are extracted, in the grand scheme it's just an issue of cargo test --all not running all tests.

[dogweather]

@joshtriplett Take a look at how Elixir does doc tests: in the form of an interactive REPL session:

defmodule MyModule do
  @doc """
  Given a number, returns true if the number is even and else otherwise.

  ## Example
    iex> MyModule.even?(2)
    true
    iex> MyModule.even?(3)
    false
  """
  def even?(number) do
    rem(number, 2) == 0
  end
end

This would be a big change, and implies a REPL that everyone is familiar with. But it's a very low-friction way to write tests. Maybe as a custom testing framework when that hits stable...

[samwightt]

Just hit this roadblock today with a smaller bin project that I was working on and it was very annoying. Doc tests should be standard across all project types, not just library.

[huntc]

I'm seeing this issue cropping up when I do have a library, albeit a static library...:

% cargo test --doc --verbose      
warning: doc tests are not supported for crate type(s) `staticlib` in package `mylib`
error: no library targets found in package `mylib`

Should I raise this as a separate issue? Thanks.