Go: cmd/go: write empty coverage report when there are no tests

[jspiro]

tl;dr

• go test -cover should output coverage for packages with .go files that lack tests to accurately report that there is no test coverage for that package
• go test should have a flag that causes it to fail if a package is found without any tests, and/or should fail if a .go file is found without associated _test.go

---

It (sort of, out of lazy convenience) makes sense that go test ./... succeeds if it finds a package with at least one .go file, but without tests, skipping it. But the same behavior doesn't make sense for coverage -- negative coverage reports should be created for packages without a single test but with go files.

The problem: Say you have 100 packages of Go code (each has approximately the same number of statements), and only one has tests, and those tests cover 100% of that package's code, then coverage for the repo will be 100% (only one coverage report is created). But it's really 1% coverage.

If you add an empty _test.go file to each package, you'll suddenly get negative coverage reports for every package, and your coverage will be correct. This is silly, you shouldn't have to remember to do this everywhere to get accurate coverage reports.

Proposal
Either (or both)

• Add a flag to go test that will fail if a package is discovered without tests (alerting us to the problem so we can add a test, and always get proper coverage)
• Have go test -cover output negative coverage repots even if no tests are found (or add a flag that will do this)

Nice to have

• A go test flag that fails if any .go file doesn't have an accompanying _test.go -- I'm sure some people would welcome this level of strictness

Up for debate

• Whether package main files must also have _test.go -- not sure how prevalent it is to have tests for these. I noticed that go test seems to try running these files if a test file exists; this causes my tests to fail since main packages tend to be commands that fail without input.
• Another caveat is that generated code like protobufs and mocks generally don't come with tests, so having a way to exclude a package might be necessary. Or maybe you just need to test your code!

What version of Go are you using (go version)?

go version go1.7 linux/amd64

[rsc]

When I run go test in a directory with no code, I get:

$ go test
?       rsc.io/tmp/asmvet   [no test files]
$ go test -cover
?       rsc.io/tmp/asmvet   [no test files]
$ 

It's true that the command succeeds, but the output is clear that no tests were run (and has been for at least the last few releases). Making this fail (even optionally) seems like overkill.

We could probably arrange for an empty cover profile to be written from the go command in this case though.

[rsc]

OK, so if we just write an empty cover profile file (no output change, no exit status change), is that enough, @jspiro? That seems reasonable to me.

[jspiro]

@rsc That should work as long as it marks all lines in the package as uncovered.

Two questions remain for debate:

• Should package main have an empty cover profile? IME they don't usually tested; but this is inconsistent and possibly stylistic.
• Generated protobuf code typically doesn't come with tests (since it's implicitly tested). Though there isn't way to determine that by package.

Neither is a real problem, since technically those lines of code have no coverage. They're just unlikely to ever get coverage.

[rsc]

Whether those matter depends on what your program does with the 0% coverage profile we're talking about writing out. That's unclear to me, but it seems fine to make sure that 'go test -cover' always writes a coverage profile.

[rsc]

Accepted for writing an empty cover profile file when there are no tests.

[JelteF]

Would be really great to have this included. Right now I work around this by adding an empty file (except for the package line) called xxx_test.go to packages without tests to work around this.

I would like to note though that there's an important difference between an empty cover profile and a cover profile containing all lines of the files with counts of 0. The one with 0 counts is what is useful, otherwise there is still no data for the cover tools to use. The 0 counts are what my above workaround results in so I doubt it would be hard to achieve that automatically.

[jspiro]

In my experience all I had to do was emit the package name to an otherwise empty file to get zero counts on everything.

On Apr 19, 2017, 6:08 AM -0700, Jelte Fennema [e-mail], wrote:
>

Would be really great to have this included. Right now I work around this by adding an empty file (except for the package line) called xxx_test.go to packages without tests to work around this.

I would like to note though that there's an important difference between an empty cover profile and a cover profile containing all lines of the files with counts of 0. The one with 0 counts is what is useful, otherwise there is still no data for the cover tools to use. The 0 counts are what my above workaround results in so I doubt it would be hard to achieve that automatically.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub (https://github.com/golang/go/issues/18909#issuecomment-295263446), or mute the thread (https://github.com/notifications/unsubscribe-auth/AAONwA2KrMBcYKTJ7SfMfwJUgHbQdNCqks5rxgdYgaJpZM4L11hy).

[gopherbot]

Change https://golang.org/cl/76875 mentions this issue: cmd/go: allow -coverprofile with multiple packages being tested