Go: cmd/go: go test -cover & go test -coverprofile should always output a coverage

[gustafj]

Description

Now in 1.10 when go test -cover supports multiple packages, I would expect it to print out a percentage for all packages (including those missing tests).
And for go test -coverprofile, I would expect all packages to be included in the calculated total.

Currently only packages that have at least one test (can be a *_test.go with only the package declaration) is included, see pkg2 below.

What version of Go are you using (go version)?

go version go1.10 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

Linux, amd64

What did you do?

go test ./... -cover
go test ./... -coverprofile cover.out; go tool cover -func cover.out

What did you expect to see?

?      path/to/pkg1   0.001s  coverage:   0.0% of statements [no test files]
ok     path/to/pkg2   0.019s  coverage:   0.0% of statements [no tests to run]
ok     path/to/pkg3   0.371s  coverage: 100.0% of statements

path/to/pkg1/pkg1.go:5: String 0.0%
path/to/pkg2/pkg2.go:5: String 0.0%
path/to/pkg3/pkg3.go:5: String 100.0%
total: (statements) 33.3%



md5-7f399644b05edafc01272b37636ff994



?      path/to/pkg1   [no test files]
ok     path/to/pkg2   0.019s  coverage:   0.0% of statements [no tests to run]
ok     path/to/pkg3   0.371s  coverage: 100.0% of statements



md5-ef691bc173e40c7f0b28ac7c398e23fb



path/to/pkg2/pkg2.go:5: String 0.0%
path/to/pkg3/pkg3.go:5: String 100.0%
total: (statements) 50.0%

[gopherbot]

Change https://golang.org/cl/115095 mentions this issue: cmd/go/internal/test: always output a coverage

[mvdan]

@kyroy noticed that this issue is very similar to #25492. It seems to me like both should be fixed at once, as they suggest changes in opposite directions. The other issue wants to change a 0.0% with a [no statements], and this one wants to change [no test files] with 0.0%.

/cc @bcmills @egonk

[bcmills]

they suggest changes in opposite directions.

25492 is about the case where there are no statements to cover (and thus the percentage is undefined). That's materially different from the case here, where pkg1 does have a statement and it is not covered: 0.0% coverage is well-defined and correct for a package with one statement and no tests.

[mvdan]

My reading of this issue is that go test -cover should always output a percentage if it succeeded. The other issue is precisely about removing a 0.0% output in favor of something else.

[bcmills]

My reading of this issue is that go test -cover should always output a percentage if it succeeded.

That's the current issue title, but that behavior seems clearly wrong if there are no statements to cover: 0/0 is not a well-defined percentage.

(I guess we could output NaN%, but that seems strictly less helpful than [no statements].)

[kyroy]

I like the idea of a consistent output that would be created by just implementing this issue.

Another approach is, as in #25492, for all statements it is true that they are covered by the tests. So one could argue for 100% which is obviously mathematically incorrect.

I am also the creator of the CL. Happy to implement a solution that results from the discussion :)

[gopherbot]

Change https://golang.org/cl/122518 mentions this issue: cmd/go: revert "output coverage report even if there are no test files"

[ianlancetaylor]

CL 115095 was reverted, so reopening this issue.

[agnivade]

FYI, f7248f05946c1804b5519d0b3eb0db054dc9c5d6 brought back the pkg{1,2,3} folders inside cmd/go/testdata/testcover that was originally reverted by https://github.com/golang/go/commit/7254cfc37b3a93a6e83dae22c4bfd6f777edb97e. Those files are unused as of now.

[Nerzal]

Any news here?

[rabadin]

One can use -coverpkg=./... to get accurate results:

$ tree .
.
├── coverage.out
├── package0.go
├── package0_test.go
├── package1
│   ├── package1.go
│   └── package1_test.go
└── package2
    └── package2.go

2 directories, 6 files

Without -coverpkg=./...:

$ go test -v -coverprofile=coverage.out ./...
=== RUN   TestMin
--- PASS: TestMin (0.00s)
PASS
coverage: 100.0% of statements
ok      github.com/rabadin/brokencoverage   0.007s  coverage: 100.0% of statements
=== RUN   TestAdd
--- PASS: TestAdd (0.00s)
PASS
coverage: 100.0% of statements
ok      github.com/rabadin/brokencoverage/package1  0.006s  coverage: 100.0% of statements
?       github.com/rabadin/brokencoverage/package2  [no test files]

$ go tool cover -func=coverage.out
github.com/rabadin/brokencoverage/package0.go:3:        min     100.0%
github.com/rabadin/brokencoverage/package1/package1.go:3:   add     100.0%
total:                              (statements)    100.0%

With -coverpkg=./...:

$ rm coverage.out

$ go test -v -coverpkg=./... -coverprofile=coverage.out ./...
=== RUN   TestMin
--- PASS: TestMin (0.00s)
PASS
coverage: 33.3% of statements in ./...
ok      github.com/rabadin/brokencoverage   0.006s  coverage: 33.3% of statements in ./...
=== RUN   TestAdd
--- PASS: TestAdd (0.00s)
PASS
coverage: 33.3% of statements in ./...
ok      github.com/rabadin/brokencoverage/package1  0.006s  coverage: 33.3% of statements in ./...
?       github.com/rabadin/brokencoverage/package2  [no test files]

$ go tool cover -func=coverage.out
github.com/rabadin/brokencoverage/package0.go:3:        min     100.0%
github.com/rabadin/brokencoverage/package1/package1.go:3:   add     100.0%
github.com/rabadin/brokencoverage/package2/package2.go:3:   mult        0.0%
total:                              (statements)    66.7%

[puellanivis]

Using -coverpkg does not actually help if the package is never called in any test at all. -coverpkg is a cover-up that allows packages from a different package to provide coverage for this package.

So, say package a has no tests, and thus no coverage. But package b has tests, and those tests call into a. As a result, if you start using a -coverpkg that covers both packages, the tests from package b will report its coverage of package a as coverage for a.

However, if we have a package c that is never called by either a or b then that package will still be left out in the dry, even if it would match the -coverpkg given.

project$ find . -name "*_test.go"
project$ go test -coverpkg=./... -coverprofile=coverage.out ./...
?       project [no test files]
?       project/sub/dir1    [no test files]
?       project/sub/dir2    [no test files]
project$ cat coverage.out
mode: set
project$

Additionally, it is misleading to say that because b has called into a that that coverage of a counts as tests on a, because they are not unit tests of a’s functionality, and b is fundamentally _not responsible_ for testing of a.

[rabadin]

@puellanivis:

Using -coverpkg does not actually help if the package is never called in any test at all.

I'm not sure this is true. Here is the code from the example above:

find . -name "*.go" -exec echo '******** {}' \; -exec cat {} \;
******** ./package2/package2.go
package package2

func mult(a int, b int) int {
    return a * b
}
******** ./package0_test.go
package brokencoverage

import "testing"

func TestMin(t *testing.T) {
    if min(3,5) != 3-5 {
        t.Errorf("min is broken")
    }
}
******** ./package1/package1_test.go
package package1

import "testing"

func TestAdd(t *testing.T) {
    if add(3,5) != 3+5 {
        t.Errorf("add is broken")
    }
}
******** ./package1/package1.go

package package1

func add(a int, b int) int {
    return a + b
}
******** ./package0.go
package brokencoverage

func min(a int, b int) int {
    return a - b
}

as you can see nothing is calling package2/package2.go:mult.

[puellanivis]

I don’t know exactly what is different, but even with -coverpkg=./... my project with no tests still reports no packages:

project-with-no-tests$ go test -v -coverpkg=./... -coverprofile=coverage.out ./...
?       project-with-no-tests   [no test files]
?       project-with-no-tests/subpackage    [no test files]
project-with-no-tests$ go tool cover -func=coverage.out
total:  (statements)    0.0%

While adding just a single _test.go with just the package name yields coverage details for a package.

Meanwhile, if I add a reference to package2.Mult in package0_test.go suddenly, with -coverpkg=./... we report that there is coverage of package2.Mult, even though it is not actually covered by any unit tests. (It would make sense to include such coverage in integration tests though, as there you are testing the system as a whole, but unit tests should never count coverage provided by any other package.)

broken-coverage$ cat package0_test.go 
package brokencoverage

import (
    "testing"
    "github.com/puellanivis/broken-coverage/package2"
)

func TestMin(t *testing.T) {
    if min(3,5) != package2.Mult(-1, 2) {
        t.Errorf("min is broken")
    }
}
broken-coverage$ go test -v -coverpkg=./... -coverprofile=coverage.out ./...
=== RUN   TestMin
--- PASS: TestMin (0.00s)
PASS
coverage: 66.7% of statements in ./...
ok      github.com/puellanivis/broken-coverage  0.001s  coverage: 66.7% of statements in ./...
=== RUN   TestAdd
--- PASS: TestAdd (0.00s)
PASS
coverage: 33.3% of statements in ./...
ok      github.com/puellanivis/broken-coverage/package1 0.004s  coverage: 33.3% of statements in ./...
?       github.com/puellanivis/broken-coverage/package2 [no test files]
broken-coverage$ go tool cover -func=coverage.out
github.com/puellanivis/broken-coverage/package0.go:3:       min     100.0%
github.com/puellanivis/broken-coverage/package1/package1.go:3:  add     100.0%
github.com/puellanivis/broken-coverage/package2/package2.go:3:  Mult        100.0%
total:                              (statements)    100.0%

[sallyruthstruik]

I wrote this simple script to get average metric from coverage.out file: https://gist.github.com/sallyruthstruik/72019ba0041deaa1a2c0a817b4603403

[luklss]

It would be really nice to get this forward.

[mvdan]

@luklss practically everyone agrees this should be fixed, but the details of the fix are unclear. Someone needs to put in the time and effort to figure that out.