Rules_go: Coverage of go test

[GinFungYJF]

Does the go rule still have no support for calculating the coverage of go test now? If so, is there any way to calculate it?

[yugui]

It does not support coverage right now.
Probably we need to wait for https://github.com/bazelbuild/bazel/issues/1118.

[GinFungYJF]

Any update on this? Or is there any usable workaround like
https://github.com/bazelbuild/bazel/issues/1118 ?

[pmbethe09]

AFAIK no one is working on this. If it is something you need, you may have
to be the one to implement it.

On Thu, Dec 1, 2016 at 7:18 AM, GinFungYJF [e-mail] wrote:

Any update on this? Or is there any usable workaround like
bazelbuild/bazel#1118 https://github.com/bazelbuild/bazel/issues/1118 ?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/rules_go/issues/140#issuecomment-264159841,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ALF-0gypcrlX564I8vd_Om9fLXppZ3mAks5rDrskgaJpZM4KVagd
.

[ghasemloo]

I think we can fix this directly, go tool itself support coverage independent of bazel.
I think what needs to be done is adding something similar to the block regarding coverage from go test main (https://golang.org/src/cmd/go/test.go?h=cover+flag#L1522)
to generate test main template: (https://github.com/bazelbuild/rules_go/blob/master/go/tools/generate_test_main.go#L129).

I tried it but I don't fully understand how flags are parsed in the go test, they are defined in
https://golang.org/src/cmd/go/testflag.go

[GinFungYJF]

@ghasemloo When I run go test -cover -x pkg, I see that it will do some preproccessing for the files that are not *_test.go files in the pkg before they are compiled.

go/pkg/tool/linux_amd64/cover -mode set -var GoCover_0 -o .A.go pkg/A.go
go/pkg/tool/linux_amd64/cover -mode set -var GoCover_0 -o .B.go pkg/B.go
...

Is it helpful to you?

[linuxerwang]

It would be great if the future bazel-go-coverage also supports external-test besides standard unit tests. Here are a few references of this technique:

https://blog.cloudflare.com/go-coverage-with-external-tests/
https://husobee.github.io/golang/test/coverage/2015/11/17/external-test-coverage.html
https://www.elastic.co/blog/code-coverage-for-your-golang-system-tests

I hope I could find time to work on it.

[pmbethe09]

@linuxerwang -- sounds great. Feel free to send some ideas here and then
of course send a PR .

On Sat, Jan 21, 2017 at 7:41 AM, linuxerwang [e-mail]
wrote:

It would be great if the future bazel-go-coverage also supports
external-test besides standard unit tests. Here are a few references of
this technique:

https://blog.cloudflare.com/go-coverage-with-external-tests/
https://husobee.github.io/golang/test/coverage/2015/11/
17/external-test-coverage.html
https://www.elastic.co/blog/code-coverage-for-your-golang-system-tests

I hope I could find time to work on it.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/bazelbuild/rules_go/issues/140#issuecomment-274259366,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ALF-0kVhZC4CZ0nFDvf92v2h8TbnemMdks5rUfz1gaJpZM4KVagd
.

[GinFungYJF]

Anybody has any idea about implementing it ? I don't know where to start :(

[jayconrod]

@GinFungYJF We haven't given this a lot of thought. It seems clear this will be a lot of work though.

Bazel has some support for exposing coverage information to Skylark. ctx.coverage_instrumented() would be called inside go_library_impl to check whether coverage instrumentation is needed. An instrumented_files provider should be returned (I'm not sure how Bazel uses that information though).

The Go code itself needs to be modified to include coverage instrumentation. The cover story provides some information on how this works. It looks like go test inserts coverage instrumentation at the source level before the compiler sees it. I'm not familiar with the implementation of go test, but test.go is the starting point. A lot of that implementation is private, and if it looks like we would need to duplicate it, we should talk to the Go Libraries team about providing a stable API we could use.

If you decide to work on this, please post a design doc that we can discuss and comment on.

[jayconrod]

@GinFungYJF I spoke with @alandonovan, and he filled in some gaps in my knowledge here.

go tool cover is the tool that adds coverage instrumentation to Go sources. That's already part of the standard distribution, so we wouldn't need to re-implement that. We'd just need to generate actions and files inside the rules to make sure that sources are processed with that before they are compiled when coverage is enabled.

Bazel expects coverage information to be in lcov format. We have some code in Blaze (Google-internal version of Bazel) that produces lcov data at the end of a Go program using the counters added by the instrumentation. This is linked into instrumented binaries. We should be able to open source this or re-implement it without too much work.

[linuxerwang]

@jayconrod Thank you for providing these info, although I've figured them out by myself. What held me back was actually I could not find a way to elegantly enable it in rules_go:

Suppose we have pkg deps like: A (main) -> B -> C. If we want to enable test coverage for A, we have to add corresponding BUILD targets also for B and C.

go_binary(name = "A",deps = ["B", "C"]
go_binary(name = "A_cover", test_coverage = true, deps = ["B_cover", "C_cover"])
go_library(name = "B")
go_library(name = "B_cover", test_coverage = true)
go_library(name = "C")
go_library(name = "C_cover", test_coverage = true)

This is too cumbersome. The ideal way is as follows:

go_binary(name = "A",deps = ["B", "C"]
go_test_coverage(name = "A_cover", deps = ["A"])

But it's probably impossible in bazel, since if A has been built before A_cover, A will not be triggered by A_cover to rebuild.

[pmbethe09]

This might be possible with aspects

https://bazel.build/versions/master/docs/skylark/aspects.html

[jayconrod]

@linuxerwang Ideally, you shouldn't need to define separate targets with and without coverage; it should be a configuration option. Bazel already has a coverage command that runs tests in coverage mode. We just need to wire up the rules to instrument code when that option is enabled and generate output in the appropriate format.

[linuxerwang]

@pmbethe09, @jayconrod Cool! I'll invest some time on it this weekend.

[linuxerwang]

Alright, I made an early stage experiment, the instrumentation is working. But we need more work on enable the coverage hookup in the generated test main func.

https://github.com/linuxerwang/rules_go/commit/42f50569362ab70cfc11037ff265db5eac45f33b

You can run "bazel coverage examples/lib:lib_test". It simply passes, without coverage collected. And the test binary has not coverage enabled:

$ ./bazel-bin/examples/lib/lib_test -test.coverprofile=report.txt
testing: cannot use -test.coverprofile because test binary was not built with coverage enabled

Of course, we also need more work on lcov format mentioned by @jayconrod.

External test coverage might be feasible through aspects, but I didn't get a chance to try.

[jayconrod]

@linuxerwang Thanks for working on this, it looks promising. A couple comments:

• It would be better to run go tool cover through an action for each file. This will let Bazel parallelize the coverage transformation and cache the results. Basically, define a new Skylark function which takes a list of sources as an argument, creates new files for the transformed sources (ctx.new_file), runs go tool cover for each of them (ctx.action), then returns the transformed files. _emit_go_compile_action can call this function if coverage is enabled and use the result as a replacement for srcs, before calling symlink_tree_commands.
• Any new Skylark functions in def.bzl should be private (name starts with '_'). This prevents other projects from depending on implementation details.

[ghasemloo]

what I was suggesting was something similar to what I did for benchmark:
https://github.com/ghasemloo/rules_go/commit/fa2088cd8d3e367902d6d896d1deccfec5da76c2
Essentially modify the main file that is generated by bazel for go tests (https://github.com/bazelbuild/rules_go/blob/master/go/tools/generate_test_main.go) to be similar to what go cover does.

I looked at it a bit but it seemed complicated than I had to time to dig into. I still feels it should be possible to just modify https://github.com/bazelbuild/rules_go/blob/master/go/tools/generate_test_main.go
to make bazel test with something like -- --cover work but I am not completely sure it is really viable or even the right thing to do.

[jayconrod]

In the long run, it's important for us to be compatible with bazel coverage. That means transforming code in response to ctx.coverage_instrumented() and generating data in lcov format. Bazel has some (rather sparse) documentation on controlling instrumentation in Skylark.

I'm not sure what the current status of this in Bazel is right now. It doesn't seem like it works yet. I'm watching bazelbuild/bazel#1118 for updates though.

[linuxerwang]

Finished the change as @jayconrod suggested. Created pull request https://github.com/bazelbuild/rules_go/pull/455.

Will work on generate_test_main.go in a later pull request.

[linuxerwang]

Actually, added the change to test main generator in the same pull request.

For an example run, execute:

$ bazel coverage examples/lib:lib_test

It generates the following files:

$ ls -l bazel-out/local-fastbuild/bin/examples/lib/
total 2016
-r-xr-xr-x 1 linuxerwang linuxerwang     471 May 14 20:57 asm_GoCover_0.go <====
drwxrwxr-x 3 linuxerwang linuxerwang    4096 Mar 18 17:49 bazel-out
drwxrwxr-x 4 linuxerwang linuxerwang    4096 May 14 12:42 deep
drwxrwxr-x 3 linuxerwang linuxerwang    4096 May 13 21:01 github.com
-r-xr-xr-x 1 linuxerwang linuxerwang     623 May 14 20:57 lib_GoCover_1.go <====
-r-xr-xr-x 1 linuxerwang linuxerwang 1931232 May 14 20:57 lib_test
drwxrwxr-x 2 linuxerwang linuxerwang    4096 May 13 21:01 lib_test.dir
-r-xr-xr-x 1 linuxerwang linuxerwang     630 May 14 20:57 lib_test.GoTestGenTest.params
-r-xr-xr-x 1 linuxerwang linuxerwang   22936 May 14 20:57 lib_test_main_test.a
-r-xr-xr-x 1 linuxerwang linuxerwang     876 May 14 20:57 lib_test_main_test.a.GoLinkFile.params
-r-xr-xr-x 1 linuxerwang linuxerwang    1940 May 14 20:57 lib_test_main_test.go
-r-xr-xr-x 1 linuxerwang linuxerwang   22637 May 14 20:57 lib_test_main_test.o
-r-xr-xr-x 1 linuxerwang linuxerwang   32892 May 14 20:57 lib_test.o
drwxrwxr-x 3 linuxerwang linuxerwang    4096 Mar 18 17:49 lib_test.runfiles
-r-xr-xr-x 1 linuxerwang linuxerwang     183 May 14 20:57 lib_test.runfiles_manifest

The instrumented files are named in such a way that the generated test main can extract cover vars easily. The code logic in generate_test_main.go is borrowed shamelessly from $GOROOT/src/cmd/go/test.go and related files.

$ ./bazel-out/local-fastbuild/bin/examples/lib/lib_test
2 + 3 = 5
PASS
coverage: 40.0% of statements

It has the following features unimplemented:

1) make -test.coverprofile=report.txt work
2) add lcov output support
3) make cover mode configurable
4) add aspect to make external coverage tests work

[linuxerwang]

I took a short time trying the coverprofile output. The currently generated test binary has all the mechanism outputting everything:

$ ./bazel-bin/examples/lib/lib_test -test.coverprofile=a.out
2 + 3 = 5
PASS
coverage: 40.0% of statements

$ cat examples/lib/a.out 
mode: set
bazel-out/local-fastbuild/bin/examples/lib/asm_GoCover_0.cover.go:8.22,10.2 1 1
bazel-out/local-fastbuild/bin/examples/lib/asm_GoCover_0.cover.go:12.22,14.2 1 0
bazel-out/local-fastbuild/bin/examples/lib/lib_GoCover_1.cover.go:27.20,29.2 1 1
bazel-out/local-fastbuild/bin/examples/lib/lib_GoCover_1.cover.go:34.23,36.2 1 0
bazel-out/local-fastbuild/bin/examples/lib/lib_GoCover_1.cover.go:39.25,41.2 1 0

However, I am stuck at two problems:

1) how to pass the argument "-test.coverprofile=a.out" to "bazel coverage" run.
I can use flag.Value.Set(), but probably a bad idea.

2) how to put "a.out" in bazel-testlogs/examples/lib/lib_test.

Can bazel team please shed some light on it?

Thanks.

[jayconrod]

If bazel coverage is run with no other arguments, I think it should communicate with Bazel in whatever format it expects (lcov?). I don't think coverage is really well-supported in Bazel yet though: bazelbuild/bazel#1118 is still open, although there has been some activity in the last few months.

You can still explicitly set the -test.coverageprofile flag using --test_arg. I don't think this should be set automatically though.

[linuxerwang]

@jayconrod Question is, after running bazel coverage --test_arg=test.coverprofile=cover.out examples/lib:lib_test, where is the file cover.out generated? I could not find it anywhere.

[Mistobaan]

Any progress on how to run coverage profiling using bazel? is it possible as of today? I can run bazel coverage on my tests but then is not clear how I inspect the results.

[jayconrod]

No change on this recently. We've been focusing on toolchains, proto support, and dependency management lately. I'm hoping to get coverage into a better state by the end of the year.

[dkt1412]

So I ran this (quotes are significant!)
bazel coverage \<test-target\> --test_arg="-test.coverprofile=coverage.out"
and this generated the coverage file (woohoo) at
bazel-bin/\<test-target-path\>/\<target-test\>.runfiles/\<test-target-path\>/coverage.out
The problem I have now is how to turn this into a html report. I can't seem to use go tool cover --html=\<path-to-coverage.out\> b/c main.go doesn't actually live in my GOPATH (which I also can't set up b/c our bazel project doesn't have the go-required src/ setup).

Is there a way to get bazel to run go tool cover --html for us? I assume we'd have to add some sort of actions to link to go tool cover like @linuxerwang did, but it's unclear to me where that needs to go.

[jayconrod]

Sorry, we still don't have a good story for this yet. I hope we'll be able to get to it soon, but cross-compilation and dependency management are bigger pain points right now.

You could try using the go_path rule, which builds a GOPATH-like source tree. Maybe cover could be run in there. See BUILD.bazel for an example of that.

[carlfriess]

Any idea when this might become available? Being able to use bazel coverage for go projects would be super useful.

[jayconrod]

@carlfriess This is largely blocked on Bazel improving coverage support for Starlark rules. You can follow the Bazel roadmap (last updated in 2018) and Bazel issues labeled coverage.

rules_go already instruments packages for coverage when using bazel coverage, but it records data in the Go format. Bazel needs a standard format we can convert the data to, it needs to collect that data and merge it across multiple targets, and it needs to be able to generate useful reports.

[carlfriess]

Thanks for clarifying! Is there anywhere I can find the recorded data? Using --test_arg="-test.coverprofile=coverage.out" from above just made the test fail.

[jayconrod]

For now, coverage data is stored in bazel-testlogs/<target>/coverage.dat. Bazel sets that location through the COVERAGE_OUTPUT_FILE environment variable. It's in Go's coverage format for now, but it will likely change in the future when there's a standardized coverage format for Bazel.

[carlfriess]

Cool, it does seem to create that file but it's empty. Maybe I'm missing something. Is there some example on how to correctly configure a target for this to work?

[jayconrod]

There are some tests in //tests/core/coverage, but I can't help much without knowing more about your project. It would be better to discuss that in a new issue though.

[carlfriess]

Oh I found the problem. I didn't realize I needed to include the source files in the go_test rule using embed rather than srcs. It's correctly writing the coverage file now. Thanks for your help!

[sluongng]

I have been hacking on this recently.

The main pain point with trying to make bazel coverage //... works from a user perspective is the lack of documentation of what are the current state of things. So in hope of making a quick summary for 2021, here is what I have so far:

1. Currently bazel coverage //... works, but there are a few problem:
   a. The output coverage data is using golang coverage profile format and not the Bazel-expected XML format.
   b. The output are created on the package level and not on the project level (which many early adopter might be expecting)

2. Using the coverage output:

   • If you have the need to get workspace-level coverage data, you need something to combine multiple packages coverage profile data files into 1 big file. Following some of the PR/Issue discussions, I came up with this genrule

   genrule(
   name = "combined-coverage-data",
   srcs = [":coverage-data"],
   outs = ["coverage.out"],
   cmd = "$(locations @com_github_wadey_gocovmerge//:gocovmerge) $(locations //:coverage-data) > $@",
   tags = [
       "local",
       "manual",
   ],
   tools = ["@com_github_wadey_gocovmerge//:gocovmerge"],
   )
   

   This can be bazel build //:combined-coverage-data after bazel coverage //... have ran and will combine all the files into 1 big bazel-bin/coverage.out for later usage.

   • Consuming the go coverage profile file:
     Most scriptures I found either recommend using go tool cover -html=coverage.out -o=$@ or using some tooling such as github.com/axw/gocov/gocov and/or github.com/elliotmr/gocover-cobertura. The problem is that most of these tools require 1 way or the other to have all the packages under a declared GOPATH somewhere, or GOROOT must be defined, which overall is hard to setup (and I don't know how to wire into a genrule). There is a go_path rule available to experiment further.

---

Right now my intuition is telling me that I should be generating the coverage profile file using bazel, but after that I should use go tool cover -html=cov.out -o=cov.html or bazel run @go_sdk//:bin/go -- tool cover -html=bazel-bin/coverage.dat -o=test2.html

🤔 What am I missing?

[sluongng]

I have some time today after taking a good break from bazel.
Made small progress:

# Generate test coverage data in Cobertura's XML format
#
# Running `bazel coverage //...` is required *before* building this target.
#
# This is the same as:
#   gocovmerge bazel-out/**/coverage.dat | <set env vars> gocover-cobertura >bazel-bin/coverage.dat
genrule(
    name = "cobertura-coverage-data",
    srcs = [":coverage-data"],
    outs = ["cobertura.xml"],
    cmd = " ".join([
        "$(locations @com_github_wadey_gocovmerge//:gocovmerge)",
        "$(locations //:coverage-data)",
        "|",
        "PATH=$$PATH:$$(dirname $(locations @go_sdk//:bin/go))",
        "GOPROXY=https://my-internal-go-proxy/",
        "GONOSUMDB=vcs.org.com",
        "GOCACHE=/tmp",
        "HOME=/tmp",
        "$(locations @com_github_boumenot_gocover_cobertura//:gocover-cobertura)",
        ">$@",
    ]),
    tags = [
        "local",
        "manual",
    ],
    tools = [
        "@go_sdk//:bin/go",
        "@com_github_wadey_gocovmerge//:gocovmerge",
        "@com_github_boumenot_gocover_cobertura//:gocover-cobertura",
    ],
)

The above worked, but setting those environment variables are hacky at best 🤔

The problem is mainly that gocover-cobertura https://github.com/boumenot/gocover-cobertura/blob/7bc1b925ff29267be04ee0693f39a099ea54f387/gocover-cobertura.go#L18 depends on golang.org/x/tools/go/packages which shell out to collect packages information using go list -json (?).

For that reason:

• go binary needs to be available in $PATH
• GOCACHE and HOME needed to be set
• GOPROXY and GONOSUMDB is for the whole thing to work with my internal packages

After a very quick read, I think I may be able to decouple gocover-cobertura away from tools/go/packages by making some naive assumption about my setup 🤔 thus simplify how this is generated.