Go: proposal: cmd/go: support compiling all tests without running

[bradfitz]

I want this to work:

$ go test -c std cmd
cannot use -c flag with multiple packages

But I don't actually care about the binaries. I just want to test that all the tests can compile, and how fast. @randall77 also wants this for SSA coverage reasons.

[cespare]

Is go test -run xxxxx std cmd a reasonable workaround for at least some of those use cases?

[josharian]

And I want this, but I do want the test binary. I'd use it to compile tests, change the compiler, compile tests again, and then run before/after benchmarks. (All in a script, of course.) Like toolstash for std tests. I currently do something similar but with go list std and a lot of bookkeeping and go test -c invocations.

It seems to me we could generate a testmain that invokes all the tests in all the included packages. There are some questions: How do you know from the -v output which package this particular instance of TestReader is from? How does the -run filter work? And so on. My hand-waving answer is: Mimic the UI for subtests and sub benchmarks, which already have this nesting problem; the nesting is just up a level rather than down.

[minux]

we can't generate one binary for tests in multiple packages
because there is no guarantee that package A's test doesn't
interfere with package B's use of package A.

[josharian]

What if we leave that up to the user to diagnose? If package A's test interferes with package B's use of package A, then don't test them together. This sort of issue can already arise in a single package.

[minux]

how could the user know that one can't test package A and package B
together?

consider "go test -c ./...", which could easily contain hundreds of
packages.

[josharian]

They'll know when the test fails. If the tests pass when run independently (i.e. go test ./..., no -c), then there's clearly an interaction problem.

[minux]

Even if they could know the bad interaction between A & B (it's
not that easy because given that B's tests failed, you still don't
know which of B's dependent [both directly and indirectly] might
be the cause. What's more, the behavior also depends on the
order tests for different packages are run.) because cmd/go
doesn't support negative patterns, there is no way to use ./...
and exclude B.

Given a large enough GOPATH, it's very likely there are bad
interactions between the package tests, and then basically
the new feature is useless (in the worse case, the user has
to name all the packages on the command line, and it's no
better than status quo.)

We might be able to do this if multiple packages are compiled
into a single package from the beginning, but I don't think we
can add this feature now.

[josharian]

I can't imagine someone wanting to run go test -c ./... on their entire GOPATH unless they had a very good reason. I can, however, see this for a particular project or set of packages, in which case this becomes very useful. Think e.g. of cross-compiling all the tests for your project and having to only scp a single binary.

cmd/go doesn't support negative patterns

Yes, this is unfortunate.

I apologize, but I don't quite understand the resistance here. It seems like the resistance is "something could go wrong". But this doesn't really seem like a big footgun to me. And when this is helpful, like cross-compiling all tests for a big project or keeping around an old version for benchmarking or testing compilation speed, it is very helpful, and users will be motivated to make it work (or not use it). And it is entirely backwards compatible.

[minux]

I think this issue gives one reason for do go test -c ./... for the
whole GOPATH: test the SSA compiler code generation.

The key point is most tests are not prepared for this.
How many packages support go test -count=2?
Even some of the std packages don't, until they're
fixed not long ago.

And how do you propose to handle conflicting flags added
by tests and multiple TestMains? Again, these features
are just not designed to combine tests for multiple packages
into a single executable.

Also note that tests for a package needs to run at the
directory for the package, how to handle that when
tests for multiple packages are combined into one binary?

(FTR, I have been thinking about this feature a long time
ago, but my conclusion was and still is that the whole thing
is not designed for this.)

[josharian]

How many packages support go test -count=2? Even some of the std packages don't, until they're
fixed not long ago.

Exactly. And they got fixed because people cared. Same situation here, I'd say.

conflicting flags added by tests

Sorry, what do you mean by this?

multiple TestMains

Refuse to compile, with a lucid error message. Rare.

tests for a package needs to run at the directory for the package

os.Chdir? Note that the go command has the directory available at compilation time. This is a bit ugly for the cross-compilation case, but we could print a warning instead of failing when os.Chdir fails.

I have been thinking about this feature a long time
ago

Yeah, I recall there being another issue about this but couldn't find it.

I agree that there are limitations, I just see them as limitations rather than showstoppers.

[minux]

By conflicting flags added by tests, consider two
packages that both add -update flags to update
golden output of their test data.

The code will compile, but will fail at runtime with
a flag redefined panic.

There are just too many global states, hence I
said the whole system is not designed for this.

[mattklein123]

I'm new to Go, but FWIW, the lack of this feature is one of the most frustrating things that I have found so far about the tooling. I tend to write production code and test code at the same time, and during my normal dev cycle I want to see if things compile many more times than I want to run the tests. So right now I'm doing roughly the same thing as stated above inside my project with a few packages: basically

for i in go list ./...; do go test -c $i; done

It would be great if this could be done in one pass with a single link and have it be up to the user to deal with conflicts as suggested by @bradfitz

[davecheney]

If you want to compile but run not tests,

 go test -run=nope ./...

Replace nope with another pattern if you have test cases that match.

On Fri, 27 May 2016, 03:45 Matt Klein [email protected] wrote:

I'm new to Go, but FWIW, the lack of this feature is one of the most
frustrating things that I have found so far about the tooling. I tend to
write production code and test code at the same time, and during my normal
dev cycle I want to see if things compile many more times than I want to
run the tests. So right now I'm doing roughly the same thing as stated
above inside my project with a few packages: basically

for i in go list ./...; do go test -c $i; done

It would be great if this could be done in one pass with a single link and
have it be up to the user to deal with conflicts as suggested by @bradfitz
https://github.com/bradfitz

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
https://github.com/golang/go/issues/15513#issuecomment-221943239

[mattklein123]

@davecheney I'm probably doing something stupid but running that command still causes all the tests to execute. I tried:

go test -run nope ./...

Also still runs tests.

[davecheney]

Please try -run=nope

On Fri, 27 May 2016, 09:56 Matt Klein [email protected] wrote:

@davecheney https://github.com/davecheney I'm probably doing something
stupid but running that command still causes all the tests to execute. I
tried:

go test -run nope ./...

Also still runs tests.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/15513#issuecomment-222027132, or mute
the thread
https://github.com/notifications/unsubscribe/AAAcA8xNrwQtBDQEF3AeZEF-377nOLMUks5qFjMwgaJpZM4IV2e-
.

[mattklein123]

I tried both:

mklein@vm:~/Source/go/src/github.com/<redacted>$ go test -run=nope ./...
ok      github.com/<redacted>   0.025s

[cespare]

-run=nope and -run nope are equivalent.

@mattklein123 no tests are being run (probably). Use -v to confirm.

Please use the mailing list if you have further questions about go test.

[davecheney]

That is correct. The test binary is built and executed, but none of the
test cases will be executed because they don't match the -run filter

On Fri, May 27, 2016 at 10:11 AM, Caleb Spare [email protected]
wrote:

-run=nope and -run nope are equivalent.

@mattklein123 https://github.com/mattklein123 no tests are being run
(probably). Use -v to confirm.

Please use the mailing list if you have further questions about go test.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/15513#issuecomment-222029023, or mute
the thread
https://github.com/notifications/unsubscribe/AAAcA8u95PM2MDjAGCytHBIVmp0py0RIks5qFjaxgaJpZM4IV2e-
.

[jurij]

i prefer -run ^$

[bradfitz]

My motivation is for the builders, and test sharding.

The problem with go test -run=^$ std cmd is that it deletes the *.test binaries afterwards. I want the test binaries, to ship around the network and shard out isolated test execution. But I want to burn CPU as hard as possible where it's not latency sensitive. (That is, I can run compilation jobs at 100% CPU, but tests are often flaky at 100% if they sleep or depend on time.)

[minux]

@bradfitz, I think for your purpose, having multiple go test -c processes
is better than figuring out where to store the compiled binaries when
building multiple tests (the only safe way is to put the compiled test
binary in their own package directories, but then you need to collect them.
It's probably easier to just go test -c with explicitly output filename).

It won't do much more work either (assuming go install std cmd has passed.)

[bradfitz]

I can do it myself, but my concern is that I can't keep the CPU busy as well as the cmd/go binary could.

cmd/go can load the dependency graph once, come up with a plan, and keep a certain number of worker processes running at all times.

Any coordination I do trying to run multiple cmd/go binaries wouldn't be as good.

Definitely low priority, though.

[rsc]

There are at least three different things people might want here (and have asked for either here or other places):

• Build all binaries and don't save them.
• Build all binaries and do save them (maybe in a directory tree).
• Compile but don't link (for quick turnaround in save hooks).

I propose we keep accumulating these kinds of go command issues and try to take a coherent look at them in the first half of next year.

I'll create a new GoCommand label. Feel free to label other (proposal or non-proposal) issues with that too.

Thanks.

[gopherbot]

CL https://golang.org/cl/42531 mentions this issue.

[FlorianUekermann]

It would be great if this enabled "buildall.bash" to also check if all tests compile. That would have saved me some embarrassment when submitting CLs.

[shivakumargn]

I have a large C based codebase which has Go wrappers. I would like to write Go test packages (only test *.go code) as test drivers for the C code, using the go wrappers, create go test binary and ship the binary with the installed software as a test suite for the environment. This is not a typical test suite in a dev machine.
The structure of go source code and the godoc helps in organizing the test suite nicely out of the box.

[titpetric]

Just a note for future travellers:

• if you're doing something like go test ./... it will continue with all package tests even if one of them fails to compile (they effectively run individually, as in O(N) not O(1)),
• the flag -failfast doesn't seem to have any effect on this (1.12), all package tests will run,
• note that the trick -run=^$ from above still runs ANY TestMain() which you may have - it's false to say that you are only compiling tests. This is especially dangerous with integration tests as you may populate an external service with data from TestMain,...

Overall: ouch

[azr]

Hi there, I think this could make multiplatform testing more simple/easy. (At least for me):

1/ have your ci in docker
2/ cross build your test binaries for the systems you care about.
3/ link & test

This should be fairly simply doable on circle-ci for example, since they recently added macos and windows support.

This way I don't have to install go for every OS I want to test upon.

I wouldn't mind that multiple binaries could be generated, if I can get a parsable list of generated binaries.
In my case I think the argument of pkg.test A making pkg.test B fail is not an issue as it would be super easy to compartmentalise were these tests are being run.

And for now I will do this manually using that good old go list ./... 🙂

Cheers !

PS: Happy to contribute if need be !

[earthelf]

-run ^$ not works, -run=nope not perfect. I prefer -count=0

[robert-zaremba]

Any hope that we will be able to compile all tests and cache the results without running the tests?

[donutloop]

Any progress? I would like to have this feature. Thank you.

[billinghamj]

I would highlight again the desire to be able to build all tests into a single binary. If go test ./... works, it seems strange that it'd be impossible to make a single binary which does the same thing

[ianlancetaylor]

go test ./... works by building and running several different binaries. In particular, test binaries assume that they are run in the directory of the package being tested, so that they can refer directly to testdata files. That seems awkward if we build a single binary for multiple packages.