Go: cmd/go: document exit codes of a process executing `go test`

[kostix]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.10.3

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/user/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/user/devel/go"
GORACE=""
GOROOT="/home/user/devel/golang1.10"
GOTMPDIR=""
GOTOOLDIR="/home/user/devel/golang1.10/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build141686204=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I run go help test and try to search for keywords "return", "exit", "code" in the output.

What did you expect to see?

I expect the possible exit codes for running go test be documented
in order to make sensible decisions about running this command in scripts
(and from Makefiles, by a CI system etc).

What did you see instead?

No menton of the return/exit code.

---

I'd expect the following cases to be documented if implemented:

• Successful run—all tests compiled and passed OK—exit code 0.
• Failed run—the compilation of at least one package failed or at least a single test failed—a non-zero exit code.
• Invalid parameters passed and/or one of the listed packages wasn't found—a non-zero exit code,
  different from that of the former case.

I'd also think that the cached results should effectively "memorize" the exit codeused ATM the results were generated (and cached).

See also #23716.

[GTB3NW]

Ignore me I'm an idiot. t.Error != t.Fail

I agree with the OP however, it should have a none 0 status, 1 for error, 2 for fail maybe?

[ianlancetaylor]

Right now the exit status is 1 for most failures, 2 if the build was interrupted.

[smasher164]

Do we distinguish between flag parsing errors where different exit codes are output? For example, passing in "-c=invalid" returns a 2

$ go test -c=invalid
go test: illegal bool flag value invalid
run "go help test" or "go help testflag" for more information
$ echo $?
2

while passing in "-covermode=invalid" returns a 1.

$ go test -covermode=invalid
invalid flag argument for -covermode: "invalid"
$ echo $?
1

[kostix]

@GTB3NW note that your particular problem is most possibly not related to the essense of this issue which is about documenting the implemented behaviour which appears to be OK.
Here at my $dayjob, in our CI pipeline, we call go test multiple times with different command-line options (say, including -race for nightly tests and also -bench … etc), and go test has always exited with a non-zero exit code on error since, I reckon, at least v1.6, and I really have no reason to think it was otherwise before.

I've created this issue merely because such things ought to be documented for reference purposes (say, for DevOps folks which may be not too familiar with Go to implement the CI pipeline).

Try to figure out what may mask the exit code of your go test process.
Note that all /bin/sh-style shells found in commodity OSes by default run in a mode which ignores any errors, that means, if you have a script like

go test ...
echo Whatever

this code won't ever exit if go test fails.
(And these shells did this since forever; and this is even mandated by POSIX.)
You either need to re-write your scripts so that they run with set -e -u (almost always a good thing to do) or explicitly test whether a command of interest failed, like in

go test ...
rc=$?
if [ $rc -ne 0 ]; then
  echo "testing failed" >&2
  exit $rc
fi

---

Updated @GTB3NW, ah, I see you got sorted it out :-)
Sorry for the noise then—should have re-read the thread rather than relying on comment e-mailing (which ignores edits).

[GTB3NW]

Okay here's one for you then, I've read the documentation, perhaps I'm missing something here, but the following should FAIL and return error code 1 should it not?

blahtest/blah_test.go:

package blahtest

import "testing"

func TestBlah(t *testing.T) {
    t.Fatal("Blah")
}

go test <redacted>/blahtest -v

=== RUN   TestBlah
--- FAIL: TestBlah (0.00s)
        blah_test.go:6: Blah
FAIL
ok      <redacted>/blahtest      0.004s

Should that not be a FAIL on /blahtest too?

The aim is to just run:

go test -v -short $(go list ${PKG}./... | grep -v /vendor/)

Test all the packages and if any one of them fails the whole test fails, surely that's a common requirement. Either I'm misunderstanding the testing facilities/there's a bug/go test is lacking something critical.

Could anyone set me in the right direction please I'd really appreciate it :)

[bcmills]

@GTB3NW, if you're seeing a 0 exit code or ok summary for a failed test, please file a separate issue with steps to reproduce.

[bcmills]

Before we document the error codes, I think we will probably need to do some work to make them consistent. That's a pretty big investigation, and probably ought to happen as part of a broader go test cleanup (which I hope to do for Go 1.14 or 1.15, but don't have a specific timeline).