Go: x/tools: tests consistently failing on linux-amd64-longtest builder

[bcmills]

There are two x/tools tests consistently failing in the longtest builder (example).

golang.org/x/tools/go/internal/gcimporter:

haserrors/haserrors.go:2:22: cannot convert "" (untyped string constant) to untyped int
haserrors/haserrors.go:3:18: undeclared name: undefined
cgo failed: [go tool cgo -objdir /workdir/tmp/os_signal_internal_pty_C916411543 -- -I /workdir/tmp/os_signal_internal_pty_C916411543 pty.go]: signal: killed
cgo failed: [go tool cgo -objdir /workdir/tmp/net_C637398552 -- -I /workdir/tmp/net_C637398552 cgo_linux.go cgo_resnew.go cgo_socknew.go cgo_unix.go]: signal: killed
Assembler messages:
Fatal error: can't create /workdir/tmp/os_signal_internal_pty_C916411543/_cgo_.o: No such file or directory
Assembler messages:
Fatal error: can't create /workdir/tmp/net_C637398552/_cgo_.o: No such file or directory
FAIL    golang.org/x/tools/go/internal/gcimporter   27.795s

golang.org/x/tools/go/ssa:

signal: killed
FAIL    golang.org/x/tools/go/ssa   46.257s

Consistently-failing tests can mask real regressions, and the longtest builder is especially important because it is often the only builder running those tests.

CC @ianthehat

[bcmills]

The go/ssa failure seems to be fixed, but go/internal/gcimporter is still failing consistently (https://build.golang.org/log/7aad7eab99b2982e013ba0dbf6bb1ff11fe02be2).

@ianthehat, this is one of the few remaining builder failures on release-branch.go1.13 ahead of the 1.13 release (#11811). Could you get someone to look into it?

[heschik]

@dmitshur

The gcimporter tests use ~2G and ~2.5G of RAM respectively. I strongly suspect they're getting OOM killed but I don't know how to prove it.

[gopherbot]

Change https://golang.org/cl/192677 mentions this issue: go/internal/gcimporter: disable gcimporter test on linux-amd64-longtest

[bcmills]

Is it feasible to reduce the footprint of the test? 2–2.5GiB seems awfully high...

[heschik]

That would seem like a good idea, but not as a release blocker.

Looking at a gomote, it has 8 cores and 8G RAM. The go command runs one command per core, and I have no trouble believing it eats up all the memory with 8 concurrent fairly heavy tests. Unless there's some way to reduce the parallelism, skipping them on the builders seems appropriate to me.

[bcmills]

The go command has a -p flag, but I'm not sure how to plumb it down to the builders.

(@dmitshur, @toothrot, @bradfitz?)

[dmitshur]

It looks like the longtest builder unintentionally has 7.2 GB RAM instead of 14.4. That is half the amount compared to normal linux/amd64 builders that have 15 GB. I'm sending a CL to fix that soon.

[gopherbot]

Change https://golang.org/cl/192679 mentions this issue: dashboard: change linux-amd64-longtest machine to n1-highcpu-16

[bcmills]

Looks like Dmitri's change worked and there were no other longtest failures being masked — the builder seems to be happy now.