Go: cmd/compile, x/sync/errgroup: TestWithContext failing after devirtualization CL

[zikaeroh]

What version of Go are you using (go version)?

$ go version
go version devel +fe70a3a0fd Thu Oct 29 21:46:54 2020 +0000 linux/amd64

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

go env Output

$ go env

GO111MODULE=""

GOARCH="amd64"

GOBIN=""

GOCACHE="/home/jake/.cache/go-build"

GOENV="/home/jake/.config/go/env"

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="linux"

GOINSECURE=""

GOMODCACHE="/home/jake/go/pkg/mod"

GONOPROXY=""

GONOSUMDB=""

GOOS="linux"

GOPATH="/home/jake/go"

GOPRIVATE=""

GOPROXY="https://proxy.golang.org,direct"

GOROOT="/home/jake/sdk/gotip"

GOSUMDB="sum.golang.org"

GOTMPDIR=""

GOTOOLDIR="/home/jake/sdk/gotip/pkg/tool/linux_amd64"

GCCGO="gccgo"

AR="ar"

CC="gcc"

CXX="g++"

CGO_ENABLED="1"

GOMOD="/home/jake/zikaeroh/sync/go.mod"

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build875916662=/tmp/go-build -gno-record-gcc-switches"

GOROOT/bin/go version: go version devel +fe70a3a0fd Thu Oct 29 21:46:54 2020 +0000 linux/amd64

GOROOT/bin/go tool compile -V: compile version devel +fe70a3a0fd Thu Oct 29 21:46:54 2020 +0000

uname -sr: Linux 5.9.1-zen2-1-zen

/usr/lib/libc.so.6: GNU C Library (GNU libc) release release version 2.32.

gdb --version: GNU gdb (GDB) 9.2

What did you do?

In the golang.org/x/sync/errgroup package, run go test.

What did you expect to see?

Passing tests.

What did you see instead?

--- FAIL: TestWithContext (0.00s)
panic: interface conversion: context.Context is *context.cancelCtx, not *context.emptyCtx [recovered]
    panic: interface conversion: context.Context is *context.cancelCtx, not *context.emptyCtx

goroutine 29 [running]:
testing.tRunner.func1.2(0x5b0e80, 0xc0000a4e70)
    /home/jake/sdk/gotip/src/testing/testing.go:1123 +0x332
testing.tRunner.func1(0xc000083080)
    /home/jake/sdk/gotip/src/testing/testing.go:1126 +0x4b6
panic(0x5b0e80, 0xc0000a4e70)
    /home/jake/sdk/gotip/src/runtime/panic.go:965 +0x1b9
golang.org/x/sync/errgroup_test.TestWithContext(0xc000083080)
    /home/jake/zikaeroh/sync/errgroup/errgroup_test.go:166 +0x652
testing.tRunner(0xc000083080, 0x5f4c88)
    /home/jake/sdk/gotip/src/testing/testing.go:1173 +0xef
created by testing.(*T).Run
    /home/jake/sdk/gotip/src/testing/testing.go:1218 +0x2b3
exit status 2
FAIL    golang.org/x/sync/errgroup  0.005s

https://build.golang.org/log/8759ffb3dd087325958f6d2d67773f3dd2f70cf2
https://build.golang.org/log/32b25547fddc3b46c217c9efffb9aa124df3766e

The dashboard shows this test consistently failing after the devirtualization CL, but this differs from #42279 as it affects runtime behavior.

cc @mdempsky

[dmitshur]

Now that #42279 is resolved, is this still an issue? I'm not sure I understand how this issue is different.

[zikaeroh]

See https://build.golang.org/?repo=golang.org%2fx%2fsync; the page is all red on tip.

[dmitshur]

Thanks. I can reproduce locally:

$ gotip version
go version devel +f7e26467b Fri Oct 30 01:31:10 2020 +0000 darwin/amd64
$ gotip test golang.org/x/sync/errgroup
--- FAIL: TestWithContext (0.00s)
panic: interface conversion: context.Context is *context.cancelCtx, not *context.emptyCtx [recovered]
    panic: interface conversion: context.Context is *context.cancelCtx, not *context.emptyCtx

goroutine 14 [running]:
testing.tRunner.func1.2(0x11eab00, 0xc00007ae70)
    /Users/dmitshur/sdk/gotip/src/testing/testing.go:1123 +0x332
testing.tRunner.func1(0xc000154000)
    /Users/dmitshur/sdk/gotip/src/testing/testing.go:1126 +0x4b6
panic(0x11eab00, 0xc00007ae70)
    /Users/dmitshur/sdk/gotip/src/runtime/panic.go:965 +0x1b9
golang.org/x/sync/errgroup_test.TestWithContext(0xc000154000)
    /Users/dmitshur/go/src/golang.org/x/sync/errgroup/errgroup_test.go:166 +0x652
testing.tRunner(0xc000154000, 0x1233b40)
    /Users/dmitshur/sdk/gotip/src/testing/testing.go:1173 +0xef
created by testing.(*T).Run
    /Users/dmitshur/sdk/gotip/src/testing/testing.go:1218 +0x2b3
FAIL    golang.org/x/sync/errgroup  0.323s
FAIL

[gopherbot]

Change https://golang.org/cl/266518 mentions this issue: cmd/compile: mark inlined functions's results as reassigned

[mdempsky]

Sorry, missed this yesterday. Investigating.

[mdempsky]

A workaround for the issue would be to to modify errgroup.WithContext like this:

 func WithContext(ctx context.Context) (*Group, context.Context) {
-       ctx, cancel := context.WithCancel(ctx)
-       return &Group{cancel: cancel}, ctx
+       ctx1, cancel := context.WithCancel(ctx)
+       return &Group{cancel: cancel}, ctx1
 }

It's a compiler bug that the current code (i.e., reassigning ctx) is miscompiled, but this change would be desirable anyway because it would play better with the current devirtualization code.

[zikaeroh]

It's a compiler bug that the current code (i.e., reassigning ctx) is miscompiled, but this change would be desirable anyway because it would play better with the current devirtualization code.

Reassigning over ctx is probably the most common thing to do with contexts; it'd be a shame if things didn't work properly without making variable name changes. But, maybe in fixing this bug it'll fix that too.

[gopherbot]

Change https://golang.org/cl/266618 mentions this issue: cmd/compile: fix reassignVisitor

[mdempsky]

Reassigning over ctx is probably the most common thing to do with contexts; it'd be a shame if things didn't work properly without making variable name changes.

Once we move escape analysis to SSA, this should be straightforward to handle. But right now, the Node AST doesn't make it easy to do. We'd have to effectively reimplement SSA renaming in the frontend.

[zikaeroh]

Ah, I had no idea this was at the AST level where you couldn't distinguish it. Clearly I'm not paying enough attention during your streams. :slightly_smiling_face:

[cespare]

@mdempsky I'm really excited that we're getting these devirtualization optimizations and there are several places in my own code where I anticipate them making a big difference. That said, it's a little concerning when an optimization applies only when code is written in a non-standard (and arguably worse) way. Depending on how many release cycles the devirtualization handles the ctx1 case but not the ctx case, it's the kind of thing that can work its way into the performance lore. I really hope in a year or two I don't start seeing high-performance code has been manually SSA-ified to satisfy the devirtualizer :)

[mdempsky]

@cespare That's a fair point. There's certainly a trade off though: the more effort put into improving the early devirtualization code is effort taken away from moving escape analysis to SSA. The current early devirtualization pass is basically just a minor extension of the inlining / escape analysis improvements implemented for #41474

If you do see that practice developing, let us know and we can revisit whether there are more cases that we can handle without too much trouble.