Go: plugin: net/http.Get fatals with "runtime: unexpected return pc for runtime.goexit called"

[nzlov]

What version of Go are you using (go version)?

go version devel +94d7c884c3 Thu Dec 14 14:57:01 2017 +0000 darwin/amd64

Does this issue reproduce with the latest release?

YES

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/qipeng/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/qipeng/workspace/go"
GORACE=""
GOROOT="/Users/qipeng/program/go"
GOTMPDIR=""
GOTOOLDIR="/Users/qipeng/program/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/bs/9120vpt14jd7rkhg27kdbj480000gn/T/go-build502747584=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Call http.Get in plugin.
Like:
plugin.go

func Init() {
    res, err := http.Get("https://www.google.com")
    if err != nil {
        fmt.Println("Init Error:", err)
    }
    res.Body.Close()
}

main.go

    p, err := plugin.Open("post.so")
    if err != nil {
        panic(err)
    }
    add, err := p.Lookup("Init")
    if err != nil {
        panic(err)
    }
    add.(func())()

DEMO

What did you expect to see?

Get Google Website

What did you see instead?

command-line-arguments
runtime: unexpected return pc for runtime.goexit called from 0x5466c00
fatal error: unknown caller pc

runtime stack:
runtime.throw(0x52e35a4, 0x11)
        /Users/qipeng/program/go/src/runtime/panic.go:616 +0x81
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc42008c900, 0x0, 0x0, 0x7fffffff, 0x52f3658, 0x7fff5fbff078, 0x0, ...)
        /Users/qipeng/program/go/src/runtime/traceback.go:286 +0x1a72
runtime.copystack(0xc42008c900, 0x1000, 0x7fff5fbf0001)
        /Users/qipeng/program/go/src/runtime/stack.go:891 +0x26e
runtime.newstack()
        /Users/qipeng/program/go/src/runtime/stack.go:1056 +0x304
runtime.morestack()
        /Users/qipeng/program/go/src/runtime/asm_amd64.s:480 +0x89

goroutine 19 [copystack]:
runtime.newobject(0x52b0280, 0x0)
        /Users/qipeng/program/go/src/runtime/malloc.go:838 +0x51 fp=0xc420052af8 sp=0xc420052af0 pc=0x5012571
net/http.(*Transport).dialConn(0x5466c00, 0x531aec0, 0xc4200aa030, 0xc42014e080, 0x52e4bf8, 0x5, 0xc4200ec1c0, 0x12, 0x0, 0x0, ...)
        /Users/qipeng/program/go/src/net/http/transport.go:1082 +0x55 fp=0xc420052f38 sp=0xc420052af8 pc=0x522a205
net/http.(*Transport).getConn.func4(0x5466c00, 0x531aec0, 0xc4200aa030, 0xc4200a6f00, 0xc420098180)
        /Users/qipeng/program/go/src/net/http/transport.go:943 +0x76 fp=0xc420052fb8 sp=0xc420052f38 pc=0x5233c56
runtime.goexit()
        /Users/qipeng/program/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420052fc0 sp=0xc420052fb8 pc=0x5051951
created by net/http.(*Transport).getConn
        /Users/qipeng/program/go/src/net/http/transport.go:942 +0x361

goroutine 1 [select]:
net/http.(*Transport).getConn(0x5466c00, 0xc4200a6ed0, 0xc42014e080, 0x52e4bf8, 0x5, 0xc4200ec1c0, 0x12, 0x0, 0x0, 0xc4200aa3f0)
        /Users/qipeng/program/go/src/net/http/transport.go:948 +0x552
net/http.(*Transport).RoundTrip(0x5466c00, 0xc420150000, 0x5466c00, 0x0, 0x0)
        /Users/qipeng/program/go/src/net/http/transport.go:400 +0x607
net/http.send(0xc420150000, 0x53154a0, 0x5466c00, 0x0, 0x0, 0x0, 0xc4200aeed8, 0xf8, 0xc420067c70, 0x1)
        /Users/qipeng/program/go/src/net/http/client.go:252 +0x185
net/http.(*Client).send(0x546b260, 0xc420150000, 0x0, 0x0, 0x0, 0xc4200aeed8, 0x0, 0x1, 0x5012558)
        /Users/qipeng/program/go/src/net/http/client.go:176 +0xfa
net/http.(*Client).Do(0x546b260, 0xc420150000, 0x52e4bf8, 0x16, 0x0)
        /Users/qipeng/program/go/src/net/http/client.go:615 +0x298
net/http.(*Client).Get(0x546b260, 0x52e4bf8, 0x16, 0xc4200a6090, 0xc4200ca240, 0xc4200ca240)
        /Users/qipeng/program/go/src/net/http/client.go:396 +0x9d
net/http.Get(0x52e4bf8, 0x16, 0x1, 0xffffffffffffffff, 0xc420067ef8)
        /Users/qipeng/program/go/src/net/http/client.go:370 +0x44
plugin/unnamed-b216a47df41c38c43adcbf9feb7d3876f589ad7d.Init()
        /Users/qipeng/workspace/go/src/github.com/nzlov/testplugin/post.go:9 +0x3a
main.main()
        /Users/qipeng/workspace/go/src/github.com/nzlov/testplugin/main.go:17 +0xb7

goroutine 18 [finalizer wait]:
runtime.gopark(0x52f3860, 0x40d4f48, 0x52e274a, 0xe, 0x14, 0x1)
        /Users/qipeng/program/go/src/runtime/proc.go:291 +0x126
runtime.goparkunlock(0x40d4f48, 0x52e274a, 0xe, 0x14, 0x1)
        /Users/qipeng/program/go/src/runtime/proc.go:297 +0x5e
runtime.runfinq()
        /Users/qipeng/program/go/src/runtime/mfinal.go:175 +0xbe
runtime.goexit()
        /Users/qipeng/program/go/src/runtime/asm_amd64.s:2361 +0x1
exit status 2

[mvdan]

Perhaps related to https://github.com/golang/go/issues/20846? The traces are different though, and this is involving plugins.

/cc @aclements @ianlancetaylor

[ianlancetaylor]

This seems to work for me on GNU/Linux.

It's weird that the number 0x5466c00 appears both in the error message and in the call to net/http.(*Transport).getConn on goroutine 1. That makes it seem that the traceback is messed up somehow.

[ianlancetaylor]

CC @crawshaw

[taiidani]

I am seeing this as well in a similar situation.

Version: go version go1.10beta1 darwin/amd64

My plugin uses https://github.com/atlassian/go-sentry-api which also appears to use net/http. Would be happy to provide more info if the original post doesn't provide enough.

[juhwany]

This still happend in 1.10 rc1 darwin.amd64

MacBook-Pro:testplugin juhwany$ go version
go version go1.10rc1 darwin/amd64

KAKAOui-MacBook-Pro:~ juhwany$ go env
GOARCH="amd64"
GOBIN="/Users/juhwany/go_workspace/bin"
GOCACHE="/Users/juhwany/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/juhwany/go_workspace"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/mk/2tyywdsn0jq6l0vl8094tss80000gp/T/go-build422996234=/tmp/go-build -gno-record-gcc-switches -fno-common"
KAKAOui-MacBook-Pro:~ juhwany$

I could reproduce this issue using this demo(https://github.com/nzlov/testplugin)

KAKAOui-MacBook-Pro:testplugin juhwany$ ./testplugin
runtime: unexpected return pc for runtime.goexit called from 0x4460ce0
stack: frame={sp:0xc420055fb8, fp:0xc420055fc0} stack=[0xc420055000,0xc420056000)
000000c420055eb8:  0000000000000000  0000000000000000
000000c420055ec8:  0000000000000000  0000000000000000
000000c420055ed8:  0000000000000000  0000000000000000
000000c420055ee8:  0000000000000000  0000000000000000
000000c420055ef8:  0000000000000000  0000000000000000
000000c420055f08:  0000000000000000  0000000000000000
000000c420055f18:  0000000000000000  0000000000000000
000000c420055f28:  000000c420043fa8  0000000005378b96 <net/http.(*Transport).getConn.func4+118>
000000c420055f38:  0000000004460ce0  00000000043154e0
000000c420055f48:  000000c4200160d0  0000000000000000
000000c420055f58:  0000000005429ff7  0000000000000005
000000c420055f68:  000000c420014300  0000000000000012
000000c420055f78:  0000000000000000  0000000000000000
000000c420055f88:  0000000000000000  0000000000000000
000000c420055f98:  0000000000000000  0000000000000000
000000c420055fa8:  0000000000000000  0000000005192201 <runtime.goexit+1>
000000c420055fb8: <0000000004460ce0 >00000000043154e0
000000c420055fc8:  000000c4200160d0  000000c420094f00
000000c420055fd8:  000000c42008a180  0000000000000000
000000c420055fe8:  0000000000000000  0000000000000000
000000c420055ff8:  0000000000000000
fatal error: unknown caller pc

runtime stack:
runtime: unexpected return pc for runtime.morestack called from 0x0
stack: frame={sp:0x7fff5fbff978, fp:0x7fff5fbff980} stack=[0x7fff5fb80520,0x7fff5fbff9a0)
00007fff5fbff878:  0000000004465be0  00007fff5fbff8c8
00007fff5fbff888:  000000000515653c <runtime.(*mcentral).grow+236>  000000c41fff59ff
00007fff5fbff898:  000000c400000000  0000000004591440
00007fff5fbff8a8:  0000000004591440  0000000000000011
00007fff5fbff8b8:  000000c420001980  000000c420043af8
00007fff5fbff8c8:  000000000536ef75 <net/http.(*Transport).dialConn+85>  000000c420001980
00007fff5fbff8d8:  0000000000000000  0000000000000000
00007fff5fbff8e8:  0000000000000000  0000000000000000
00007fff5fbff8f8:  000000c420001980  000000c420001901
00007fff5fbff908:  0000000004465b01  00007fff5fbff930
00007fff5fbff918:  0000000005179555 <runtime.selparkcommit+133>  0000000004465be0
00007fff5fbff928:  000000c4200721e0  00007fff5fbff960
00007fff5fbff938:  00000000051724eb <runtime.park_m+187>  000000c420000180
00007fff5fbff948:  0000000000000000  000000000458a001
00007fff5fbff958:  0000000004465be0  000000c420065328
00007fff5fbff968:  000000c420043f28  000000000518f859 <runtime.morestack+137>
00007fff5fbff978: <0000000000000000 >00007fff5fbff9d8
00007fff5fbff988:  00000000040549b3 <runtime.rt0_go+515>  00007fff5fbff9d8
00007fff5fbff998:  00000000040549ba <runtime.rt0_go+522>
runtime.throw(0x542898e, 0x11)
    /usr/local/go/src/runtime/panic.go:619 +0x81
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc420001980, 0x0, 0x0, 0x7fffffff, 0x5438a88, 0x7fff5fbff6b8, 0x0, ...)
    /usr/local/go/src/runtime/traceback.go:292 +0x1bb2
runtime.copystack(0xc420001980, 0x1000, 0x7fff5fbff801)
    /usr/local/go/src/runtime/stack.go:891 +0x26e
runtime.newstack()
    /usr/local/go/src/runtime/stack.go:1063 +0x310
runtime: unexpected return pc for runtime.morestack called from 0x0
stack: frame={sp:0x7fff5fbff978, fp:0x7fff5fbff980} stack=[0x7fff5fb80520,0x7fff5fbff9a0)
00007fff5fbff878:  0000000004465be0  00007fff5fbff8c8
00007fff5fbff888:  000000000515653c <runtime.(*mcentral).grow+236>  000000c41fff59ff
00007fff5fbff898:  000000c400000000  0000000004591440
00007fff5fbff8a8:  0000000004591440  0000000000000011
00007fff5fbff8b8:  000000c420001980  000000c420043af8
00007fff5fbff8c8:  000000000536ef75 <net/http.(*Transport).dialConn+85>  000000c420001980
00007fff5fbff8d8:  0000000000000000  0000000000000000
00007fff5fbff8e8:  0000000000000000  0000000000000000
00007fff5fbff8f8:  000000c420001980  000000c420001901
00007fff5fbff908:  0000000004465b01  00007fff5fbff930
00007fff5fbff918:  0000000005179555 <runtime.selparkcommit+133>  0000000004465be0
00007fff5fbff928:  000000c4200721e0  00007fff5fbff960
00007fff5fbff938:  00000000051724eb <runtime.park_m+187>  000000c420000180
00007fff5fbff948:  0000000000000000  000000000458a001
00007fff5fbff958:  0000000004465be0  000000c420065328
00007fff5fbff968:  000000c420043f28  000000000518f859 <runtime.morestack+137>
00007fff5fbff978: <0000000000000000 >00007fff5fbff9d8
00007fff5fbff988:  00000000040549b3 <runtime.rt0_go+515>  00007fff5fbff9d8
00007fff5fbff998:  00000000040549ba <runtime.rt0_go+522>
runtime.morestack()
    /usr/local/go/src/runtime/asm_amd64.s:480 +0x89

goroutine 5 [copystack]:
runtime.newobject(0x42aba80, 0x0)
    /usr/local/go/src/runtime/malloc.go:838 +0x51 fp=0xc420055af8 sp=0xc420055af0 pc=0x5152411
net/http.(*Transport).dialConn(0x4460ce0, 0x43154e0, 0xc4200160d0, 0x0, 0x5429ff7, 0x5, 0xc420014300, 0x12, 0x0, 0x0, ...)
    /usr/local/go/src/net/http/transport.go:1096 +0x55 fp=0xc420055f38 sp=0xc420055af8 pc=0x536ef75
net/http.(*Transport).getConn.func4(0x4460ce0, 0x43154e0, 0xc4200160d0, 0xc420094f00, 0xc42008a180)
    /usr/local/go/src/net/http/transport.go:957 +0x76 fp=0xc420055fb8 sp=0xc420055f38 pc=0x5378b96
runtime: unexpected return pc for runtime.goexit called from 0x4460ce0
stack: frame={sp:0xc420055fb8, fp:0xc420055fc0} stack=[0xc420055000,0xc420056000)
000000c420055eb8:  0000000000000000  0000000000000000
000000c420055ec8:  0000000000000000  0000000000000000
000000c420055ed8:  0000000000000000  0000000000000000
000000c420055ee8:  0000000000000000  0000000000000000
000000c420055ef8:  0000000000000000  0000000000000000
000000c420055f08:  0000000000000000  0000000000000000
000000c420055f18:  0000000000000000  0000000000000000
000000c420055f28:  000000c420043fa8  0000000005378b96 <net/http.(*Transport).getConn.func4+118>
000000c420055f38:  0000000004460ce0  00000000043154e0
000000c420055f48:  000000c4200160d0  0000000000000000
000000c420055f58:  0000000005429ff7  0000000000000005
000000c420055f68:  000000c420014300  0000000000000012
000000c420055f78:  0000000000000000  0000000000000000
000000c420055f88:  0000000000000000  0000000000000000
000000c420055f98:  0000000000000000  0000000000000000
000000c420055fa8:  0000000000000000  0000000005192201 <runtime.goexit+1>
000000c420055fb8: <0000000004460ce0 >00000000043154e0
000000c420055fc8:  000000c4200160d0  000000c420094f00
000000c420055fd8:  000000c42008a180  0000000000000000
000000c420055fe8:  0000000000000000  0000000000000000
000000c420055ff8:  0000000000000000
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420055fc0 sp=0xc420055fb8 pc=0x5192201
created by net/http.(*Transport).getConn
    /usr/local/go/src/net/http/transport.go:956 +0x36d

goroutine 1 [select]:
net/http.(*Transport).getConn(0x4460ce0, 0xc420094ed0, 0x0, 0x5429ff7, 0x5, 0xc420014300, 0x12, 0x0, 0x0, 0x4288e00)
    /usr/local/go/src/net/http/transport.go:962 +0x55e
net/http.(*Transport).RoundTrip(0x4460ce0, 0xc420148000, 0x4460ce0, 0x0, 0x0)
    /usr/local/go/src/net/http/transport.go:409 +0x62e
net/http.send(0xc420148000, 0x545a980, 0x4460ce0, 0x0, 0x0, 0x0, 0xc4201329e0, 0xf8, 0xc420065c70, 0x1)
    /usr/local/go/src/net/http/client.go:252 +0x185
net/http.(*Client).send(0x55afbc0, 0xc420148000, 0x0, 0x0, 0x0, 0xc4201329e0, 0x0, 0x1, 0x51523f8)
    /usr/local/go/src/net/http/client.go:176 +0xfa
net/http.(*Client).Do(0x55afbc0, 0xc420148000, 0x5429ff7, 0x16, 0x0)
    /usr/local/go/src/net/http/client.go:615 +0x298
net/http.(*Client).Get(0x55afbc0, 0x5429ff7, 0x16, 0xc420094cf0, 0xc4200ad0e0, 0xc4200ad0e0)
    /usr/local/go/src/net/http/client.go:396 +0x9d
net/http.Get(0x5429ff7, 0x16, 0x1, 0xffffffffffffffff, 0xc420065ef8)
    /usr/local/go/src/net/http/client.go:370 +0x44
plugin/unnamed-71262ac8d344fb916e3c94d62f776ef5713970c9.Init()
    /Users/juhwany/go_workspace/src/testplugin/post.go:9 +0x3a
main.main()
    /Users/juhwany/go_workspace/src/testplugin/main.go:17 +0xb7

[odeke-em]

I've retitled the bug a little and @juhwany, it's cool to see you are already here as I just closed your duplicate issue in favor of this already existent one.

[juhwany]

This issue still happens in 1.10. and Go 1.10 release notes says that it supports plugin build on Darwin.

[randall77]

It does look like there is something strange going on. When I add this line to goroutine creation:

    println("newg goexit", hex(goexitPC), hex(funcPC(goexit)))

Most of the time, it prints:

newg goexit 0x404da50 0x404da50

But sometimes, it prints:

newg goexit 0x404da50 0x704d6e0

Those two values should always be the same (startup does goexitPC = funcPC(goexit), and they are never subsequently modified), but they aren't.

The first value printed is correct, it is the address of the goexit function in the main binary. The second value, when wrong, is the address of the goexit function in the plugin. Which is extra weird, because it's not just 2 copies of the runtime. Instead, a single line of code is using a global variable from one runtime (goexitPC) and a static closure from another (goexit refers to the runtime.goexit.f symbol).

On Darwin, I see the following symbols:

$ go tool nm main | grep -w goexit
 404da50 T runtime.goexit
 407db18 R runtime.goexit.f
$ go tool nm plugin.so | grep -w goexit
   4d6e0 T runtime.goexit
   eb9b8 D runtime.goexit.f

On Linux, there's something different:

$ go tool nm main | grep -w goexit
  4823a0 t local.runtime.goexit
  4823a0 T runtime.goexit
  4b35c0 R runtime.goexit·f
$ go tool nm plugin.so | grep -w goexit
  126cb0 t local.runtime.goexit
  126cb0 T runtime.goexit
  3e7b28 D runtime.goexit·f

I'm not sure if that's relevant.

@crawshaw @ianlancetaylor , any ideas?

[ianlancetaylor]

I'm not familiar with how dynamic linking works on MacOS. But this looks like the failing code is running inside the plugin. It's essential for correctness that the main executable and a shared library agree on the location of a global variable. So it's not surprising that the plugin gets the value of goexitPC correct. It's much less essential for the main executable and a shared library to agree on the address of a function. For a function it's important that all calls go to the same place, and in shared libraries that is normally done by having the shared library use a Procedure Linkage Table to branch to the function in the main executable if it is defined there, and to use the copy in the shared library if it is not. But that is only calls, not addresses. ELF dynamic linkers go to considerable effort to ensure that equality comparisons of functions in executables and shared libraries resolve as is required by C. It's possible that the Darwin dynamic linker does not. In that case, the reference to the function in the plugin may be retrieving the address of the PLT entry, not the address of the function in the main executable. In particular funcPC is going to pull apart the goexit.f value to fetch the PC.

If my guess is correct then the simple fix is to introduce a package scope variable initialized to funcPC(goexit) and use that instead of calling funcPC(goexit) each time. The main executable will set the value to the address in the main executable, and the reference in the shared library will reliably retrieve that value from the main executable. Since the shared library PLT stub should always branch to the main executable version, the global variable should match what is on the stack.

[randall77]

If my guess is correct then the simple fix is to introduce a package scope variable initialized to funcPC(goexit) and use that instead of calling funcPC(goexit) each time.

That was the fix I was thinking of, but I was worried that there might be a larger issue that this was just the tip of the iceberg for. But maybe you're right, the only issue is function address equality, and as long as we don't rely on that across packages, it's ok if the plugin and main program are using two different runtimes with identical contents. As long as all the mutable global variables are shared.

goexitPC is already that package scope variable you describe. We just have to use that instead of funcPC(goexit) everywhere. And audit all other uses of funcPC.

[randall77]

Audit of uses of funcPC:

1. funcPC results passed to race detector. Those should all be ok, they will just be used to map back to function names.
2. funcPC results passed to routines that call the result. pthread_create, bsdthread_create, asmcgocall, sigaction, sysvicall0, stdcallX, onosstack, setsig. Also ok, as it doesn't matter which copy of the funcPCd function we call.
3. funcPC results used for profile samples. This may end up with attributing those samples to multiple routines of the same name, but that's going to happen anyway.
4. funcPC results used for various traceback special cases. These are not fine. It turns out that the goexit one is easy to fix, because we write the pc explicitly. Lots of others, though, the pc is written implicitly as part of a call. If there are multiple instances of, say, the function jmpdefer, then comparing the pc against a single value obtained with funcPC(jmpdefer) will miss one of those instances. There are a dozen or so such functions: jmpdefer, mcall, morestack, mstart, rt0_go, asmcgocall, sigpanic, .... Some of these are never called directly, only by function pointer, for example sigpanic. For those we could always use a single cached version of funcPC(sigpanic) everywhere, so we always call a particular version. For others like mcall, though, we don't control which version is called. We could call all such functions indirectly instead, but that gets ugly quickly.

Long story short, the goexit fix is easy, but there are other bugs lurking here. I think we really need to fix whatever is different about the linking to make sure we use only a single copy of the runtime's functions.

[ianlancetaylor]

I don't see any particular problem with jmpdefer. The only comparison there is comparing it against f.entry. f.entry is going to come from the symbol table. The value in the symbol table ought to be correct in all cases. If it is not, we should clearly fix that.

The problem with goexit is different. The value is set in newproc1 with a call funcPC(goexit). Calls to funcPC can, I think, return different values in different shared modules on Darwin. Calls to f.entry ought to return the same value in all cases. Similarly, the actual PC value on the stack ought to be the same in all cases; actually calling the function in the plugin should call a stub that jumps to the code in the main executable. The only problematic case (and, I think, only problematic on Darwin, not GNU/Linux) is calling funcPC in the plugin.

[ianlancetaylor]

Or to put it another way, in effect only a single copy of the runtime functions will ever be used, with one exception: explicit calls to funcPC.

[randall77]

I don't think that last statement is correct. We are clearly using two copies of runtime.newproc1, at least. If I add the line

println(hex(funcPC(goexit)), hex(getcallerpc()))

to runtime.newproc1, I get output lines like this:

0x404df50 0x404b6f6
0x404df50 0x404b6f6
0x704d6e0 0x704b206
0x704d6e0 0x704b206

Sometimes it is the main executable's runtime.newproc1, in which case it gets the address of the main executable's goexit. Other times, it's the plugin's runtime.newproc1, in which case it gets the address of the plugin's goexit. I don't think trampolines are in play here; we really are executing whole functions out of the plugin's copy of the runtime (0x704b206 is the address following the call to newproc1 in the anonymous function declared in newproc).

0x704d6e0 is the address of the goexit code in the plugin. It isn't the address of a trampoline or PLT/GOT entry or whatever.

Which means that the actual PC of runtime functions on the stack won't be the same in plugin vs main instances. And I suspect, but haven't confirmed, that we get separate symtab entries with different .entry fields when we use those different PCs (the PC is the key, and symtab entries cover a contiguous range of pcs, how could it be otherwise?).

[ianlancetaylor]

Thanks for checking. I clearly do not understand how shared libraries work on Darwin.

[randall77]

We are both in the same (sinking) boat.

[gopherbot]

Change https://golang.org/cl/100739 mentions this issue: runtime: identify special functions by flag instead of address

[nzlov]

@randall77

go version devel +9c312245ac Tue Mar 20 03:45:04 2018 +0000 darwin/amd6

Still wrong...

command-line-arguments
runtime: bad pointer in frame runtime.selectgo at 0xc000042e20: 0x2
fatal error: invalid pointer found on stack

runtime stack:
runtime.throw(0x72f8e2f, 0x1e)
    /Users/nzlov/program/go/src/runtime/panic.go:598 +0x72 fp=0x7fff5fbfea48 sp=0x7fff5fbfea18 pc=0x702b892
runtime.adjustpointers(0xc000042da0, 0x7fff5fbfeb40, 0x7fff5fbfef00, 0x73739e0, 0x7458e80)
    /Users/nzlov/program/go/src/runtime/stack.go:592 +0x23c fp=0x7fff5fbfeab8 sp=0x7fff5fbfea48 pc=0x70412cc
runtime.adjustframe(0x7fff5fbfee10, 0x7fff5fbfef00, 0x7458e80)
    /Users/nzlov/program/go/src/runtime/stack.go:663 +0x325 fp=0x7fff5fbfeb70 sp=0x7fff5fbfeab8 pc=0x7041615
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc000066c00, 0x0, 0x0, 0x7fffffff, 0x7305118, 0x7fff5fbfef00, 0x0, ...)
    /Users/nzlov/program/go/src/runtime/traceback.go:310 +0x127b fp=0x7fff5fbfee78 sp=0x7fff5fbfeb70 pc=0x704a54b
runtime.copystack(0xc000066c00, 0x1000, 0x1)
    /Users/nzlov/program/go/src/runtime/stack.go:891 +0x26c fp=0x7fff5fbff030 sp=0x7fff5fbfee78 pc=0x70420fc
runtime.newstack()
    /Users/nzlov/program/go/src/runtime/stack.go:1063 +0x30f fp=0x7fff5fbff1c8 sp=0x7fff5fbff030 pc=0x704250f
runtime.morestack()
    /Users/nzlov/program/go/src/runtime/asm_amd64.s:481 +0x8f fp=0x7fff5fbff1d0 sp=0x7fff5fbff1c8 pc=0x705018f

goroutine 21 [copystack]:
runtime.heapBitsSetType(0xc0001303c0, 0x60, 0x60, 0x4072c20)
    /Users/nzlov/program/go/src/runtime/mbitmap.go:911 +0xa9d fp=0xc000042bc8 sp=0xc000042bc0 pc=0x701650d
runtime.mallocgc(0x60, 0x4072c20, 0x1, 0x0)
    /Users/nzlov/program/go/src/runtime/malloc.go:915 +0x57f fp=0xc000042c68 sp=0xc000042bc8 pc=0x700ceaf
runtime.newobject(0x4072c20, 0x0)
    /Users/nzlov/program/go/src/runtime/malloc.go:1014 +0x38 fp=0xc000042c98 sp=0xc000042c68 pc=0x700d518
runtime.acquireSudog(0xc00007c348)
    /Users/nzlov/program/go/src/runtime/proc.go:330 +0x2b4 fp=0xc000042d08 sp=0xc000042c98 pc=0x702d724
runtime.selectgo(0xc000031728, 0xc00007c360)
    /Users/nzlov/program/go/src/runtime/select.go:366 +0xb13 fp=0xc000042ee0 sp=0xc000042d08 pc=0x703aeb3
net.(*netFD).connect.func2(0x732ea80, 0xc000130300, 0xc000134180, 0xc00007c3c0, 0xc00007c360)
    /Users/nzlov/program/go/src/net/fd_unix.go:129 +0xdc fp=0xc000042fb8 sp=0xc000042ee0 pc=0x7180ccc
runtime.goexit()
    /Users/nzlov/program/go/src/runtime/asm_amd64.s:1385 +0x1 fp=0xc000042fc0 sp=0xc000042fb8 pc=0x70520a1
created by net.(*netFD).connect
    /Users/nzlov/program/go/src/net/fd_unix.go:128 +0x28b

goroutine 1 [select]:
runtime.gopark(0x7305328, 0x0, 0x72f23cb, 0x6, 0x18, 0x1)
    /Users/nzlov/program/go/src/runtime/proc.go:291 +0x111 fp=0xc00005f348 sp=0xc00005f328 pc=0x702d331
runtime.selectgo(0xc00005f768, 0xc0001300c0)
    /Users/nzlov/program/go/src/runtime/select.go:392 +0xe6e fp=0xc00005f520 sp=0xc00005f348 pc=0x703b20e
net/http.(*Transport).getConn(0x74835c0, 0xc00008aed0, 0xc000134080, 0x72f6341, 0x5, 0xc0000b6200, 0x12, 0x0, 0x0, 0x413d4c8)
    /Users/nzlov/program/go/src/net/http/transport.go:962 +0x54f fp=0xc00005f898 sp=0xc00005f520 pc=0x72393ff
net/http.(*Transport).RoundTrip(0x74835c0, 0xc000136000, 0x74835c0, 0x0, 0x0)
    /Users/nzlov/program/go/src/net/http/transport.go:409 +0x620 fp=0xc00005fad8 sp=0xc00005f898 pc=0x7235c90
net/http.send(0xc000136000, 0x7328fe0, 0x74835c0, 0x0, 0x0, 0x0, 0xc000092ee8, 0x100, 0xc00005fc70, 0x1)
    /Users/nzlov/program/go/src/net/http/client.go:252 +0x17b fp=0xc00005fc00 sp=0xc00005fad8 pc=0x720b86b
net/http.(*Client).send(0x7487d00, 0xc000136000, 0x0, 0x0, 0x0, 0xc000092ee8, 0x0, 0x1, 0xc00005fcf0)
    /Users/nzlov/program/go/src/net/http/client.go:176 +0xfa fp=0xc00005fc80 sp=0xc00005fc00 pc=0x720b54a
net/http.(*Client).Do(0x7487d00, 0xc000136000, 0x72f6341, 0x16, 0x0)
    /Users/nzlov/program/go/src/net/http/client.go:615 +0x28c fp=0xc00005fe08 sp=0xc00005fc80 pc=0x720cbfc
net/http.(*Client).Get(0x7487d00, 0x72f6341, 0x16, 0xc00008a090, 0xc0000b0240, 0xc0000b0240)
    /Users/nzlov/program/go/src/net/http/client.go:396 +0x9d fp=0xc00005fe60 sp=0xc00005fe08 pc=0x720c72d
net/http.Get(0x72f6341, 0x16, 0x1, 0xffffffffffffffff, 0xc00005fef8)
    /Users/nzlov/program/go/src/net/http/client.go:370 +0x44 fp=0xc00005fea0 sp=0xc00005fe60 pc=0x720c654
plugin/p1-1521532347.Init()
    /Users/nzlov/workspace/go/src/github.com/nzlov/testplugin/post.go:9 +0x3a fp=0xc00005ff08 sp=0xc00005fea0 pc=0x724fa8a
main.main()
    /Users/nzlov/workspace/go/src/github.com/nzlov/testplugin/main.go:17 +0xb7 fp=0xc00005ff88 sp=0xc00005ff08 pc=0x4054c57
runtime.main()
    /Users/nzlov/program/go/src/runtime/proc.go:198 +0x207 fp=0xc00005ffe0 sp=0xc00005ff88 pc=0x4028ca7
runtime.goexit()
    /Users/nzlov/program/go/src/runtime/asm_amd64.s:1385 +0x1 fp=0xc00005ffe8 sp=0xc00005ffe0 pc=0x404bde1

goroutine 2 [force gc (idle)]:
runtime.gopark(0x407b998, 0x40b9cd0, 0x4077582, 0xf, 0x407b814, 0x1)
    /Users/nzlov/program/go/src/runtime/proc.go:291 +0x105 fp=0xc000034768 sp=0xc000034748 pc=0x40290d5
runtime.goparkunlock(0x40b9cd0, 0x4077582, 0xf, 0x14, 0x1)
    /Users/nzlov/program/go/src/runtime/proc.go:297 +0x5e fp=0xc0000347a8 sp=0xc000034768 pc=0x402919e
runtime.forcegchelper()
    /Users/nzlov/program/go/src/runtime/proc.go:248 +0xca fp=0xc0000347e0 sp=0xc0000347a8 pc=0x4028f2a
runtime.goexit()
    /Users/nzlov/program/go/src/runtime/asm_amd64.s:1385 +0x1 fp=0xc0000347e8 sp=0xc0000347e0 pc=0x404bde1
created by runtime.init.4
    /Users/nzlov/program/go/src/runtime/proc.go:237 +0x35

goroutine 3 [GC sweep wait]:
runtime.gopark(0x407b998, 0x40b9dc0, 0x40772b7, 0xd, 0x401cc14, 0x1)
    /Users/nzlov/program/go/src/runtime/proc.go:291 +0x105 fp=0xc000034f60 sp=0xc000034f40 pc=0x40290d5
runtime.goparkunlock(0x40b9dc0, 0x40772b7, 0xd, 0x14, 0x1)
    /Users/nzlov/program/go/src/runtime/proc.go:297 +0x5e fp=0xc000034fa0 sp=0xc000034f60 pc=0x402919e
runtime.bgsweep(0xc000020070)
    /Users/nzlov/program/go/src/runtime/mgcsweep.go:52 +0xa2 fp=0xc000034fd8 sp=0xc000034fa0 pc=0x401cc72
runtime.goexit()
    /Users/nzlov/program/go/src/runtime/asm_amd64.s:1385 +0x1 fp=0xc000034fe0 sp=0xc000034fd8 pc=0x404bde1
created by runtime.gcenable
    /Users/nzlov/program/go/src/runtime/mgc.go:216 +0x58

goroutine 18 [finalizer wait]:
runtime.gopark(0x73052f0, 0x40d6040, 0x72f3dd6, 0xe, 0x14, 0x1)
    /Users/nzlov/program/go/src/runtime/proc.go:291 +0x111 fp=0xc000030718 sp=0xc0000306f8 pc=0x702d331
runtime.goparkunlock(0x40d6040, 0x72f3dd6, 0xe, 0x14, 0x1)
    /Users/nzlov/program/go/src/runtime/proc.go:297 +0x5e fp=0xc000030758 sp=0xc000030718 pc=0x702d3fe
runtime.runfinq()
    /Users/nzlov/program/go/src/runtime/mfinal.go:175 +0xbd fp=0xc0000307e0 sp=0xc000030758 pc=0x701822d
runtime.goexit()
    /Users/nzlov/program/go/src/runtime/asm_amd64.s:1385 +0x1 fp=0xc0000307e8 sp=0xc0000307e0 pc=0x70520a1
created by runtime.createfing
    /Users/nzlov/program/go/src/runtime/mfinal.go:156 +0x65

goroutine 19 [running]:
    goroutine running on other thread; stack unavailable
created by net/http.(*Transport).getConn
    /Users/nzlov/program/go/src/net/http/transport.go:956 +0x35b

goroutine 20 [syscall]:
runtime.notetsleepg(0x40bc9c0, 0x6fc1f164d, 0x1)
    /Users/nzlov/program/go/src/runtime/lock_sema.go:280 +0x4b fp=0xc000030f60 sp=0xc000030f20 pc=0x700be6b
runtime.timerproc(0x40bc9a0)
    /Users/nzlov/program/go/src/runtime/time.go:261 +0x2e7 fp=0xc000030fd8 sp=0xc000030f60 pc=0x7046da7
runtime.goexit()
    /Users/nzlov/program/go/src/runtime/asm_amd64.s:1385 +0x1 fp=0xc000030fe0 sp=0xc000030fd8 pc=0x70520a1
created by runtime.(*timersBucket).addtimerLocked
    /Users/nzlov/program/go/src/runtime/time.go:160 +0x105
exit status 2

[randall77]

Ok, thanks, I'll keep looking.

[andybons]

CL 102793 OK for Go 1.10.1

[gopherbot]

Change https://golang.org/cl/102793 mentions this issue: [release-branch.go1.10] runtime: identify special functions by flag instead of address

[nzlov]

@andybons @randall77 why closed this issue?
macos 10.12.6 go1.10.1 don't fix this issue....
demo

command-line-arguments
runtime: bad pointer in frame crypto/tls.(*clientHandshakeState).handshake at 0xc42017ed68: 0x4
fatal error: invalid pointer found on stack

runtime stack:
runtime.throw(0x52eca09, 0x1e)
    /Users/nzlov/program/go/src/runtime/panic.go:616 +0x81 fp=0x70000772a708 sp=0x70000772a6e8 pc=0x502a9f1
runtime.adjustpointers(0xc42017ed60, 0x70000772a800, 0x70000772abc0, 0x53f8b80, 0x5441f60)
    /Users/nzlov/program/go/src/runtime/stack.go:592 +0x23e fp=0x70000772a778 sp=0x70000772a708 pc=0x504051e
runtime.adjustframe(0x70000772aad0, 0x70000772abc0, 0x5441f60)
    /Users/nzlov/program/go/src/runtime/stack.go:663 +0x325 fp=0x70000772a830 sp=0x70000772a778 pc=0x5040865
runtime.gentraceback(0xffffffffffffffff, 0xffffffffffffffff, 0x0, 0xc420064480, 0x0, 0x0, 0x7fffffff, 0x52f8ab0, 0x70000772abc0, 0x0, ...)
    /Users/nzlov/program/go/src/runtime/traceback.go:310 +0x12d3 fp=0x70000772ab38 sp=0x70000772a830 pc=0x5049703
runtime.copystack(0xc420064480, 0x1000, 0x70000772ad01)
    /Users/nzlov/program/go/src/runtime/stack.go:891 +0x26e fp=0x70000772acf0 sp=0x70000772ab38 pc=0x504134e
runtime.newstack()
    /Users/nzlov/program/go/src/runtime/stack.go:1063 +0x310 fp=0x70000772ae80 sp=0x70000772acf0 pc=0x5041760
runtime.morestack()
    /Users/nzlov/program/go/src/runtime/asm_amd64.s:480 +0x89 fp=0x70000772ae88 sp=0x70000772ae80 pc=0x504f699

goroutine 35 [copystack]:
runtime.(*mcache).nextFree(0x41326c8, 0xc42015611b, 0x0, 0x0, 0x0)
    /Users/nzlov/program/go/src/runtime/malloc.go:545 +0x254 fp=0xc42017eb10 sp=0xc42017eb08 pc=0x5011924
runtime.mallocgc(0xc0, 0x4062a60, 0x1, 0x0)
    /Users/nzlov/program/go/src/runtime/malloc.go:710 +0x7e5 fp=0xc42017ebb0 sp=0xc42017eb10 pc=0x5012115
runtime.makeslice(0x4062a60, 0xb6, 0xb6, 0xc420168060, 0x3, 0xc420024570)
    /Users/nzlov/program/go/src/runtime/slice.go:61 +0x77 fp=0xc42017ebe0 sp=0xc42017ebb0 pc=0x503ef57
crypto/tls.(*clientHelloMsg).marshal(0xc420178000, 0x20, 0x20, 0x20)
    /Users/nzlov/program/go/src/crypto/tls/handshake_messages.go:117 +0x1a1 fp=0xc42017ecd8 sp=0xc42017ebe0 pc=0x51b69c1
crypto/tls.(*clientHandshakeState).handshake(0xc420174670, 0xc420178000, 0x0)
    /Users/nzlov/program/go/src/crypto/tls/handshake_client.go:187 +0x52 fp=0xc42017edf0 sp=0xc42017ecd8 pc=0x51b1722
crypto/tls.(*Conn).clientHandshake(0xc42016e000, 0x52f8d80, 0xc42016e120)
    /Users/nzlov/program/go/src/crypto/tls/handshake_client.go:168 +0x395 fp=0xc42017ef28 sp=0xc42017edf0 pc=0x51b1465
crypto/tls.(*Conn).Handshake(0xc42016e000, 0x0, 0x0)
    /Users/nzlov/program/go/src/crypto/tls/conn.go:1329 +0x189 fp=0xc42017ef78 sp=0xc42017ef28 pc=0x51b00a9
net/http.(*persistConn).addTLS.func2(0x0, 0xc42016e000, 0xc42016a050, 0xc42014a120)
    /Users/nzlov/program/go/src/net/http/transport.go:1068 +0x42 fp=0xc42017efc0 sp=0xc42017ef78 pc=0x5238c82
runtime.goexit()
    /Users/nzlov/program/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc42017efc8 sp=0xc42017efc0 pc=0x5052041
created by net/http.(*persistConn).addTLS
    /Users/nzlov/program/go/src/net/http/transport.go:1064 +0x1a7

goroutine 1 [select]:
runtime.gopark(0x52f8cf8, 0x0, 0x52e6190, 0x6, 0x18, 0x1)
    /Users/nzlov/program/go/src/runtime/proc.go:291 +0x126 fp=0xc420057338 sp=0xc420057318 pc=0x502c396
runtime.selectgo(0xc420057760, 0xc42013e0c0)
    /Users/nzlov/program/go/src/runtime/select.go:392 +0xe96 fp=0xc420057510 sp=0xc420057338 pc=0x503a336
net/http.(*Transport).getConn(0x546c6a0, 0xc420098ed0, 0xc420142080, 0x52ea054, 0x5, 0xc4200de1c0, 0x12, 0x0, 0x0, 0xc42009c3f0)
    /Users/nzlov/program/go/src/net/http/transport.go:962 +0x55e fp=0xc420057890 sp=0xc420057510 pc=0x522e18e
net/http.(*Transport).RoundTrip(0x546c6a0, 0xc420144000, 0x546c6a0, 0x0, 0x0)
    /Users/nzlov/program/go/src/net/http/transport.go:409 +0x62e fp=0xc420057ad8 sp=0xc420057890 pc=0x522a95e
net/http.send(0xc420144000, 0x531a980, 0x546c6a0, 0x0, 0x0, 0x0, 0xc4200a0eb8, 0xf8, 0xc420057c70, 0x1)
    /Users/nzlov/program/go/src/net/http/client.go:252 +0x185 fp=0xc420057c00 sp=0xc420057ad8 pc=0x51fff45
net/http.(*Client).send(0x5470ce0, 0xc420144000, 0x0, 0x0, 0x0, 0xc4200a0eb8, 0x0, 0x1, 0x50124c8)
    /Users/nzlov/program/go/src/net/http/client.go:176 +0xfa fp=0xc420057c80 sp=0xc420057c00 pc=0x51ffc1a
net/http.(*Client).Do(0x5470ce0, 0xc420144000, 0x52ea054, 0x16, 0x0)
    /Users/nzlov/program/go/src/net/http/client.go:615 +0x298 fp=0xc420057e08 sp=0xc420057c80 pc=0x5201328
net/http.(*Client).Get(0x5470ce0, 0x52ea054, 0x16, 0xc420098090, 0xc4200be240, 0xc4200be240)
    /Users/nzlov/program/go/src/net/http/client.go:396 +0x9d fp=0xc420057e60 sp=0xc420057e08 pc=0x5200e2d
net/http.Get(0x52ea054, 0x16, 0x1, 0xffffffffffffffff, 0xc420057ef8)
    /Users/nzlov/program/go/src/net/http/client.go:370 +0x44 fp=0xc420057ea0 sp=0xc420057e60 pc=0x5200d54
plugin/unnamed-97878f87bae92dff48fcbbe4b48fa01e3564416f.Init()
    /Users/nzlov/workspace/go/src/github.com/nzlov/testplugin/post.go:9 +0x3a fp=0xc420057f08 sp=0xc420057ea0 pc=0x5244a8a
main.main()
    /Users/nzlov/workspace/go/src/github.com/nzlov/testplugin/main.go:17 +0xb7 fp=0xc420057f88 sp=0xc420057f08 pc=0x40555b7
runtime.main()
    /Users/nzlov/program/go/src/runtime/proc.go:198 +0x212 fp=0xc420057fe0 sp=0xc420057f88 pc=0x4028d22
runtime.goexit()
    /Users/nzlov/program/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420057fe8 sp=0xc420057fe0 pc=0x404c751

goroutine 2 [force gc (idle)]:
runtime.gopark(0x407b8c8, 0x40b8cb0, 0x4077536, 0xf, 0x407b714, 0x1)
    /Users/nzlov/program/go/src/runtime/proc.go:291 +0x11a fp=0xc420034768 sp=0xc420034748 pc=0x402916a
runtime.goparkunlock(0x40b8cb0, 0x4077536, 0xf, 0x14, 0x1)
    /Users/nzlov/program/go/src/runtime/proc.go:297 +0x5e fp=0xc4200347a8 sp=0xc420034768 pc=0x402921e
runtime.forcegchelper()
    /Users/nzlov/program/go/src/runtime/proc.go:248 +0xcc fp=0xc4200347e0 sp=0xc4200347a8 pc=0x4028fac
runtime.goexit()
    /Users/nzlov/program/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4200347e8 sp=0xc4200347e0 pc=0x404c751
created by runtime.init.4
    /Users/nzlov/program/go/src/runtime/proc.go:237 +0x35

goroutine 3 [GC sweep wait]:
runtime.gopark(0x407b8c8, 0x40b8da0, 0x4077241, 0xd, 0x401cb14, 0x1)
    /Users/nzlov/program/go/src/runtime/proc.go:291 +0x11a fp=0xc420034f60 sp=0xc420034f40 pc=0x402916a
runtime.goparkunlock(0x40b8da0, 0x4077241, 0xd, 0x14, 0x1)
    /Users/nzlov/program/go/src/runtime/proc.go:297 +0x5e fp=0xc420034fa0 sp=0xc420034f60 pc=0x402921e
runtime.bgsweep(0xc420020070)
    /Users/nzlov/program/go/src/runtime/mgcsweep.go:52 +0xa3 fp=0xc420034fd8 sp=0xc420034fa0 pc=0x401cbd3
runtime.goexit()
    /Users/nzlov/program/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420034fe0 sp=0xc420034fd8 pc=0x404c751
created by runtime.gcenable
    /Users/nzlov/program/go/src/runtime/mgc.go:216 +0x58

goroutine 18 [finalizer wait]:
runtime.gopark(0x52f8cc0, 0x40d4f38, 0x52e7b80, 0xe, 0x14, 0x1)
    /Users/nzlov/program/go/src/runtime/proc.go:291 +0x126 fp=0xc420030718 sp=0xc4200306f8 pc=0x502c396
runtime.goparkunlock(0x40d4f38, 0x52e7b80, 0xe, 0x14, 0x1)
    /Users/nzlov/program/go/src/runtime/proc.go:297 +0x5e fp=0xc420030758 sp=0xc420030718 pc=0x502c44e
runtime.runfinq()
    /Users/nzlov/program/go/src/runtime/mfinal.go:175 +0xbe fp=0xc4200307e0 sp=0xc420030758 pc=0x5016d7e
runtime.goexit()
    /Users/nzlov/program/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4200307e8 sp=0xc4200307e0 pc=0x5052041
created by runtime.createfing
    /Users/nzlov/program/go/src/runtime/mfinal.go:156 +0x66

goroutine 19 [chan receive]:
runtime.gopark(0x52f8cc0, 0xc42014a178, 0x52e7426, 0xc, 0x5045917, 0x3)
    /Users/nzlov/program/go/src/runtime/proc.go:291 +0x126 fp=0xc420053808 sp=0xc4200537e8 pc=0x502c396
runtime.goparkunlock(0xc42014a178, 0x52e7426, 0xc, 0x17, 0x3)
    /Users/nzlov/program/go/src/runtime/proc.go:297 +0x5e fp=0xc420053848 sp=0xc420053808 pc=0x502c44e
runtime.chanrecv(0xc42014a120, 0xc420053a30, 0x1, 0x522ead7)
    /Users/nzlov/program/go/src/runtime/chan.go:518 +0x304 fp=0xc4200538e0 sp=0xc420053848 pc=0x5006924
runtime.chanrecv1(0xc42014a120, 0xc420053a30)
    /Users/nzlov/program/go/src/runtime/chan.go:400 +0x2b fp=0xc420053910 sp=0xc4200538e0 pc=0x500660b
net/http.(*persistConn).addTLS(0xc4200bf560, 0xc4200de1c0, 0xe, 0x0, 0x0, 0x1000)
    /Users/nzlov/program/go/src/net/http/transport.go:1074 +0x1d0 fp=0xc420053af8 sp=0xc420053910 pc=0x522eb00
net/http.(*Transport).dialConn(0x546c6a0, 0x53203c0, 0xc42009c030, 0xc420142080, 0x52ea054, 0x5, 0xc4200de1c0, 0x12, 0x0, 0x0, ...)
    /Users/nzlov/program/go/src/net/http/transport.go:1224 +0xb90 fp=0xc420053f38 sp=0xc420053af8 pc=0x522faa0
net/http.(*Transport).getConn.func4(0x546c6a0, 0x53203c0, 0xc42009c030, 0xc420098f00, 0xc42008a180)
    /Users/nzlov/program/go/src/net/http/transport.go:957 +0x76 fp=0xc420053fb8 sp=0xc420053f38 pc=0x5238b86
runtime.goexit()
    /Users/nzlov/program/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420053fc0 sp=0xc420053fb8 pc=0x5052041
created by net/http.(*Transport).getConn
    /Users/nzlov/program/go/src/net/http/transport.go:956 +0x36d

goroutine 20 [syscall]:
runtime.notetsleepg(0x40bb980, 0x6fc1f069e, 0x1)
    /Users/nzlov/program/go/src/runtime/lock_sema.go:280 +0x4b fp=0xc420030f60 sp=0xc420030f20 pc=0x50113eb
runtime.timerproc(0x40bb960)
    /Users/nzlov/program/go/src/runtime/time.go:261 +0x2f4 fp=0xc420030fd8 sp=0xc420030f60 pc=0x5045f84
runtime.goexit()
    /Users/nzlov/program/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420030fe0 sp=0xc420030fd8 pc=0x5052041
created by runtime.(*timersBucket).addtimerLocked
    /Users/nzlov/program/go/src/runtime/time.go:160 +0x10b

goroutine 34 [syscall]:
runtime.notetsleepg(0x40bb900, 0x2540b37eb, 0x0)
    /Users/nzlov/program/go/src/runtime/lock_sema.go:280 +0x4b fp=0xc420031760 sp=0xc420031720 pc=0x50113eb
runtime.timerproc(0x40bb8e0)
    /Users/nzlov/program/go/src/runtime/time.go:261 +0x2f4 fp=0xc4200317d8 sp=0xc420031760 pc=0x5045f84
runtime.goexit()
    /Users/nzlov/program/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4200317e0 sp=0xc4200317d8 pc=0x5052041
created by runtime.(*timersBucket).addtimerLocked
    /Users/nzlov/program/go/src/runtime/time.go:160 +0x10b
exit status 2

[randall77]

Sorry, the closing is just from the automated point release process.
I'll reopen. Part of this issue has been fixed, but not all of it.

[nzlov]

@randall77 Thanks.

[andybons]

Sorry about that. Moved to 1.10.2 milestone as well.

[FiloSottile]

@gopherbot please open a 1.10 backport tracking issue. Go 1.10.1 only included a partial fix.

[gopherbot]

Backport issue(s) opened: #25038 (for 1.10).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[chen-keinan]

I have confirmed that this issue has been resolved on GO 1.11 beta

[cherrymui]

Thanks.

[odeke-em]

@FiloSottile @ianlancetaylor @andybons, @randall77's first CL 100739 was already backported to Go1.10 with CL 102793 but @randall77 reopened this issue in https://github.com/golang/go/issues/23133#issuecomment-377415066 mentioning that his CL fixed a part of this issue. How do we proceed with closing the requested backport issue https://github.com/golang/go/issues/25038?

[cherrymui]

I think the other half is CL https://go-review.googlesource.com/c/go/+/104715. So CL https://go-review.googlesource.com/c/go/+/122116 will do.

[odeke-em]

Cool, thanks for finding and reporting that @cherrymui! I don't think we should tamper with editing that backport message in CL 122116, to state that it finishes up this one, so perhaps this conversation in here suffices.