Go: runtime: can't call from Go to c-shared Go library on Darwin

[khaitranvan96kt]

What version of Go are you using (go version)?

$ go version
go version go1.13.10 darwin/amd64

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

go env Output

$ go env

GO111MODULE=""

GOARCH="amd64"

GOBIN=""

GOCACHE="/Users/trankhai/Library/Caches/go-build"

GOENV="/Users/trankhai/Library/Application Support/go/env"

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="darwin"

GONOPROXY=""

GONOSUMDB=""

GOOS="darwin"

GOPATH="/Users/trankhai/go"

GOPRIVATE=""

GOPROXY="https://proxy.golang.org,direct"

GOROOT="/usr/local/go"

GOSUMDB="sum.golang.org"

GOTMPDIR=""

GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"

GCCGO="gccgo"

AR="ar"

CC="clang"

CXX="clang++"

CGO_ENABLED="1"

GOMOD="/Users/trankhai/Documents/GitHub/2hat_contest2/go.mod"

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/2_/9794qxcj4l56p3k1z5_y9v040000gn/T/go-build055049177=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I wrote a program that combines golang vs python. I call from golang to python and from python call to golang. It works great in Ubuntu but can't work in macos.

Then run the program in macOS Catalina v10.15.3:

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x6238648]

goroutine 20 [running, locked to thread]:
runtime.throw(0x62f3e2d, 0x2a)
    /usr/local/go/src/runtime/panic.go:774 +0x72 fp=0xc0000e71d8 sp=0xc0000e71a8 pc=0x620d382
runtime.sigpanic()
    /usr/local/go/src/runtime/signal_unix.go:378 +0x47c fp=0xc0000e7208 sp=0xc0000e71d8 pc=0x621ffbc
runtime.memmove(0x0, 0xc00006f800, 0x1)
    /usr/local/go/src/runtime/memmove_amd64.s:146 +0x108 fp=0xc0000e7210 sp=0xc0000e7208 pc=0x6238648
runtime.heapBitsSetType(0xc00006f800, 0x400, 0x400, 0x62d5640)
    /usr/local/go/src/runtime/mbitmap.go:1400 +0x45d fp=0xc0000e72e8 sp=0xc0000e7210 pc=0x61f66fd
runtime.mallocgc(0x400, 0x62d5640, 0xc0000d4001, 0x8)
    /usr/local/go/src/runtime/malloc.go:1052 +0x53e fp=0xc0000e7388 sp=0xc0000e72e8 pc=0x61ede1e
runtime.makeslice(0x62d5640, 0x8, 0x8, 0x0)
    /usr/local/go/src/runtime/slice.go:49 +0x6c fp=0xc0000e73b8 sp=0xc0000e7388 pc=0x6220b2c
sync.(*Pool).pinSlow(0x63bbce0, 0x0, 0x0)
    /usr/local/go/src/sync/pool.go:227 +0xfa fp=0xc0000e7458 sp=0xc0000e73b8 pc=0x623fefa
sync.(*Pool).pin(0x63bbce0, 0x1, 0xc0000e74d8)
    /usr/local/go/src/sync/pool.go:206 +0x5e fp=0xc0000e7480 sp=0xc0000e7458 pc=0x623fdce
sync.(*Pool).Get(0x63bbce0, 0x1c025ae5de0, 0xc0000e7558)
    /usr/local/go/src/sync/pool.go:128 +0x2f fp=0xc0000e74c8 sp=0xc0000e7480 pc=0x623fa8f
encoding/json.newEncodeState(0x1c01e2dad20)
    /usr/local/go/src/encoding/json/encode.go:286 +0x31 fp=0xc0000e74f0 sp=0xc0000e74c8 pc=0x629e151
encoding/json.Marshal(0x62ca280, 0x1c0000921e0, 0xc0000e75f8, 0x62ca280, 0x1c0000921e0, 0x40cdb01, 0xc0001121b0)
    /usr/local/go/src/encoding/json/encode.go:159 +0x26 fp=0xc0000e7568 sp=0xc0000e74f0 pc=0x629da66
main.GetTopics(0x60369b4, 0x4, 0x0)
    /Users/trankhai/Documents/GitHub/2hat_contest2/hybrid_go.go:46 +0xf9 fp=0xc0000e7618 sp=0xc0000e7568 pc=0x62ad369
main._cgoexpwrap_d079111cff13_GetTopics(0x60369b4, 0x4, 0x0)
    _cgo_gotypes.go:64 +0x72 fp=0xc0000e7678 sp=0xc0000e7618 pc=0x62acf32
runtime.call32(0x0, 0x7ffeefbff1b0, 0x7ffeefbff240, 0x18)
    /usr/local/go/src/runtime/asm_amd64.s:539 +0x3b fp=0xc0000e76a8 sp=0xc0000e7678 pc=0x62356cb
runtime.cgocallbackg1(0x0)
    /usr/local/go/src/runtime/cgocall.go:314 +0x1b7 fp=0xc0000e7790 sp=0xc0000e76a8 pc=0x61e5847
runtime.cgocallbackg(0x0)
    /usr/local/go/src/runtime/cgocall.go:191 +0xc1 fp=0xc0000e77f8 sp=0xc0000e7790 pc=0x61e55f1
runtime: unexpected return pc for runtime.cgocallback_gofunc called from 0x405ae42
stack: frame={sp:0xc0000e77f8, fp:0xc0000e7818} stack=[0xc0000e6000,0xc0000e8000)
000000c0000e76f8:  0000000000000006  00000000000001ff 
000000c0000e7708:  0000000000000008  000000c0000e76a8 
000000c0000e7718:  00000000061e57f5 <runtime.cgocallbackg1+357>  00000000062f5ba0 
000000c0000e7728:  0000000000000000  000000c0000e7980 
000000c0000e7738:  000000c0000e76ef  0000000000000002 
000000c0000e7748:  0000000000000019  000000c0000e7780 
000000c0000e7758:  0000000006215ed0 <runtime.exitsyscall+416>  000000c000078300 
000000c0000e7768:  0000000200000003  000000c000078300 
000000c0000e7778:  000000c000078300  000000c0000e77e8 
000000c0000e7788:  00000000061e55f1 <runtime.cgocallbackg+193>  0000000000000000 
000000c0000e7798:  0000000004005f65  0000000004004d9b 
000000c0000e77a8:  0000000000000000  0000000000000000 
000000c0000e77b8:  0000000000000000  0000000000000000 
000000c0000e77c8:  0000000000000000  0000000000000000 
000000c0000e77d8:  000000c0000e7820  000000c000078300 
000000c0000e77e8:  00007ffeefbff170  0000000006236c9b <runtime.cgocallback_gofunc+155> 
000000c0000e77f8: <0000000000000000  000000000458c5c0 
000000c0000e7808:  000000c0000e7820 !000000000405ae42 
000000c0000e7818: >0000000004004dbf  0000000004285c60 
000000c0000e7828:  000000c0000e7888  000000c0000e7878 
000000c0000e7838:  0000000004005405  000000000458c5c0 
000000c0000e7848:  000000c0000e7878  00000000040e455e 
000000c0000e7858:  0000000004285c60  000000c0000e7888 
000000c0000e7868:  000000c000026500  000000c0000e7888 
000000c0000e7878:  000000c0000e78c8  00000000040e5d26 
000000c0000e7888:  0000000006033e60  00000000057e0690 
000000c0000e7898:  00000000057e8398  0000000000000000 
000000c0000e78a8:  0000000000000000  00000000057e8398 
000000c0000e78b8:  00000000057e0690  0000000006033e60 
000000c0000e78c8:  000000c0000e7900  00000000040e5283 
000000c0000e78d8:  000000c0000d4050  000000c0000d4058 
000000c0000e78e8:  000000c0000d4068  000000c0000d4068 
000000c0000e78f8:  00000000057e8398  000000c0000e79e0 
000000c0000e7908:  000000000428334c  000000c0000d4050 
runtime.cgocallback_gofunc(0x4004dbf, 0x4285c60, 0xc0000e7888, 0xc0000e7878)
    /usr/local/go/src/runtime/asm_amd64.s:793 +0x9b fp=0xc0000e7818 sp=0xc0000e77f8 pc=0x6236c9b

goroutine 2 [force gc (idle)]:
runtime.gopark(0x62f5a30, 0x63bbbc0, 0x1411, 0x1)
    /usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0x1c000044fb0 sp=0x1c000044f90 pc=0x620f0d0
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:310
runtime.forcegchelper()
    /usr/local/go/src/runtime/proc.go:253 +0xb7 fp=0x1c000044fe0 sp=0x1c000044fb0 pc=0x620ef87
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0x1c000044fe8 sp=0x1c000044fe0 pc=0x62373c1
created by runtime.init.5
    /usr/local/go/src/runtime/proc.go:242 +0x35

goroutine 3 [GC sweep wait]:
runtime.gopark(0x62f5a30, 0x63bbcc0, 0x140c, 0x1)
    /usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0x1c0000457a8 sp=0x1c000045788 pc=0x620f0d0
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:310
runtime.bgsweep(0x1c00006a000)
    /usr/local/go/src/runtime/mgcsweep.go:89 +0x131 fp=0x1c0000457d8 sp=0x1c0000457a8 pc=0x6202c01
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0x1c0000457e0 sp=0x1c0000457d8 pc=0x62373c1
created by runtime.gcenable
    /usr/local/go/src/runtime/mgc.go:210 +0x5c

goroutine 4 [GC scavenge wait]:
runtime.gopark(0x62f5a30, 0x63bbe60, 0x140d, 0x1)
    /usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0x1c000045f40 sp=0x1c000045f20 pc=0x620f0d0
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:310
runtime.bgscavenge(0x1c00006a000)
    /usr/local/go/src/runtime/mgcscavenge.go:374 +0x3b3 fp=0x1c000045fd8 sp=0x1c000045f40 pc=0x62024c3
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0x1c000045fe0 sp=0x1c000045fd8 pc=0x62373c1
created by runtime.gcenable
    /usr/local/go/src/runtime/mgc.go:211 +0x7e

goroutine 18 [finalizer wait]:
runtime.gopark(0x62f5a30, 0x63d71a8, 0x1410, 0x1)
    /usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0x1c000044758 sp=0x1c000044738 pc=0x620f0d0
runtime.goparkunlock(...)
    /usr/local/go/src/runtime/proc.go:310
runtime.runfinq()
    /usr/local/go/src/runtime/mfinal.go:175 +0xa3 fp=0x1c0000447e0 sp=0x1c000044758 pc=0x61f8be3
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0x1c0000447e8 sp=0x1c0000447e0 pc=0x62373c1
created by runtime.createfing
    /usr/local/go/src/runtime/mfinal.go:156 +0x61

goroutine 19 [GC worker (idle)]:
runtime.gopark(0x62f58a8, 0x1c000016090, 0x1418, 0x0)
    /usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0x1c000040760 sp=0x1c000040740 pc=0x620f0d0
runtime.gcBgMarkWorker(0x1c000024000)
    /usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0x1c0000407d8 sp=0x1c000040760 pc=0x61fc46f
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0x1c0000407e0 sp=0x1c0000407d8 pc=0x62373c1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 34 [GC worker (idle)]:
runtime.gopark(0x62f58a8, 0x1c0000160a0, 0x1418, 0x0)
    /usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0x1c0013c2760 sp=0x1c0013c2740 pc=0x620f0d0
runtime.gcBgMarkWorker(0x1c000026500)
    /usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0x1c0013c27d8 sp=0x1c0013c2760 pc=0x61fc46f
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0x1c0013c27e0 sp=0x1c0013c27d8 pc=0x62373c1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 5 [GC worker (idle)]:
runtime.gopark(0x62f58a8, 0x1c0000160b0, 0x1418, 0x0)
    /usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0x1c000046760 sp=0x1c000046740 pc=0x620f0d0
runtime.gcBgMarkWorker(0x1c000028a00)
    /usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0x1c0000467d8 sp=0x1c000046760 pc=0x61fc46f
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0x1c0000467e0 sp=0x1c0000467d8 pc=0x62373c1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 6 [GC worker (idle)]:
runtime.gopark(0x62f58a8, 0x1c0000ac090, 0x1418, 0x0)
    /usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0x1c000046f60 sp=0x1c000046f40 pc=0x620f0d0
runtime.gcBgMarkWorker(0x1c00002af00)
    /usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0x1c000046fd8 sp=0x1c000046f60 pc=0x61fc46f
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0x1c000046fe0 sp=0x1c000046fd8 pc=0x62373c1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 35 [GC worker (idle)]:
runtime.gopark(0x62f58a8, 0x1c0013c8000, 0x1418, 0x0)
    /usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0x1c0013c2f60 sp=0x1c0013c2f40 pc=0x620f0d0
runtime.gcBgMarkWorker(0x1c00002d400)
    /usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0x1c0013c2fd8 sp=0x1c0013c2f60 pc=0x61fc46f
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0x1c0013c2fe0 sp=0x1c0013c2fd8 pc=0x62373c1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 36 [GC worker (idle)]:
runtime.gopark(0x62f58a8, 0x1c0000ac0a0, 0x1418, 0x0)
    /usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0x1c0013c3760 sp=0x1c0013c3740 pc=0x620f0d0
runtime.gcBgMarkWorker(0x1c00002f900)
    /usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0x1c0013c37d8 sp=0x1c0013c3760 pc=0x61fc46f
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0x1c0013c37e0 sp=0x1c0013c37d8 pc=0x62373c1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 7 [GC worker (idle)]:
runtime.gopark(0x62f58a8, 0x1c0000160c0, 0x1418, 0x0)
    /usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0x1c000047760 sp=0x1c000047740 pc=0x620f0d0
runtime.gcBgMarkWorker(0x1c000032000)
    /usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0x1c0000477d8 sp=0x1c000047760 pc=0x61fc46f
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0x1c0000477e0 sp=0x1c0000477d8 pc=0x62373c1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1794 +0x77

goroutine 8 [GC worker (idle)]:
runtime.gopark(0x62f58a8, 0x1c0013c8010, 0x1418, 0x0)
    /usr/local/go/src/runtime/proc.go:304 +0xe0 fp=0x1c000047f60 sp=0x1c000047f40 pc=0x620f0d0
runtime.gcBgMarkWorker(0x1c000034500)
    /usr/local/go/src/runtime/mgc.go:1846 +0xff fp=0x1c000047fd8 sp=0x1c000047f60 pc=0x61fc46f
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1357 +0x1 fp=0x1c000047fe0 sp=0x1c000047fd8 pc=0x62373c1
created by runtime.gcBgMarkStartWorkers
    /usr/local/go/src/runtime/mgc.go:1794 +0x77
exit status 2

What did you expect to see?

Program successfully

What did you see instead?

Program panic

[randall77]

Can you share the program?
Have you tried 1.14.2?
Does it pass the race detector?

@mknyszek

[khaitranvan96kt]

Can you share the program?
Have you tried 1.14.2?
Does it pass the race detector?

@mknyszek

• Sorry I couldn't share the code because it was part of the company code. But I have used a combination of the 2 articles below:
  
  
  
  • https://www.datadoghq.com/blog/engineering/cgo-and-python/
  
  
  • https://medium.com/learning-the-go-programming-language/calling-go-functions-from-other-languages-4c7d8bcc69bf
  
  
• I tried 1.14.2 but things didn't change
• how to use the race detector?

Please, help me

[randall77]

how to use the race detector?

Use the -race flag on the go tool.

https://golang.org/doc/articles/race_detector.html

[khaitranvan96kt]

Use the -race flag on the go tool.

Then run the program:
go test -race -v -run TestInit

Python version: 2.7.16 (default, Dec 13 2019, 18:00:32) 
[GCC 4.2.1 Compatible Apple LLVM 11.0.0 (clang-1100.0.32.4) (-macos10.15-objc-s

runtime: unexpected return pc for runtime.cgocallback_gofunc called from 0x406a152
stack: frame={sp:0xc00013bd78, fp:0xc00013bd98} stack=[0xc00013a000,0xc00013c000)
000000c00013bc78:  0000000000000000  00007ffeefbfede0 
000000c00013bc88:  00007ffeefbfee77  0000000000000000 
000000c00013bc98:  000000002f38ca8b <runtime.exitsyscallfast+171>  000000c00013bcb0 
000000c00013bca8:  010000c00013bd78  000000002f3acd50 <runtime.exitsyscallfast.func1+0> 
000000c00013bcb8:  000201c00013bcaf  000000c000024000 
000000c00013bcc8:  000000c000000180  000000c00013bd00 
000000c00013bcd8:  000000c000000180  000000c00013bcbd 
000000c00013bce8:  000000002f4625a0  0000000000000000 
000000c00013bcf8:  0000000000000000  000000c00013bd68 
000000c00013bd08:  000000002f355f61 <runtime.cgocallbackg+193>  0000000000000000 
000000c00013bd18:  0000000000000010  000000000400550a 
000000c00013bd28:  0000000000000000  0000000000000000 
000000c00013bd38:  0000000000000000  0000000000000000 
000000c00013bd48:  0000000000000000  0000000000000000 
000000c00013bd58:  000000c00013bda0  000000c000000180 
000000c00013bd68:  00007ffeefbfeda0  000000002f3af14b <runtime.cgocallback_gofunc+155> 
000000c00013bd78: <0000000000000000  00000000047fd920 
000000c00013bd88:  000000c00013bda0 !000000000406a152 
000000c00013bd98: >000000000400552e  0000000004421c30 
000000c00013bda8:  000000c00013be08  00000000044b46a0 
000000c00013bdb8:  00000000041e7f01  00000000047fd920 
000000c00013bdc8:  000000c00013bdf8  00000000041e83cc 
000000c00013bdd8:  0000000004421c30  000000c00013be08 
000000c00013bde8:  000000c000052038  000000c00013be08 
000000c00013bdf8:  000000c00013be90  00000000041ea3d3 
000000c00013be08:  0000000005403bf0  0000000000000000 
000000c00013be18:  0000000005403bf0  0000000000000000 
000000c00013be28:  000000c00013be08  00000000041ea3bd 
000000c00013be38:  000000c000052030  0000000000000000 
000000c00013be48:  0000000000000000  0000000000000000 
000000c00013be58:  0000000000000000  0000000000000000 
000000c00013be68:  0000000000000000  0000000000000000 
000000c00013be78:  0000000000000000  0000000005403bf0 
000000c00013be88:  0000000005413650  000000c00013bf18 
fatal error: unknown caller pc

runtime stack:
runtime.throw(0x2f45b745, 0x11)
    /usr/local/go/src/runtime/panic.go:1116 +0x72
runtime.gentraceback(0x2f356188, 0xc00013bc78, 0x0, 0xc000000180, 0x0, 0x0, 0x7fffffff, 0x7ffeefbff7b0, 0x0, 0x0, ...)
    /usr/local/go/src/runtime/traceback.go:273 +0x19fb
runtime.addOneOpenDeferFrame.func1()
    /usr/local/go/src/runtime/panic.go:721 +0x91
runtime.systemstack(0x7ffeefbff830)
    /usr/local/go/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
    /usr/local/go/src/runtime/proc.go:1041

goroutine 1 [running, locked to thread]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:330 fp=0xc00013bac0 sp=0xc00013bab8 pc=0x2f3ad7c0
runtime.addOneOpenDeferFrame(0xc000000180, 0x0, 0x0)
    /usr/local/go/src/runtime/panic.go:720 +0x7b fp=0xc00013bb00 sp=0xc00013bac0 pc=0x2f381dbb
panic(0x2f43b820, 0x2f528c80)
    /usr/local/go/src/runtime/panic.go:971 +0x383 fp=0xc00013bbb8 sp=0xc00013bb00 pc=0x2f3826d3
runtime.panicmem(...)
    /usr/local/go/src/runtime/panic.go:212
runtime.sigpanic()
    /usr/local/go/src/runtime/signal_unix.go:695 +0x3da fp=0xc00013bbe8 sp=0xc00013bbb8 pc=0x2f396bfa
log.SetOutput(0x2f47e340, 0x0)
    /usr/local/go/src/log/log.go:277 +0x3c fp=0xc00013bc18 sp=0xc00013bbe8 pc=0x2f420aec
main.Setup()
    /Users/trankhai/Documents/GitHub/2hat_contest2/hybrid_go.go:19 +0x39 fp=0xc00013bc38 sp=0xc00013bc18 pc=0x2f420fa9
main._cgoexpwrap_cb1e17e59065_Setup()
    _cgo_gotypes.go:61 +0x20 fp=0xc00013bc48 sp=0xc00013bc38 pc=0x2f420dd0
runtime.call32(0x0, 0x7ffeefbfede0, 0x7ffeefbfee77, 0x0)
    /usr/local/go/src/runtime/asm_amd64.s:539 +0x3b fp=0xc00013bc78 sp=0xc00013bc48 pc=0x2f3adb7b
runtime.cgocallbackg1(0x0)
    /usr/local/go/src/runtime/cgocall.go:332 +0x188 fp=0xc00013bd10 sp=0xc00013bc78 pc=0x2f356188
runtime.cgocallbackg(0x0)
    /usr/local/go/src/runtime/cgocall.go:207 +0xc1 fp=0xc00013bd78 sp=0xc00013bd10 pc=0x2f355f61
runtime: unexpected return pc for runtime.cgocallback_gofunc called from 0x406a152
stack: frame={sp:0xc00013bd78, fp:0xc00013bd98} stack=[0xc00013a000,0xc00013c000)
000000c00013bc78:  0000000000000000  00007ffeefbfede0 
000000c00013bc88:  00007ffeefbfee77  0000000000000000 
000000c00013bc98:  000000002f38ca8b <runtime.exitsyscallfast+171>  000000c00013bcb0 
000000c00013bca8:  010000c00013bd78  000000002f3acd50 <runtime.exitsyscallfast.func1+0> 
000000c00013bcb8:  000201c00013bcaf  000000c000024000 
000000c00013bcc8:  000000c000000180  000000c00013bd00 
000000c00013bcd8:  000000c000000180  000000c00013bcbd 
000000c00013bce8:  000000002f4625a0  0000000000000000 
000000c00013bcf8:  0000000000000000  000000c00013bd68 
000000c00013bd08:  000000002f355f61 <runtime.cgocallbackg+193>  0000000000000000 
000000c00013bd18:  0000000000000010  000000000400550a 
000000c00013bd28:  0000000000000000  0000000000000000 
000000c00013bd38:  0000000000000000  0000000000000000 
000000c00013bd48:  0000000000000000  0000000000000000 
000000c00013bd58:  000000c00013bda0  000000c000000180 
000000c00013bd68:  00007ffeefbfeda0  000000002f3af14b <runtime.cgocallback_gofunc+155> 
000000c00013bd78: <0000000000000000  00000000047fd920 
000000c00013bd88:  000000c00013bda0 !000000000406a152 
000000c00013bd98: >000000000400552e  0000000004421c30 
000000c00013bda8:  000000c00013be08  00000000044b46a0 
000000c00013bdb8:  00000000041e7f01  00000000047fd920 
000000c00013bdc8:  000000c00013bdf8  00000000041e83cc 
000000c00013bdd8:  0000000004421c30  000000c00013be08 
000000c00013bde8:  000000c000052038  000000c00013be08 
000000c00013bdf8:  000000c00013be90  00000000041ea3d3 
000000c00013be08:  0000000005403bf0  0000000000000000 
000000c00013be18:  0000000005403bf0  0000000000000000 
000000c00013be28:  000000c00013be08  00000000041ea3bd 
000000c00013be38:  000000c000052030  0000000000000000 
000000c00013be48:  0000000000000000  0000000000000000 
000000c00013be58:  0000000000000000  0000000000000000 
000000c00013be68:  0000000000000000  0000000000000000 
000000c00013be78:  0000000000000000  0000000005403bf0 
000000c00013be88:  0000000005413650  000000c00013bf18 
runtime.cgocallback_gofunc(0x400552e, 0x4421c30, 0xc00013be08, 0x44b46a0)
    /usr/local/go/src/runtime/asm_amd64.s:793 +0x9b fp=0xc00013bd98 sp=0xc00013bd78 pc=0x2f3af14b
exit status 2

Can it help you?

[khaitranvan96kt]

There are things I do not understand, my code runs well in ubuntu but not in macos, what's the difference here?

[randall77]

Well that looks like a completely different panic trace.
Something is definitely clobbered here, it's crashing because log.std is nil, which can't happen (unless Python is calling back into Go before Go has been initialized)? Looks like memory corruption.

my code runs well in ubuntu but not in macos, what's the difference here?

I do not know. Without a way for us to reproduce it's going to be hard to find out.

[khaitranvan96kt]

I do not know. Without a way for us to reproduce it's going to be hard to find out.

I have uploaded it to the driver, Can you give me your gmail?

[cuonglm]

I do not know. Without a way for us to reproduce it's going to be hard to find out.

Please let me know after you've watched it, I will delete it because I don't want to be public
Demo

You can send it directly to @randall77 via email.

[andybons]

Marking as release-blocker until we figure out what’s going on.

[khaitranvan96kt]

Marking as release-blocker until we figure out what’s going on.

How do I solve this problem? Please

[ianlancetaylor]

We don't know what the problem is.

We will most likely need the ability to reproduce this problem ourselves.

[khaitranvan96kt]

We will most likely need the ability to reproduce this problem ourselves.

I have uploaded it to the driver, You can see it here

[randall77]

I downloaded your code and tried to run it, but ran into problems. We need exact instructions on how to get to the error posted. Show us every command you issued.

$ go run main.go
# pkg-config --cflags  -- python-2.7
pkg-config: exec: "pkg-config": executable file not found in $PATH

[khaitranvan96kt]

I downloaded your code and tried to run it, but ran into problems. We need exact instructions on how to get to the error posted. Show us every command you issued.

$ go run main.go
# pkg-config --cflags  -- python-2.7
pkg-config: exec: "pkg-config": executable file not found in $PATH

Thank you for watching, to resolve this error for macos

• Fix error: pkg-config: exec: "pkg-config": executable file not found in $PATH
  ```shell script
  brew install pkg-config

- Fix error: Perhaps you should add the directory containing `python-2.7.pc'
```bash
export PKG_CONFIG_PATH=$PKG_CONFIG_PATH:/System/Library/Frameworks/Python.framework/Versions/2.7/lib/pkgconfig

[khaitranvan96kt]

Show us every command you issued

• The package is compiled using the -buildmode=c-shared build flag to create the shared object binary:
  ```shell script
  go build -o hybrid_go.so -buildmode=c-shared hybrid_go.go

- Run webserver at http://localhost:8080
```shell script
go run main.go

• Run test
  shell script cd test/ && \ go run main.go

[randall77]

Ok, I can reproduce the problem.

Not sure what is going on yet. One thing to try would be to move the python initialization code out of an init function and into main. That way, we make sure that all packages are initialized before calling into python and back to Go again.

[khaitranvan96kt]

Not sure what is going on yet. One thing to try would be to move the python initialization code out of an init function and into main. That way, we make sure that all packages are initialized before calling into python and back to Go again.

I have tried all possible cases. But does not solve this problem on mac. Everything at Ubuntu is still very good

[khaitranvan96kt]

After finding a lot of ways, the only possible way is to work as sleel in python. You can see it below

def filter_text(text):
    time.sleep(0.5)
    results = {}
    print('Text:', text)
    words = text.split(' ')
    try:
        for word in words:
            msg = GoString(word.encode(), len(word))
            result = ''
            for i in range(100):
                result = lib.GetTopics(msg, len(word))
            result = result.decode()
            # print("lib.GetTopics(%s) = %s" % (word, result))
            if result == '':
                continue
            results.update(json.loads(result))
    except Exception as e:
        traceback.print_exc()
    time.sleep(0.5)
    return json.dumps(results)

But it's slow :))

[cuonglm]

I got completely different stack trace with this demo program.

Click to see stack trace

flag 0
Python version: 2.7.16 (default, Feb 29 2020, 01:55:37)
[GCC 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.29.20) (-macos10.15-objc-

fatal error: bad sweepgen in refill

runtime stack:
runtime.throw(0x60b406a, 0x16)
    /Users/cuonglm/sources/go/src/runtime/panic.go:1116 +0x72
runtime.(*mcache).refill(0x465f7d0, 0xc)
    /Users/cuonglm/sources/go/src/runtime/mcache.go:132 +0x126
runtime.(*mcache).nextFree(0x465f7d0, 0x5efcd0c, 0x5f29840, 0x475fd10, 0x7ffeefbfe6c0)
    /Users/cuonglm/sources/go/src/runtime/malloc.go:878 +0x87
runtime.mallocgc(0x50, 0x60a8a60, 0x3801, 0x3874)
    /Users/cuonglm/sources/go/src/runtime/malloc.go:1058 +0x792
runtime.newdefer.func2()
    /Users/cuonglm/sources/go/src/runtime/panic.go:417 +0x9f
runtime.newdefer(0x8, 0x8)
    /Users/cuonglm/sources/go/src/runtime/panic.go:415 +0x13b
runtime.addOneOpenDeferFrame.func1.1(0x7ffeefbfea70, 0x0, 0x617e440)
    /Users/cuonglm/sources/go/src/runtime/panic.go:757 +0xec
runtime.gentraceback(0x5feb7ea, 0xc00005bc48, 0x0, 0xc000000180, 0x0, 0x0, 0x7fffffff, 0x7ffeefbfeb30, 0x0, 0x0, ...)
    /Users/cuonglm/sources/go/src/runtime/traceback.go:334 +0x10c8
runtime.addOneOpenDeferFrame.func1()
    /Users/cuonglm/sources/go/src/runtime/panic.go:721 +0x91
runtime.systemstack(0x7ffeefbff590)
    /Users/cuonglm/sources/go/src/runtime/asm_amd64.s:370 +0x66
runtime.mstart()
    /Users/cuonglm/sources/go/src/runtime/proc.go:1094

goroutine 1 [running, locked to thread]:
runtime.systemstack_switch()
    /Users/cuonglm/sources/go/src/runtime/asm_amd64.s:330 fp=0xc00005bb50 sp=0xc00005bb48 pc=0x6005290
runtime.addOneOpenDeferFrame(0xc000000180, 0x5feb7ea, 0xc00005bc48)
    /Users/cuonglm/sources/go/src/runtime/panic.go:720 +0x7b fp=0xc00005bb90 sp=0xc00005bb50 pc=0x5fd78bb
panic(0x6092900, 0x6189900)
    /Users/cuonglm/sources/go/src/runtime/panic.go:929 +0xfc fp=0xc00005bc48 sp=0xc00005bb90 pc=0x5fd7f3c
runtime.panicmem(...)
    /Users/cuonglm/sources/go/src/runtime/panic.go:212
runtime.sigpanic()
    /Users/cuonglm/sources/go/src/runtime/signal_unix.go:718 +0x3ca fp=0xc00005bc78 sp=0xc00005bc48 pc=0x5feb7ea
log.SetOutput(0x60daba0, 0x0)
    /Users/cuonglm/sources/go/src/log/log.go:277 +0x3c fp=0xc00005bca8 sp=0xc00005bc78 pc=0x6077afc
main.Setup()
    /Users/cuonglm/Downloads/demo/hybrid_go.go:19 +0x39 fp=0xc00005bcc8 sp=0xc00005bca8 pc=0x6077ef9
main._cgoexpwrap_cb1e17e59065_Setup()
    _cgo_gotypes.go:61 +0x20 fp=0xc00005bcd8 sp=0xc00005bcc8 pc=0x6077d80
runtime.call32(0x0, 0x7ffeefbfebb0, 0x7ffeefbfec47, 0x0)
    /Users/cuonglm/sources/go/src/runtime/asm_amd64.s:540 +0x3e fp=0xc00005bd08 sp=0xc00005bcd8 pc=0x600561e
runtime.cgocallbackg1(0x0)
    /Users/cuonglm/sources/go/src/runtime/cgocall.go:332 +0x19b fp=0xc00005bda0 sp=0xc00005bd08 pc=0x5fac27b
runtime.cgocallbackg(0x0)
    /Users/cuonglm/sources/go/src/runtime/cgocall.go:207 +0xc1 fp=0xc00005be08 sp=0xc00005bda0 pc=0x5fac041
runtime: unexpected return pc for runtime.cgocallback_gofunc called from 0x4062e52
stack: frame={sp:0xc00005be08, fp:0xc00005be28} stack=[0xc00005b000,0xc00005c000)
000000c00005bd08:  0000000000000000  00007ffeefbfebb0
000000c00005bd18:  00007ffeefbfec47  0000000000000000
000000c00005bd28:  0000000005fe23b7 <runtime.exitsyscallfast+199>  000000c000028800
000000c00005bd38:  0000000005c11bb8  000000c00005bd68
000000c00005bd48:  020201c00005bd68  0000000000000200
000000c00005bd58:  00000000045ae9d8  000000c00005bd90
000000c00005bd68:  000000c000000180  000000c00005bd4d
000000c00005bd78:  00000000060ba0e8  0000000000000000
000000c00005bd88:  0000000000000000  000000c00005bdf8
000000c00005bd98:  0000000005fac041 <runtime.cgocallbackg+193>  0000000000000000
000000c00005bda8:  000000c00008e010  00000000040056ab
000000c00005bdb8:  00000000045869e0  0000000000000000
000000c00005bdc8:  0000000000000000  0000000000000000
000000c00005bdd8:  0000000000000000  0000000000000000
000000c00005bde8:  000000c00005be30  000000c000000180
000000c00005bdf8:  00007ffeefbfeb70  0000000006006c1b <runtime.cgocallback_gofunc+155>
000000c00005be08: <0000000000000000  0000000004586f40
000000c00005be18:  000000c00005be30 !0000000004062e52
000000c00005be28: >00000000040056cf  0000000004287660
000000c00005be38:  000000c00005be98  000000000400edf8
000000c00005be48:  0000000000000010  0000000004586f40
000000c00005be58:  000000c00005be88  00000000040e7cfa
000000c00005be68:  0000000004287660  000000c00005be98
000000c00005be78:  00000000042dbbc0  000000c00005be98
000000c00005be88:  000000c00005bed0  00000000040e9083
000000c00005be98:  0000000004717e80  0000000000000000
000000c00005bea8:  0000000004717e80  0100000004718fe0
000000c00005beb8:  0000000000000007  0000000004717e80
000000c00005bec8:  000000c00008e010  000000c00005bf48
000000c00005bed8:  0000000004281e5a  0000000004325e50
000000c00005bee8:  000000000000000d  0000000000000000
000000c00005bef8:  0000000000000002  0000000000000002
000000c00005bf08:  0000000000000007  0000000000000000
000000c00005bf18:  0000000000000000  000000000404343a
runtime.cgocallback_gofunc(0x40056cf, 0x4287660, 0xc00005be98, 0x400edf8)
    /Users/cuonglm/sources/go/src/runtime/asm_amd64.s:794 +0x9b fp=0xc00005be28 sp=0xc00005be08 pc=0x6006c1b

goroutine 1 [runnable, locked to thread]:
unicode.init()
    /Users/cuonglm/sources/go/src/unicode/tables.go:9 +0x2c1
exit status 2

[khaitranvan96kt]

I got completely different stack trace with this demo program.

I changed the code a bit to reproduce the error, but my problem didn't change

[randall77]

Here's a simple reproducer.

foo.go:

package main

import "C"
import "os"

//export Foo
func Foo() {
     println(os.Stdout)
}
func main() {
}

main.go:

package main

/*
#include <dlfcn.h>
#cgo LDFLAGS: -ldl
void f() {
   void *x = dlopen("./libfoo.so", RTLD_LAZY);
   void (*Foo)() = (void(*)())dlsym(x, "Foo");
   Foo();
}
*/
import "C"

func main() {
     C.f()
}

$ go build -o libfoo.so -buildmode=c-shared foo.go
$ go run main.go

This prints 0x0 on a mac. It prints something nonzero on linux.

Something is wrong with the Go->C->Go path. When we get back to Go, we have an uninitialized environment.

[ianlancetaylor]

It would be nice if this worked but I'm not terribly surprised that it doesn't. -buildmode=c-shared gives you a program that can be dlopen'ed by a C program. Trying to dlopen it in a Go program is going to run into symbol name collisions between the main program and the shared library. As far as I know nobody has tried to fix these collisions.

[khaitranvan96kt]

It would be nice if this worked but I'm not terribly surprised that it doesn't. -buildmode=c-shared gives you a program that can be dlopen'ed by a C program. Trying to dlopen it in a Go program is going to run into symbol name collisions between the main program and the shared library. As far as I know nobody has tried to fix these collisions.

Will my problem not be solved?

[networkimprov]

cc @odeke-em @eliasnaur for ideas on a solution.

[ianlancetaylor]

@khaitranvan96kt I hope that someone will solve your problem. But it is hard in the general case, and I do not know of anybody working on it. Sorry.

[khaitranvan96kt]

@khaitranvan96kt I hope that someone will solve your problem. But it is hard in the general case, and I do not know of anybody working on it. Sorry.

thanks you

[JohnStarich]

@ianlancetaylor Any insight into what it might take to fix (or work around) the darwin symbol collision issue?

I'm certainly not an expert in this area, but I'm not finding any solid resources on the darwin-specific dlopen behavior. Perhaps it's related to this "two-level namespace" concept I keep seeing. If you have any pointers, I can dig into it a bit further.

[ianlancetaylor]

I'm sorry, I don't have any useful pointers. I'm not familiar with the details of shared libraries on macOS.

I want to clarify that this is not an intended use case. The -buildmode=c-shared option is intended to build a Go shared library that is opened by a program written in C (or C++ or some other non-Go language). Using -buildmode=c-shared is not intended for a Go library that is opened by a Go program. People who need Go shared libraries that are opened by Go programs are expected to use either -buildmode=plugin or -buildmode=shared, depending on what they are trying to do.

That said, I would not be surprised if those options also have trouble on macOS. My point in mentioning this is that if you want to work on fixing something here, I encourage you to focus on -buildmode=plugin or -buildmode=shared before you focus on -buildmode=c-shared.

[JohnStarich]

@ianlancetaylor Thanks for the clarification 👍 I do find that position interesting, since I would expect a dlopened library to have the same behavior, regardless of which language it was written in. I understand that could be additional maintenance burden in the long run, so still a perfectly reasonable position.

Your doc on the different buildmodes was very helpful. If I'm understanding them properly, plugin requires identical versions of runtime and other packages to load, and shared appears to be a similar story. While these make it possible for Go to load shared Go libraries, the strict requirements of identical standard library packages make it more difficult to share libraries.

I was inspired by the power of c-shared to try loading and interacting with a separate Go program in-process. In other words, loading a Go shared library with a different toolchain inside another Go program makes development and distribution simpler.

As an example, I created a project to make writing Bash plugins easier by abstracting away the C-specific APIs and loading other programs as plugins. Unfortunately, since the runtime needs to be identical that means the program, plugins, and Go dist must all be built together in one shot. (Similar issues as found in https://github.com/golang/go/issues/27751)
Failing that, I tried c-shared instead, which led me here 😄

Based on your last point, focusing more on improving plugin or shared, is the best path forward to enable relaxed requirements on plugin so they're more portable?

[ianlancetaylor]

I would expect a dlopened library to have the same behavior, regardless of which language it was written in.

That relies on a particular semantics for symbols that occur in both the executable and the shared library, or that occur in multiple shared libraries. ELF provides fairly tight control for how to handle multiple symbol definitions, and Go tries to take advantage of that on ELF systems. But I don't know how that works on macOS.

I think it would be useful to remove the requirement that buildmode=plugin and buildmode=shared use exactly the same Go version. But I don't see any plausible way to do it. I guess it might be slightly easier to tackle for plugins, but it would require inventing a completely new mechanism for plugins, and not using the system dynamic linker at all. But we are talking about months of work.

[JohnStarich]

I see, thank you for your responses. Maybe if someone has some intimate knowledge of the macOS dynamic library situation they can chime in. (I may poke around in my spare time, who knows. 🙂)

Based on the source of plugin, it looks like the runtimes must match so that all shared symbols may also match. I agree that does sound like a lot of work for a new approach.
It'd also be interesting if there could somehow be multiple runtimes coordinating with each other, but I'm just musing here. That'd probably break all sorts of prior assumptions we wouldn't want to get into for Go.