Go: runtime: some Go executables broken with macOS 10.12.4 / Xcode 8.3 but fine with Xcode 8.2

[dmaclach]

See https://github.com/couchbase/sync_gateway/issues/2417 for repro steps.

System:

MacBook Pro (x86_64)
macOS 10.12.3 (Sierra)
go version go1.8 darwin/amd64
Xcode 8.3; clang -v
_Apple LLVM version 8.1.0 (clang-802.0.38)
Target: x86_64-apple-darwin16.4.0
Thread model: posix_

What did you do?

See description at
https://github.com/couchbase/sync_gateway/issues/2417

I also found this with some code I am unable to share.

My code does work fine with Xcode 8.2:
clang -v
_Apple LLVM version 8.0.0 (clang-800.0.42.1)
Target: x86_64-apple-darwin16.4.0
Thread model: posix_

and from reading the bug linked above it works fine there as well.

[favadi]

I have the same problem as OP but only for some binaries, not all. When trying to start, it is killed instantly.

$ ./test
Killed: 9

I can't figure out the common pattern for broken programs.

[rasky]

Reproduction:

package main

import (
    "fmt"

    _ "github.com/veandco/go-sdl2/sdl"
)

func init() {
    fmt.Println("init")
}

func main() {
    fmt.Println("Hello, world")
}

 $ ./testx
 fish: './testx' terminated by signal SIGKILL (Forced quit)

Importing go-sdl2 is enough to make it crash, the prints are just to show that the runtime is unable to get to main.init. go-sdl2 has a single init() function, but commenting it out doesn't change anything, it still crashes right away.

I installed SDL itself via homebrew, the bottled version, a few weeks ago.

[rasky]

Basically, just using cgo and linking against sdl2 causes the crash:

package sdl

// #cgo linux freebsd darwin pkg-config: sdl2
import "C"

If you make the above file the whole go-sdl package, the binary will still crash.

[rasky]

Uhm it looks like lldb is even unable to spawn the binary:

(lldb) run
error: error: ::posix_spawnp ( pid => 24893, path = '/Users/rasky/Sources/go/src/ndsemu/testx/testx', file_actions = 0x7fff555225e8, attr = 0x7fff555225f8, argv = 0x7fd1e92a8150, envp = 0x7fd1e92ad520 ) err = Cannot allocate memory (0x0000000c)

so it seems like the binary is being linked in a way that the OS refuses to run it.

/cc @ianlancetaylor

[fighterlyt]

just the same, cgo make it crash

[crazytyper]

same here but with github.com/shirou/gopsutil/cpu instead of sdl2.

trying to run the binary with dtruss gives Unknown error.

[rgeronimi]

Same here, very simple to reproduce:

$ go test github.com/blevesearch/cld2
signal: killed
FAIL    github.com/blevesearch/cld2 0.003s

Investigating the MacOS system logs, it seems to be an issue with taskgated:
taskgated MacOS error: -67062
taskgated no signature for pid=4766 (cannot make code: UNIX[No such process])

Maybe a code signature issue around cgo...

[rasky]

I'll notice that all Go binaries (including working ones) generate the -67062 error, but I think that's normal because they are not codesigned. The errors I get only when running these crashing cgo binaries are:

default 14:32:47.099750 +0200   taskgated   UNIX error exception: 3
default 14:32:47.099853 +0200   taskgated   no signature for pid=28180 (cannot make code: UNIX[No such process])

[rgeronimi]

@rasky
On my environment (MacOS 10.12.14, Xcode8.3), non-cgo go programs do not generate -67062 errors. In my tests I performed, this error was strictly correlated to (1) crashes with the "signal: killed" message (2) cgo programs.
Additionally, I also get the "UNIX error exception: 3" message in the system logs

[margerum]

getting signal: killed when trying to use go run *.go

go build works, but then i get "Killed: 9" when I try to run the app

[aprilchild]

same here, dowloaded Xcode 8.2.1, replaced in /Applications and it works again..

[repustate]

For everyone running into this issue, I've been able to build binaries by adding -ldflags -s during the go build phase.

[ianlancetaylor]

By default on Darwin the linker will invoke dsymutil to put the debug info directly into the executable. This invocation of dsymutil is disabled by the linker's -s option. Perhaps this is really a bug in dsymutil, or in how dsymutil deals with files created by the Go linker. Could somebody with an affected system please try modifying the code in Link.hostlink in cmd/link/internal/ld/lib.go to skip invoking dsymutil, and see if that fixes the problem? Thanks.

[josharian]

@ianlancetaylor that does fix the problem. Tested on 1.8.

[ianlancetaylor]

The call to dsymutil was added to fix #8973. I suppose the next question is: if we don't call dsymutil, what happens when you try to debug a Go program built with cgo?

[cosnicolaou]

I'm seeing this without using cgo, what's the suggested work around - use xcode 8.2?

[bradfitz]

@cosnicolaou, see above (https://github.com/golang/go/issues/19734#issuecomment-289822644). Does that not work for you?

[cosnicolaou]

thanks, it works, from reading the comments I thought this was a cgo only problem/fix. It's also painful for our build/automation for obvious reasons.

[travissanderson-wf]

would adding these flags to build scripts have any negative repercussions that you're aware of? performance implications or anything like that?

[cosnicolaou]

no, it's just a pain to find all of the invocations etc.

[travissanderson-wf]

agreed :) thanks for verifying they are safe to add though

[cosnicolaou]

oh, and all of our reg tests that build go programs also have to changed to work with this... that's a lot of code for us. When do you expect to have a 'transparent' fix done/out? Thanks.

[ianlancetaylor]

I don't have a Mac. I'm waiting for an answer to my question about #8973.

[rasky]

I've never used gdb/lldb on a Go program, so it's hard for me to spot patterns that are different from "usual". Is there some specific test that you would like to be performed (e.g.: specific things that you want us to check whether they work or not after dsymutil is taken out), or is it a really a generic question and it requires someone familiar with the matter that can do some sessions and find out if there's something wrong?

[ianlancetaylor]

It's really a generic question, thanks.

[kchristidis]

For building binaries, https://github.com/golang/go/issues/19734#issuecomment-289822644 is the way to go.

How do you go about go test-ing things though, without the signal: kill error?

[repustate]

@kchristidis The same, just do go test -ldflags -s ...

[dmitshur]

I suppose the next question is: if we don't call dsymutil, what happens when you try to debug a Go program built with cgo?

@ianlancetaylor, using which debugger?

[ianlancetaylor]

@shurcooL I have no idea. I don't use MacOS. I'm just trying to help people connect the dots from #8973 to this problem, and trying to understand whether we can just stop calling dsymutil or whether that will cause #8973 to reappear. #8973 mentions both gdb and lldb.

[repustate]

@ianlancetaylor Built 1.8 from source, commenting out the dsymutil section, ran gdb on a simple program that uses cgo, got this:

Reading symbols from $HOME/go/simple...
warning: `/var/folders/84/p363xh452_b1_f909jm5yw780000gn/T/go-build096684628/runtime/cgo/_obj/gcc_amd64.o': can't open to read symbols: No such file or directory.

warning: `/var/folders/84/p363xh452_b1_f909jm5yw780000gn/T/go-build096684628/runtime/cgo/_obj/gcc_context.o': can't open to read symbols: No such file or directory.

warning: `/var/folders/84/p363xh452_b1_f909jm5yw780000gn/T/go-build096684628/runtime/cgo/_obj/gcc_darwin_amd64.o': can't open to read symbols: No such file or directory.

warning: `/var/folders/84/p363xh452_b1_f909jm5yw780000gn/T/go-build096684628/runtime/cgo/_obj/gcc_libinit.o': can't open to read symbols: No such file or directory.

warning: `/var/folders/84/p363xh452_b1_f909jm5yw780000gn/T/go-build096684628/runtime/cgo/_obj/gcc_setenv.o': can't open to read symbols: No such file or directory.

warning: `/var/folders/84/p363xh452_b1_f909jm5yw780000gn/T/go-build096684628/runtime/cgo/_obj/gcc_util.o': can't open to read symbols: No such file or directory.

warning: `/var/folders/84/p363xh452_b1_f909jm5yw780000gn/T/go-build185537422/mypackage/_obj/_cgo_export.o': can't open to read symbols: No such file or directory.

warning: `/var/folders/84/p363xh452_b1_f909jm5yw780000gn/T/go-build185537422/mypackage/_obj/mypackage.cgo2.o': can't open to read symbols: No such file or directory.

warning: `/var/folders/84/p363xh452_b1_f909jm5yw780000gn/T/go-link-879290217/go.o': can't open to read symbols: No such file or directory.
(no debugging symbols found)...done.

[lyonel]

Quick and dirty workaround: have a "fake" dsymutil before /usr/bin/dsymutil in your path

$ ln -s /usr/bin/true /usr/local/bin/dsymutil

(of course, this will prevent you from using gdb)

[rsc]

I let my Mac update to Xcode 8.3 and now various binaries are killed during all.bash. So consider this reproduced. I'll try to figure it out, since I can't get any work done anymore.

[rsc]

CL 38747 should fix the problem (by not using dsymutil's output), and we'll get that backported into Go 1.8.1. Until then use -ldflags=-s as the workaround.

I filed #19772 to track a non-workaround fix. If that ends up being very simple we might consider it for a backport, but I'm not holding my breath.

[gopherbot]

CL https://golang.org/cl/38747 mentions this issue.

[nicksnyder]

in case anyone else ran into this, go install -ldflags -s doesn't work.
You have to go build -ldflags -s and then run that binary.

[rsc]

If anyone has a working Xcode 8.2 system, can you please run

cd go/src/runtime/testdata/testprogcgo
go build

and then post the 'testprogcgo' binary that is built? Here is fine if you can get Github to accept it; anywhere else is fine too. Thanks.

[bradfitz]

@rsc, testprogcgo.gz

[gopherbot]

CL https://golang.org/cl/38853 mentions this issue.

[gopherbot]

CL https://golang.org/cl/38854 mentions this issue.

[rsc]

Be the bot you want to see in the world.

[l0n3r4n83r]

I used https://developer.apple.com/download/more/?name=Xcode to download and install Command line tools 8.2 as work around. Remove /Library/Developer/CommandLineTools before installing.

[johnsogg]

For any gb users who stumble on this, you can apply the same fix from the comment above, except I need to add in the equal sign:

gb test -ldflags=-s

[thejacobtaylor]

Thank you for the workaround. I was totally stuck until I looked in my update log and noticed XCode had updated yesterday.

Kill signal 9

[samus]

I was able to get around this while keeping the latest Xcode installed. Download Xcode 8.2 from Apple and unpack it if your browser hasn't already. Rename the extracted bundle, I renamed it to Xcode8.2 then copy it over to /Applications. From a terminal use xcode-select to set 8.2 as toolchain to use.

sudo xcode-select --switch /Applications/Xcode8.2.app/

[jekriske-lilly]

@nicksnyder go install -ldflags -s does work for me but only if I delete the binary in my bin folder first.

[rsc]

@jekriske-lilly That's right, go install doesn't include the gcflags or ldflags when deciding something is out of date. We're planning to change that at some point, but for now that's how it is.

[gopherbot]

CL https://golang.org/cl/39602 mentions this issue.

[gopherbot]

CL https://golang.org/cl/39603 mentions this issue.

[rsc]

Cherry-picked.

[drewlandis]

I ran into this problem as well. Simply deleting Xcode 8.3 and replacing it with Xcode 8.2.1 did not downgrade the CommandLineTools. I needed to first remove /Library/Developer/CommandLineTools before installing Xcode 8.2.1. Another (probably better) option is to remove only /Library/Developer/CommandLineTools and download _only_ the 8.2.1 CommandLineTools. See comment by @l0n3r4n83r above. Here is the link again: https://developer.apple.com/download/more/?name=Xcode

[DomT4]

Apologies for the commit spam in this thread, I forgot GitHub would do that 😓. Are there any plans to backport the fix beyond 1.8? I can seem to reproduce this using this test code right back to at least 1.5.

[ianlancetaylor]

At present there are no plans to make any more releases of anything earlier than 1.8. We'll have to see how it goes.

[rsc]

There are no plans to backport this to Go 1.7 or earlier. Per https://golang.org/doc/devel/release.html#policy, Go 1.7 is no longer supported, since Go 1.8 is released.

If you must still use Go 1.7, the workaround is to compile with -ldflags=-s or to sudo ln -s /usr/bin/true /usr/local/bin/dsymutil, assuming /usr/local/bin is ahead of /usr/bin on your $PATH.

[bradfitz]

@rsc, I thought in #19069 we decided to support the past two releases. Isn't that what your 86abfbb931312db022ed8437a89219791ced6435 says?

[rsc]

I have been assuming that takes effect with Go 1.8: Go 1.9 - the first release with that policy - won't invalidate Go 1.8; Go 1.10 will.

[bradfitz]

Well, as one datapoint, Kubernetes still hasn't moved from Go 1.7 to Go 1.8, so this means you can't run Kubernetes (even Kubernetes 1.6) on a Mac now.

[bradfitz]

Er, build, not run. Maybe that's okay.

[rsc]

Right, build, and even then only build with newer Xcode. It's not like the gettimeofday crashes.

[bradfitz]

This may impact their point release builds, but I'm not sure how Kubernetes does their releases (on an actual Mac, or cross-compiled).

Running file on https://storage.googleapis.com/kubernetes-release/release/v1.6.1/bin/darwin/amd64/kubectl just says kubectl: Mach-O 64-bit executable x86_64. (no ldd... I don't know Mac tools)

[rsc]

They can always add -ldflags=-s or update to Go 1.8.1 like everyone else. Also, kubectl at least doesn't appear to have any cgo code in it (otool -l kubectl | grep nundefsym).

[abh]

Yeah, kubectl seems to be cross compiled (it doesn't use the macOS DNS resolver).

[vyorkin]

none of the above workarounds worked for me :(

mac os x sierra 10.12.4
xcode 8.2 (8C38)
command line tools: http://adcdownload.apple.com/Developer_Tools/Command_Line_Tools_macOS_10.12_for_Xcode_8.2/Command_Line_Tools_macOS_10.12_for_Xcode_8.2.dmg

before installing command line tools I've removed /Library/Developer/CommandLineTools directory

UPD: it works now! after uninstalling the previous version with brew uninstall ethereum and installing a new one from the develop branch with brew install ethereum --devel

[pawelkowalak]

Update to Go 1.8.1, it is fixed now.

[rodrigo-brito]

Go 1.8.1 Works! 🎉

[OSMeteor]

use go1.8.1 fixed it good