Go: cmd/link: use libsystem_kernel.dylib or libSystem.dylib for syscalls on macOS

[copumpkin]

As I understand it, Go currently has its own syscall wrappers for Darwin. This is explicitly against what Apple recommends, precisely because they're not willing to commit to a particular syscall ABI. This leads to issues like https://github.com/golang/go/issues/16570, and although we've been lucky in that things have generally been backward-compatible so far, there's no guarantee that it'll continue to happen. It doesn't seem inconceivable to me that we'd at some point end up having to specify "build for macOS 10.13+" vs. "build for 10.12 and below", for example.

Linking against libsystem_kernel.dylib (or the broader libSystem.dylib) would put Go back in line with Apple's recommendations for the platform, and the library is guaranteed to exist on all macOS boxes.

Apologies if this has been suggested before and people have already gone over good reasons not to do it, but I'm currently struggling to figure out what appears to be another Sierra incompatibility related to #16570 above (still haven't figured it out enough to post a reproducible bug) and am wishing we didn't have to deal with these issues.

[ianlancetaylor]

I agree that we have to do this. I'm not sure who is actually going to do it, though.

[quentinmit]

Note that the trouble with #16570 comes from fetching the current time, and that is something that is so performance critical that we might want to continue to use our own implementation even if we use libSystem for other syscall.

[crawshaw]

We got into a discussion about how to do this over on #17200. I believe this can be done without forcing external linking. But I also don't know who is going to do it, my plate is full for the foreseeable future.

[rsc]

Agree we should do this at some point. It's not terribly high priority though.

[rsc]

This is about using the system calls. We already link against libSystem.dylib when cgo is in use.

[hirochachacha]

Hi, I made a CL for x86. https://go-review.googlesource.com/c/50290/
Could someone review it (after development cycle open) ?
If the overall design is OK, I'd like do something in go 1.10. Thank you.

[gopherbot]

CL https://golang.org/cl/50290 mentions this issue.

[gopherbot]

Change https://golang.org/cl/54871 mentions this issue: cmd/internal/obj/x86: don't apply workaround for solaris to darwin

[rsc]

Despite the release-blocker label I don't see this happening for Go 1.10. Moving to Go 1.11. If we really want to do this we need to make sure work starts at the beginning of the cycle.

[ianlancetaylor]

Nick Kledzik suggested in e-mail:

There is a middle ground that some JIT creators use, which is to wrap their JITed code in a “boilerplate” mach-o.

For the Go case, since you already generate the wrapper mach-o structure for executables, you expand that a bit, and standard content for linking to libSystem.dylib. This would include the stubs (PLT) section and lazy pointer section (GOT), and the LINKEDIT stuff to bind them for all the POSIX calls Go supports. The Go compiler then calls the stub (PLT) instead of directly inlining the syscall.

The new mach-o content would be the same for all Go executables, so you don’t need a linker. You write a C program that calls all POSIX functions Go supports and compile/link that once, and copy the stubs/lazy pointer sections and LINKEDIT content for use as your boilerplate.

This won’t solve the thread local issue, but will make Go executables true dynamic executables and free Apple to evolve the private libSystem/kernel syscall interface.

[gopherbot]

Change https://golang.org/cl/108679 mentions this issue: runtime,cmd/ld: on darwin, create theads using libc

[gopherbot]

Change https://golang.org/cl/110475 mentions this issue: runtime: implement darwin raise with pthread_self and pthread_kill

[gopherbot]

Change https://golang.org/cl/110437 mentions this issue: runtime: move more syscalls to libc on Darwin

[gopherbot]

Change https://golang.org/cl/110438 mentions this issue: runtime: move open/close/read/write from syscall to libc on Darwin

[gopherbot]

Change https://golang.org/cl/110655 mentions this issue: runtime: use libc for nanotime on Darwin

[andlabs]

Probably the wrong place to ask this, but I am curious: what does MOVL $0xf1, 0xf1 do to trigger a crash? Does it just get assembled to a bunch of icebp instructions?

[ianlancetaylor]

MOVL $0xf1, 0xf1 assembles into exactly what it looks like: a store of a value to the memory address 0xf1. Since that memory address is not mapped, it causes a bus error, crashing the program. The advantage of using 0xf1 is that the address will be printed in the stack backtrace, giving a clue where the problem happened.

[gopherbot]

Change https://golang.org/cl/111015 mentions this issue: cmd/link,runtime: move syscalls to to libc on iOS

[gopherbot]

Change https://golang.org/cl/111258 mentions this issue: runtime: fix darwin 386/amd64 stack switches

[gopherbot]

Change https://golang.org/cl/116875 mentions this issue: runtime: use libc's signal functions on Darwin

[gopherbot]

Change https://golang.org/cl/118736 mentions this issue: runtime: move semaphore ops from system calls to libc calls on Darwin

[randall77]

Package runtime now calls out to libSystem for all its syscall needs. That should fix our dependence on the syscall API for most Go binaries.

Direct use of package syscall still does raw syscalls; it does not use libSystem. There are ~100 exported functions to deal with there, so it's a bunch of additional work. I'm going to leave this issue open for that task, but bump to 1.12. In case anyone sees

1. An easier way to do it
2. A reason to do it
3. Someone to do it

we can continue this work.

[gopherbot]

Change https://golang.org/cl/141639 mentions this issue: syscall: implement syscalls on Darwin using libSystem

[gopherbot]

Change https://golang.org/cl/148257 mentions this issue: syscall: move uses of Syscall to libSystem on darwin

[ianlancetaylor]

@randall77 Any reason to keep this issue open?

[randall77]

Yes, I still need to do x/sys/unix and vendor it into the stdlib.
I think everything else is done.

[gopherbot]

Change https://golang.org/cl/154179 mentions this issue: x/sys/unix: convert Darwin syscalls from raw to libSystem

[gopherbot]

Change https://golang.org/cl/154660 mentions this issue: unix: remove raw syscall from Getfsstat

[gopherbot]

Change https://golang.org/cl/154661 mentions this issue: unix: remove raw syscall from getattrlistTimes

[gopherbot]

Change https://golang.org/cl/154662 mentions this issue: unix: remove raw syscall from getAttrList

[gopherbot]

Change https://golang.org/cl/154663 mentions this issue: unix: remove raw syscall from Sendfile

[gopherbot]

Change https://golang.org/cl/155737 mentions this issue: cmd: vendor x/sys/unix into the stdlib

[gopherbot]

Change https://golang.org/cl/156346 mentions this issue: unix: make Fcntl* routines use libSystem on Darwin

[gopherbot]

Change https://golang.org/cl/156365 mentions this issue: cmd: vendor x/sys/unix into the stdlib

[gopherbot]

Change https://golang.org/cl/197938 mentions this issue: runtime: fix darwin syscall performance regression

[gopherbot]

Change https://golang.org/cl/200103 mentions this issue: [release-branch.go1.13] runtime: fix darwin syscall performance regression

[gopherbot]

Change https://golang.org/cl/227037 mentions this issue: crypto/x509: use Security.framework without cgo for roots on macOS