Go: all: ensure that Go toolchain meets Apple’s notarization requirements

[artyom]

What version of Go are you using (go version)?

$ go version
go version go1.13.3 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env

GO111MODULE="on"

GOARCH="amd64"

GOBIN=""

GOCACHE="/Users/artyom/Library/Caches/go-build"

GOENV="/Users/artyom/Library/Application Support/go/env"

GOEXE=""

GOFLAGS="-ldflags=-w -trimpath"

GOHOSTARCH="amd64"

GOHOSTOS="darwin"

GONOPROXY=""

GONOSUMDB=""

GOOS="darwin"

GOPATH="/Users/artyom/go"

GOPRIVATE=""

GOPROXY="https://proxy.golang.org,direct"

GOROOT="/Users/artyom/Library/go"

GOSUMDB="sum.golang.org"

GOTMPDIR=""

GOTOOLDIR="/Users/artyom/Library/go/pkg/tool/darwin_amd64"

GCCGO="gccgo"

AR="ar"

CC="clang"

CXX="clang++"

CGO_ENABLED="1"

GOMOD="/Users/artyom/Repositories/artyom/rex/go.mod"

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/lb/3rk8rqs53czgb4v35w_342xc0000gn/T/go-build499526677=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

Installed go from https://dl.google.com/go/go1.13.3.darwin-amd64.tar.gz and then tried to use it to build project. This is on macOS Catalina 10.15:

Darwin MacBook-Air.local 19.0.0 Darwin Kernel Version 19.0.0: Wed Sep 25 20:18:50 PDT 2019; root:xnu-6153.11.26~2/RELEASE_X86_64 x86_64

What did you expect to see?

Go works as usual.

What did you see instead?

On first go call I'm presented with OS pop-up that "macOS cannot verify that this app is free from malware", with options either to discard this window (the process is killed then) or move the binary to trash. After whitelisting this binary via Preferences → Security & Privacy → General → Allow Anyway, go can be called fine.

Next I tried to build project with go install and got the came pop-up for "compile" tool.

[artyom]

To be able to run updated Go 1.13.3 and compile my project with it, I had to whitelist so far these binaries:

• go
• compile
• asm
• cgo
• link

Whitelisting each takes the following:

1. presented with os pop-up about binary not being verified;
2. click cancel, at this moment binary is killed;
3. go to Preferences → Security & Privacy → General, click "Allow Anyway" button;
4. try running the same command again;
5. presented with os pop-up about binary not being verified, but now with another option to run binary anyway;
6. click "open" button on that pop-up.

[bradfitz]

Good times.

/cc @golang/osp-team

[bradfitz]

Related: #34748 (we need a Catalina builder)

[networkimprov]

Discussion on Hacker News: https://news.ycombinator.com/item?id=21179970

Also, Catalina is reported to be a disastrous upgrade.

[andybons]

I’ll take a look at this.

[dmitshur]

I can reproduce on my personal MBP with macOS 10.15 (19A602) when installing from the .pkg installer too, however it's a different (less intrusive) experience.

The "macOS cannot verify that this app is free from malware" prompt appears when first running the installer. After accepting and installing, Go binaries such as go, compile, do not produce the same prompt. Also, re-running the .pkg installer doesn't ask again.

[andybons]

In the process of submitting the installer to Apple's notary service, but the process is very slow due to a service issue on Apple's side and our internal tooling keeps timing out. Diagnosing this has been difficult.

It may be that we will need to enable the Hardened Runtime which likely requires a newer version of Xcode that we're using to build releases and an audit of whatever entitlements are required for our binaries to run.

@aclements can you take a look at the link above and let me know which you think we'll need (if any)?

[aclements]

@andybons, I don't think we need any of the runtime exceptions from that page. The only one that we may even potentially want is "Disable Library Validation Entitlement", which would let users use the the plugin package without signing their plugins, but it seems like developers probably should sign their plugins.

[vp2177]

The (.pkg) installer doesn't even start for me. Is that the same issue, or unrelated?

"go1.13.3.darwin-amd64.pkg can't be opened because Apple cannot check it for malicious software."

Meanwhile, the older Go version I had installed from before the OS X Catalina update, continues to work fine, no warning dialogs.

[andybons]

@vp2177 it’s the same issue.

[andybons]

Good news: according to Apple, we don’t have to enable the hardened runtime until January 2020. That said, we’ll likely have to start signing the binaries themselves (not just the installer as we do now), so this issue should stay open even if we fix the problem in the short term via notarization of the installer.

[andybons]

@dmitshur to follow up on your experience, do you have SIP (System Integrity Protection) turned on? csrutil status should give you that info.

@artyom do you have SIP turned on? Thanks.

[dmitshur]

@andybons Yes.

$ csrutil status
System Integrity Protection status: enabled.

[artyom]

@andybons yes, it's enabled:

¶ csrutil status
System Integrity Protection status: enabled.

[andybons]

Our installer is getting stuck in limbo with the Notarization service. We’ve contacted Apple and are awaiting a response. It’s unclear what the ETA on that will be, though.

[andybons]

https://developer.apple.com/system-status/ now shows an issue with the Notary service:

Performance: Today, 10:10 AM - ongoing

Some users are affected

Users are experiencing a problem with this service. We are working to resolve this issue.

[andybons]

Notarization of the installer package has completed successfully. We are testing locally to ensure no scary popups are shown.

[andybons]

While we believe we have the installer package issue resolved, we have to deal with the .tar.gz bundles, which require notarization of each individual binary. Working on this.

[andybons]

As an update, this is what we are going to do:

• Sign all Mach-O binaries within $GOROOT/bin/ and $GOROOT/pkg/tool/darwin_amd64 (without the Hardened Runtime enabled as it’s not required just yet)
• Construct the .pkg installer with the signed binaries included
• Sign the .pkg installer
• Send the .pkg installer off to the notarization service and staple the resulting ticket to the installer

Since the notarization process is recursive (and the bits being distributed in the .tar.gz file are the same that will be in the installer), this should solve the warnings for users that choose to install using either the .pkg file or the .tar.gz file.

I will update this thread once this work is complete.

[andybons]

The initial work mentioned in https://github.com/golang/go/issues/34986#issuecomment-546067381 is now complete. We will likely issue a point release this week with the changes after some further testing.

[virskor]

same issue

csrutil status
System Integrity Protection status: disabled.

go version
go version go1.13.3 darwin/amd64

osversion
10.15.1 (19B88)

[andybons]

New point releases have been released (https://golang.org/dl/#go1.13.4) that have signed and notarized binaries/installers. From testing locally I did not see any issues but feel free to comment if there are further problems.

Will continue the rest of the work (enabling the Hardened Runtime, for example) on #31918.

[andybons]

Oh, and Happy Halloween! 👻

[andybons]

Re-opening as #31918 should be tracking whether binaries created by the Go toolchain remain acceptable to Apple’s notarization service, which is a different issue than the toolchain itself being acceptable.

Remaining work:

• [ ] Enable the Hardened Runtime for the go, gofmt, link, etc. binaries (and figure out whatever entitlements are needed)
• [x] Figure out what to do about binaries shipped (within testdata) that will never be signed but the notary will reject

[andybons]

Below is derived from the log returned by Apple’s notarization service. I expect any warning to turn into an error come January 2020:

Warnings we intend to fix by enabling the Hardened Runtime on the listed binaries:

[warning]: (go/bin/go): The executable does not have the hardened runtime enabled.
[warning]: (go/bin/gofmt): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/vet): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/objdump): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/asm): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/trace): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/pprof): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/dist): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/pack): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/addr2line): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/compile): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/link): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/cgo): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/nm): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/fix): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/test2json): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/doc): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/buildid): The executable does not have the hardened runtime enabled.
[warning]: (go/pkg/tool/darwin_amd64/cover): The executable does not have the hardened runtime enabled.

Warnings we’ll have to fix another way:

[warning]: (go/src/compress/gzip/testdata/issue6550.gz): b'gunzip: data stream error\ngunzip: /tmp/tmpd_lw2hk0/issue6550.gz.unpacked_00/issue6550.gz: uncompress failed\n'
[warning]: (go/src/archive/zip/testdata/go-no-datadesc-sig.zip): b"ditto: foo.txt: No such file or directory\nditto: Couldn't read pkzip signature.\n"
[warning]: (go/src/cmd/internal/buildid/testdata/a.macho): The binary is not signed.
[warning]: (go/src/cmd/internal/buildid/testdata/a.macho): The signature does not include a secure timestamp.
[warning]: (go/src/cmd/internal/buildid/testdata/a.macho): The executable does not have the hardened runtime enabled.
[warning]: (go/src/debug/macho/testdata/gcc-386-darwin-exec): The binary is not signed.
[warning]: (go/src/debug/macho/testdata/gcc-386-darwin-exec): The signature does not include a secure timestamp.
[warning]: (go/src/debug/macho/testdata/gcc-386-darwin-exec): The executable does not have the hardened runtime enabled.
[warning]: (go/src/debug/macho/testdata/clang-386-darwin-exec-with-rpath): The binary is not signed.
[warning]: (go/src/debug/macho/testdata/clang-386-darwin-exec-with-rpath): The signature does not include a secure timestamp.
[warning]: (go/src/debug/macho/testdata/clang-386-darwin-exec-with-rpath): The executable does not have the hardened runtime enabled.
[warning]: (go/src/debug/macho/testdata/clang-amd64-darwin-exec-with-rpath): The binary is not signed.
[warning]: (go/src/debug/macho/testdata/clang-amd64-darwin-exec-with-rpath): The signature does not include a secure timestamp.
[warning]: (go/src/debug/macho/testdata/clang-amd64-darwin-exec-with-rpath): The executable does not have the hardened runtime enabled.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The binary is not signed.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The signature does not include a secure timestamp.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The executable does not have the hardened runtime enabled.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The binary is not signed.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The signature does not include a secure timestamp.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The executable does not have the hardened runtime enabled.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The binary is not signed.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The signature does not include a secure timestamp.
[warning]: (go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec): The executable does not have the hardened runtime enabled.
[warning]: (go/src/debug/macho/testdata/gcc-amd64-darwin-exec): The binary is not signed.
[warning]: (go/src/debug/macho/testdata/gcc-amd64-darwin-exec): The signature does not include a secure timestamp.
[warning]: (go/src/debug/macho/testdata/gcc-amd64-darwin-exec): The executable does not have the hardened runtime enabled.

De-duping from the above, the following binaries need to be dealt with (through obfuscation or some other means):

go/src/debug/macho/testdata/clang-amd64-darwin-exec-with-rpath
go/src/archive/zip/testdata/go-no-datadesc-sig.zip
go/src/debug/macho/testdata/fat-gcc-386-amd64-darwin-exec
go/src/debug/macho/testdata/gcc-amd64-darwin-exec
go/src/compress/gzip/testdata/issue6550.gz
go/src/debug/macho/testdata/clang-386-darwin-exec-with-rpath
go/src/cmd/internal/buildid/testdata/a.macho
go/src/debug/macho/testdata/gcc-386-darwin-exec

[aclements]

Is it saying the binaries need any particular hardened runtime entitlements, or just that all binaries need to be signed as having a hardened runtime?

[andybons]

@aclements just that the binaries must have the hardened runtime enabled. It doesn’t (and likely won‘t/can’t?) say what entitlements a binary needs.

[gopherbot]

Change https://golang.org/cl/208118 mentions this issue: all: base64-encode binaries that will cause Apple notarization to fail

[gopherbot]

Change https://golang.org/cl/208219 mentions this issue: [release-branch.go1.13] all: base64-encode binaries that will cause Apple notarization to fail

[gopherbot]

Change https://golang.org/cl/208220 mentions this issue: [release-branch.go1.12] all: base64-encode binaries that will cause Apple notarization to fail

[andybons]

@gopherbot please open a backport for 1.12 and 1.13 since otherwise Apple will reject future point releases of those versions

[gopherbot]

Backport issue(s) opened: #35747 (for 1.12), #35748 (for 1.13).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[gopherbot]

Backport issue(s) opened: #35747 (for 1.12), #35748 (for 1.13).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[nightlyone]

I wonder why tests and test data need to be shipped in releases. People interested in that can always retrieve it from the source repository. Has that been considered?

[bradfitz]

@nightlyone, it has, several times. The main reason we continue to include them is that when a user reports a problem, it's nice to be able to ask them "What does go test syscall or go test -v os say?". That's a question they can easily answer. If we make them do a new download, build, and parallel install, they're very unlikely to do so. Admittedly this debugging procedure doesn't happen too often, though.

[bcmills]

It also seems desirable for go test all to have the same behavior regardless of how the Go toolchain was installed, particularly in module mode (where all is actually a reasonable thing to want to test).

[gopherbot]

Change https://golang.org/cl/208227 mentions this issue: [release-branch.go1.12] cmd/vendor: remove _test.go and testdata files

[gopherbot]

Change https://golang.org/cl/208266 mentions this issue: cmd/release, cmd/releasebot: clean up unused code

[andybons]

I've run an altered all.bash that signs (with the hardened runtime enabled) the toolchain binaries and all tests passed on macOS 10.15.1.

We plan to enable the hardened runtime for the 1.14 beta (scheduled for early December) and see if anything breaks. If everything looks good, we'll enable it for all releases moving forward.

[toothrot]

Just to reiterate what Andy said, the hardened runtime will be enabled for the go1.14 beta release.

[cagedmantis]

I've confirmed with @bcmills that https://github.com/golang/go/issues/36572 should not be considered a blocker for go1.14rc1. I'm going to recommend that we close this issue. @andybons @dmitshur @toothrot

[dmitshur]

Sounds good. With the hardened runtime enabled as of Go 1.14 beta 1, I don't believe there's anything left to do here other than to watch for any issues if they're reported, but new issues can be opened for that.

I'll let one of @andybons or @toothrot confirm and close this issue since they're currently assigned to it.

[andybons]

Yes this looks to be fixed. If we discover any issues during the notarization stage of a release we'll open a new issue.