Go: cmd/go: MacOS binaries invalid for eventual Apple Notary

[macetw]

What version of Go are you using (go version)?

1.12

Does this issue reproduce with the latest release?

Yes. (1.12)

What operating system and processor architecture are you using (go env)?

go env Output

GOARCH="amd64"

GOBIN=""

GOCACHE="/Users/macet/Library/Caches/go-build"

GOEXE=""

GOFLAGS=""

GOHOSTARCH="amd64"

GOHOSTOS="darwin"

GOOS="darwin"

GOPATH="/Users/macet/go"

GOPROXY=""

GORACE=""

GOROOT="/usr/local/go"

GOTMPDIR=""

GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"

GCCGO="gccgo"

CC="clang"

CXX="clang++"

CGO_ENABLED="1"

GOMOD=""

CGO_CFLAGS="-g -O2"

CGO_CPPFLAGS=""

CGO_CXXFLAGS="-g -O2"

CGO_FFLAGS="-g -O2"

CGO_LDFLAGS="-g -O2"

PKG_CONFIG="pkg-config"

GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/9v/6zqn9ncn39x63s0j25sqh7z00000gn/T/go-build033883546=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I applied my executable for an Apple Notary, required for release with the App Store on MacOS 10.14. Apple Notary gives users the assurance that their application is safe, but binaries must have the notary approval "stapled" to their .app. Apple Notary requirements include "runtime" option with a code signature and the MacOS SDK be "10.9" or newer. This is seen with "otool -l." With go, binaries are "10.7" based.

Here is the result of submission to Apple, presented in JSON format:
{"severity": "error", "code": null, "path": "mygobasedapplication.dmg/my/go/based/application", "message": "The binary uses an SDK older than the 10.9 SDK.", "docUrl": null}

What did you expect to see?

  version 10.9
      sdk 10.9

What did you see instead?

% otool -l main | tail -n4
      cmd LC_VERSION_MIN_MACOSX
  cmdsize 16
  version 10.7
      sdk 10.7

[macetw]

This is an update on top of issue #12941

For Apple documentation of this, see:
Url: https://developer.apple.com/documentation/security/notarizing_your_app_before_distribution/resolving_common_notarization_issues
Section: "Use the macOS 10.9 SDK or Later"

[randall77]

Is there anything we need to do other than change the constant?

[cherrymui]

This is set in the linker
https://go.googlesource.com/go/+/refs/heads/master/src/cmd/link/internal/ld/macho.go#665

It seems this is set to the minimum version Go required at the time this code was written.

[macetw]

At this time, I don't have full proof that this is the only change, but based on other evidence we have from a suite of binaries submitted, this is the only change: to change that constant in macho.go.

There are other requirements for Apple Notary, but those requirements fall outside of the scope of the compiler. They are more around the codesign step. Go binaries can be signed like any other binary.

[networkimprov]

Seems like a back-port candidate?

[macetw]

@networkimprov yes, this is viable for back-port also. I'm so far unfamiliar with how I would approach that.

[ianlancetaylor]

@gopherbot please open backport issues

[gopherbot]

Backport issue(s) opened: #30525 (for 1.11), #30526 (for 1.12).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[macetw]

I plan to submit code changes for this, including the backports.

[macetw]

How can I be recognized as the "assignee?"

[macetw]

Also, how can we recognize this no longer as "Needs Investigation?"

[macetw]

Haha. I just get @randall77 to do all my work. Thank you, Keith!

[networkimprov]

Gopherbot listens for commands re label manipulation, e.g.

@gopherbot add NeedsFix

Someone pls refresh my memory re the delete/drop command...

[networkimprov]

Can this patch provide a script to pull the version string from a canonical source, if networking is available?

[macetw]

@networkimprov help me understand your question.
That "10.7" or "10.9" (or whatever) version numbers (not strings) are in linked binaries. They do not appear in source.

[macetw]

At this time, I have a "No Contributor Agreement on file for user Tyler Mace" error, but I just got one completed this morning (via Autodesk). How long is the lag time for this to take effect? Is there a better forum for this question, other than my issue ticket?

[ianlancetaylor]

@macetw If you configure GitHub such that your e-mail address is @autodesk.com then I think the CLA robot will recognize that you are approved. If you don't want to do that then, if you are willing, I recommend that you sign the personal CLA as well, using the same e-mail address as your GitHub account.

[macetw]

@ianlancetaylor the google-group had both my private email and my corporate email, but in the contrib guide, it was said to use consistent email addresses. (I had already done work as [email protected]). You're suggesting that I not be consistent, but if that's okay, my employer would actually prefer it I use my corporate email.

[ianlancetaylor]

@macetw Are you using Gerrit or GitHub (go-review.googlesource.com or github.com)? I believe that Gerrit lets you have multiple e-mail addresses.

If you've already contributed under the e-mail address [email protected] then I don't know why that would not still work today.

[macetw]

@ianlancetaylor I haven't contributed lines of code yet. I've merely contributed via github issue conversations, like here. I'm new today on the CLA.

[networkimprov]

@macetw I should have written "hex constant".

Seems like its value should be pulled from an online source or config file via //go: generate

[randall77]

@networkimprov That seems overkill. Just a comment with a web link to where to find a recent number would be fine.

[ianlancetaylor]

@macetw Ah, OK, I got confused when you said you had already done work as [email protected]. The Google Group is irrelevant here. All that matters is your Gerrit or GitHub identity. It doesn't matter whether that identity is consistent with the Google Group. We just suggest that you use a single Gerrit or GitHub identity for all your contributions. For Gerrit that means a single main e-mail address although I believe that you can add secondary e-mail addresses. So go ahead and use your corporate e-mail address in Gerrit if that is OK with you. Thanks.

[networkimprov]

@randall77 then this issue should be a permanent release-blocker, so it's checked for every release.

[randall77]

@networkimprov That would be fine. We have a few such issues already, e.g. #12042.

(A //go:generate comment would not run every release anyway, for what it is worth.)

[macetw]

FYI, adding that link to my future PR. It is:
https://developer.apple.com/documentation/security/notarizing_your_app_before_distribution/resolving_common_notarization_issues

[macetw]

I still have a CLA problem on my corporate email, with git codereview....
% git codereview mail
fatal: remote error: No Contributor Agreement on file for user Tyler Mace [e-mail] (id=31443)

[networkimprov]

@gopherbot add release-blocker

[cherrymui]

From the document @macetw linked, macOS 10.9 or later is required "because of significant differences in the way code signing works prior to macOS 10.9". So it seems this doesn't need to change very frequently, therefore not necessarily release-blocker on every release.

[networkimprov]

The point is that it should be checked for every release, not that it will change.

[tmm1]

Note that if https://golang.org/cl/168321 is merged, one way to override the minimum version set by golang would be by using CGO_ENABLED=1 CGO_CFLAGS=-mmacosx-version-min=10.9 go build

[bcmills]

@macetw, are you still having CLA trouble?

[pieterclaerhout]

Any idea when this will be released?

We're actually waiting for this fix to be able to notarize our binaries for macOS. Since Apple will be pushing this after WWDC beginning of june, it would be awesome to be ready…

[ianlancetaylor]

As far as I know this isn't fixed yet. I'm not sure what is happening here. We've currently marked this as a release blocker for Go 1.13, which will be released around August 1.

[pieterclaerhout]

:-(

Is there any way to get this fixed earlier? I presume we're not the only ones running into this, right?

[ianlancetaylor]

The first step is to get it fixed at all. Once we have a fix we can discuss backporting that fix to a release branch. Right now, as far as I know, it is not fixed.

[networkimprov]

It's a minor change to a pair of constant expressions, and already marked for backport to 1.11 & 1.12.

https://github.com/golang/go/tree/master/src/cmd/link/internal/ld/macho.go#L418

This issue should perhaps be reassigned?

EDIT: mistakenly added "help" and "wanted" whilst trying to add "help wanted"

[ianlancetaylor]

If anyone has a patch to send that fixes this problem, please go ahead.

[tmm1]

FYI as of https://github.com/golang/go/commit/277609f844ed9254d25e975f7cf202d042beecc6 the version number is copied from external objects when available, so compiling with CGO_ENABLED=1 MACOSX_DEPLOYMENT_TARGET=10.14 will produce a notary compatible binary.

[pieterclaerhout]

FYI as of 277609f the version number is copied from external objects when available, so compiling with CGO_ENABLED=1 MACOSX_DEPLOYMENT_TARGET=10.14 will produce a notary compatible binary.

In which version of Go will that be available?

[ianlancetaylor]

That CL was part of the work for #22395 and it will be available in the 1.13 release.

@eliasnaur @cherrymui Would it be appropriate to backport https://golang.org/cl/168321 to a Go 1.12 release?

[eliasnaur]

CL 168321 is not entirely trivial and I believe it requires CL 168459 to apply. It doesn't even fix the root cause: the 10.7 version is still hardcoded in tip. That the CL enables a workaround as described by @tmm1 is accidental.

I propose someone mails the trivial change discussed earlier in this issue that bumps the version to 10.9 in tip, fixing this issue. The scope of the change is much narrower and can be manually backported. Would that be acceptable?

[ianlancetaylor]

I propose someone mails the trivial change discussed earlier in this issue that bumps the version to 10.9 in tip, fixing this issue. The scope of the change is much narrower and can be manually backported. Would that be acceptable?

Yes, I think so. Thanks.

[gopherbot]

Change https://golang.org/cl/175918 mentions this issue: cmd/link/internal/ld: bump macOS and macOS SDK version to 10.9

[pieterclaerhout]

Can you make sure this ends up in 1.12 as well?

[ianlancetaylor]

@pieterclaerhout The 1.12 backport is #30526.

[networkimprov]

Ian, could you reopen this? It needs to be reviewed before each release, as the value will change again.

Also its milestone should be set appropriately.

[ianlancetaylor]

@networkimprov Would you mind opening a new issue for that? This issue is pretty cluttered, and I wouldn't want people to have to read through all of it for each release. Thanks.

[networkimprov]

Filed #31918, thanks!