Go: cmd/go: working directory affects binaries even with -trimpath

[dsnet]

Using go1.13rc1

After #16860 was closed, I was hoping that I could build byte-for-byte identical builds regardless of what machine was used to build the binaries. At the present moment, building binaries are nearly identical regardless of the machine used, except for the following bytes:

$ diff <(hexdump -Cv linux1/protoc-gen-go.v1.19.0-devel.linux.amd64) <(hexdump -Cv linux2/protoc-gen-go.v1.19.0-devel.linux.amd64)
251,254c251,254
< 00000fa0  53 00 00 00 04 00 00 00  47 6f 00 00 76 72 6e 46  |S.......Go..vrnF|
< 00000fb0  43 55 48 32 42 74 57 4e  53 4c 6a 62 54 78 47 35  |CUH2BtWNSLjbTxG5|
< 00000fc0  2f 54 49 30 2d 6d 6e 55  4f 45 49 52 5f 41 48 6c  |/TI0-mnUOEIR_AHl|
< 00000fd0  36 57 39 66 65 2f 6b 62  30 43 44 67 66 72 38 64  |6W9fe/kb0CDgfr8d|
---
> 00000fa0  53 00 00 00 04 00 00 00  47 6f 00 00 61 69 6a 4b  |S.......Go..aijK|
> 00000fb0  48 6b 71 33 2d 41 36 37  64 6a 63 77 50 4b 48 79  |Hkq3-A67djcwPKHy|
> 00000fc0  2f 65 66 2d 78 37 5f 34  72 4c 4d 5f 73 68 36 73  |/ef-x7_4rLM_sh6s|
> 00000fd0  79 56 75 47 31 2f 6b 62  30 43 44 67 66 72 38 64  |yVuG1/kb0CDgfr8d|

There seems to be some type of build ID encoded into the binary.

Steps to reproduce:

$ git clone https://go.googlesource.com/protobuf
$ cd protobuf
$ git checkout 08ff73004835d46c133844f593fe74596d3e8194
$ go test -mod=vendor -timeout=60m -count=1 integration_test.go -buildRelease
$ md5sum bin/protoc-gen-go.v1.19.0-devel.linux.amd64

Unfortunately, that command will take a while to run since it needs to build the C++ protoc binary as a dependency.

This need not block go1.13 release.

[dsnet]

Here's a shorter repro without needing to build protoc:
$ $ go get golang.org/dl/go1.13rc1 $ go1.13rc1 download $ git clone https://go.googlesource.com/protobuf $ cd protobuf $ git checkout 08ff73004835d46c133844f593fe74596d3e8194 $ GOOS=linux GOARCH=amd64 go1.13rc1 build -trimpath -ldflags "-s -w" -o protoc-gen-go ./cmd/protoc-gen-go

Running that above on different machines produces different binaries, even if both machines are the same host OS.

[dsnet]

It seems that the current working directory has some affect on the build ID embedded in the binary:

/tmp/protobuf $ GOOS=linux GOARCH=amd64 go1.13rc1 build -trimpath -ldflags "-s -w" -o protoc-gen-go ./cmd/protoc-gen-go && md5sum protoc-gen-go && rm protoc-gen-go
04b0ff070cc56ec8ed899cf986a97f50  protoc-gen-go
/tmp/protobuf $ GOOS=linux GOARCH=amd64 go1.13rc1 build -trimpath -ldflags "-s -w" -o protoc-gen-go ./cmd/protoc-gen-go && md5sum protoc-gen-go && rm protoc-gen-go
04b0ff070cc56ec8ed899cf986a97f50  protoc-gen-go
/tmp/protobuf $ cd .. && mv protobuf protobuf2 && cd protobuf2
/tmp/protobuf2 $ GOOS=linux GOARCH=amd64 go1.13rc1 build -trimpath -ldflags "-s -w" -o protoc-gen-go ./cmd/protoc-gen-go && md5sum protoc-gen-go && rm protoc-gen-go
47091a2082a8ef696c0b76c1ab983c62  protoc-gen-go
/tmp/protobuf2 $ GOOS=linux GOARCH=amd64 go1.13rc1 build -trimpath -ldflags "-s -w" -o protoc-gen-go ./cmd/protoc-gen-go && md5sum protoc-gen-go && rm protoc-gen-go
47091a2082a8ef696c0b76c1ab983c62  protoc-gen-go

[bcmills]

CC @jayconrod

[jrick]

This difference is due to the .note.go.buildid section added by the linker. It can be set to something static e.g. -ldflags=-buildid= (empty string) to gain reproducibility.

Thanks to @neutralinsomniac for working with me to find this.

[hugelgupf]

This difference is due to the .note.go.buildid section added by the linker. It can be set to something static e.g. -ldflags=-buildid= (empty string) to gain reproducibility.

Thanks to @neutralinsomniac for working with me to find this.

I just hit the same issue -- is there somewhere this can be documented better, and/or maybe given a -reproducible flag that == "-trimpath -ldflags=-buildid="?

[hugelgupf]

cc @rsc who fixed #16860

[gopherbot]

Change https://golang.org/cl/195318 mentions this issue: cmd/go: don't include package dir in cache key when -trimpath is set

[FiloSottile]

I'd like this to be considered for backport to Go 1.13 (@gopherbot, please do the honors).

There is no real workaround to get the functionality of -trimpath without this fix, which AFAICT was added specifically for reproducible builds.

[gopherbot]

Backport issue(s) opened: #34326 (for 1.13).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

[gopherbot]

Change https://golang.org/cl/198259 mentions this issue: [release-branch.go1.13] cmd/go: don't include package dir in cache key when -trimpath is set