Rules_go: internal compiler errors using go1.7.6

[ixdy]

bazel 0.5.4, rules_go 473417ec48310325e1fcb1c154621a83197a17fe

I updated the rules_go reference on one the older kubernetes release branches (release-1.6) to support newer versions of bazel, and now when trying to build with go1.7.6, I'm getting weird internal compiler errors:

$ bazel build pkg/credentialprovider/aws/...
INFO: Analysed 4 targets (18 packages loaded).              
INFO: Found 4 targets...                                    
ERROR: [home]/code/kubernetes/src/k8s.io/kubernetes/pkg/credentialprovider/aws/BUILD:25:1: GoCompile pkg/credentialprovider/aws/~lib~/go_default_test.o failed (Exit 1): compile failed: er
ror executing command                                       
  (cd [home]/.cache/bazel/_bazel_jgrafton/75e5737b0f848bf1b7c3ab3303c8bf72/execroot/__main__ && \
  exec env - \                                              
    GOARCH=amd64 \                                          
    GOOS=linux \                                            
    GOROOT=[home]/.cache/bazel/_bazel_jgrafton/75e5737b0f848bf1b7c3ab3303c8bf72/external/go1_7_6_linux_amd64 \
  bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/compile external/go1_7_6_linux_amd64/bin/go -src pkg/credentialprovider/aws/aws_credentials.go -src pkg/credentialprovider/aws/aws_credentials_tes
t.go -dep k8s.io/kubernetes/pkg/credentialprovider -dep github.com/aws/aws-sdk-go/aws -dep github.com/aws/aws-sdk-go/service/ecr -dep k8s.io/kubernetes/pkg/credentialprovider -dep github.com/aws/aws-sdk-go/aws -d
ep github.com/aws/aws-sdk-go/aws/request -dep github.com/aws/aws-sdk-go/aws/session -dep github.com/aws/aws-sdk-go/service/ecr -dep github.com/golang/glog -o 'bazel-out/local-fastbuild/bin/pkg/credentialprovider/
aws/~lib~/go_default_test.o' -trimpath . -I . -I 'bazel-out/local-fastbuild/bin/pkg/credentialprovider/~lib~/' -I 'bazel-out/local-fastbuild/bin/vendor/~lib~/' -I 'bazel-out/local-fastbuild/bin/vendor/~lib~/' -I 
'bazel-out/local-fastbuild/bin/pkg/credentialprovider/~lib~/' -I 'bazel-out/local-fastbuild/bin/vendor/~lib~/' -I 'bazel-out/local-fastbuild/bin/vendor/~lib~/' -I 'bazel-out/local-fastbuild/bin/vendor/~lib~/' -I 
'bazel-out/local-fastbuild/bin/vendor/~lib~/' -I 'bazel-out/local-fastbuild/bin/vendor/~lib~/' --)

Use --sandbox_debug to see verbose messages from the sandbox
pkg/credentialprovider/aws/aws_credentials.go:28: internal compiler error: importer: expected package tag, found tag = -28
goroutine 1 [running]:                                      
runtime/debug.Stack(0x0, 0x0, 0x0)                          
        /usr/local/go/src/runtime/debug/stack.go:24 +0x79   
cmd/compile/internal/gc.Fatalf(0x90c78e, 0x2e, 0xc42079a5d8, 0x1, 0x1)
        /usr/local/go/src/cmd/compile/internal/gc/subr.go:165 +0x248
cmd/compile/internal/gc.(*importer).pkg(0xc42049a000, 0x32) 
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:221 +0xa2a
cmd/compile/internal/gc.(*importer).fieldName(0xc42049a000, 0x0)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:603 +0x397
cmd/compile/internal/gc.(*importer).field(0xc42049a000, 0x5)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:545 +0x3d
cmd/compile/internal/gc.(*importer).fieldList(0xc42049a000, 0xffffffffffffff16, 0xc4208ddf40, 0x90)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:536 +0x8f
cmd/compile/internal/gc.(*importer).typ(0xc42049a000, 0xc4208ddc20)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:483 +0xc77
cmd/compile/internal/gc.(*importer).typ(0xc42049a000, 0xffffffffffffff12)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:407 +0x148
cmd/compile/internal/gc.(*importer).typ(0xc42049a000, 0xffffffffffffff14)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:487 +0xfde
cmd/compile/internal/gc.(*importer).typ(0xc42049a000, 0xc420952690)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:474 +0xbf4
cmd/compile/internal/gc.(*importer).field(0xc42049a000, 0xc420977710)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:546 +0x55
cmd/compile/internal/gc.(*importer).fieldList(0xc42049a000, 0xffffffffffffff16, 0xc4208dd450, 0x3f)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:536 +0x8f
cmd/compile/internal/gc.(*importer).typ(0xc42049a000, 0xc4208dd180)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:483 +0xc77
cmd/compile/internal/gc.(*importer).typ(0xc42049a000, 0x0)  
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:407 +0x148
cmd/compile/internal/gc.(*importer).obj(0xc42049a000, 0xfffffffffffffffd)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:282 +0x967
cmd/compile/internal/gc.Import(0xc42049e000)                
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:97 +0x3b2
cmd/compile/internal/gc.importfile(0xc420424800, 0x0, 0x0, 0x0)
        /usr/local/go/src/cmd/compile/internal/gc/main.go:843 +0x9b1
cmd/compile/internal/gc.(*parser).importdcl(0xc4204247e0)   
        /usr/local/go/src/cmd/compile/internal/gc/parser.go:341 +0x178
cmd/compile/internal/gc.(*parser).import_(0xc4204247e0)     
        /usr/local/go/src/cmd/compile/internal/gc/parser.go:300 +0x71
cmd/compile/internal/gc.(*parser).file(0xc4204247e0)        
        /usr/local/go/src/cmd/compile/internal/gc/parser.go:268 +0x5e
cmd/compile/internal/gc.parse_file(0xc420074240)            
        /usr/local/go/src/cmd/compile/internal/gc/parser.go:31 +0x54
cmd/compile/internal/gc.Main()                              
        /usr/local/go/src/cmd/compile/internal/gc/main.go:334 +0x14f4
cmd/compile/internal/amd64.Main()                           
        /usr/local/go/src/cmd/compile/internal/amd64/galign.go:93 +0x2fa
main.main()                                                 
        /usr/local/go/src/cmd/compile/main.go:33 +0x2a3     

The BUILD files are a bit of a mess in this branch (we have one BUILD file for all of vendor/), but updating to use multiple BUILD files doesn't seem to help.

I can reproduce on kubernetes master as well, with a slightly different error:

$ git diff
diff --git a/build/root/WORKSPACE b/build/root/WORKSPACE
index f92e09dded..95e6e362c7 100644
--- a/build/root/WORKSPACE
+++ b/build/root/WORKSPACE
@@ -43,7 +43,7 @@ load("@io_bazel_rules_go//go:def.bzl", "go_repositories")
 load("@io_bazel_rules_docker//docker:docker.bzl", "docker_repositories", "docker_pull")

 go_repositories(
-    go_version = "1.8.3",
+    go_version = "1.7.6",
 )

 docker_repositories()

$ bazel build pkg/credentialprovider/aws/...
INFO: Analysed 4 targets (1 packages loaded).
INFO: Found 4 targets...
ERROR: [home]/code/kubernetes/src/k8s.io/kubernetes/pkg/credentialprovider/aws/BUILD:22:1: GoCompile pkg/credentialprovider/aws/~lib~/go_default_test.o failed (Exit 1): compile failed: error executing command
  (cd [home]/.cache/bazel/_bazel_jgrafton/75e5737b0f848bf1b7c3ab3303c8bf72/execroot/__main__ && \
  exec env - \
    GOARCH=amd64 \
    GOOS=linux \
    GOROOT=[home].cache/bazel/_bazel_jgrafton/75e5737b0f848bf1b7c3ab3303c8bf72/external/go1_7_6_linux_amd64 \
  bazel-out/host/bin/external/io_bazel_rules_go/go/tools/builders/compile external/go1_7_6_linux_amd64/bin/go -src pkg/credentialprovider/aws/aws_credentials.go -src pkg/credentialprovider/aws/aws_credentials_test.go -dep k8s.io/kubernetes/pkg/credentialprovider -dep github.com/aws/aws-sdk-go/aws -dep github.com/aws/aws-sdk-go/service/ecr -dep k8s.io/kubernetes/pkg/credentialprovider -dep github.com/aws/aws-sdk-go/aws -dep github.com/aws/aws-sdk-go/aws/request -dep github.com/aws/aws-sdk-go/aws/session -dep github.com/aws/aws-sdk-go/service/ecr -dep github.com/golang/glog -o 'bazel-out/local-fastbuild/bin/pkg/credentialprovider/aws/~lib~/go_default_test.o' -trimpath . -I . -I 'bazel-out/local-fastbuild/bin/pkg/credentialprovider/~lib~/' -I 'bazel-out/local-fastbuild/bin/vendor/github.com/aws/aws-sdk-go/aws/~lib~/' -I 'bazel-out/local-fastbuild/bin/vendor/github.com/aws/aws-sdk-go/service/ecr/~lib~/' -I 'bazel-out/local-fastbuild/bin/pkg/credentialprovider/~lib~/' -I 'bazel-out/local-fastbuild/bin/vendor/github.com/aws/aws-sdk-go/aws/~lib~/' -I'bazel-out/local-fastbuild/bin/vendor/github.com/aws/aws-sdk-go/aws/request/~lib~/' -I 'bazel-out/local-fastbuild/bin/vendor/github.com/aws/aws-sdk-go/aws/session/~lib~/' -I 'bazel-out/local-fastbuild/bin/vendor/github.com/aws/aws-sdk-go/service/ecr/~lib~/' -I 'bazel-out/local-fastbuild/bin/vendor/github.com/golang/glog/~lib~/' --)

Use --sandbox_debug to see verbose messages from the sandbox
panic: runtime error: index out of range          

goroutine 1 [running]:                            
panic(0x8b82c0, 0xc4200100d0)                     
        /usr/local/go/src/runtime/panic.go:500 +0x1a1
cmd/compile/internal/gc.(*importer).pkg(0xc420474000, 0x38)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:216 +0xb1
cmd/compile/internal/gc.(*importer).fieldName(0xc420474000, 0x0)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:603 +0x397
cmd/compile/internal/gc.(*importer).field(0xc420474000, 0x5)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:545 +0x3d
cmd/compile/internal/gc.(*importer).fieldList(0xc420474000, 0xffffffffffffff16, 0xc420999ea0, 0x90)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:536 +0x8f
cmd/compile/internal/gc.(*importer).typ(0xc420474000, 0xc420999d60)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:483 +0xc77
cmd/compile/internal/gc.(*importer).typ(0xc420474000, 0xffffffffffffff12)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:407 +0x148
cmd/compile/internal/gc.(*importer).typ(0xc420474000, 0xffffffffffffff14)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:487 +0xfde
cmd/compile/internal/gc.(*importer).typ(0xc420474000, 0xc4208e09a0)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:474 +0xbf4
cmd/compile/internal/gc.(*importer).field(0xc420474000, 0xc420a56870)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:546 +0x55
cmd/compile/internal/gc.(*importer).fieldList(0xc420474000, 0xffffffffffffff16, 0xc420999810, 0x90)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:536 +0x8f
cmd/compile/internal/gc.(*importer).typ(0xc420474000, 0xc4209995e0)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:483 +0xc77
cmd/compile/internal/gc.(*importer).typ(0xc420474000, 0x0)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:407 +0x148
cmd/compile/internal/gc.(*importer).obj(0xc420474000, 0xfffffffffffffffd)
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:282 +0x967
cmd/compile/internal/gc.Import(0xc420060180)      
        /usr/local/go/src/cmd/compile/internal/gc/bimport.go:97 +0x3b2
cmd/compile/internal/gc.importfile(0xc420410800, 0x0, 0x0, 0x0)
        /usr/local/go/src/cmd/compile/internal/gc/main.go:843 +0x9b1
cmd/compile/internal/gc.(*parser).importdcl(0xc4204107e0)
        /usr/local/go/src/cmd/compile/internal/gc/parser.go:341 +0x178
cmd/compile/internal/gc.(*parser).import_(0xc4204107e0)
        /usr/local/go/src/cmd/compile/internal/gc/parser.go:300 +0x71
cmd/compile/internal/gc.(*parser).file(0xc4204107e0)
        /usr/local/go/src/cmd/compile/internal/gc/parser.go:268 +0x5e
cmd/compile/internal/gc.parse_file(0xc420060240)  
        /usr/local/go/src/cmd/compile/internal/gc/parser.go:31 +0x54
cmd/compile/internal/gc.Main()                    
        /usr/local/go/src/cmd/compile/internal/gc/main.go:334 +0x14f4
cmd/compile/internal/amd64.Main()                 
        /usr/local/go/src/cmd/compile/internal/amd64/galign.go:93 +0x2fa
main.main()                                       
        /usr/local/go/src/cmd/compile/main.go:33 +0x2a3

[ianthehat]

That looks like the kind of error you get when you try to read a .a file built by the wrong version of the compiler, but I have no clue how you would have managed to do that.

[jayconrod]

Reproduced at rules_go 473417e and at 0.5.4. There are different errors at rules_go master related to go_prefix that I'll look into today as well.

[twpayne]

543 is possibly related.

[jayconrod]

This was a fun rabbit-hole to explore. This is actually a bug in the Go 1.7.6 toolchain. I think we can work around this by producing .a files directly when we invoke the compiler.

The problematic target is //vendor:github.com/aws/aws-sdk-go/service/ecr. If you build this with Bazel, you'll get an .a file with invalid export data. If you copy it somewhere and run gcexportdata (built from main.go in golang.org/x/tools/go/gcexportdata), it will report a similar error.

I compared this with what go build does by adding the Kubernetes vendor folder to my GOPATH, then running with -x to print commands. When go build invokes the compiler, it does so with the -pack flag, which produces an .a file directly. The .a contains two files: __.PKGDEF (type information, used by the compiler on dependent libraries), and _go_.o (compiled code; not ELF format; used by the linker).

In the Bazel Go rules, we don't use the -pack flag when invoking the compiler. We produce the .o file directly, then run go tool pack, which also produces an .a file with __.PKGDEF and an .o file. In general, you need to run go tool pack if there are multiple .o files in a package (because of assembly files or cgo). We always run it even if we have one .o file to reduce complexity in our code.

You might be wondering how go tool pack materializes the __.PKGDEF file if it's not produced by the compiler. When the compiler is run without -pack, it includes the content of that file in the generated .o file. See dumpobj1 in cmd/compile/internal/gc/obj.go. The type information is separated from the compiled code by a single ! by itself on a line. When go tool pack creates an .a file out of an .o file, it looks for that line and splits the data into two files in the generated archive.

It appears that for this library, __.PKGDEF actually contains a ! on a single line. This is part of the content, not intended to be a separator. go tool pack treats this as the separator and splits the data early (after about 5K instead of 42K). That causes the error. I'm not familiar with the binary format, so I don't know under what circumstances the ! is allowed to be there.

I'm not sure whether this is a problem for newer Go releases. It could have been fixed, or it could be working by coincidence. I'll report it upstream so we can get a better understanding.

In any case, rules_go should go back to creating an .a file directly with the compiler. This will avoid this bug on older SDKs, and it will be slightly faster in the common case where there's only one .o file per package. We'll need to do a copy-and-append action if there are more .o files.