Go: x/build/cmd/release: linux-armv6l release tests aren't passing

[dmitshur]

As part of the plan in https://github.com/golang/go/issues/40561#issuecomment-731482962, the linux-armv6l binary release has changed from using the "linux-arm" builder (on Scaleway) to the "linux-arm-aws" builder in CL 276034. Previously, tests were skipped for that target. Since the "linux-arm-aws" builder had more resource allocated, we tried to remove the test skip:

 {
+   GoQuery: ">= go1.16beta1",
    OS:      "linux",
    Arch:    "arm",
-   Builder: "linux-arm",
-   Goarm:   6, // for compatibility with all Raspberry Pi models.
-   // The tests take too long for the release packaging.
-   // Much of the time the whole buildlet times out.
-   SkipTests: true,
+   Builder: "linux-arm-aws",
+   Goarm:   6, // For compatibility with all Raspberry Pi models.
 },

The tests are not passing when using a recent tip commit with CL 276454 applied on top.

This can be reproduced if your account has permissions needed to run releasebot (documented here) with:

# (The release command uses builders to create a release artifact locally.
#  It does not publish anything, so it's safe to run for testing needs.)
$ release -target=linux-armv6l -version=go1.16beta123 -watch \
          -rev=456cd66e5d73d6971ab40ce6c57b1c9cf9444e1f  # or -rev=master for latest commit

[...]
ok      cmd/internal/src    0.012s
ok      cmd/internal/sys    0.029s
ok      cmd/internal/test2json  0.103s
--- FAIL: TestPIESize (0.00s)
    --- FAIL: TestPIESize/TestPieSize-external (10.99s)
        elf_test.go:289: [/workdir/go/bin/go build -o /workdir/tmp/go-link-TestPieSize-external736090553/pieexternal -buildmode=pie -ldflags=-linkmode=external pie.go]
        elf_test.go:289: [/workdir/go/bin/go build -o /workdir/tmp/go-link-TestPieSize-external736090553/exeexternal -buildmode=exe -ldflags=-linkmode=external pie.go]
        elf_test.go:411: real size difference 0xfffffffffffffd87, expected 0xfffffffffffffb58
        elf_test.go:414: PIE unexpectedly large: got difference of 18446744073709550983 (2693752 - 2692792), expected difference 18446744073709550424
FAIL
FAIL    cmd/link    18.954s
ok      cmd/link/internal/benchmark 0.043s
--- FAIL: TestRuntimeTypeAttrExternal (1.54s)
    dwarf_test.go:1030: DWARF type offset was 0x3b50+0x744c8, but test program said 0xa47018
FAIL
FAIL    cmd/link/internal/ld    6.289s
ok      cmd/link/internal/loader    0.041s
--- FAIL: TestExternalLinkerCgoExec (3.04s)
    nm_test.go:191: want 0x638140 address for main.main symbol, but have 0xb4140
    nm_test.go:191: want 0x6efa8c address for main.testdata symbol, but have 0x16ba8c
    nm_test.go:191: want 0x638164 address for main.testfunc symbol, but have 0xb4164
FAIL
FAIL    cmd/nm  4.917s
ok      cmd/objdump 6.293s
ok      cmd/pack    2.633s
ok      cmd/trace   0.248s
ok      cmd/vet 11.428s
FAIL
go tool dist: Failed: exit status 1

Full Log

Building Go cmd/dist using /usr/local/go-bootstrap. (go1.15.1 linux/arm)
Building Go toolchain1 using /usr/local/go-bootstrap.
Building Go bootstrap cmd/go (go_bootstrap) using Go toolchain1.
Building Go toolchain2 using go_bootstrap and Go toolchain1.
Building Go toolchain3 using go_bootstrap and Go toolchain2.
Building packages and commands for linux/arm.

##### Testing packages.
ok      archive/tar 0.055s
ok      archive/zip 0.462s
ok      bufio   0.164s
ok      bytes   5.752s
ok      compress/bzip2  0.091s
ok      compress/flate  1.125s
ok      compress/gzip   0.119s
ok      compress/lzw    0.069s
ok      compress/zlib   0.739s
ok      container/heap  0.025s
ok      container/list  0.012s
ok      container/ring  0.053s
ok      context 0.041s
ok      crypto  0.003s
ok      crypto/aes  0.025s
ok      crypto/cipher   0.021s
ok      crypto/des  0.020s
ok      crypto/dsa  0.021s
ok      crypto/ecdsa    0.250s
ok      crypto/ed25519  0.168s
ok      crypto/elliptic 0.062s
ok      crypto/hmac 0.037s
ok      crypto/internal/subtle  0.003s
ok      crypto/md5  0.036s
ok      crypto/rand 0.017s
ok      crypto/rc4  0.040s
ok      crypto/rsa  0.202s
ok      crypto/sha1 0.009s
ok      crypto/sha256   0.024s
ok      crypto/sha512   0.026s
ok      crypto/subtle   0.021s
ok      crypto/tls  2.004s
ok      crypto/x509 2.473s
ok      database/sql    0.659s
ok      database/sql/driver 0.022s
ok      debug/dwarf 0.049s
ok      debug/elf   0.059s
ok      debug/gosym 0.044s
ok      debug/macho 0.005s
ok      debug/pe    0.035s
ok      debug/plan9obj  0.018s
ok      embed/internal/embedtest    0.011s
ok      encoding/ascii85    0.019s
ok      encoding/asn1   0.023s
ok      encoding/base32 0.047s
ok      encoding/base64 0.028s
ok      encoding/binary 0.003s
ok      encoding/csv    0.031s
ok      encoding/gob    0.038s
ok      encoding/hex    0.004s
ok      encoding/json   0.193s
ok      encoding/pem    0.015s
ok      encoding/xml    0.023s
ok      errors  0.011s
ok      expvar  0.024s
ok      flag    0.115s
ok      fmt 0.073s
ok      go/ast  0.005s
ok      go/build    1.414s
ok      go/constant 0.037s
ok      go/doc  0.053s
ok      go/format   0.021s
ok      go/importer 0.156s
ok      go/internal/gccgoimporter   0.020s
ok      go/internal/gcimporter  3.207s
ok      go/internal/srcimporter 8.038s
ok      go/parser   0.031s
ok      go/printer  0.273s
ok      go/scanner  0.011s
ok      go/token    0.028s
ok      go/types    2.441s
ok      hash    0.021s
ok      hash/adler32    0.012s
ok      hash/crc32  0.032s
ok      hash/crc64  0.027s
ok      hash/fnv    0.004s
ok      hash/maphash    0.062s
ok      html    0.003s
ok      html/template   0.127s
ok      image   0.111s
ok      image/color 0.034s
ok      image/draw  0.070s
ok      image/gif   0.487s
ok      image/jpeg  0.168s
ok      image/png   0.062s
ok      index/suffixarray   1.468s
ok      internal/cpu    0.006s
ok      internal/fmtsort    0.020s
ok      internal/poll   0.150s
ok      internal/profile    0.007s
ok      internal/reflectlite    0.089s
ok      internal/singleflight   0.017s
ok      internal/trace  0.075s
ok      internal/unsafeheader   0.004s
ok      internal/xcoff  0.013s
ok      io  0.052s
ok      io/fs   0.020s
ok      io/ioutil   0.014s
ok      log 0.006s
ok      log/syslog  1.217s
ok      math    0.028s
ok      math/big    2.042s
ok      math/bits   0.014s
ok      math/cmplx  0.042s
ok      math/rand   0.411s
ok      mime    0.034s
ok      mime/multipart  0.208s
ok      mime/quotedprintable    0.033s
ok      net 14.877s
ok      net/http    11.030s
ok      net/http/cgi    0.647s
ok      net/http/cookiejar  0.029s
ok      net/http/fcgi   0.039s
ok      net/http/httptest   0.085s
ok      net/http/httptrace  0.025s
ok      net/http/httputil   0.065s
ok      net/http/internal   0.012s
ok      net/http/pprof  5.139s
ok      net/internal/socktest   0.003s
ok      net/mail    0.039s
ok      net/rpc 0.020s
ok      net/rpc/jsonrpc 0.006s
ok      net/smtp    0.029s
ok      net/textproto   0.010s
ok      net/url 0.013s
ok      os  1.039s
ok      os/exec 0.988s
ok      os/signal   2.205s
ok      os/user 0.004s
ok      path    0.005s
ok      path/filepath   0.011s
ok      plugin  0.020s
ok      reflect 0.344s
ok      regexp  0.164s
ok      regexp/syntax   0.579s
ok      runtime 55.410s
ok      runtime/debug   0.058s
ok      runtime/internal/atomic 0.109s
ok      runtime/internal/math   0.026s
ok      runtime/internal/sys    0.006s
ok      runtime/metrics 0.016s
ok      runtime/pprof   8.553s
ok      runtime/race    0.021s
ok      runtime/trace   0.831s
ok      sort    0.104s
ok      strconv 0.425s
ok      strings 5.003s
ok      sync    0.483s
ok      sync/atomic 0.186s
ok      syscall 0.183s
ok      testing 1.260s
ok      testing/fstest  0.010s
ok      testing/iotest  0.008s
ok      testing/quick   0.042s
ok      text/scanner    0.007s
ok      text/tabwriter  0.017s
ok      text/template   0.049s
ok      text/template/parse 0.014s
ok      time    2.447s
ok      unicode 0.011s
ok      unicode/utf16   0.021s
ok      unicode/utf8    0.021s
ok      cmd/addr2line   6.385s
ok      cmd/api 9.669s
ok      cmd/asm/internal/asm    1.364s
ok      cmd/asm/internal/lex    0.025s
ok      cmd/compile 11.035s
ok      cmd/compile/internal/gc 28.827s
ok      cmd/compile/internal/logopt 0.164s
ok      cmd/compile/internal/ssa    0.568s
ok      cmd/compile/internal/syntax 0.012s
ok      cmd/compile/internal/test   0.024s
ok      cmd/compile/internal/types  0.014s
ok      cmd/cover   2.363s
ok      cmd/doc 0.094s
ok      cmd/fix 3.569s
ok      cmd/go  7.115s
ok      cmd/go/internal/auth    0.034s
ok      cmd/go/internal/cache   0.079s
ok      cmd/go/internal/fsys    0.041s
ok      cmd/go/internal/generate    0.015s
ok      cmd/go/internal/get 0.089s
ok      cmd/go/internal/imports 0.035s
ok      cmd/go/internal/load    0.061s
ok      cmd/go/internal/lockedfile  0.175s
ok      cmd/go/internal/lockedfile/internal/filelock    0.045s
ok      cmd/go/internal/modconv 0.016s
ok      cmd/go/internal/modfetch    0.034s
ok      cmd/go/internal/modfetch/codehost   0.053s
ok      cmd/go/internal/modfetch/zip_sum_test   0.050s
ok      cmd/go/internal/modload 0.047s
ok      cmd/go/internal/mvs 0.028s
ok      cmd/go/internal/par 0.053s
ok      cmd/go/internal/renameio    0.030s
ok      cmd/go/internal/search  0.012s
ok      cmd/go/internal/str 0.028s
ok      cmd/go/internal/test    0.035s
ok      cmd/go/internal/txtar   0.021s
ok      cmd/go/internal/vcs 0.033s
ok      cmd/go/internal/web 0.054s
ok      cmd/go/internal/work    0.028s
ok      cmd/gofmt   0.055s
ok      cmd/internal/archive    0.818s
ok      cmd/internal/buildid    0.333s
ok      cmd/internal/dwarf  0.009s
ok      cmd/internal/edit   0.022s
ok      cmd/internal/goobj  0.027s
ok      cmd/internal/moddeps    1.699s
ok      cmd/internal/obj    0.026s
ok      cmd/internal/obj/arm64  0.130s
ok      cmd/internal/obj/ppc64  0.171s
ok      cmd/internal/obj/riscv  0.100s
ok      cmd/internal/obj/s390x  0.017s
ok      cmd/internal/obj/x86    4.505s
ok      cmd/internal/objabi 0.005s
ok      cmd/internal/pkgpath    0.077s
ok      cmd/internal/src    0.012s
ok      cmd/internal/sys    0.029s
ok      cmd/internal/test2json  0.103s
--- FAIL: TestPIESize (0.00s)
    --- FAIL: TestPIESize/TestPieSize-external (10.99s)
        elf_test.go:289: [/workdir/go/bin/go build -o /workdir/tmp/go-link-TestPieSize-external736090553/pieexternal -buildmode=pie -ldflags=-linkmode=external pie.go]
        elf_test.go:289: [/workdir/go/bin/go build -o /workdir/tmp/go-link-TestPieSize-external736090553/exeexternal -buildmode=exe -ldflags=-linkmode=external pie.go]
        elf_test.go:411: real size difference 0xfffffffffffffd87, expected 0xfffffffffffffb58
        elf_test.go:414: PIE unexpectedly large: got difference of 18446744073709550983 (2693752 - 2692792), expected difference 18446744073709550424
FAIL
FAIL    cmd/link    18.954s
ok      cmd/link/internal/benchmark 0.043s
--- FAIL: TestRuntimeTypeAttrExternal (1.54s)
    dwarf_test.go:1030: DWARF type offset was 0x3b50+0x744c8, but test program said 0xa47018
FAIL
FAIL    cmd/link/internal/ld    6.289s
ok      cmd/link/internal/loader    0.041s
--- FAIL: TestExternalLinkerCgoExec (3.04s)
    nm_test.go:191: want 0x638140 address for main.main symbol, but have 0xb4140
    nm_test.go:191: want 0x6efa8c address for main.testdata symbol, but have 0x16ba8c
    nm_test.go:191: want 0x638164 address for main.testfunc symbol, but have 0xb4164
FAIL
FAIL    cmd/nm  4.917s
ok      cmd/objdump 6.293s
ok      cmd/pack    2.633s
ok      cmd/trace   0.248s
ok      cmd/vet 11.428s
FAIL
go tool dist: Failed: exit status 1

The release command sets some environment variables that may not be set when post-submit tests run on the linux-arm-aws builder, perhaps that's relevant?

if b.Goarm > 0 {
    env = append(env, fmt.Sprintf("GOARM=%d", b.Goarm))
    env = append(env, fmt.Sprintf("CGO_CFLAGS=-march=armv%d", b.Goarm))
    env = append(env, fmt.Sprintf("CGO_LDFLAGS=-march=armv%d", b.Goarm))
}

CC @golang/release, @ianlancetaylor.

[cherrymui]

I can confirm that it is indeed those flags. On the builder, it passes without CGO_LDFLAGS,

root@13fe5f0e6a12:/workdir/go/src# ../bin/go test cmd/nm
ok      cmd/nm  1.117s

and fails with CGO_LDFLAGS,

root@13fe5f0e6a12:/workdir/go/src# CGO_LDFLAGS=-march=armv6 ../bin/go test cmd/nm
--- FAIL: TestExternalLinkerCgoExec (0.61s)
    nm_test.go:191: want 0x7df4cc address for main.main symbol, but have 0xa74cc
    nm_test.go:191: want 0x88ca5c address for main.testdata symbol, but have 0x154a5c
    nm_test.go:191: want 0x7df4f0 address for main.testfunc symbol, but have 0xa74f0
FAIL
FAIL    cmd/nm  1.008s
FAIL

I'll look into it.

[gopherbot]

Change https://golang.org/cl/278592 mentions this issue: cmd/link: pass arch-specific flags to external linker when testing supported flag