Go: cmd/compile, cmd/link: can't build large arm binaries with external linking, Kubernetes now too big

[luxas]

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go1.6.3

What operating system and processor architecture are you using (go env)?

linux/arm

What did you do?

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.
Just run
docker run -it gcr.io/google_containers/kube-apiserver-arm:v1.4.0-alpha.3 /usr/local/bin/kube-apiserver

What did you expect to see?

kube-apiserver starting

What did you see instead?

unexpected fault address 0x40d62ec
fatal error: fault
[signal 0xb code=0x2 addr=0x40d62ec pc=0x40d62ec]

goroutine 1 [running, locked to thread]:
runtime.throw(0x2cc9010, 0x5)
    /usr/local/go/src/runtime/panic.go:547 +0x78 fp=0x1482beec sp=0x1482bee0
runtime.sigpanic()
    /usr/local/go/src/runtime/sigpanic_unix.go:27 +0x280 fp=0x1482bf18 sp=0x1482beec
k8s.io/kubernetes/vendor/github.com/docker/engine-api/types/versions.init()
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/engine-api/types/versions/compare.go:62 +0x4c fp=0x1482bf20 sp=0x1482bf1c
k8s.io/kubernetes/vendor/github.com/docker/engine-api/types/filters.init()
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/engine-api/types/filters/parse.go:295 +0x5c fp=0x1482bf34 sp=0x1482bf20
k8s.io/kubernetes/vendor/github.com/docker/engine-api/types.init()
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/docker/engine-api/types/types.go:473 +0x5c fp=0x1482bf38 sp=0x1482bf34
k8s.io/kubernetes/pkg/credentialprovider.init()
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/credentialprovider/provider.go:123 +0x7c fp=0x1482bf6c sp=0x1482bf38
k8s.io/kubernetes/pkg/credentialprovider/aws.init()
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/credentialprovider/aws/aws_credentials.go:232 +0x70 fp=0x1482bf70 sp=0x1482bf6c
k8s.io/kubernetes/pkg/cloudprovider/providers/aws.init()
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/aws/sets_ippermissions.go:146 +0xb0 fp=0x1482bf90 sp=0x1482bf70
k8s.io/kubernetes/pkg/cloudprovider/providers.init()
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/cloudprovider/providers/providers.go:30 +0x4c fp=0x1482bf94 sp=0x1482bf90
k8s.io/kubernetes/cmd/kube-apiserver/app.init()
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:310 +0x4c fp=0x1482bf98 sp=0x1482bf94
main.init()
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/apiserver.go:53 +0x5c fp=0x1482bf9c sp=0x1482bf98
runtime.main()
    /usr/local/go/src/runtime/proc.go:177 +0x274 fp=0x1482bfc4 sp=0x1482bf9c
runtime.goexit()
    /usr/local/go/src/runtime/asm_arm.s:990 +0x4 fp=0x1482bfc4 sp=0x1482bfc4

goroutine 5 [chan receive]:
k8s.io/kubernetes/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x462d1f8)
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x60
created by k8s.io/kubernetes/vendor/github.com/golang/glog.init.1
    /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x2cc

Seems like Kubernetes with it's deps just grew too big for arm :(
Any help here would be really appreciated. We're releasing Kubernetes in about 10 days

I need help from some Go guru here that knows how the internals work!
I assume this is a Go issue rather than a Kubernetes issue, since the file that's segfaulting doesn't have an init()

Please take a look as quickly as possible!
-> @lavalamp @smarterclayton @ixdy @rsc @davecheney @wojtek-t @jfrazelle @bradfitz

[lavalamp]

The file in question is: https://github.com/kubernetes/kubernetes/blob/master/vendor/github.com/docker/engine-api/types/versions/compare.go

@luxas have you done a git bisect to try and figure out what introduced this? I am guessing that the go team would probably appreciate a more specific problem to look at.

[bradfitz]

There's always at least an implicit init function in packages to run all the assignments and function calls at the package (global) scope, and to call the init functions of dependent packages.

Does the problem reproduce with Go 1.7.1?

Seems like Kubernetes with it's deps just grew too big for arm :(

What makes you say that?

[luxas]

@lavalamp
hyperkube was working at v1.4.0-alpha.1, broke at v1.4.0-alpha.2
kube-apiserver was working at v1.4.0-alpha.2, broke at v1.4.0-alpha.3

breakage at current HEAD with go 1.7.1:

unexpected fault address 0x408cce4
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x2 addr=0x408cce4 pc=0x408cce4]

goroutine 1 [running, locked to thread]:
runtime.throw(0x275f8c2, 0x5)
        /usr/local/go/src/runtime/panic.go:566 +0x78 fp=0x13f1b6cc sp=0x13f1b6c0
runtime.sigpanic()
        /usr/local/go/src/runtime/sigpanic_unix.go:27 +0x280 fp=0x13f1b6f8 sp=0x13f1b6cc
k8s.io/kubernetes/vendor/golang.org/x/net/internal/timeseries.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/internal/timeseries/timeseries.go:526 +0x4c fp=0x13f1b700 sp=0x13f1b6fc
k8s.io/kubernetes/vendor/golang.org/x/net/trace.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/golang.org/x/net/trace/trace.go:1064 +0x80 fp=0x13f1b748 sp=0x13f1b700
k8s.io/kubernetes/vendor/google.golang.org/grpc.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/google.golang.org/grpc/trace.go:120 +0x6c fp=0x13f1b75c sp=0x13f1b748
k8s.io/kubernetes/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/etcdserver/etcdserverpb/rpc.pb.gw.go:1867 +0x70 fp=0x13f1bf18 sp=0x13f1b75c
k8s.io/kubernetes/vendor/github.com/coreos/etcd/clientv3.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/coreos/etcd/clientv3/watch.go:715 +0x58 fp=0x13f1bf3c sp=0x13f1bf18
k8s.io/kubernetes/pkg/storage/etcd3.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/storage/etcd3/watcher.go:352 +0x58 fp=0x13f1bf40 sp=0x13f1bf3c
k8s.io/kubernetes/pkg/storage/storagebackend/factory.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/storage/storagebackend/factory/factory.go:44 +0x70 fp=0x13f1bf44 sp=0x13f1bf40
k8s.io/kubernetes/pkg/registry/generic.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/registry/generic/storage_decorator.go:61 +0x6c fp=0x13f1bf48 sp=0x13f1bf44
k8s.io/kubernetes/pkg/registry/generic/registry.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/pkg/registry/generic/registry/store.go:937 +0x60 fp=0x13f1bf68 sp=0x13f1bf48
k8s.io/kubernetes/cmd/kube-apiserver/app/options.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/options/options.go:131 +0x5c fp=0x13f1bf6c sp=0x13f1bf68
k8s.io/kubernetes/cmd/kube-apiserver/app.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/app/server.go:311 +0xb8 fp=0x13f1bf70 sp=0x13f1bf6c
main.init()
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/cmd/kube-apiserver/apiserver.go:55 +0x5c fp=0x13f1bf74 sp=0x13f1bf70
runtime.main()
        /usr/local/go/src/runtime/proc.go:172 +0x224 fp=0x13f1bf9c sp=0x13f1bf74
runtime.goexit()
        /usr/local/go/src/runtime/asm_arm.s:998 +0x4 fp=0x13f1bf9c sp=0x13f1bf9c

goroutine 5 [chan receive]:
k8s.io/kubernetes/vendor/github.com/golang/glog.(*loggingT).flushDaemon(0x3a05e90)
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:879 +0x60
created by k8s.io/kubernetes/vendor/github.com/golang/glog.init.1
        /go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/golang/glog/glog.go:410 +0x1ec

What makes you say that?

It guess it's hitting some kind of limit in the golang arm implementation right now
The panic error seems to change with every commit.

I don't know much about the lower-level Golang ARM implementation, can you think about something that would panic there?

[jessfraz]

can you bisect for the exact sha that started the breakage, not the tag

[luxas]

Trying to find the commit kube-apiserver broke on

[laboger]

Your faulting address is the pc (program counter) and it is greater than 2**26. I'm not familiar with arm, but could it be that the 32 bit arm platform has a limit on the size of the programs it can run and these programs are exceeding it? These are the same programs we have issues with on ppc64le due to their size.

[luxas]

@laboger Thanks a lot!
Yeah, there's the issue.
0x408cce4 == 67685604 which is 576740 bigger than 67108864 (2**26)

also broke with pc=0x4018784 == 67209092 which is 100228 bigger than 2**26

at commit kubernetes/kubernetes@a81732ba76944ae7015c2d55d61b2f0b05900109 it didn't break, but with kubernetes/kubernetes@ff9980e2035210f14a4a9a3f6c756f28c23c883a it does break.

The difference there is only 300 commits, but the binary clearly passes the 0x4000000 (2**26) mark just there.

Well, now we know what the problem is.
Do anyone have suggestions on what to do? Can we change something in golang to solve it or is it just a hard limit for 32-bit overall?

[luxas]

Other interesting observations:

• running the arm binary on an arm64 host (pine64) segfaults
• running the arm64 version of kube-apiserver v1.4.0-alpha.3 works

So it might be a limitation in golang...

[davecheney]

running a 32 bit binary on a 64 bit system will run it in 32 bit emulation
mode with the same limitations.

On Fri, Sep 9, 2016 at 7:53 AM, Lucas Käldström [email protected]
wrote:

Other interesting observations:

• running the arm binary on an _arm64_ host (pine64) segfaults
• running the arm64 version of kube-apiserver v1.4.0-alpha.3 works

So it might be a limitation in golang...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/17028#issuecomment-245753611, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA7z4Oc_DEn2ZVkXpW9q3ZfOEIe40ks5qoIPKgaJpZM4J4WnZ
.

[bradfitz]

/cc @minux @cherrymui

[minux]

A hard problem to fix because we do instruction encoding in
the compiler front end and at that time there is no way for
the compiler to know if the target is too far away.

Splitting text sections might help, but a real solution is perhaps
to make the front end delay choosing the encoding for a jump
until link time.

Could we modify cmd/internal/obj so that it does instruction
encoding on a basic block basis and let the linker choose
the encoding for jump instructions?

A short term solution is make our liker generate trampolines
if the target is too far away.

[bradfitz]

/cc @randall77 @crawshaw @ianlancetaylor

[luxas]

Thanks everyone for helping with this!

I know @laboger has a patch for ppc64le that can be cherrypicked upon go1.7
Is it possible to create a such patch for this problem that we can temporarily use for the v1.4 release (go1.7 + 2 patches), only when building arm and ppc64le.
I think I can make the Kubernetes build infra work, but I can't make the arm patch.
Can someone try to make a fast prototype I can test?

[crawshaw]

The CL @laboger sent may cover this with minimal changes. I haven't investigated how the external linker is rewriting the jump, it may not work on ARM, but it's worth a try. @luxas, can you patch in CL 27790, modify cmd/link/internal/ld/data.go:1912 to look for either PPC64 or ARM, and see if it works?

[laboger]

Has any investigation been done to determine if the binaries built by Kubernetes could be reduced in size? I suggested this when the problem was first reported. That would not only resolve your issue with ppc64le and arm but improve compile and link times, save space for your binaries, etc. etc. I looked briefly at the packages included in hyperkube and it appeared to have many duplicate path names rooted at different locations. Could there be new versions of some packages being added without removing the old ones that are no longer used? Or could the binaries get split in any way?
I can send you more detail on the package path names I am referring to if you want @luxas.

[cherrymui]

While we are solving the linker problem, in the meantime, is using Go tip an option for you? The SSA compiler generates smaller code that does not fault.

$ size -A kube-apiserver-*
kube-apiserver-nossa  :
section                  size       addr
.text                34873308      69632
.rodata               8542712   34942976
.typelink              130564   43485688
.itablink               11288   43616252
.gosymtab                   0   43627540
.gopclntab           15408474   43627544
.noptrdata            1475636   59039744
.data                   34424   60515384
.bss                    85392   60549808
.noptrbss               22672   60635200
.debug_abbrev             255   60661760
.debug_line           2956588   60662015
.debug_frame          2925588   63618603
.debug_pubnames        966925   66544191
.debug_pubtypes       4387464   67511116
.debug_aranges             32   71898580
.debug_gdb_scripts         48   71898612
.debug_info          20743066   71898660
.note.go.buildid           56      69576
Total                92564492


kube-apiserver-ssa  :
section                  size       addr
.text                28344912      69632
.rodata               8210247   28418048
.typelink              129448   36628296
.itablink               11288   36757744
.gosymtab                   0   36769032
.gopclntab           16949001   36769032
.noptrdata            1475636   53719040
.data                   34424   55194680
.bss                    85392   55229104
.noptrbss               22664   55314496
.debug_abbrev             255   55341056
.debug_line           6825244   55341311
.debug_frame          2914972   62166555
.debug_pubnames        966478   65081527
.debug_pubtypes       4012779   66048005
.debug_aranges             32   70060784
.debug_gdb_scripts         48   70060816
.debug_info          18903367   70060864
.note.go.buildid           56      69576
Total                88886243

[cherrymui]

What about we introduce a -largemode flag or like, when set, always emits "load address into register; make indirect call"? This seems a simpler fix, although it makes large program even larger... There might be some optimizations that reduce the overhead (when we know for sure it is safe, make direct call). And the linker should signal and hint to use -largemode when it sees a jump-too-far.

[laboger]

I think that would be a good solution for all platforms that could have this problem, and would fix internal and external linking. My CL only works for external linking, because it depends on the GNU linker to handle the long calls.

I believe it needs to be an option because it will degrade call performance, so we wouldn't want to do it all the time but there should be opportunities for optimizing. @minux made similar suggestions in the corresponding issue for ppc64le #15823.

[laboger]

One other note, on arm isn't this a runtime problem when executing the program that has been built, not a build problem? So while I think the problem is due to the large program size the symptoms on arm 32 bit are different than ppc64le.

[cherrymui]

I think it is a build problem: it emits a pc-relative jump and the offset overflows, and the linker did not signal, so the program jumped to a faulty address.

[crawshaw]

I don't think a new compiler mode is the answer. That won't fit well with the go tool.

If there's no space in the text for the linker to rewrite the pc-relative jump, than as long as no single function is too large, the linker can insert a trampoline.

[laboger]

That means your suggestion is for internal linking only? Because I assume you mean the internal linker would insert the trampoline.

[luxas]

@laboger Can you provide a version of your CL as cherrypickable upon go1.6.3 or go1.7.1 so I can test if it fixes this issue?

[crawshaw]

@laboger The linker has plenty of opportunities to insert trampolines for both external and internal linking (though it may take some wiring).

[laboger]

@luxas If you get the patch from CL 23963, that should apply against go 1.7. That was my previous patch before I had to rebase it for master and ended up with a new CL.

Please note that this patch depends on using external linking so that the external linker makes the necessary changes to allow for long jumps or calls.

[luxas]

Does the arm compiler link internally or externally by default?

[laboger]

@crawshaw But when you say "the linker" you mean the golang linker would generate the trampolines before writing out the elf text sections that are passed on to the external linker?

[crawshaw]

Yes. (Of course it may be that the external linker is already smart enough to do that itself, I haven't checked.)

[luxas]

@cherrymui Is ssa available for arm at HEAD?

[bradfitz]

@luxas, yes.

[laboger]

@luxas In your Kubernetes build, programs that are nonstatic are linked externally, programs that are static are linked internally. You might need to force the programs that are too large to be built nonstatic.

[cherrymui]

Yes, SSA is available at Go HEAD
Go on linux/arm by default does internal linking for non-cgo programs, external linking if uses cgo.

[cherrymui]

Send CL https://go-review.googlesource.com/c/28857/ for "large mode", for discussion and play. With it kube-apiserver at least passes init's. (I don't know how I should use that program)

$ ./kube-apiserver 
F0909 15:22:25.921914   12409 etcd_validation.go:26] --etcd-servers must be specified

[gopherbot]

CL https://golang.org/cl/28857 mentions this issue.

[luxas]

@cherrymui So you mean that with CL 28857 applied to Go HEAD, this issue is solved?
That sounds as a great start!
Can you test to compile hyperkube with that patch as well and see if it also works?

If possible, I'd prefer if I could cherry-pick CL 28857 and 27790 on go1.7.1 for getting a temporary, somewhat stable version in order to release ppc64le and arm for v1.4 (which is only ~a week away!)

I think we definitely should fix this properly, but I'm also willing to go with a "fast path" that works for now... it's commercially unsupported architectures for Kubernetes anyway (see https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/multi-platform.md for more info)

cc @david-mcmahon @ixdy prepare yourselves that we're gonna build the arm and maybe ppc64le v1.4 binary release with a custom (patched) golang. I'm gonna help creating the temporary release infra code that's required for this edge case.

[cherrymui]

@luxas Yes, at least it doesn't panic in init's. Hyperkube (I assume it is k8s.io/kubernetes/cmd/hyperkube) also doesn't panic,

$ ./hyperkube 
Error: No server specified

This is an all-in-one binary that can run any of the various Kubernetes servers.

Usage

  hyperkube <server> [flags]

Servers

  kubectl
    Kubernetes command line client

  apiserver
    The main API entrypoint and interface to the storage system. The API server
    is also the focal point for all authorization decisions.

  controller-manager
    A server that runs a set of active components. This includes replication
    controllers, service endpoints and nodes.

  scheduler
    Implements a Kubernetes scheduler. This will assign pods to kubelets based
    on capacity and constraints.

  kubelet
    The kubelet binary is responsible for maintaining a set of containers on a

      particular node. It syncs data from a variety of sources including a
      Kubernetes API server, an etcd cluster, HTTP endpoint or local file. It then
      queries Docker to see what is currently running.  It synchronizes the
      configuration data, with the running set of containers by starting or stopping
      Docker containers.
  proxy
    The Kubernetes proxy server is responsible for taking traffic directed at

      services and forwarding it to the appropriate pods. It generally runs on
      nodes next to the Kubelet and proxies traffic from local pods to remote pods.
      It is also used when handling incoming external traffic.
  federation-apiserver
    The API entrypoint for the federation control plane

  federation-controller-manager
    Controller manager for federation control plane. Manages federation service
    endpoints and controllers

Call 'hyperkube --make-symlinks' to create symlinks for each server in the local directory.
Call 'hyperkube <server> --help' for help on a specific server.

(same output as on AMD64)

I believe the CL should apply to Go 1.7. There is currently a trybot failure, but it seems to happen only with SSA, so it might be an SSA bug :(

[laboger]

Are you planning to include a patch to fix ppc64le as well? If so which one?

[luxas]

Yes, I'm planning to cherrypick CL 28857 and 27790 onto go1.7.1 for releasing both arm and ppc64le in v1.4.

@laboger Can you provide a somewhat stable ppc64le patch for me to use upon go1.7.1?

[laboger]

@luxas, were you able to try the latest patch from CL 23963 or should I create a new CL for you?

[luxas]

Hmm, isn't your CL 27790 the latest one?
I've already ported @cherrymui's CL 28857 to https://github.com/luxas/go/tree/k8s_patch on top of go1.7.1

Can you fork golang/go on github and try to make a new commit from your latest CL and upload it to a branch of laboger/go so I can cherrypick that into luxas/go:k8s_patch so I can use that for building k8s?

Right now CL 28857 can't be merged on top of go1.7.1...

[laboger]

OK, 27790 doesn't apply to go 1.7 which is why I suggested using 23963, but you are correct that is not quite the latest. So I will backport 27790 to go 1.7 and make it available on github after testing it out.

[cherrymui]

@luxas in case you need it, CL 28857 is now rebased onto Go tip. Also added some fix so that both normal and -largemodel (except -shared) builds are happy.

[luxas]

@laboger Thanks, ping me when you have it.

@cherrymui Great, I'll check out the latest

[luxas]

See the linked PR for Kubernetes here: https://github.com/kubernetes/kubernetes/issues/32517

[laboger]

@luxas Here is the fix for go1.7.1.
https://github.com/powertechpreview/go.git
git checkout ibm1.7

Original go1.7 fix from CL 23963:
commit d6beea7f9ea1aa2ae5abca7fccb252767820aa13
Author: Lynn Boger [email protected]
Date: Tue Jul 26 08:51:10 2016 -0500

Backport of latest updates from CL27790:
commit 9a526305b4d92748b397aca3f7bedd7e5b55f21d
Author: Lynn Boger [email protected]
Date: Mon Sep 12 13:53:39 2016 -0400

One other note, you will need to use external linking with the large programs to avoid the errors. The easiest way would be to add -linkmode=external to the goldflags used by hack/lib/golang.sh, although you could also change the script to only use that option for those that are too big.

[gopherbot]

CL https://golang.org/cl/29397 mentions this issue.

[luxas]

@cherrymui Many thanks for the CLs you've made.
Thanks to your first patch, kube-apiserver will now work on arm in k8s v1.4.1!!

I assume CL 29397 is the "real" one, which can be merged at some point, right?

We should definitely have a proper fix for this as soon as possible.
So, what's the status of this right now?
Can you describe the issue and what you're proposing to do?

[cherrymui]

@luxas Yes, I think CL 29397 will be the "real" one.
I just need to refactor the code as pointed in the review.

Can you describe the issue and what you're proposing to do?

What issue?

[luxas]

What issue?

I meant this linker issue. Can you write a quick recap of what you're doing in CL 29397?

[cherrymui]

In short, the issue is that when resolving a direct jump (or call), if the target is too far so that the offset cannot be encoded into the instruction, the linker simply truncated the address, and the generated binary jumped wild. The fix is to let the linker detect too-far jumps and automatically insert trampolines when needed.

[laboger]

Isn't this fix for internal linking only?

[cherrymui]

I believe it is not too hard to make it also work for external linking. But maybe not in that CL.

[luxas]

Thanks, what's the ETA on CL 29397?
I'd love to have external linking support as well, otherwise kubelet will break next.

But one step at a time.

[cherrymui]

Thanks, what's the ETA on CL 29397?

Hopefully by this week.

[luxas]

@cherrymui Thanks for the patch!

Still I think we need to get external linking working, so can you reopen this issue @bradfitz @cherrymui @crawshaw?

Do you have any plan to get external linking working or am I missing something?

[crawshaw]

I would have thought a similar approach to ppc64 should work, or including the trampolines in the section given to the external linker. I don't know if anyone is working on it.

[luxas]

@laboger is working on ppc64le support for this as well, see #16665

[cherrymui]

I will try to bring up external linking support on ARM soon.

[laboger]

Clarification on the ppc64le fixes: the fix for this problem when using external linking is already upstream. That is the fix to split text sections if they get too large #15823, because then the GNU linker can create and insert trampolines or long branches as needed. The fix for #16665 solves the problem on ppc64le when using internal linking.

[luxas]

@laboger But arm still needs this external linking fix?

[laboger]

Yes it sounds like @cherrymui is working on it.

[gopherbot]

CL https://golang.org/cl/31143 mentions this issue.