Go: cmd/go: drop runtime from plugins

[rasky]

It looks like plugins contain a copy of the runtime. I guess that copy is unused, and could probably be dropped, reducing binary size.

[rasky]

/cc @crawshaw

[crawshaw]

Yup, I had a TODO about this I lost somewhere in the CL patchsets.

It should be straightforward, it's already possible to build -linkshared binaries without a runtime. I expect most of the work is careful argument construction in cmd/go.

[rsc]

@crawshaw, still want to do this for Go 1.8?

[crawshaw]

Pushing to 1.9, so I can spend what cycles I have on bugs.

[gopherbot]

Change https://golang.org/cl/62352 mentions this issue: cmd/link: correct symbol table order for plugin-use code on darwin/amd64

[crawshaw]

@hirochachacha at one point you had a CL exploring removing the runtime from plugins. Are you still working on it?

[hirochachacha]

@crawshaw No, I'm afraid not.

[montao]

I think I have this issue too in my code https://gitlab.com/montao/forbasile

The background is that we want to generate a large number of plugins, and statically linked objects would take too much space. If the runtime is removed from the plugin then I think that the problem would be solved and we could use Go. Right now we are looking at other choices of implementation (C, Rust, custom...) instead.

[bstarynk]

For what it is worth, my manydl.c is generating "randomly" the code of many plugins in C and compiling it and loading these many plugins. That program demonstrates that on Linux, one can have hundreds of thousands of dlopen-ed plugins. I see no reason why that would not be possible in Go (and I guess that writing the equivalent manydl.go program is reasonably easy)
And generating plugins at runtime is sometimes interesting. My old GCC MELT project did that (generating lots of plugins in C++). My current bismon project is doing that (generating lots of plugins in C), and it would be nice to be able to start a project generating lots of plugins in Go (which has, as a language for generated code, some advantages over C and C++)

So I believe it would be great if Go would practically support many plugins (in practice generated ones). The current implementation does that poorly (it is designed for a few, hand-written, plugins in mind).
In its current implementation, each generated *.so would have its own copy of common standard functions (e.g. fmt.Printf), so if we generate thousands of plugins, the binary code of fmt.Printf gets duplicated a thousand times in the virtual address space. That does not scale.

[montao]

@hirochachacha at one point you had a CL exploring removing the runtime from plugins. Are you still working on it?

I found this CL, I think: https://go-review.googlesource.com/c/go/+/61090 Not sure what is the problem with it but it was abandoned. Any clues?

[hirochachacha]

I don't remember exactly, but I couldn't recommend investing that approach since that is a half-baked solution after all. It only cover runtime package, it doesn't cover other standard libraries. For example, if there are two plugin A and B that use strings package, both plugins still need to bundle strings package. Ideally, this issue should be solved by re-implementing plugin system on top of buildmode=shared. I guess that it requires hard work though.