Go: plugin: Add support for closing plugins

[zacps]

Adding support for closing plugins would give a simple way to do hot code reloading in long running applications like web servers.

This would add another method to src/plugin/plugin.go and a new file src/plugin/dl_close.go that would handle closing the dlhandle and destroying the go symbol map.
It would also add the handle as an un-exported unsafe pointer to the go plugin struct.

The main issue is how to signal that a plugin has been closed without breaking API compatibility. Currently the plugin has a channel called loaded which is closed when the plugin is loaded. The simplest solution is to add another channel which is closed when the plugin is closed, however this doesn't seem very 'neat' to me.

[davecheney]

I don't see how this could be implemented safely. Other Go code could have references to functions and symbols loaded from the plugin-would unloading the plugin make those dangling references, or delay the unloading of a plugin until all the references have been reclaimed? Neither sound workable.

[zacps]

Would it be possible to make close fail until the reference count of all symbols in the plugin reached 1, so that the plugin itself was the only structure that held a reference?

Alternatively you could have a different function which checked whether references to the symbols existed and if there were none close the dlhandle, leaving the destruction of the plugin object and any other references up to the program. While this is far from ideal I think with adequate caution it could be a workable solution.

[davecheney]

Would it be possible to make close fail until the reference count of all symbols in the plugin reached 1, so that the plugin itself was the only structure that held a reference?

This is spinny and racy.

Alternatively you could have a different function which checked whether references to the symbols existed and if there were none close the dlhandle, leaving the destruction of the plugin object and any other references up to the program. While this is far from ideal I think with adequate caution it could be a workable solution.

This sounds like a finaliser, with the associated problems of finaliser non determinism.

[zacps]

Could you be a bit more specific about the second point?

[davecheney]

Your proposal is to add a way to free resources associated with a plugin. If this is delegated to a finaliser it becomes non deterministic when this freeing occurs as finalisers are not guaranteed to run promptly, or indeed, at all.

[zacps]

I'm not suggesting that it's run automatically by the GC, or in a defer (unfortunately).

I'm suggesting that if the needs to be unloaded that the developer will keep track of references to it's symbols then when they no longer need them destroy the references then explicitly call close.

In that situation they will either forget to call close, leaving the handle open, close successfully or attempt to close when there are still references which will fail. It could possibly fail with a panic rather than a plain error as attempting the close a resource that still has a reference is a pretty big bug.

I don't see what issues this approach has beyond reliance on the developer to not make mistakes. It does introduce a need to keep track of references which is far from perfect and not something I'd like to introduce to go. However I think that it's worth having this for those that want to use it and understand the additional complexity it introduces.

[ianlancetaylor]

What if the plugin started a goroutine?

Go as a language and a runtime environment tends to avoid providing facilities that are easy to get wrong when getting them wrong causes the program to crash or behave unpredictably. I don't see how to make this at all reliable, so to me it doesn't seem to be a good fit for Go.

[zacps]

goroutines would have to be handled in the same way, requiring the developer to close them.

I can understand not wanting to introduce functionality that's easy to misuse. I just think that it's better to have the ability to do it if you know what you're doing. I guess it does go against the philosophy of go though.

[davecheney]

How can you close a goroutine? The runtime doesn't give you any way to get
a handle or an identifier for a goroutine?

On Tue, 23 May 2017, 09:54 Zac Pullar-Strecker [e-mail]
wrote:

goroutines would have to be handled in the same way, requiring the
developer to close them.

I can understand not wanting to introduce functionality that's easy to
misuse I just think that it's better to have the ability to do it if you
know what you're doing. I guess it does go against the philosophy of go
though.

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/20461#issuecomment-303251214, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA7TSJ5h7YYZU1PZDXlrNZfjSaaifks5r8iBGgaJpZM4Ni-0b
.

[zacps]

I took a quick look at runtime but couldn't find where exactly the goroutine is created. Are goroutines not closed when they return?

[davecheney]

Yup, a goroutine dies/quits/goes away when the function that was passed to
the go statement returns. So that leaves the problem of ensuring every
goroutine that may have access to a symbol from a plugin must exit before
the plugin can be unloaded. This means either finalisers, which won't give
you a timely response, or manual bookkeeping, which as Ian said is
unreliable.

On Tue, May 23, 2017 at 12:16 PM, Zac Pullar-Strecker <
[email protected]> wrote:

I took a quick look at runtime but couldn't find where exactly the
goroutine is created. Are goroutines not closed when they return?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/20461#issuecomment-303271038, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA-7gIyQn1lrDQJPC5XHyF1Oe4u_Mks5r8kGDgaJpZM4Ni-0b
.

[zacps]

Ok, that's what I thought.

I personally think that manual bookkeeping wouldn't be too bad. If you're writing something designed to be loaded/unloaded you're probably going to make sure you keep track of references and goroutines as is.

I'm trying to think whether there's a better way to do this. I guess it's not something other languages have had problems with.

[zacps]

I have another idea for how this could be solved, it's not simple though and I'm not sure whether it would make sense.

Basically I think it would be possible to implement a compile-time check that ensured there was no possibility for a dangling reference when a program was closed.

The compiler could check to see if close was called on any plugin object in the program. If it was we'd find all of the lookup calls associated with that plugin and determine whether there were any paths that could leave a live reference when or after closed was called.

I have no idea how you'd implement this or whether it'd be fast enough to be viable. Just an idea.

[DemiMarie]

What about having the GC close any plugins that are no longer referenced?

That's what the JVM does. And forcing a full GC whenever Go code must be unloaded is not hard.

[davecheney]

That's a finaliser.

On Sun, 4 Jun 2017, 11:45 Demi Marie Obenour [e-mail]
wrote:

What about having the GC close any plugins that are no longer referenced?

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/20461#issuecomment-306012057, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcAyloFURN5wCFLtkaTGCb3LPug7v7ks5sAgwmgaJpZM4Ni-0b
.

[zacps]

@davecheney Any comment on making it a compile-time check?

[davecheney]

It feels like somewhere between rusts ownership system and a finaliser.

On Sun, 4 Jun 2017, 13:48 Zac Pullar-Strecker [e-mail]
wrote:

@davecheney https://github.com/davecheney Any comment on making it a
compile-time check?

—
You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/20461#issuecomment-306016154, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA5tJxnFTW952qctOd7bRsZVmylDHks5sAikQgaJpZM4Ni-0b
.

[zacps]

I don't see how it's like a finaliser, at least in the aspect of non-determinism.

It is pretty similar to rust's general approach; I don't think that's a bad thing though.

[tarndt]

If I load a plugin foobar-v1.so that has foobar() I could wrap it in my own functionfoobarWapper() that increments/decrements a waitgroup on every call. Things that want to call foobar do so accessing foobarWraper atomically and calling it. Later on I load foobar-v2.so, wrap it in a new wrapper, and atomically swap it out for out for the old foobarWrapper. Now I can use the old foobarWraper's wait group to make sure no one is active and when that is the case call dlclose using cgo.

This all would depend on the user not creating goroutines, but that could be checked for at the source level if you are building plugins from user submitted code.

@zacps Maybe something like the above would work for you.

[zacps]

Wouldn't you have to either use code generation or lose type safety though?

[tarndt]

Wouldn't you have to either use code generation or lose type safety though?

@zacps I'm not sure how type safety is related to this...

This all would depend on the user not creating goroutines, but that could be checked for at the source level if you are building plugins from user submitted code.

So using my approach you would write a program to vet the user code before building the plugin. Probably by having a white-list of importable packages, and searching the code to get for invocations of "go X" and any functions in white listed packages you want to also blacklist.

[DemiMarie]

I know it's​ a finalizer. It is also what the JVM does with JNI code, and
the only safe solution I know of without Rust-esque lifetime tracking,
generating proxies for each plugin function, or requiring reflection for
every invocation.

On Sat, Jun 3, 2017, 9:47 PM Dave Cheney [e-mail] wrote:

That's a finaliser.

On Sun, 4 Jun 2017, 11:45 Demi Marie Obenour [e-mail]
wrote:

What about having the GC close any plugins that are no longer referenced?

—
You are receiving this because you commented.

Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/20461#issuecomment-306012057, or
mute
the thread
<
https://github.com/notifications/unsubscribe-auth/AAAcAyloFURN5wCFLtkaTGCb3LPug7v7ks5sAgwmgaJpZM4Ni-0b

.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/20461#issuecomment-306012133, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AGGWBxUSgkhkS3oeftTUu49MXgnWoeFfks5sAgzNgaJpZM4Ni-0b
.

[jaekwon]

I second starting with an unsafe way to load and unload plugins, with the expectation that the user will sanitize code before compiling it. It's OK if attempting to unload a plugin panics when there exist dangling pointers/references, and it's also OK if the panic happens later nondeterministically when the dangling pointers/references are accessed.

[Azareal]

I currently use code generation for templates. I have a template transpiler which transpiles text/template templates to Pure Go (it's a little coupled with my application right now, but I could always split it off and it currently misses a few features).

It would be nice, if I could reload these templates on the fly without restarting the entire instance, something impossible without the ability to close plugins or rolling my own JIT compiler.

[DemiMarie]

Here is a thought: what about having plugins export their API as a first-class object, such that the plugin gets unloaded when the API object is garbage collected?

[davecheney]

@DemiMarie this approach is problematic, please see my comments about finalisers earlier in this thread.

[pjebs]

Objective C/Swift use automatic reference counting. Maybe that's the approach to take (manual book keeping - but with compiler's assistance)

[pjebs]

Indirectly related for some use cases: https://www.datadoghq.com/blog/engineering/cgo-and-python/

[AlbinoGeek]

This whole thread seems to be discussing a non-issue

To quote the man pages on dlclose(3)

       A successful return from dlclose() does not guarantee that the
       symbols associated with handle are removed from the caller's address
       space.  In addition to references resulting from explicit dlopen()
       calls, a shared object may have been implicitly loaded (and reference
       counted) because of dependencies in other shared objects.  Only when
       all references have been released can the shared object be removed
       from the address space.

So why would we argue about trying to enforce something stronger than the underlying implementation?

---

Golang has no way to force you to lose your reference to something, nor do I think it should. It is and always has been you (the programmer's) responsibility to not use a resource that you have explicitly asked the system to dispose of.

[tarndt]

Its one thing for the app to know it is no longer using references to a plugin that was loaded and think its safe to call dlclose, but what about the runtime, is the GC still going to try to scan that address space and blow up?

[DemiMarie]

In that case the GC is the only valid way to close a plugin.

On Tue, May 8, 2018, 4:32 PM Tylor Arndt [e-mail] wrote:

Its one thing for the app to know it is no longer using references to a
plugin that was loaded and think its safe to call dlclose, but what about
the runtime, is the GC still going to try to scan that address space and
blow up?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/20461#issuecomment-387532940, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AGGWBy-ZaW2bDBn9CFud33FLB2RdGgalks5twgD4gaJpZM4Ni-0b
.

[qaisjp]

Would it be possible to make close fail until the reference count of all symbols in the plugin reached 1, so that the plugin itself was the only structure that held a reference?

This is spinny and racy.

@davecheney what does "spinny" mean?

[ianlancetaylor]

@qaisjp https://en.wikipedia.org/wiki/Busy_waiting

[mehrvarz]

Its one thing for the app to know it is no longer using references to a plugin that was loaded and think its safe to call dlclose, but what about the runtime, is the GC still going to try to scan that address space and blow up?

You stop all goroutines in and towards the plugin and give up all other resources. Then you call plugin close. The runtime will enforce a GC and once this is done it will call dlclose. It shouldn't matter if this takes some time. In the meantime you can load your new plugin and use it via a new handle.

If the plugin cannot be fully GCed, dlclose will not be called.

I do not understand why it should be impossible to resolve this issue. It would be good to see a short summary of the reasoning.

[buchanae]

From what I've read, the desire to close plugins comes primarily from developers building a fast code reloading system. For example, I have a Go program that...

1. Compiles file_x.go into a plugin, loads it, and calls a function loaded from the plugin.
2. Watches file_x.go for changes
3. When the file changes, rebuild the plugin, reload it, and call the function again.

This has two issues:

1. Loading a plugin consumes memory which cannot be freed, so overall memory usage increases with each change to file_x.go
2. Plugins have identity, and it's not possible to load a plugin twice. To workaround this, it's possible to give each version of the plugin a unique identity. Instead of reloading a plugin, you load a new version with a unique identity. This feels hacky.

Regarding issue 1: perhaps the garbage collector, not the program, should be responsible for releasing the memory associated with a plugin. #28783 seems related.

It's worth calling out that if the garbage collector is responsible for freeing plugin resources, then there's no way for a program to know when those resources have been freed, and that's a good thing because it removes complexity and matches the overall design of memory management in Go.

Regarding issue 2: perhaps there's a better way to manage the identity of plugins that have many, rapid iterations. Or, maybe the existing hack is enough. Seems like we should define this problem clearly and explore solutions before we decide that implementing Plugin.Close is the best solution for that particular issue.

I'd like to note: I think this is an important issue. I don't consider fast code reloading to be an obscure edge case. There are many projects trying to build Go interpreters and REPLs and test runners and other interesting ways to iterate quickly on Go code. "Building a better Go linker" alludes to an interesting future where Go can compile, link, and load code directly into memory; that would open some interesting doors. Many projects would benefit significantly by having a way to collect unused plugin resources.

Example reloading code: https://github.com/buchanae/go-code-reload

[pjebs]

One solution is to forget about the plugin package and use wasm: https://medium.com/wasmer/announcing-the-fastest-webassembly-runtime-for-go-wasmer-19832d77c050

[tarndt]

@pjebs Every project I have used plugins on, for the use case @buchanae called out above, made the move because having to keep large pools of Lua or V8 runtimes around and call them via cgo to process externally loaded per transaction logic was too slow, used lots of memory and was dangerous. What you are proposing doesn't help at all. Its just another run-time to have to try to manage that happens to run WASM.

[pjebs]

There are pure go wasm runtimes

[samplesizeofone]

Seems unfortunate not to have a nice embedded no third party stuff
solution. Possibly a minor nitpick.

On Mon, Mar 9, 2020 at 2:52 PM pj [e-mail] wrote:

There are pure go wasm runtimes

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/golang/go/issues/20461?email_source=notifications&email_token=AADYSWNEKLTQLKWMKJNZRHTRGVQIZA5CNFSM4DML5UN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEOJGO4Y#issuecomment-596797299,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AADYSWOIZBSRBFUHXBYDWI3RGVQIZANCNFSM4DML5UNQ
.

[porfirion]

Oberon's implementation BlackBox has ability to unload modules and able to hotswap them. When you ask runtime to unload module - it marks it as unreachable and when there are no pointers to it's memory - it is removed. If someone asks for types from unloaded module - runtime tries lo load it again, but it can be a newer version. So it is possible to have multiple instances of module at the same time. When all links to old one are removed - there will be only one!
I think it would be killer feature for web development or long running applications.