Telegraf: Support External Plugins using Go 1.8 plugin feature
We should support building and loading of external plugins as dynamic libraries, using Go's new "plugin" feature, available in Go 1.8: https://tip.golang.org/pkg/plugin
For this we will likely want to maintain an external repository where we can aggregate "external" plugins which are not part of the telegraf statically-linked build.
Telegraf will be able to load all ".so" plugins from a directory, adding them to the regular plugin registry before the config file is loaded.
sparrc
All 27 comments
we may have experimental support for this in Telegraf 1.2, but since Go 1.8 will not be officially released at that point we will probably only see full support in Telegraf 1.3.
sparrc
on 18 Nov 2016
@sparrc I have to do some reading on Go 1.8's feature, but have you seen or evaluated this as an alternative? https://github.com/hashicorp/go-plugin
johnrengelman
on 21 Nov 2016
yes, that has been evaluated, IIRC that library doesn't give us anything we don't already have (plugins statically-compiled into a single binary)
sparrc
on 21 Nov 2016
I believe you can use it to load plugins as external binaries that communicate over network RPC. That's how Terraform used to work (as separate binaries per plugin) before they changed to bundling everything into a single binary. I believe external plugins are still loaded via this mechanism.
johnrengelman
on 21 Nov 2016
johnrengelman
on 21 Nov 2016
We could do external binaries, but there really is no need in that case for a plugin framework, IMHO, why not just communicate over plain grpc or text at that point?
the decision to go with dynamic libraries was specifically to avoid managing external processes.
sparrc
on 22 Nov 2016
Sounds good. Just wanted to offer up some options since plugins would be super beneficial for me. +1 to this plan. :)
johnrengelman
on 22 Nov 2016
Intel snap telemetry project also uses external processes via grpc.
daledude
on 3 Feb 2017
Would this supersede the exec plugin moving forward? Or would that stick around?
sebito91
on 4 Feb 2017
exec will definitely stick around, these .so files will only ever support plugins written in Go, for one thing
sparrc
on 4 Feb 2017
Does this pr will allow dynamic loading of parser and serializer plugins ? (we are using custom metrics format in our company with custom serializer and parser)
Does all the actual plugins will be provided as external libraries ? Each one in a separate repository ? This will reduce the size of the telegraf binary.
seuf
on 2 May 2017
Answers based on #2373. It is still up in the air if we will merge this, though at this time it is on hold due to issues with shared object support in go.
Does this pr will allow dynamic loading of parser and serializer plugins ?
No.
Does all the actual plugins will be provided as external libraries ?
Only the ones that are specifically made to be external.
Each one in a separate repository ?
Probably all in one repo, but in theory yes you could do this.
danielnelson
on 2 May 2017
Does this pr will allow dynamic loading of parser and serializer plugins ?
No.
Why not ? It can be very usefull for us...
seuf
on 3 May 2017
Why not ? It can be very useful for us...
Because this is more of a POC and it would also be significantly more complex given the way that parser and serializer plugins are loaded and configured in telegraf.
sparrc
on 3 May 2017
Such feature would be very useful for us. For some internal projects we had to develop very custom and specific input/output plugins. We currently have the burden to make our own internal build of telegarf and container image and keep everything in sync with the upstream project.
palmamartin
on 20 Sep 2017
We have the same problem as mazebuhu.
szuki
on 20 Sep 2017
I think we will try to add support for gRPC plugins due to usability issues with the Go plugins. I think everyone will be happier with this solution in the end.
danielnelson
on 20 Sep 2017
I certainly would. I really like Go as a language but I'm far from proficient in it. gRPC would enable us to do our custom plugins in languages that we are more familiar with. Would also enable us to fairly easily port our existing monitoring tools.
What issues/PRs should I track to stay up to date regarding external plugins?
BartVB
on 20 Feb 2018
I just opened #3813 to track the gRPC idea.
danielnelson
on 20 Feb 2018
Both approaches to extension have their merits; possibly in the long-run both mechanisms should be implemented. One tradeoff the gRPC process-based plugin approach brings is increased resource utilization. In trying snaptel for example, I found each plugin enabled added roughly a dozen MiB of resident memory utilization to the overall installation. There each plugin is hosting its own isolated language runtime, gRPC server, etc. That adds up quickly with even a few 'basic' plugins enabled and can become untenable on smaller virtual servers. Also data flowing between the core agent process and plugins must be marshalled multiple times along the pipeline, incurring CPU overhead. I've found Telegraf's lighter footprint to be one of its distinguishing features and believe .so based extensions would help better maintain that.
esstone7
on 20 Apr 2018
Is this patch the latest work that has been done on this front?
Timidger
on 13 Jun 2019
Yes, no work has been done on this for several years.
danielnelson
on 13 Jun 2019
I'm currently evaluating if it's possible if this feature would enable us to use other languages (namely Rust) to collect metrics. In our use case that would mean our .so Go plugin would have a statically linked in Rust code that can be called over the C ABI.
We don't want to use gRPC because we are targeting devices that have low space requirements. This feature would be better on our space usage.
The major question I have is what data can be handed back to Go. The fields are type punned but it looks like it's just basic types (based on this code). Is this correct?
Timidger
on 14 Jun 2019
I got a toy example put together that works, but would the project be willing to use the C ABI for the shared library?
It would mean we wouldn't have to maintain any shim Go code (which I would currently have to do since the change as proposed so far expects something implementing the Plugin interface, which I can't express in Rust). As well, more projects could benefit since every language under the sun can talk C, and it is an alternative to using a full gRPC server if instead protobuf is used as the marshaling format.
It would necessitate writing some unsafe Go code, but with the cgochecks and if protobuf is used it won't be very difficult to use.
Timidger
on 14 Jun 2019
I don't think that would be something we would merge as it would require cgo enabled, and I don't think this is an API we want to maintain officially. Wouldn't mind seeing what you have though if you can link to the code.
The best methods for integrating other languages currently are either the exec plugin (for polling data), or for event driven data or for efficiency by running an external process sending to the socket_listener plugin. You can send in any of the supported data formats but InfluxDB line protocol has the best support.
danielnelson
on 14 Jun 2019
Ok. I'm going to add the Go plugin feature then and keep that CGo shim out of tree. We are going to publish it in a Rust crate so if you're interested I can link it here once it's complete.
Timidger
on 14 Jun 2019
Closed in #6024
danielnelson
on 30 Jul 2019
Related issues
fahimeh2010
路
3Comments
Xiol
路
3Comments
Isonami
路
3Comments
robert-gomes
路
3Comments
yn1v
路
3Comments
Most helpful comment
Such feature would be very useful for us. For some internal projects we had to develop very custom and specific input/output plugins. We currently have the burden to make our own internal build of telegarf and container image and keep everything in sync with the upstream project.