Telegraf: Ability to use Telegraf as a library

[collinlambert]

What is the reason for using the /internal directory for agent configurations? I ask because it would be nice if one could instantiate the agent within their own source (treat it as a library). This would allow configurations and such to be populated via code instead of on disk files or unique filters/plugins (not relevant to others) to be created and used.

https://golang.org/s/go14internal

[sparrc]

The main reason would be that we've never intended telegraf to be a library, I suppose we could get rid of the internal directory.

This could be a question of whether I want to spend time supporting telegraf as a library though....I'm trying to think of something that may go wrong treating it as a library but nothing is really coming to mind at the moment...

@titilambert might be interested in that too for his telegraf "poller" use-case

[collinlambert]

I can understand the concern for maintaining as if it's a library. A potential argument for it, though, is that there are numerous custom or unique services and environments that one would want to integrate with but are unable to because only the shell of telegraf is accessible. I understand something like the exec plugin helps alleviate some of this but it's not as performant nor streamlined as a custom in-process plugin.

[sparrc]

@collinlambert but couldn't you just fork telegraf and write your own plugin? I guess I'm still not sure how making telegraf a library helps with that specific use-case?

To me forking and writing a plugin seems easier than writing your own telegraf binary

[collinlambert]

Agree that forking is a possible solution but there are obvious downsides (syncing, conflicts, etc.)

The specific use-case I'm thinking of is getting configurations from a centralized config store (programmatically) and hitting my custom service that exposes metrics in an unorthodox way. The thing is, I could probably get away with writing my own equivalent service that does this but there are other components of telegraf (cpu metrics, network metrics, etc) that make it appealing as a universal metrics consumer/propagater.

Not to mention, the way you've organized/written telegraf is very conducive to writing your own binary (minus the /internal portion 😀).

[sparrc]

Fair enough, I can definitely understand the benefits of exposing telegraf as a library. Just so I have it clear, the only problem right now is basically just the config package (getting imported here: https://github.com/influxdata/telegraf/blob/master/cmd/telegraf/telegraf.go#L13)

From what I can tell this is the only package that is currently "internal" that you would need?

[collinlambert]

Yeah, from what I can tell that's the only thing stopping me from instantiating and running an agent. I can dig deeper tomorrow, though, to make sure that's the case.

[titilambert]

@collinlambert Does this PR https://github.com/influxdata/telegraf/pull/651 could, kind of, cover your use-case ?

[collinlambert]

@titilambert I don't think so. The way I read that PR is it allows you to read configurations from etcd as opposed to a file. I'm looking to read configurations from anywhere, against any custom configuration store. The only means of doing that is by exposing telegraf as a library. It also doesn't address the custom plugin issue I brought up.

@sparrc Looking a little deeper exposing config should allow for programmatic execution but I see one shortfall that makes it less than ideal:

func (c *Config) addInput(name string, table *ast.Table) error {

func (c *Config) addOutput(name string, table *ast.Table) error {

Because these methods are not exposed, to instantiate and populate a config object we need to go through the config.LoadDirectory or config.LoadConfig path. This requires the serialization of a configuration just to load it.

It's still a path forward. Long term, though, we can look into exposing an AddOutput/AddInput method that is public and takes something a bit more user friendly than an *ast.Table. I'd be happy to put together any or all of these PR's, just let me know.

[sparrc]

Can you explain how you would make addInput & addOutput more user-friendly? I agree that passing in an *ast.Table is not ideal, but marshalling text config files into the plugin structs is a thorny problem

[collinlambert]

It could be something as straightforward as taking an internal_models.RunningInput or internal_models.RunningOutput and run some validation against it.

As an example, I'm capable of building out and configuring a kafka.Kafka output and *output.OutputConfig on my own. Why not just take that and call AddOuput? Of course, it would require moving the internal_models out of /internal.

This path requires a bit more to change but even without it we can still begin to get to programmatic configuration and execution by using the config.LoadConfig method, which is my ideal goal here.

Thoughts?

[sparrc]

yes, I see what you mean. The current configuration assumes that all we would want to do is apply generic text to static structures, but you are saying that you would like to take the structures and set them up manually within your code.

[titilambert]

@collinlambert Do you want to load telegraf deamon directly in your own soft ? Then Telegraf will run directly inside you soft ? (like cadvisor in kubelet?)

[collinlambert]

@titilambert Yeah, that's correct. I don't have experience with cadvisor or kubelet but your description of running the telegraf daemon within my software is correct.

[titilambert]

@collinlambert ok ! With this method, are you limited to golang soft only ?

[collinlambert]

I don't have experience creating bindings between languages so I can't really say. My use case is only around golang (but I don't know if it's limited to that).

[titilambert]

@collinlambert And what you do think about Prometheus approach ? This means, each application expose a http endpoint to expose metrics and register themselves in a discovery service... See here: http://prometheus.io/docs/introduction/overview/

[collinlambert]

@titilambert It sounds like a solid approach. Are you asking because my request would allow for this to happen?

[titilambert]

@collinlambert I don't know I'm just curious about all approaches around monitoring solutions.
In my opinion, your use case could be cover with the Prometheus approach.
Your soft could expose en http endpoint (:/metrics) with Prometheus format and register itself in etcd/consul/... Then Telegraf/Prometheus can gathers metrics.
What do you think about ? This is pulling but it's language agnostic.

Maybe I'm wrong, but you just want your soft send data to your influxdb/opentsdb, don't you ?
So, you don't need the inputs parts of Telegraf, do you ?

[collinlambert]

@titilambert I see what you're saying now.

The problem lies more with the fact that I already have this discovery/configuration environment built up. Though Prometheus looks promising it's not the route I went down and changing direction would be a big jump. That's sort of why I like the idea of making Telegraf more extensible, for people in situations like my own.

I think I misinterpreted your reference to "soft". In my world my services do expose metrics through an endpoint, I don't want Telegraf running in the same process space as them. I just want the logic for finding and adapting to these services to be wrapped around the Telegraf daemon.

@sparrc would it make sense to fork this into a new issue? "Telegraf Library" maybe?

[sparrc]

sure, I can just rename this issue

[netixen]

+1
Right now I have a local telegraf fork, just so It can integrate with my stack (service discovery, key-value store, etc). But keeping it synced is an unnecessary chore since all I do is add some integration boilerplate and cut the modules I don't use.

[bharley]

I would also like to voice my support for some kind of sane plugin system. Right now we've also had to resort to forking and syncing Telegraf to write a plugin for an internal service, and it's not an ideal situation to be in.

[sparrc]

We're getting into a bit of feature creep with this issue. I don't think making telegraf a library would help with users trying to maintain their own forks with internal plugins.

If you want some sort of runtime-defined plugin set then that should be a separate feature request.

[LarryCN]

Hi, would telegraf as a library happen?
Right now, I`m also looking at telegraf. It has lots of plugins, which is nice. As mentioned above, what we need is "internal/config", I guess. Then, we could integrate telegraf within our own project. In this way, we could keep config file format, and use selected plugins, as well as use our own specific "plugins". So if telegraf could be imported as a library, it gonna provide a lot of flexibility.