Fabio: Support for prometheus

[Equilibri0]

Hi there,

it would be great to get the fabio metrics also to a prometheus instance.
Is there any chance for ?

greetz

Equilibri0

[magiconair]

Somehow I knew that this was going to come :) I'm working on another metrics ticket right now and thought the same thing. I'll add it to the list.

[raben2]

+1

[pipozzz]

+1

[mdirkse]

+1

[deuch]

+1 for me too !

[agonzalezandino]

+1 that'd be cool!

[nanoz]

Any news on the Prometheus /metrics endpoint ?

This feature would be very helpful to us to measure things like requests throughput, call latencies and status codes, globally and maybe per entrypoint/endpoint as well ? :)

[csawyerYumaed]

I'd love to see prom support, but in the meantime you should be able to work-around it with statsd in fabio and https://github.com/prometheus/statsd_exporter

[magiconair]

I need to wrap my head around on how to configure the naming and to expose the tags. The issue is that some metrics system want a single string as name which flattens all parts of the name into a single value. Others have a base name and other parts as a tag. What makes this a bit more complex is that some of the names are auto-generated and the pattern is configurable. So either this mechanism needs to work for all providers or I need to come up with something else. It probably isn't that difficult but that's the hold-up. Same for #165 .

[drawks]

Any progress on updating the internal Registry api to support labels? It would be a boon to both prometheus support as well as datadog style statsd+labels and others.

IMHO, part of the solution could be as simple as passing into the Registry's update functions a context object which contains the same information that is currently being used to flatten out into a name. And leave it up to the Registry to do the work of either flattening in accordance to the configured template OR using the attributes of the context in whatever other manner makes sense.

[tbouvet]

+1

[magiconair]

work on consul 1.0 has kept me quite busy. I plan to go through open issues soon.

[nuriel77]

+1

[magiconair]

working on it.

[drawks]

Any updates here?

[magiconair]

I've started and got distracted with family and personal stuff. Sorry man. I'll give it another push.

[marcosnils]

@magiconair I'd love to use this feature natively instad of the statsd prometheus exporter. Do you need a hand with this?. Do you have a branch with this feature where I can collaborate?.

On a different subject Hashicorp's Nomad uses their on go-metrics (https://github.com/armon/go-metrics) package that translates metrics directly to graphite, statsd and prometheus. I haven't evaluated how difficult would be to change the current go-metrics package for his one, but seems like this would solve a lot of issues and remove some extra dependencies as well.

[magiconair]

@marcosnils I'd love to have some help here to push this forward. I have an idea on how to do this since the problem is a bit bigger than just supporting prometheus. Look for the issues tagged theme-metrics. In short, none of the existing metrics libs solve all the problems and there needs to be some fabio specific glue code. My current thinking is to write a layer which handles the naming issues and configuration and then uses go-kit/metrics drivers since they are complete and seem to be maintained. Shall we do a conf call to discuss the approach and then document it here? Where are you located?

[magiconair]

If someone else wants to help with this, too then pls just shout out.

[alexebird]

@magiconair @marcosnils I'd like to help out with this if it's still needed.

[rileyje]

@magiconair has there been any progress on labeled metrics? I'm also willing to help with this.

[magiconair]

I've finally got something that I think is worth working on. Please see #476 in the metrics4 branch. Right now I'm trying to validate the approach. So if someone wants to write a proof-of-concept provider for prometheus using the go-kit/metrics driver then this would be awesome. This would then allow us to integrate all existing go-kit/metrics drivers which covers most of the open issues.

Lets keep the discussion on the metrics in #476 since other backends are also affected.

I'm not sure how we can collaborate on a PR on Github. I assume you fork the branch and create a patch. I'll try to incorporate patches into the metrics4 branch so that we all have the same base.

Thanks for your patience. This has been bugging me for a while so lets solve this properly.

[marcosnils]

I'm not sure how we can collaborate on a PR on Github

We can fork your code and open a PR to the metrics4 branch directly. If you merge your PR will get updated automatically.

Thanks for the hard work @magiconair . I'll give it a look this week and contribute in whatever I can.

[ketzacoatl]

@marcosnils did you have any luck diving it?

[marcosnils]

Not really, been extremely busy with some other stuff. Might be able to take a look some time this week. Can't promise anything though

[KHiis]

Any updates on this?

[ketzacoatl]

Until support for Prometheus lands, what is the current recommended method for getting stats from fabio to prometheus?

[KHiis]

@marcosnils Do you have any answers?

[marcosnils]

@KHiis not really. I'm currently using fabio statsd support and statsd_exporter (https://github.com/prometheus/statsd_exporter) to push metrics to prom. Would love to add native prom support it's just that I don't have the time.

[magiconair]

See https://github.com/fabiolb/fabio/pull/476#issuecomment-434048436

[kikdevops]

@marcosnils can you provide statsd_exporter mappings? Can't figure out why my mappings don't working.

[marcosnils]

@kikdevops here's my mapping config

mappings:
- match: ^([^.]*)\.([^.]*)--http.status.([^.]*).count
  match_type: "regex"
  name: "fabio_http_status_count"
  labels:
    code: "$3"
    instance: "$1"
- match: .
  match_type: "regex"
  action: drop
  name: dropped

[kikdevops]

@marcosnils Thanks a lot.