metrics
Chaps, I'm am trying to evaluate how to instrument Kong and to my honest surprise found that there is no native metric exporter for prometheus.
The only metrics plugins available are "paid for" things like data-dog etc...
I would actually like to politely suggest that someone makes this a priority, fact is that there are 0 open source metrics plugins I can see...
starkers
All 18 comments
I'm currently instrumenting Kong for Prometheus using https://github.com/knyar/nginx-lua-prometheus
thenayr
on 6 Mar 2017
That being said, a native plugin would be super useful, although the customization and metrics would pose an interesting challenge.
My assumption is there would be a few "default" metrics exposed (upstream response times, connection counters etc.) with some mechanism of defining additional ones on your own.
thenayr
on 6 Mar 2017
@thenayr I had a quick look at the lib you mentioned, seems a lot like statsd on a first glance, maybe that plugin could provide some inspiration? see https://github.com/Mashape/kong/tree/master/kong/plugins/statsd
Tieske
on 7 Mar 2017
@Tieske I had enabled Kong`s statsd plugin, I plan to send kong metrics to statsd_exporter, But I did not seen any data about kong,
I guess something wrong with statsd_mapping.conf
JoshuaAndrew
on 10 Apr 2017
Any news?
hugodlg
on 27 Apr 2017
hello, any changes in the work ?
ghost
on 12 Jul 2017
Nope, sorry... Kong was literally the hardest thing to run inside k8s that we ever tried I'm afraid..
starkers
on 12 Jul 2017
@starkers Wondering why that is the case? It works beautifully for us inside of k8s
thenayr
on 12 Jul 2017
Just had a quick look again, but looks to me that the statsd plugin should suffice if you use the official Prometheus Statsd exporter.
@JoshuaAndrew any luck in getting that to work?
Tieske
on 27 Jul 2017
Would be nice to have things in /cluster as metrics exposed via prometheus as welll
caarlos0
on 19 Sep 2017
馃憤 馃憤 馃憤
chenrui333
on 6 Jan 2018
A quick google will find it for you.... https://github.com/yciabaud/kong-plugin-prometheus , there are issues with prom as a pull against an application like kong (which the author discloses) which is why pushing to a statsd server is safer/ideal.
jeremyjpj0916
on 7 Jan 2018
Have setup https://github.com/nijikokun/kong-plugin-prometheus and its going ok so far. I'd been keen to see 1) if there are any caveats or issues in usage of a kong plugin to provide a prom exporter; 2) related to that if it works fine when kong is horizontally scaled, and 3) what we can contribute to enhance the plugin. @coopr be good to get an update from Mashape.
chris-scentregroup
on 5 Mar 2018
@chris-scentregroup It looks like that plugin stores values in a shared dict, similar to the yciabaud/kong-plugin-prometheus plugin linked in the previous comment. If that's the case then I imagine it will be subject to the same caveats that yciabaud notes for their plugin, which are also noted in the underlying knyar/nginx-lua-prometheus library.
It looks like both plugins use gauges, counters, and histograms. If Kong were horizontally scaled then each server would only be able to observe its own metrics, but you'd be able aggregate them on the Prometheus server using avg/min/max/histogram_quantile, etc.
cosmopetrich
on 5 Mar 2018
Thanks @cosmopetrich. In general I'm also wondering about data stores for metric like data in general in kong. Before looking at plugins/extras and the like; I noticed for example that wherever konga gets its metrics for kong in the dashboard this will reset if I replace the containers. I presume this may well be a global problem and currently there isn't any basic metric data stored back to the db?
Excuse my ignorance here, I'm pretty new. If none of its going back to the db, happy to use data containers/volumes to persist 'counts' if I can get some pointers where these are stored.
chris-scentregroup
on 5 Mar 2018
No worries at all. Prometheus' approach is a bit different to statsd/datadog/etc.
Both the plugins (and the underlying kynar/nginx-lua-prometheus module) store metrics in memory in a variable with a predefined maximum size. That's a pretty common approach for Prometheus metrics, since it's assumed that Prometheus will be retrieving and storing metrics approximately every 60 seconds. Kong will reset its metrics when it restarts, but Prometheus will miss 60s of data at most, and it will automatically detect that the metrics have reset.
The caveat is that Nginx will need to read the entire metrics variable whenever Prometheus retrieves metrics from Kong. Nginx won't the variable to be written to until it's finished reading it. That might not be a problem, but it depends on a few things. How often the variable is written to and how fast a write needs to be will depend on the amount of traffic Kong receives. Kong's client will already have received a response, assuming that metrics are updated during the log phase. However, a delay in this phase could still be detrimental if Kong is receiving a large amount of traffic.
The amount of time it takes for the variable to be read will depend on the number of metrics that are stored, and how many values there are for each metric's labels. The nijikokun plugin appears to store quite a large number of labels. This will mean that its metrics provide a lot of granularity, but that they're more resource-intensive on Kong and on Prometheus itself. The presence of remote_addr as a label could easily generate tens of thousands of metrics on a Kong server that's exposed to the public internet. The yciabaud plugin offers a much more conservative set of labels. These won't allow you to "drill down" as much as the other plugin, but they're far less likely to cause problems for Kong or Prometheus.
Storing metrics in the database could help, though that approach introduces problems of its own. Database read/writes are expensive compared to updating in-memory variables, particularly if there's a need to update tens of variables for each request that Kong processes.
cosmopetrich
on 5 Mar 2018
@cosmopetrich thank you for the detailed explanation. Do you think the yciabaud plugin is the best/move practical option at the moment or is adding in use of statsd the better solution?
chris-scentregroup
on 6 Mar 2018
There are multiple solutions out there now. I think we can close this.
thibaultcha
on 11 Apr 2018
Related issues
vikaskmr1008
路
3Comments
carnei-ro
路
3Comments
xiaochai
路
3Comments
z843248880
路
3Comments
rucciva
路
3Comments
Most helpful comment
No worries at all. Prometheus' approach is a bit different to statsd/datadog/etc.
Both the plugins (and the underlying kynar/nginx-lua-prometheus module) store metrics in memory in a variable with a predefined maximum size. That's a pretty common approach for Prometheus metrics, since it's assumed that Prometheus will be retrieving and storing metrics approximately every 60 seconds. Kong will reset its metrics when it restarts, but Prometheus will miss 60s of data at most, and it will automatically detect that the metrics have reset.
The caveat is that Nginx will need to read the entire metrics variable whenever Prometheus retrieves metrics from Kong. Nginx won't the variable to be written to until it's finished reading it. That might not be a problem, but it depends on a few things. How often the variable is written to and how fast a write needs to be will depend on the amount of traffic Kong receives. Kong's client will already have received a response, assuming that metrics are updated during the log phase. However, a delay in this phase could still be detrimental if Kong is receiving a large amount of traffic.
The amount of time it takes for the variable to be read will depend on the number of metrics that are stored, and how many values there are for each metric's labels. The
nijikokunplugin appears to store quite a large number of labels. This will mean that its metrics provide a lot of granularity, but that they're more resource-intensive on Kong and on Prometheus itself. The presence ofremote_addras a label could easily generate tens of thousands of metrics on a Kong server that's exposed to the public internet. Theyciabaudplugin offers a much more conservative set of labels. These won't allow you to "drill down" as much as the other plugin, but they're far less likely to cause problems for Kong or Prometheus.Storing metrics in the database could help, though that approach introduces problems of its own. Database read/writes are expensive compared to updating in-memory variables, particularly if there's a need to update tens of variables for each request that Kong processes.