Telegraf: HTTP status 406 returned on prometheus input plugin

[KavyaKanupuru]

Bug report

Relevant telegraf.conf:

[[inputs.prometheus]]
urls = ["http://localhost:8080/metrics"]

System info:

Telegraf v1.4.2

Steps to reproduce:

1. To config add the above configuration , mapping prometheus url (metrics in prometheus format)

Expected behavior:

Metrics should be available on the outputs defined

Actual behavior:

Error in plugin [inputs.prometheus]: http://localhost:8080/metrics returned HTTP status 406

Additional info:

Running the application on open stack server

[danielnelson]

The prometheus input uses this accept header:

Accept: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited;q=0.7,text/plain;version=0.0.4;q=0.3

Can you find out what Content-Type's the server can return?

[KavyaKanupuru]

@danielnelson The server is returning content-type= "text/plain; version=0.0.4;charset=utf-8"

[danielnelson]

Hmm, it seems like this should be acceptable. Can you tell me a little about the service you are monitoring? For instance, is it a particular application or a custom service? If it is a custom service are you using the official prometheus client libraries and if so which language/version?

[KavyaKanupuru]

@danielnelson I am using official prometheus for a spring boot application. Language is Java 8.
Below are the prometheus dependencies I have included in my build.gradle.
compile group: 'io.prometheus', name: 'simpleclient_spring_boot', version: '0.0.21'
compile group: 'io.prometheus', name: 'simpleclient_hotspot', version: '0.0.21'
compile group: 'io.prometheus', name: 'simpleclient_servlet', version: '0.0.21'
compile group: 'io.prometheus', name: 'simpleclient_dropwizard', version: '0.0.21'

[danielnelson]

I think this is your issue: https://github.com/prometheus/client_java/issues/265

[danielnelson]

I wonder if adding a charset would work around this issue until you can update. Would you be able to test using something like curl and see if sending the header like this helps:

Accept: application/vnd.google.protobuf;proto=io.prometheus.client.MetricFamily;encoding=delimited;q=0.7,text/plain;version=0.0.4;charset=utf-8;q=0.3

[KavyaKanupuru]

Hi Daniel, I tried to curl the url with 0.0.21 Prometheus version. Even the curl command was throwing 406 error.As suggested in the issues you redirected me,I have changed the version to 0.0.24 and after changing the version I could curl the url with header content type 'text/plain' without any error.

[danielnelson]

Okay, glad you have it working, I'm going to close this issue then since I don't think we can do anything here.