Consul: Unable to scrape consul prometheus metrics

[wargamez]

Overview of the Issue

Using 1.2.1 of Consul running in docker/rancher, Prometheus is unable to scrape the prometheus metrics endpoint as it is not following the standard

Reproduction Steps

1) Enable the telemetry prometheus metrics endpoint in consul, e.g
"telemetry": {
"prometheus_retention_time": "24h"
}

2) Add scape config to prometheus e.g:
- job_name: 'consul'
metrics_path: '/v1/agent/metrics'
dns_sd_configs:
- names:
- 'consul.consul-rancher'
refresh_interval: 5s
type: A
port: 8500

Watch as prometheus logs or ui for reports of error with "no token found"

And also promtool gives warnings about wrong suffixes on metrics:
$ wget -O metrics.txt https://consul.yourdomain.com/v1/agent/metrics?format=prometheus; cat metrics.txt | promtool check metrics
Connecting to XXXX
metrics.txt 100% |*****************************| 19194 0:00:00 ETA
consul_acl_cache_hit counter metrics should have "_total" suffix
consul_acl_cache_miss counter metrics should have "_total" suffix
consul_client_api_catalog_datacenters counter metrics should have "_total" suffix
consul_client_api_success_catalog_datacenters counter metrics should have "_total" suffix
consul_memberlist_msg_alive counter metrics should have "_total" suffix
consul_consul_rancher_consul_6_runtime_free_count non-histogram and non-summary metrics should not have "_count" suffix
consul_memberlist_tcp_accept counter metrics should have "_total" suffix
consul_memberlist_tcp_connect counter metrics should have "_total" suffix
consul_memberlist_tcp_sent counter metrics should have "_total" suffix
consul_consul_rancher_consul_6_runtime_malloc_count non-histogram and non-summary metrics should not have "_count" suffix
consul_memberlist_udp_received counter metrics should have "_total" suffix
consul_memberlist_udp_sent counter metrics should have "_total" suffix
consul_raft_state_follower counter metrics should have "_total" suffix
consul_rpc_accept_conn counter metrics should have "_total" suffix
consul_rpc_raft_handoff counter metrics should have "_total" suffix
consul_rpc_request counter metrics should have "_total" suffix
consul_serf_member_join counter metrics should have "_total" suffix

[pierresouchay]

You also have to add:

params:

• format: ['prometheus']

In your Prometheus configuration as well

[linuxrebel]

I've a similar issue to wargames in that I keep getting the token issue.

• job_name: 'consul-server'
  metrics_path: '/v1/agent/metrics'
  params:
  format: ['prometheus']
  static_configs:
  
  
  
  • targets: ['10.36.0.86:8500']
  
  

I get the URL of http://10.36.0.86:8500/v1/agent/metrics?format=prometheus but I'm still getting the no token error. (Please assume the proper yaml indents.... I've not yet grokked the github markup)

[pearkes]

Looks like @pierresouchay covered the answer for the original question here. @linuxrebel seems like you have a different issue if I'm interpreting this correctly, feel free to re-open another but will close this given the original question.

[linuxrebel]

Yes my issue was different. But now found. At least found in that we know why it's not working. Seems that The helm chart for consul is not working the expected way and the Telemetry is not getting setup correctly. Thanks for the response.