Telegraf: mongodb plugin with prometheus_client output wont show member_status

[freeseacher]

Bug report

Relevant telegraf.conf:

[[inputs.mongodb]]
  ## An array of URI to gather stats about. Specify an ip or hostname
  ## with optional port add password. ie,
  ##   mongodb://user:[email protected]:27017,
  ##   mongodb://10.10.3.33:18832,
  ##   10.0.0.1:10000, etc.
  #
  # we use 127.0.0.1 hre because of it will be launched on each mongodb server
  #
  servers = ["mongodb://root:[email protected]:27017/admin"]
  gather_perdb_stats = true

# Configuration for the Prometheus client to spawn
[[outputs.prometheus_client]]
  ## Address to listen on
  listen = "0.0.0.0:9126"
  expiration_interval = "10s"

System info:

Telegraf v1.3.2 (git: release-1.3 0f419e9a0d294ab4d68bb7d1c6b28e60d362239b)

Steps to reproduce:

1. run telegraf --config mongodb.conf --test

* Plugin: inputs.mongodb, Collection 1
* Plugin: inputs.mongodb, Collection 2
> mongodb,hostname=127.0.0.1:27017,host=hostname active_writes=1i,vsize_megabytes=11588i,repl_updates_per_sec=1240i,repl_lag=0i,jumbo_chunks=0i,updates_per_sec=0i,queries_per_sec=940i,state="SECONDARY",percent_cache_dirty=0.7,repl_getmores_per_sec=0i,member_status="SEC",flushes_per_sec=0i,open_connections=1402i,ttl_passes_per_sec=0i,active_reads=0i,queued_reads=0i,repl_queries_per_sec=0i,deletes_per_sec=0i,net_in_bytes=267791i,queued_writes=0i,repl_commands_per_sec=0i,repl_inserts_per_sec=0i,percent_cache_used=6.5,resident_megabytes=8793i,inserts_per_sec=0i,ttl_deletes_per_sec=0i,repl_deletes_per_sec=0i,commands_per_sec=9i,net_out_bytes=1632198i,getmores_per_sec=2i 1497567411000000000
> mongodb_db_stats,host=hostname,db_name=admin,hostname=127.0.0.1:27017 num_extents=0i,indexes=3i,type="db_stat",collections=2i,objects=6i,ok=1i,storage_size=53248i,index_size=65536i,avg_obj_size=212,data_size=1272i 1497567411000000000
> mongodb_db_stats,hostname=127.0.0.1:27017,host=hostname,db_name=mydb indexes=501i,index_size=10194022400i,type="db_stat",num_extents=0i,ok=1i,avg_obj_size=574.4335489103034,data_size=59652610236i,storage_size=21825556480i,collections=187i,objects=103845972i 1497567411000000000

1. run curl http://0.0.0.0:9126/metrics

mongodb_active_reads{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 0
# HELP mongodb_active_writes Telegraf collected metric
# TYPE mongodb_active_writes untyped
mongodb_active_writes{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 0
# HELP mongodb_commands_per_sec Telegraf collected metric
# TYPE mongodb_commands_per_sec untyped
mongodb_commands_per_sec{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 21
# HELP mongodb_db_stats_avg_obj_size Telegraf collected metric
# TYPE mongodb_db_stats_avg_obj_size untyped
mongodb_db_stats_avg_obj_size{db_name="mydb",dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 574.4350308437449
# HELP mongodb_db_stats_collections Telegraf collected metric
# TYPE mongodb_db_stats_collections untyped
mongodb_db_stats_collections{db_name="mydb",dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 187
# HELP mongodb_db_stats_data_size Telegraf collected metric
# TYPE mongodb_db_stats_data_size untyped
mongodb_db_stats_data_size{db_name="mydb",dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 5.9652881888e+10
# HELP mongodb_db_stats_index_size Telegraf collected metric
# TYPE mongodb_db_stats_index_size untyped
mongodb_db_stats_index_size{db_name="mydb",dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 1.0194034688e+10
# HELP mongodb_db_stats_indexes Telegraf collected metric
# TYPE mongodb_db_stats_indexes untyped
mongodb_db_stats_indexes{db_name="mydb",dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 501
# HELP mongodb_db_stats_num_extents Telegraf collected metric
# TYPE mongodb_db_stats_num_extents untyped
mongodb_db_stats_num_extents{db_name="mydb",dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 0
# HELP mongodb_db_stats_objects Telegraf collected metric
# TYPE mongodb_db_stats_objects untyped
mongodb_db_stats_objects{db_name="mydb",dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 1.03846177e+08
# HELP mongodb_db_stats_ok Telegraf collected metric
# TYPE mongodb_db_stats_ok untyped
mongodb_db_stats_ok{db_name="mydb",dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 1
# HELP mongodb_db_stats_storage_size Telegraf collected metric
# TYPE mongodb_db_stats_storage_size untyped
mongodb_db_stats_storage_size{db_name="mydb",dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 2.1825572864e+10
# HELP mongodb_deletes_per_sec Telegraf collected metric
# TYPE mongodb_deletes_per_sec untyped
mongodb_deletes_per_sec{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 0
# HELP mongodb_flushes_per_sec Telegraf collected metric
# TYPE mongodb_flushes_per_sec untyped
mongodb_flushes_per_sec{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 0
# HELP mongodb_getmores_per_sec Telegraf collected metric
# TYPE mongodb_getmores_per_sec untyped
mongodb_getmores_per_sec{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 2
# HELP mongodb_inserts_per_sec Telegraf collected metric
# TYPE mongodb_inserts_per_sec untyped
mongodb_inserts_per_sec{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 0
# HELP mongodb_jumbo_chunks Telegraf collected metric
# TYPE mongodb_jumbo_chunks untyped
mongodb_jumbo_chunks{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 0
# HELP mongodb_net_in_bytes Telegraf collected metric
# TYPE mongodb_net_in_bytes untyped
mongodb_net_in_bytes{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 554257
# HELP mongodb_net_out_bytes Telegraf collected metric
# TYPE mongodb_net_out_bytes untyped
mongodb_net_out_bytes{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 3.244973e+06
# HELP mongodb_open_connections Telegraf collected metric
# TYPE mongodb_open_connections untyped
mongodb_open_connections{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 1404
# HELP mongodb_percent_cache_dirty Telegraf collected metric
# TYPE mongodb_percent_cache_dirty untyped
mongodb_percent_cache_dirty{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 0.6
# HELP mongodb_percent_cache_used Telegraf collected metric
# TYPE mongodb_percent_cache_used untyped
mongodb_percent_cache_used{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 6.6
# HELP mongodb_queries_per_sec Telegraf collected metric
# TYPE mongodb_queries_per_sec untyped
mongodb_queries_per_sec{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 1953
# HELP mongodb_queued_reads Telegraf collected metric
# TYPE mongodb_queued_reads untyped
mongodb_queued_reads{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 0
# HELP mongodb_queued_writes Telegraf collected metric
# TYPE mongodb_queued_writes untyped
mongodb_queued_writes{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 0
# HELP mongodb_repl_commands_per_sec Telegraf collected metric
# TYPE mongodb_repl_commands_per_sec untyped
mongodb_repl_commands_per_sec{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 0
# HELP mongodb_repl_deletes_per_sec Telegraf collected metric
# TYPE mongodb_repl_deletes_per_sec untyped
mongodb_repl_deletes_per_sec{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 1
# HELP mongodb_repl_getmores_per_sec Telegraf collected metric
# TYPE mongodb_repl_getmores_per_sec untyped
mongodb_repl_getmores_per_sec{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 0
# HELP mongodb_repl_inserts_per_sec Telegraf collected metric
# TYPE mongodb_repl_inserts_per_sec untyped
mongodb_repl_inserts_per_sec{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 2
# HELP mongodb_repl_lag Telegraf collected metric
# TYPE mongodb_repl_lag untyped
mongodb_repl_lag{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 0
# HELP mongodb_repl_queries_per_sec Telegraf collected metric
# TYPE mongodb_repl_queries_per_sec untyped
mongodb_repl_queries_per_sec{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 0
# HELP mongodb_repl_updates_per_sec Telegraf collected metric
# TYPE mongodb_repl_updates_per_sec untyped
mongodb_repl_updates_per_sec{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 2478
# HELP mongodb_resident_megabytes Telegraf collected metric
# TYPE mongodb_resident_megabytes untyped
mongodb_resident_megabytes{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 8926
# HELP mongodb_ttl_deletes_per_sec Telegraf collected metric
# TYPE mongodb_ttl_deletes_per_sec untyped
mongodb_ttl_deletes_per_sec{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 0
# HELP mongodb_ttl_passes_per_sec Telegraf collected metric
# TYPE mongodb_ttl_passes_per_sec untyped
mongodb_ttl_passes_per_sec{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 0
# HELP mongodb_updates_per_sec Telegraf collected metric
# TYPE mongodb_updates_per_sec untyped
mongodb_updates_per_sec{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 0
# HELP mongodb_vsize_megabytes Telegraf collected metric
# TYPE mongodb_vsize_megabytes untyped
mongodb_vsize_megabytes{dc="DC",env="prod",host="hostname",hostname="127.0.0.1:27017"} 11720

Expected behavior:

state="SECONDARY" and member_status="SEC" will be shown as tags

Actual behavior:

none of states is shown

[danielnelson]

This is because these are string fields, prometheus has only float values. Are you thinking we should add string fields as labels?

[freeseacher]

in my opinion that info are tags for prometheus. and later in query i want to search them as a tags not as values

[danielnelson]

If we move string fields to labels, we need to consider it for all metrics. Would love to hear from other prometheus users too.

[freeseacher]

probably there we have 3 options here

1. ignore string values. like now.
2. convert string values to maps and send them to most outputs as strings and to some as ints
3. make strings values tags.

to make a decidion we have to understand which values are currently lost in case of luck of convertion.

[danielnelson]

1. convert string values to maps and send them to most outputs as strings and to some as ints

I'm not sure what you mean by this, can you explain in a little more detail?

[danielnelson]

One more idea that I think is being worked on, a processor that can promote a field to tag and vice versa.

[freeseacher]

https://github.com/influxdata/telegraf/blob/e7f9db297eb309f722597023a73d175e5ff81431/plugins/inputs/mongodb/mongostat.go#L559

looks a little weird for me.
probably should be replaced with something like map[int]string

[freeseacher]

any news?

[freeseacher]

ping?

[danielnelson]

Hey @freeseacher, I'm actually in this code right now, I will go ahead and add the string fields as labels and we can see how it ends up looking.

[freeseacher]

oh! cool. :)