Influxdb: Prometheus remote write missing measurements name
Bug report
Metrics send from prometheus to influxdb with native influx remote write have missing measurement name.
All metrics are saved by InfluxDB to _ measurement.
> select * from _ where __name__='up' and kubernetes_io_hostname='ctl03' limit 1;
name: _
time From To __name__ action addonmanager_kubernetes_io_mode alertmanager alertname alertstate annotation_io_kubernetes_container_hash annotation_io_kubernetes_container_ports annotation_io_kubernetes_container_restartCount annotation_io_kubernetes_container_terminationMessagePath annotation_io_kubernetes_container_terminationMessagePolicy annotation_io_kubernetes_pod_terminationGracePeriod annotation_kubernetes_io_config_seen annotation_kubernetes_io_config_source annotation_kubernetes_io_created_by annotation_prometheus_io_port annotation_prometheus_io_scrape annotation_scheduler_alpha_kubernetes_io_tolerations app backend beta_kubernetes_io_arch beta_kubernetes_io_os bind branch buildDate call client cluster_name code com_docker_stack_namespace com_docker_swarm_node_id com_docker_swarm_service_id com_docker_swarm_service_name com_docker_swarm_task_id com_docker_swarm_task_name com_mirantis_image_specs_calico_bird6Url com_mirantis_image_specs_calico_birdUrl com_mirantis_image_specs_calico_birdclUrl com_mirantis_image_specs_calico_confdUrl com_mirantis_image_specs_calico_felixImage com_mirantis_image_specs_docker_imgTag com_mirantis_image_specs_gerritChangeId com_mirantis_image_specs_gerritChangeNumber com_mirantis_image_specs_gerritPatchsetNumber com_mirantis_image_specs_gerritPatchsetRevision com_mirantis_image_specs_gerritProject com_mirantis_monitoring compiler container_image container_name container_version contentType controller controller_revision_hash cpu database device dialer_name direction dockerVersion endpoint engine engine_host event exe f64 fstype gitCommit gitTreeState gitVersion goVersion goversion grpc_code grpc_method grpc_service grpc_type handler host id image input instance integration interface interval io_kubernetes_container_logpath io_kubernetes_container_name io_kubernetes_docker_type io_kubernetes_pod_name io_kubernetes_pod_namespace io_kubernetes_pod_uid io_kubernetes_sandbox_id ip_version job k8s_app kernelVersion kubernetes_io_hostname le listener_name major method minor name namespace network node_attribute_master node_host node_id node_name node_role_kubernetes_io_master node_role_kubernetes_io_node operation_type osVersion output path pattern pid platform pod_name pod_template_generation pod_template_hash port process_name protocol_instance proxy quantile queue reason refid region remote resource retentionPolicy revision role route_type rule_type scope scrape_job server service severity slice socket state state_prefix status stratum subresource sv table type unit url username verb version walPath
---- ---- -- -------- ------ ------------------------------- ------------ --------- ---------- --------------------------------------- ---------------------------------------- ----------------------------------------------- --------------------------------------------------------- ----------------------------------------------------------- --------------------------------------------------- ------------------------------------ -------------------------------------- ----------------------------------- ----------------------------- ------------------------------- ---------------------------------------------------- --- ------- ----------------------- --------------------- ---- ------ --------- ---- ------ ------------ ---- -------------------------- ------------------------ --------------------------- ----------------------------- ------------------------ -------------------------- ---------------------------------------- --------------------------------------- ----------------------------------------- ---------------------------------------- ------------------------------------------ -------------------------------------- --------------------------------------- ------------------------------------------- --------------------------------------------- ----------------------------------------------- -------------------------------------- ----------------------- -------- --------------- -------------- ----------------- ----------- ---------- ------------------------ --- -------- ------ ----------- --------- ------------- -------- ------ ----------- ----- --- --- ------ --------- ------------ ---------- --------- --------- --------- ----------- ------------ --------- ------- ---- -- ----- ----- -------- ----------- --------- -------- ------------------------------- ---------------------------- ------------------------- ---------------------- --------------------------- --------------------- ------------------------ ---------- --- ------- ------------- ---------------------- -- ------------- ----- ------ ----- ---- --------- ------- --------------------- --------- ------- --------- ------------------------------ ---------------------------- -------------- --------- ------ ---- ------- --- -------- -------- ----------------------- ----------------- ---- ------------ ----------------- ----- -------- ----- ------ ----- ------ ------ -------- --------------- -------- ---- ---------- --------- ----- ---------- ------ ------- -------- ----- ------ ----- ------------ ------ ------- ----------- -- ----- ---- ---- --- -------- ---- ------- -------
1513343319131000000 up amd64 linux 0 ctl03 kubernetes-cadvisor ctl03 true region1
Looks like labels from ALL metrics are merged, so each metric have labels (set to null) from all other metrics...
When Prometheus Remote Storage Adapter is in place:
> select * from up limit 1;
name: up
time beta_kubernetes_io_arch beta_kubernetes_io_os instance job kubernetes_io_hostname node_role_kubernetes_io_master node_role_kubernetes_io_node region value
---- ----------------------- --------------------- -------- --- ---------------------- ------------------------------ ---------------------------- ------ -----
1513585977606000000 amd64 linux ctl03 kubernetes-node ctl03 true region1 0
IMHO native support for prometheus remote write should behave similar to remote storage adapter, and use label __name__ as measurement name
InfluxDB: 1.4.2-1
Ubuntu: 16.04
Prometheus: 2.0.0
How to reproduce:
- Configure remote write in prometheus:
remote_write:
# docker_remote_write
- url: http://INFLUXDB_HOST:8086/api/v1/prom/write?u=USER&p=PASS&db=DB
bkupidura
All 15 comments
Fixing this would also make the Influxdb prometheus backend backwards compatible with the remote_storage_adapter. This is essential to use the native backend as a drop-in replacement for the adapter.
fmoessbauer
on 2 Feb 2018
+1
gangsta
on 5 Feb 2018
+1
Crapworks
on 7 Feb 2018
Since i am not sure which post i should thumbs-up, here is the obligatory +1 from me.
simonszu
on 19 Feb 2018
Hi there!
Is someone taking care of this? I might give it a try, but I am totally new to Influxdb codebase and Golang. From my initial findings, the bug is localized in prometheus/converters.go. We should adapt the write function and probably also the read. I don't know what the other fonctions do.
I am not the maintainer, but I guess any help is welcomed :)
Nodraak
on 22 Feb 2018
@Nodraak do it, try it. i think my skills are as new as yours, an i currently don't have the time, but i, for myself, am waiting for someone to fix this.
simonszu
on 23 Feb 2018
@pauldix If this change is made, please make sure to include me to doc it.
stevebang
on 1 Mar 2018
Originally, this was actually by design to have this future proof against IFQL. For now, the best way to query data from Influx that was written through the remote write gateway is to do it through the Prometheus remote read. In the new IFQL query language there's no distinction between measurements, tags, and fields. They're all just key and value pairs.
And __name__ is a tag key. Although now that I'm thinking a bit more about this it might be a better idea to actually have the measurement name match the __name__ in addition to having the __name__ tag. Would give the most flexibility while working with where things are going.
Thoughts?
pauldix
on 5 Mar 2018
+1 .
Any update on this issue?
Or any workaround?
jigarshahindia
on 7 Mar 2018
@pauldix From your comment I guess you are going to migrate to (some sort) of the prometheus metrics model, where __name__ is a tag, but also the name of the series? Is it intended to do this mapping transparently, or just by duplication of the value of the __name__ key?
If this mapping is done transparently (__name__ => measurement name), the remote storage endpoint of influx should be backwards compatible with the prometheus remote storage adapter. However this will break any measurements which already have a __name__ tag that does not match the measurement name. Nevertheless, I vote for this! This is a much smaller breaking change than by just using the __name__ value as measurement name.
In the meantime, is it possible to just add the measurement name by duplication of the __name__ value? This would be very helpful.
fmoessbauer
on 11 Mar 2018
In the meantime, is it possible to just add the measurement name by duplication of the __name__ value? This would be very helpful.
@fmoessbauer PR above does exactly this :) I can't speak to the suitability of this re IFQL mentioned above, but the duplication itself is trivial
roganartu
on 19 Mar 2018
Ok, I think I've found a solution to this problem, but bear with me for a little bit of extra history/detail.
The original reason I pushed everything into a single measurement named _ was partly because of IFQL, but also because of a limitation in our query engine. Prometheus has matchers against __name__ for != and !~ (which are standard tag matchers) which InfluxQL can't support against measurements. For the first version of the Prometheus remote read endpoint I had to use the InfluxQL engine since that's all that existed.
That limitation kept me from being able to put the Prometheus metric name in addition to having __name__ as a tag. The experimental Prom -> Influx gateway opted to throw errors for those cases: https://github.com/prometheus/prometheus/blob/master/documentation/examples/remote_storage/remote_storage_adapter/influxdb/client.go#L153-L164
However, since I wrote the original remote adapter, we've had a bunch of work done on creating a lower level query layer that bypasses the InfluxQL engine to get to storage. This is work directly related to IFQL, but I realized it could probably be used for the remote adapter. I've pushed a branch here: https://github.com/influxdata/influxdb/tree/pd-prometheus-to-new-storage
The good news is that this will probably be more performant and use less memory since it doesn't use the entirety of the InfluxQL query engine to process the request. I've tested it manually and it seems to work. That's experimental (just like this whole feature). I need to review with @stuartcarnie to figure out a less "ghetto-hack" solution.
If you're so inclined, I'd be interested in hearing if the branch works for you. This is a breaking change so you'd have to start with fresh data to really test it out.
pauldix
on 23 Apr 2018
This is fixed in 1.6. __name__ will now map to an InfluxDB measurement, and the __name__ tag is also available with the measurement name.
e-dard
on 23 Jul 2018
Is this new behaviour control by a setting or is it a new unconditional breaking change ?
Jean-Daniel
on 24 Jul 2018
I'm fairly new to InfluxDB, so please excuse me if this is obvious, but how can I take advantage of this change @e-dard? In Influx v1.6.1 (in a new database, with data fresh out of Prometheus) I still only see one measurement (_), and I cannot use Prometheus metric names as measurements in Influx queries. That is, I still have to write queries like select * from _ where __name__='up' and kubernetes_io_hostname='ctl03' limit 1; instead of the preferable select * from up limit 1;.
What am I missing?
Like @Jean-Daniel I would also like to know if this behaviour is on by default or if we need to opt-in with a setting.
AndrewC-B
on 13 Aug 2018
Related issues
jayannah
路
3Comments
deepujain
路
3Comments
udf2457
路
3Comments
dtouzeau
路
3Comments
shilpapadgaonkar
路
3Comments
Most helpful comment
Ok, I think I've found a solution to this problem, but bear with me for a little bit of extra history/detail.
The original reason I pushed everything into a single measurement named
_was partly because of IFQL, but also because of a limitation in our query engine. Prometheus has matchers against__name__for!=and!~(which are standard tag matchers) which InfluxQL can't support against measurements. For the first version of the Prometheus remote read endpoint I had to use the InfluxQL engine since that's all that existed.That limitation kept me from being able to put the Prometheus metric name in addition to having
__name__as a tag. The experimental Prom -> Influx gateway opted to throw errors for those cases: https://github.com/prometheus/prometheus/blob/master/documentation/examples/remote_storage/remote_storage_adapter/influxdb/client.go#L153-L164However, since I wrote the original remote adapter, we've had a bunch of work done on creating a lower level query layer that bypasses the InfluxQL engine to get to storage. This is work directly related to IFQL, but I realized it could probably be used for the remote adapter. I've pushed a branch here: https://github.com/influxdata/influxdb/tree/pd-prometheus-to-new-storage
The good news is that this will probably be more performant and use less memory since it doesn't use the entirety of the InfluxQL query engine to process the request. I've tested it manually and it seems to work. That's experimental (just like this whole feature). I need to review with @stuartcarnie to figure out a less "ghetto-hack" solution.
If you're so inclined, I'd be interested in hearing if the branch works for you. This is a breaking change so you'd have to start with fresh data to really test it out.