Sitespeed.io: Data export to Prometheus

[ronderksen]

My company uses (Prometheus)[https://prometheus.io] as a store for Grafana data. It would be great if the sitespeed data could be exported to Prometheus.

[tobli]

I've never used Prometheus but from what I've understood we'd push to a Pushgateway, right?

[brian-brazil]

From a quick peek at you're docs, this is intended to run as a cronjob? The pushgateway is appropriate then.

[beenanner]

@ronderksen I haven't tested it, but there is graphite-exporter available now until the pushgateway is built.

[thomasfr]

I would love to see prometheus support in sitespeed.io. The exporter would work maybe, but i do not want to run graphite AND prometheus just to export it from graphite to prometheus.

[beenanner]

@thomasfr the graphite-exporter runs independent and just mimics an endpoint for tools that want to send metrics to graphite. You would only need to run the exporter and prometheus. I'm ran it on the same box as prometheus to test and seems to work well. It does need more testing though. The initial launch of sitespeed.io 4.0 we are looking to support graphite and influxDb. Will keep this open as it would be nice to have native support built within after 4.0 goes live.

[beenanner]

For reference I think it would be a perfect use-case for an external plugin.

https://github.com/prometheus/pushgateway/blob/master/README.md

[thedaviddias]

@beenanner Any news / option from that feature?

[beenanner]

Definitely try the graphite exporter for a quick solution. The pushgateway I agree with Brian is the better more native longterm integration, but we will need to create a plugin to support pushing to it. Either of those will require you run a separate process as prometheus is designed to pull metrics and not be pushed to.

[beenanner]

Def join us on our slack channel and we can maybe collaborate if you are interested in diving into plugin creation for pushgateway support that would be neat to have as an option on top of the influxdb and graphite options.

[P6rguVyrst]

For reference I think it would be a perfect use-case for an external plugin.

https://github.com/prometheus/pushgateway/blob/master/README.md

I ended up here because someone in my office wanted to make use of this projects capabilities.

Can you tell me why you think why Prometheus integration is perfect use-case for an external plugin?
I haven't dwelt into the code base of this project yet, but wouldn't it be better to try and abstract specific implementations of TSDB's away and let the end user choose whether they choose to use Graphite, InfluxDB, Elasticsearch, Prometheus or something else?

Thank you for this amazing resource!

[soulgalore]

What @beenanner mean is that we don't want have too big codebase in the main project. We have Graphite because it was the first implementation (and the one we mainly use) and influx so we can make sure we have an alternative. Also including one/two is nice since it easier for people to start to use it.

Elasticsearch would be nice for some data that we collect (that doesn't work so good for TSDBs) but we haven't implemented anything.

[P6rguVyrst]

What @beenanner mean is that we don't want have too big codebase in the main project. We have Graphite because it was the first implementation (and the one we mainly use) and influx so we can make sure we have an alternative. Also including one/two is nice since it easier for people to start to use it.

Elasticsearch would be nice for some data that we collect (that doesn't work so good for TSDBs) but we haven't implemented anything.

Since I'm working with a different stack, I'm trying to avoid having to maintain multiple live TSDB's and worry about all the correlation and alerting logic in different systems.
However, I think I can have a disposable Graphite service with limited history in Kubernetes with Prometheus exporter exposing its data, which should be enough to avoid setting up a production grade Graphite cluster that would meet all the requirements.

Originally I was wondering if you have pondered the idea of abstracting away specific TSDB implementations and making it more flexible for end user to choose a TSDB themselves. The project could still have a default TSDB it uses out of the box, while enabling users to choose other technologies. I mentioned Elastic, because I saw an open PR with Elasticsearch plugin.

[beenanner]

I believe that's the idea around plugins. Abstracting away the TSDB implementation to make it generic for all TSDB will just result in losing any features for that specific TSDB that a user would want to leverage?

https://tsdbbench.github.io/Ultimate-TSDB-Comparison/

[soulgalore]

I'm pretty sure this should be done as a plugin outside of sitespeed.io, so I'm closing this for now.

[cablespaghetti]

In case someone else comes across this. I am using the graphite exporter for prometheus with sitespeed.io and here is my config:

mappings:
- match: 'sitespeed\.pageSummary\.([A-Za-z0-9-_]+)\.([A-Za-z0-9-_]+)\.([A-Za-z0-9-_]+)\.([A-Za-z0-9-_]+)\.(.*)'
  match_type: regex
  name: 'sitespeed_pageSummary_${5}'
  labels:
    site: ${1}
    page: ${2}
    browser: ${3}
    throttle_profile: ${4}
- match: 'sitespeed\.summary\.([A-Za-z0-9-_]+)\.([A-Za-z0-9-_]+)\.([A-Za-z0-9-_]+)\.(.*)'
  match_type: regex
  name: 'sitespeed_summary_${4}'
  labels:
    site: ${1}
    browser: ${2}
    throttle_profile: ${3}

And sitespeed.io flags:

--graphite.host graphite-exporter --graphite.port 9109 --graphite.namespace sitespeed 

[AtzeDeVries]

you can also use the influxdb_exporter, this mimics an influxdb server and can be scraped by prometheus
https://github.com/prometheus/influxdb_exporter