Wttr.in: Output Format: Prometheus

[mrueg]

Todo

• [x] observation time fields,
• [x] forecast data,
• [x] update README
• [ ] hourly forecast data

Details

I would be interested in a Prometheus-style format output similar to the json output that is already provided. https://prometheus.io/docs/concepts/data_model/

This can be used then to scrape wttr.in easily for monitoring systems.

[chubin]

It is an interesting idea.
Maybe we could add additional representation.

Would this work for the beginning?

jq -r 'keys_unsorted[] as $k | "\($k) \(.[$k])"'

[mrueg]

I don't think that it might work as the RHS needs to be numeric.
https://prometheus.io/docs/practices/naming/

temperature_celsius{location="Berlin"} 28.9

There's a python client library that probably is the best approach to do the output formatting base on the data object.
https://github.com/prometheus/client_python

[chubin]

Do I understand it right, that only current values make sense, and there is no need to export forecast values? At least at the beginning.

Question 2:
if you say that the RHS has to be numeric, do I understand it right, that non-numeric fields should not be exported at all?

[mrueg]

Forecast makes sense as well.
Non-numeric fields can be exported as dimensions.

  "current_condition": [
    {
      "FeelsLikeC": "21",
      "FeelsLikeF": "70",
      "cloudcover": "0",
      "humidity": "31",
      "localObsDateTime": "2020-05-22 11:44 AM",
      "observation_time": "09:44 AM",
      "precipMM": "0.0",
      "pressure": "1021",
      "temp_C": "21",
      "temp_F": "70",
      "uvIndex": 6,
      "visibility": "10",
      "weatherCode": "113",
      "weatherDesc": [
        {
          "value": "Sunny"
        }
      ],
      "weatherIconUrl": [
        {
          "value": ""
        }
      ],
      "winddir16Point": "SE",
      "winddirDegree": "130",
      "windspeedKmph": "19",
      "windspeedMiles": "12"
    }
  ],

could become (i left some of the items out):

FeelsLikeC{forecast="0h"} 21
FeelsLikeF{forecast="0h"} 70
cloudcover{forecast="0h"} 0
humidity{forecast="0h"}  31
localObsDateTime_epoch{forecast="0h"} 1590141901
observation_time_hour{forecast="0h"}s 584
weatherDesc{forecast="0h", description="Sunny"} 1
winddir16Point{forecast="0h", direction="SE"} 1

for the hourly prediction it could look like this:

FeelsLikeC{forecast="1h"} 21
FeelsLikeF{forecast="1h"} 70
cloudcover{forecast="1h"} 0
humidity{forecast="1h"}  31
localObsDateTime_epoch{forecast="1h"} 1590141901
observation_time_hour{forecast="1h"}s 584
weatherDesc{forecast="1h", description="Sunny"} 1
winddir16Point{forecast="1h", direction="SE"} 1

[chubin]

I've added the first trivial implementation of the Promethus output format.
Could you please take a look at it?

$ curl wttr.in/Regensburg?format=p1
FeelsLikeC(forecast="0h") 11
FeelsLikeF(forecast="0h") 53
cloudcover(forecast="0h") 100
humidity(forecast="0h") 97
precipMM(forecast="0h") 6.7
pressure(forecast="0h") 1013
temp_C(forecast="0h") 12
temp_F(forecast="0h") 54
uvIndex(forecast="0h") 1
visibility(forecast="0h") 6
winddirDegree(forecast="0h") 1
windspeedKmph(forecast="0h") 11
windspeedMiles(forecast="0h") 7
weatherDesc{forecast="0h", description="Moderate rain"} 1
winddir16Point{forecast="0h", description="N"} 1

What is missing at the moment:

• observation date/time, epoch;
• observation time, today (btw. why is it _hour in your example, if units are minutes?)
• forecast
• location name (do we need it? probably yes)
• localized weather condition name (irrelevant for wttr.in, aber ist durchaus sinnvoll für de.wttr.in)

If you are going to add it to some monitoring system (and if it will work as expected :),
could you please post a couple of screenshots here, just for better understanding
how it should look like, especially non-numerical fields

[mrueg]

Thanks a lot for the initial implementation!
I've sent in https://github.com/chubin/wttr.in/pull/468

What I would suggest is to make "EXPORTED_FIELDS" a map to comply with prometheus-style metric names: https://prometheus.io/docs/practices/naming/#metric-names

_hour in my example was a mistake, good catch. It should be minutes.

[chubin]

Merged, deployed, please test.

This with EXPORTED_FIELDS is a good idea; we should add it, I think

[mrueg]

Can confirm it's working, if you need a prometheus config to check with:

  - job_name: 'wttr_in'
    static_configs:
      - targets: ['wttr.in']
    metrics_path: '/Regensburg'
    params:
      format: ['p1']

Validated metrics output:

 curl "http://wttr.in:80/Regensburg?format=p1" 2>/dev/null | promtool check metrics                                                                                                                     


FeelsLikeC metric names should be written in 'snake_case' not 'camelCase'
FeelsLikeF metric names should be written in 'snake_case' not 'camelCase'
precipMM metric names should be written in 'snake_case' not 'camelCase'
uvIndex metric names should be written in 'snake_case' not 'camelCase'
weatherDesc metric names should be written in 'snake_case' not 'camelCase'
winddirDegree metric names should be written in 'snake_case' not 'camelCase'
windspeedMiles metric names should be written in 'snake_case' not 'camelCase'
FeelsLikeC no help text
FeelsLikeF no help text
cloudcover no help text
humidity no help text
precipMM no help text
pressure no help text
temp_C no help text
temp_F no help text
uvIndex no help text
visibility no help text
weatherDesc no help text
windspeedKmph metric names should be written in 'snake_case' not 'camelCase'
winddir16Point no help text
winddirDegree no help text
windspeedKmph no help text
windspeedMiles no help text

[chubin]

Merged, deployed, please test.

Do you have some nice screenshot with that how it is looking now?

[mrueg]

curl -s "http://wttr.in/Berlin?format=p1" | promtool check metrics

temperature_fahrenheit use base unit "celsius" instead of "fahrenheit"
temperature_feels_like_fahrenheit use base unit "celsius" instead of "fahrenheit"

looks good, I'd ignore the complaints around fahrenheit, there might be users needing it.

I don't have any dashboard setup yet, but simply ingested into prometheus.

[chubin]

Now one should add the time fields, and the forecast data,
extend the README and one can publish this new feature.
I wonder if anybody would use it, but the idea seems to be very interesting

[mrueg]

I added a few more bits here: https://github.com/chubin/wttr.in/pull/471

[chubin]

I've cleaned up the code a little bit. I hope you will like it

Again, not being able to test and probably we should consider using the prometheus python client to create those metrics instead of doing concatenations at some point

Yes sure, we can do it, but I am not sure that it will give us anything,
at least data conversion into the Prometheus format per se
is now implemented literally with one single line

Please test

[mrueg]

Thanks for the fix and cleanups, looks good to me! So the last remaining bit is transforming the hourly output, right?

[chubin]

Yes, I believe so. It should not bee too hard, because we already have almost everything for that.

What I still can't understand, why do you need Prometheus format (an how only this idea came on your mind?) if you don't even have a Prometheus dashboard?

[mrueg]

The next step is to expose it in grafana (that's where I'll create the dashboard in), but to get there I needed the exposed metrics exposed first and ingested in my prometheus instance.

[chubin]

Yes, but now it works, i.e. you can already add it to Grafana (presuming you already have it)?
Or do you want to add the hourly forecast first?

[chubin]

@mrueg Manuel, no news here?
What is yet to be done?

[mrueg]

@chubin sorry, for the late reply. Due to some private events I didn't have time to work on the grafana dashboard yet. For feature parity with the json output the hourly forecast would be perfect.

[chubin]

@mrueg Agree. I will add it