Telegraf: openweathermap update interval

[spcqike]

Hello,

i can't find anything about it, but it seems that there is a problem with the update interval of openweathermap.
the weather (now) is updated once per hour. when i run "telegraf --test" i get something like

weather,city=Berlin,city_id=2950159,country=DE,forecast=*,host=XXX cloudiness=0i,humidity=27i,pressure=1021,rain=0,sunrise=1561430648000000000i,sunset=1561491212000000000i,temperature=33.34,visibility=10000i,wind_degrees=150,wind_speed=4.6 1561471760000000000

when i use the http API (https://api.openweathermap.org/data/2.5/weather?id=2950159&APPID=XXXX) i get
{"coord":{"lon":13.41,"lat":52.52},"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01d"}],"base":"stations","main":{"temp":306.52,"pressure":1021,"humidity":27,"temp_min":304.82,"temp_max":309.26},"visibility":10000,"wind":{"speed":4.6,"deg":150},"clouds":{"all":0},"dt":1561472392,"sys":{"type":1,"id":1275,"message":0.008,"country":"DE","sunrise":1561430648,"sunset":1561491212},"timezone":7200,"id":2950159,"name":"Berlin","cod":200}

the https API is updated every 10 minutes, whereas telegraf gets an update every hour.

also, the times (sunrise, -set, now) have a weird format. so you i need to use "math (/1000000)" to get a readable value.

i use the free API and influxDB. everything is up to date.

do i miss something?

Thank you

[danielnelson]

Thanks for reporting this, it seems that it has something to do with the use of the /data/2.5/group API to get multiple locations in one query:

Single location (we don't use this)

https://api.openweathermap.org/data/2.5/weather?id=5391959&APPID=XXXX
{   
    "dt": 1561501144,

Multiple locations, what the plugin uses:

https://api.openweathermap.org/data/2.5/group?id=5391959&APPID=XXXX
{
    "list": [
        {
            "dt": 1561500568,

We might need to consider switching to the single location per call API, though it is probably worth asking OpenWeatherMap support about this first. Can you verify this is the issue you are seeing?

also, the times (sunrise, -set, now) have a weird format. so you i need to use "math (/1000000)" to get a readable value.

In Telegraf we store time as nanoseconds since the unix epoch, this level of resolution doesn't really help in openweathermap which only updates every 10 minutes by default, but it's used to be consistent with other plugins. If you are using Grafana you can set the axis units to "time, nanoseconds"

[spcqike]

Hello,

thanks for your anwser.
yes, the time reported from the group api is the same, as telegraf --test gets. so this might be the problem.

maybe it's possible to give a "switch" or path via telegraf.conf, like you can allready set the base url. so the user can decide wether to use the group or singel call.

e* as workaround i'm using a separate [inputs.http] for now. but it would be great to see this interval within the openweathermap plugin :)

[danielnelson]

I posted [this question] with OpenWeatherMap, let's give them at least a week and based on what we learn we can decide our next step.

[spcqike]

Hello danielnelson,

so it seems, that this is expected behavior for group requests.
so, as a free user can generate 60 API calls per minute, maybe its best to switch to single query requests, as long as we don't monitor more than 600 locations :)

or are there any disadvantages about multiple single requests?

[danielnelson]

Yes, I agree. Downside is just more requests/api calls but I think for most people this will be best and I don't see this plugin as being the right solution for exporting tons of locations.

[jimmybrancaccio]

Glad to see I wasn't the only one having the issue. I've just switched over to inputs.http for now.

[danielnelson]

@regel What do you think on this one?

[regel]

Hi Daniel, ok. Reading OWM's response

Database for Group request has an update once an hour for Startup and Free, so this is expected behavior.

An easy route is to document this behavior, use a 1-hour interval as the default value in the configuration, and call the single location API if the cities list parameter contains one unique item. Then we can duplicate the [[inputs.openweathermap]] section if a lower time resolution is required for more than one city.

[[inputs.openweathermap]]
  ## OpenWeatherMap API key.
  app_id = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

  ## Query interval; OpenWeatherMap weather data is updated every hour for group requests,
  ## and every 10 minutes for single city queries.
  ## Set the interval to 10 minutes if your cities list contains one unique entry.
  interval = "1h"

warning: the single city API uses a different JSON schema.

[OlliL]

Is there any update on this? The one hour update for the group list is a quite low resolution :( It took me quite some time to figure out why my influxdb does not received new values after each request.....

[fowlerjk]

Reading the thread, it looks as though a single-location API was hinted at, but not explained. Could someone expand on this?

[OlliL]

I just replaced the plugin with a small shellscript which I run via crontab every 5 minutes to get more frequent data and push it into the InfluxDB running on localhost.

#!/bin/sh

APPID=$1
PLACE=$2
INFLUX_DB=$3
INFLUX_USER=$4
INFLUX_PASS=$5

/usr/local/bin/curl 'http://api.openweathermap.org/data/2.5/weather?q='$PLACE'&units=metric&APPID='$APPID | \
        /usr/local/bin/jq -r '.main.temp, .main.humidity, .main.pressure, .dt' | \
        awk ' NR==1 {printf("'$PLACE' temp=%s,",$0)}
              NR==2 {printf("humidity=%s,",$0)}
              NR==3 {printf("pressure=%s",$0)}
              NR==4 {printf(" %d",$0*1000000000)}' | \
        /usr/local/bin/curl -i -XPOST 'http://localhost:8086/write?db='$INFLUX_DB -u $INFLUX_USER:$INFLUX_PASS --data-binary @-

[KLVN]

This is my workaround using Telegraf:

# Openweathermap
[[inputs.http]]
  urls = [
    "http://api.openweathermap.org/data/2.5/weather?id=XXXXX&units=metric&appid=XXXXX"
  ]
  name_override = "weather"
  method = "GET"
  interval = "10m"
  timeout = "15s"
  data_format = "json"

Notice that fields will have different names, e.g. main_temp instead of temperature and clouds_all instead of cloudiness.

[g-linde]

I've been using the Telegraf workaround for some time, however I'm calling the (newer?) One Call API instead. That's working well with the caveat that I am not getting the same set of tag keys.
Might it be possible to support that API in the future? I'm not Go-fluent so I likely can't do much to help create that, however I'd love to test it out if someone else can do that part.

[hansoostendorp]

Could at least the default config be updated ?

In the default config its mentioned that OpenWeatherMap updates their weather data every 10minutes. It took me a while to figure out this is not true, it only updates every hour because this plugin uses the group url. Thanks to this bug report I finally figured out what was happening. I am using the workaround @KLVN posted, thanks! I can live with the different field and key names.

[hansoostendorp]

This is my workaround using Telegraf:

# Openweathermap
[[inputs.http]]
  urls = [
    "http://api.openweathermap.org/data/2.5/weather?id=XXXXX&units=metric&appid=XXXXX"
  ]
  name_override = "weather"
  method = "GET"
  interval = "10m"
  timeout = "15s"
  data_format = "json"

Notice that fields will have different names, e.g. main_temp instead of temperature and clouds_all instead of cloudiness.

Thanks!

I am using your workaround now. I only added

json_time_key = "dt"

this way the timestamp Openweathermap returns is used in stead of the timestamp the http call is made

[KLVN]

In the default config its mentioned that OpenWeatherMap updates their weather data every 10minutes.

It used to be 10 minutes. I recently switched to a different method for gathering OpenWeatherMap data and just noticed it too.

---

this way the timestamp Openweathermap returns is used in stead of the timestamp the http call is made

Good point!