Core: NSW fuel sensors unavailable on boot (api rate error)

[GreenhouseAU]

The problem

nsw_fuel_station sensors regularly are not available on boot

A delay/retry timer should be added to this sensor. I have no programming experience so can't help :(

The only solution right now is to delay and then restart and hope that you can get all sensors up without triggering the API limit (10-20% of all boots from my 4 sensor experience).

Environment

• Home Assistant Core release with the issue: 0.116.4
• Last working Home Assistant Core release (if known): since inception of this sensor
• Operating environment (OS/Container/Supervised/Core): Container
• Integration causing this issue: nsw_fuel_station sensor
• Link to integration documentation on our website: https://www.home-assistant.io/integrations/nsw_fuel_station/

Problem-relevant configuration.yaml

  - platform: nsw_fuel_station
    #station_name: 'Kissing Pt Rd'  
    station_id: 17126
    fuel_types:
      - E10

  - platform: nsw_fuel_station
    #station_name: 'West Ryde'    
    station_id: 1441
    fuel_types:
      - E10

  - platform: nsw_fuel_station
    #station_name: 'Granville'   
    station_id: 17114
    fuel_types:
      - E10

  - platform: nsw_fuel_station
    #station_name: 'Granville2'   
    station_id: 2141
    fuel_types:
      - E10

Traceback/Error logs

Log error ends in:

requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.onegov.nsw.gov.au', port=443): Max retries exceeded with url: /FuelCheckApp/v1/fuel/prices/station/17126 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f6859395700>: Failed to establish a new connection: [Errno -3] Try again')

Additional information

[frenck]

Please fill out the Environment section of the issue template. Thanks! 👍

[GreenhouseAU]

Please fill out the Environment section of the issue template. Thanks!

Apologies - done.

[probot-home-assistant[bot]]

nsw_fuel_station documentation
nsw_fuel_station source
_{^{(message by IssueLinks)}}

[probot-home-assistant[bot]]

Hey there @nickw444, mind taking a look at this issue as its been labeled with an integration (nsw_fuel_station) you are listed as a codeowner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[nickw444]

Interesting! I haven't come across this mostly because I only monitor one or two so don't hit rate limiting. Can you provide more info about how many sensors you have? Perhaps also a configuration.yaml that is able to reproduce the issue consistently?

I think it would be more appropriate to instead implement a queue rather than retry to avoid unnecessary calls to NSW government data site. I'll take a deeper look once there's a configuration.yaml and help with a solution.

[GreenhouseAU]

Thanks for the quick reply Nick - detail added.

It has flown under the radar before: https://github.com/home-assistant/core/issues/23280

[nickw444]

Will try look at this on Wednesday 👍

[nickw444]

Hey @GreenhouseAU, I looked into this today and struggled to reproduce. I imagine this is only reproducible under the circumstance where you've restarted Home Assistant many times, maybe whilst changing old/adding new integrations?

I feel the solution to this problem is simply not worth the increased complexity it adds to the integration. Additionally, I'm not too much sure if it will even help - because if your instance has hit the rate limit, even with throttling and/or retries you'll still be waiting until the rate limit is lifted (no idea what the duration is), and therefore the next poll (1 hour) would hopefully pick up the data assuming the limit had been removed.

What are your thoughts? Maybe there is a scenario I am not considering and this is having a large impact on your workflow or setup, and if so I'd love to know more.

[nickw444]

Actually, looking into the code, I see the platform setup bails out early in the case of failure at setup - this definitely can be improved so that entities are still created

[GreenhouseAU]

No, it happens after no restarts in a month. Then I try to keep restarting to 'fix' the problem and roll the dice with api limits.

I think your solution of (possibly) no data for an hour is far more useful than 'no data until next restart/roll of the dice'. I'll owe you a beer for your fix :)

[nickw444]

The weekend got away from me, but will hopefully have some time this week to jump into this and build out a fix!

[nickw444]

Hey @GreenhouseAU, I've implemented a solution in #43039 – Turns out I re-discovered a much more optimal way to make the requests to the NSW Fuel API. In the client library I wrote to interface with the API, there is a way to request all fuel prices for all stations and all station data in a single request. So my solution is to make a single request to serve all sensors in Home Assistant - this should be the end of rate limiting once and for all!

[GreenhouseAU]

Legend! Thanks heaps 👍