Core: August integration makes too many requests and hits rate limits

[bdraco]

The problem

It appears august has recently implemented stricter rate limits.

Environment

Home Assistant 0.104.3

• Home Assistant release with the issue:
• Last working Home Assistant release (if known):
• Operating environment (Hass.io/Docker/Windows/etc.):
• Integration causing this issue: august
• Link to integration documentation on our website: https://www.home-assistant.io/integrations/august/

Problem-relevant configuration.yaml

n/a

august:
 login_method: email
 username: !secret august_username
 password: !secret august_password

Traceback/Error logs

2020-02-04 20:17:01 ERROR (SyncWorker_14) [homeassistant.components.august] Request error trying to retrieve door status for [removed] Garage Side Door. 429 Client Error: Too Many Requests for url: https://api-production.august.com/locks/[removed]/status
2020-02-04 20:17:01 ERROR (SyncWorker_14) [homeassistant.components.august] Request error trying to retrieve door details for [removed] Garage Side Door. 429 Client Error: Too Many Requests for url: https://api-production.august.com/locks/[removed]
2020-02-04 20:17:01 ERROR (MainThread) [homeassistant.helpers.entity] Update for lock.alexander_front_door fails
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 279, in async_update_ha_state
    await self.async_device_update()
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 461, in async_device_update
    await self.hass.async_add_executor_job(self.update)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/august/lock.py", line 58, in update
    self._lock.device_id, ActivityType.LOCK_OPERATION
  File "/usr/src/homeassistant/homeassistant/components/august/__init__.py", line 218, in get_latest_device_activity
    activities = self.get_device_activities(device_id, *activity_types)
  File "/usr/src/homeassistant/homeassistant/components/august/__init__.py", line 209, in get_device_activities
    self._update_device_activities()
  File "/usr/src/homeassistant/homeassistant/util/__init__.py", line 240, in wrapper
    result = method(*args, **kwargs)
  File "/usr/src/homeassistant/homeassistant/components/august/__init__.py", line 229, in _update_device_activities
    self._access_token, house_id, limit=limit
  File "/usr/local/lib/python3.7/site-packages/august/api.py", line 188, in get_house_activities
    "limit": limit,
  File "/usr/local/lib/python3.7/site-packages/august/api.py", line 296, in _call_api
    response.raise_for_status()
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 429 Client Error: Too Many Requests for url: https://api-production.august.com/houses/[removed]/activities?limit=10

Additional information

[mikesalz]

Same issue here...

[gary-reyes]

Same issue here. Sometimes if I restarted, the integration fails to load.

[bdraco]

```diff --git a/custom_components/august/__init__.py b/custom_components/august/__init__.py
index 0bb0d63..ed0ea5a 100644
--- a/custom_components/august/__init__.py
+++ b/custom_components/august/__init__.py
@@ -36,8 +36,8 @@ AUGUST_CONFIG_FILE = ".august.conf"
DATA_AUGUST = "august"
DOMAIN = "august"
DEFAULT_ENTITY_NAMESPACE = "august"
-MIN_TIME_BETWEEN_UPDATES = timedelta(seconds=5)
-DEFAULT_SCAN_INTERVAL = timedelta(seconds=5)
+MIN_TIME_BETWEEN_UPDATES = timedelta(seconds=60)
+DEFAULT_SCAN_INTERVAL = timedelta(seconds=60)
LOGIN_METHODS = ["phone", "email"]

CONFIG_SCHEMA = vol.Schema(
```

Working again with the patched py-august branch and set to 60s

[atom-bomb]

forgive my ignorance, a couple of suggestions:

it looks like each time get_lock_status is invoked, lock status and lock details are updated for all locks in the house. that probably doesn't matter much for houses with only one lock, but for multiple locks that seems likely to hit a rate limit from the server.

also, looking at py-august, it seems like lock details are mostly a set of IDs that seem unlikely to change ever, let alone over the course of a day.
the only possibly interesting part of lock details is the battery level, expressed as a percentage. that number also seems to not change very often and from my experiments, is not accurate enough to be useful. still, getting lock details once a day should be sufficient.

[bittles]

index 0bb0d63..ed0ea5a 100644
--- a/custom_components/august/__init__.py
+++ b/custom_components/august/__init__.py
@@ -36,8 +36,8 @@ AUGUST_CONFIG_FILE = ".august.conf"
 DATA_AUGUST = "august"
 DOMAIN = "august"
 DEFAULT_ENTITY_NAMESPACE = "august"
-MIN_TIME_BETWEEN_UPDATES = timedelta(seconds=5)
-DEFAULT_SCAN_INTERVAL = timedelta(seconds=5)
+MIN_TIME_BETWEEN_UPDATES = timedelta(seconds=30)
+DEFAULT_SCAN_INTERVAL = timedelta(seconds=30)
 LOGIN_METHODS = ["phone", "email"]

 CONFIG_SCHEMA = vol.Schema(

Going to give this a shot

Ya, just tried that. Along with changing those same values in the individual components. No dice still getting error 429.

[josephkiser04]

Yep having same issue here.

[atom-bomb]

oh yeah, the house activity request doesn't catch the 429 exception. something like this will make it not abort, but it could still use some mechanism to back off and try again later.

@Throttle(MIN_TIME_BETWEEN_UPDATES)
    def _update_device_activities(self, limit=ACTIVITY_FETCH_LIMIT):
        """Update data object with latest from August API."""
        _LOGGER.debug("Start retrieving device activities")
        for house_id in self.house_ids:
            _LOGGER.debug("Updating device activity for house id %s", house_id)
            try:
                activities = self._api.get_house_activities(
                    self._access_token, house_id, limit=limit
                )
                device_ids = {a.device_id for a in activities}
                for device_id in device_ids:
                    self._activities_by_id[device_id] = [
                        a for a in activities if a.device_id == device_id
                    ]
            except RequestException as ex:
                _LOGGER.error(
                    "Request error trying to retrieve activity for house id %s. %s",
                    house_id,
                    ex,
                )
            except Exception:
                raise
        _LOGGER.debug("Completed retrieving device activities")

[CLARENNE-Q]

Do you plan to increase MIN_TIME_BETWEEN_UPDATES in HA-repo ?

[macfanatic77]

Any update on this issue? I'm having this same problem and I repeatedly keep getting 429 errors. I am trying to do a new HA install and it is frustrating that my front door doesn't work with HA now.

[bdraco]

I have a branch that appears to work around the issue combined with the patch I mentioned above
https://github.com/bdraco/py-august/tree/handle_429

diff -u homeassistant/components/august/binary_sensor.py ../alexander-homeassistant/custom_components/august/binary_sensor.py
--- homeassistant/components/august/binary_sensor.py    2019-12-30 13:09:13.043996768 -0600
+++ ../alexander-homeassistant/custom_components/august/binary_sensor.py    2020-02-05 23:04:08.539088561 -0600
@@ -11,7 +11,7 @@

 _LOGGER = logging.getLogger(__name__)

-SCAN_INTERVAL = timedelta(seconds=5)
+SCAN_INTERVAL = timedelta(seconds=60)


 def _retrieve_door_state(data, lock):
diff -u homeassistant/components/august/camera.py ../alexander-homeassistant/custom_components/august/camera.py
--- homeassistant/components/august/camera.py   2019-12-30 13:09:13.043996768 -0600
+++ ../alexander-homeassistant/custom_components/august/camera.py   2020-02-05 23:04:08.540088576 -0600
@@ -7,7 +7,7 @@

 from . import DATA_AUGUST, DEFAULT_TIMEOUT

-SCAN_INTERVAL = timedelta(seconds=5)
+SCAN_INTERVAL = timedelta(seconds=60)


 def setup_platform(hass, config, add_entities, discovery_info=None):
diff -u homeassistant/components/august/__init__.py ../alexander-homeassistant/custom_components/august/__init__.py
--- homeassistant/components/august/__init__.py 2020-01-09 11:26:08.397188795 -0600
+++ ../alexander-homeassistant/custom_components/august/__init__.py 2020-02-05 22:45:22.408682708 -0600
@@ -36,8 +36,8 @@
 DATA_AUGUST = "august"
 DOMAIN = "august"
 DEFAULT_ENTITY_NAMESPACE = "august"
-MIN_TIME_BETWEEN_UPDATES = timedelta(seconds=5)
-DEFAULT_SCAN_INTERVAL = timedelta(seconds=5)
+MIN_TIME_BETWEEN_UPDATES = timedelta(seconds=60)
+DEFAULT_SCAN_INTERVAL = timedelta(seconds=60)
 LOGIN_METHODS = ["phone", "email"]

 CONFIG_SCHEMA = vol.Schema(
diff -u homeassistant/components/august/lock.py ../alexander-homeassistant/custom_components/august/lock.py
--- homeassistant/components/august/lock.py 2019-12-30 13:09:13.043996768 -0600
+++ ../alexander-homeassistant/custom_components/august/lock.py 2020-02-05 23:04:08.540088576 -0600
@@ -12,7 +12,7 @@

 _LOGGER = logging.getLogger(__name__)

-SCAN_INTERVAL = timedelta(seconds=5)
+SCAN_INTERVAL = timedelta(seconds=60)


 def setup_platform(hass, config, add_entities, discovery_info=None):
@@ -88,7 +88,12 @@
         if self._lock_detail is None:
             return None

-        return {ATTR_BATTERY_LEVEL: self._lock_detail.battery_level}
+        attributes = {ATTR_BATTERY_LEVEL: self._lock_detail.battery_level}
+
+        if self._lock_detail._keypad_detail is not None:
+            attributes["keypad_battery_level"] = self._lock_detail._keypad_detail._battery_level
+
+        return attributes

     @property
     def unique_id(self) -> str:

[altran-git]

I have a branch that appears to work around the issue combined with the patch I mentioned above
https://github.com/bdraco/py-august/tree/handle_429

I tried cloning and installing your fork, but every time I restart Home Assistant, py-august gets replaced back to version 0.7.0

EDIT:
I forgot, the component lists the requirement for 0.7.0. I'll try removing the requirement.

EDIT2:
Got the fix running. So far so good. I will continue to use and monitor the issue. Thanks @bdraco

[size9ed]

Same problem here

[dannyk81]

Hey @bdraco, perhaps submit a PR with your suggested chnages so the maintainers can weigh in?

[bdraco]

Hey @bdraco, perhaps submit a PR with your suggested chnages so the maintainers can weigh in?

I’m not particularly happy with the slow update times. I’ll try to find some time to see if there is a way to reduce the number of requests first and increase the updates a bit before sending anything.

[bdraco]

Door status and lock status look like they can be combined so the same api isn't being hit over and over. I'm doing the refactoring here https://github.com/bdraco/august

[jhbcosta]

Same issue here!

[niemyjski]

Same issue that I'm seeing where it's not setup, but I don't see any errors in my ui which is kind of shocking considering it's reporting that my lock is unlocked and it's not (and it's not saying unavailable).

[bdraco]

It also looks like there is a bug that hammers the activities endpoint

[bdraco]

Each lock puts a duplicate in the house id
self._house_ids = [d.house_id for d in self._doorbells + self._locks]

Fix is here: https://github.com/bdraco/august/commit/c3f82ee14e12407dcb456b9aa677f28070673e37

Since it would append each house id, it would re-hit the activity endpoint once per device

[bdraco]

PR to fix is here: https://github.com/home-assistant/home-assistant/pull/31558

[gary-reyes]

@bdraco
I added your change as a custom component in my setup and I still get the failure on a reboot.
I have 4 locks, 1 door bell and a keypad.

Would adjusting the scan interval help with this? If it does, would it make sense to just make the scan interval be a configuration?

[bittles]

@bdraco
Getting the same error with the pull changes. Have 2 locks here.

[bdraco]

@bittles @gary-reyes did you update py-august as well?

[gary-reyes]

@bdraco no. The pull changes did not make it seem like there was a requirement for it.

[bittles]

@bittles @gary-reyes did you update py-august as well?

Ahh no I hadn't since the manifest.json wasn't in the PR.

[bittles]

@bittles @gary-reyes did you update py-august as well?

@bdraco
Ok working so far now after pulling your branch.

[bdraco]

https://github.com/bdraco/august & the PR (https://github.com/home-assistant/home-assistant/pull/31558) have been updated to work without needing to update py-august

[thatguy55]

Total newb question - where is a link to your repository so that I can install this add-on? If this is not the case which I believe it may not be how can I get it installed on a pi running HA?

[bdraco]

Total newb question - where is a link to your repository so that I can install this add-on? If this is not the case which I believe it may not be how can I get it installed on a pi running HA?

cd <dir you have home-assistant installed>
mkdir custom_components
cd custom_components
git clone https://github.com/bdraco/august.git

Restart

[mikesalz]

Total newb question - where is a link to your repository so that I can install this add-on? If this is not the case which I believe it may not be how can I get it installed on a pi running HA?

cd <dir you have home-assistant installed>
mkdir custom_components
cd custom_components
git clone https://github.com/bdraco/august.git

Restart

Follow-up newb question! Would this remain as a custom component indefinitely, or would it eventually (hopefully soon!) be incorporated into the main HA branch?

[bdraco]

Total newb question - where is a link to your repository so that I can install this add-on? If this is not the case which I believe it may not be how can I get it installed on a pi running HA?

cd <dir you have home-assistant installed>
mkdir custom_components
cd custom_components
git clone https://github.com/bdraco/august.git

Restart

Follow-up newb question! Would this remain as a custom component indefinitely, or would it eventually (hopefully soon!) be incorporated into the main HA branch?

Ideally once it’s fixed in main line home assistant you can delete the custom version

[mikesalz]

@bdraco You sir, are a genius. Works perfectly! Thanks for taking the time to make this work!

[thatguy55]

bdraco thank you!

[EspElement]

Thank you guys, I had started a problem under #31609 and they led me here. I have put the august-master under my custom-components folder, rebooted the pi, and I am still getting the error. Is that all I am supposed to do?

---

Scratch that, I just had to re-authenticate as well! Thanks so much draco!

---

So, it worked for about 5 minutes then I restarted the server and it is back to not working :-(

[jshayden]

This allows me to authenticate and call the lock and unlock services in dev tools, but state still shows "unavailable". Any ideas?

[bdraco]

@jshayden Please try pulling the latest version ef808aeebe8b0998ef186c0b25cb5ba7eaf03a6c

[niemyjski]

@bdraco would be easier to support hacs:) but thank you

[dannyk81]

@bdraco would be easier to support hacs:) but thank you

august is an official component of HASS which is being fixed here due to changes in August's API, the instructions from @bdraco are to apply the fix in the interim, before this PR is reviewed, merged and released, there's no point in adding it to HACS since this is not a community component.

[jpenyc]

@bdraco thanks!

[dasb00ter]

Thanks but how can I get the pop up to enter the code they send me? I dont have a pop up now?NVM it had been so long I forgot where to enter it