Core: Blink (multi hub) second alarm_control_panel reverts to disarmed status when armed

[TazUk]

Home Assistant release with the issue:
v0.89.2

Last working Home Assistant release (if known):
v0.88.1
NB: seems to be related to fixes for service endpoint changes under issue 21559

Operating environment (Hass.io/Docker/Windows/etc.):
Ubuntu 18.04.1 Python 3.6 venv

Component/platform:
Blink camera: https://www.home-assistant.io/components/blink/

Description of problem:
I am using two hubs under the same account, "Indoor" and "Outdoor" with Outdoor being purchased and added to the account much later chronologically than Indoor.
Both hubs are armed and disarmed (using alarm_control_panel.) based on presence with a short interval between them.
The Indoor hub behaves as expected.
However over the last week I have observed that homeassistant loses knowledge of the Outdoor hub's armed state between 5 and 7 minutes after it is armed. The hub does remain armed according to the Blink Android app and continues to record footage and send notifications when triggered by motion.
This prevents homeassistant users disarming the hub through the front end (automations still arm and disarm correctly).

This behaviour is only occurring for the second, Outdoor, hub. It occurs regardless of configured scan_interval, how the hub is armed (by automation or front end or via mobile app), which hub is armed first (when arming both) and whether armed separately to the other hub or as part of the same service call.

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

# blink cameras - 15 minute scan interval
blink:
  username: !secret blink_usr
  password: !secret blink_pwd
  scan_interval: 900
  binary_sensors:
    monitored_conditions:
      - motion_detected

NB: also tested with default 5 minute scan_interval.

Traceback (if applicable):

N/A

Additional information:
Arming automation action:

  action:
    - service: alarm_control_panel.alarm_arm_away
      data:
         entity_id: "alarm_control_panel.blink_sync"
    - delay: '00:00:{{ (range(45, 60)|random|int) }}'
    - service: alarm_control_panel.alarm_arm_away
      data:
         entity_id: "alarm_control_panel.blink_outdoor"

Disarming automation action:

  action:
    - service: alarm_control_panel.alarm_disarm
      data:
        entity_id: "alarm_control_panel.blink_outdoor"
    - delay: '00:00:{{ (range(15, 30)|random|int) }}'
    - service: alarm_control_panel.alarm_disarm
      data:
        entity_id: "alarm_control_panel.blink_sync"

[TazUk]

NB: I also reviewed the similar (closed) issue 18341 but found no url errors in my logs.

[fronzbot]

When arming the blink cameras, the state is not immediately updated. The alarm control panel updates the state by performing a system refresh (which is throttled according to your set scan_interval).

Now, when refreshing the sync module information, it relies on a network status api call which is internally throttled... so what I THINK is happening is that the first "update" of the sync module happens, but the second one is throttled so you have to wait at LEAST one scan_interval time before the second refreshes. So, if this is correct, I think that the following should work and correctly update the states (just test to see if it works, let me know. Don't keep it in though, it's a bit aggressive):

action:
    - service: alarm_control_panel.alarm_arm_away
      data:
         entity_id: "alarm_control_panel.blink_sync"
    - delay: '00:00:{{ (range(45, 60)|random|int) }}'
    - service: alarm_control_panel.alarm_arm_away
      data:
         entity_id: "alarm_control_panel.blink_outdoor"
    - delay: '00:00:10'
    - service: blink.blink_update
    - delay: '00:00:05'
    - service: blink.blink_update

If this doesn't work...well, 🤷‍♂️

If it DOES work, the fix on my end is pretty easy to implement.

[TazUk]

Thanks for responding.
I'm somewhat wary of attempting the test given I've been banned before, and because I've seen this issue occur and persist over a very long period that would exceed two configured scan intervals.

This morning we left the house at 11:40, triggering the arm automation for both hubs at 11:50.
Both hubs then showed "armed away" states from 11:50.
The Indoor hub retained this state constantly until our return at 13:25
The Outdoor hub reverted to "disarmed" at 11:56 and did not return to "armed away" before our return.
The Blink Android app showed both hubs as armed during this period.

This was with a configured scan_interval of 15 minutes.

If I understand your test correctly, the above would suggest that the issue is not tied to the throttling of the second hub.

[fronzbot]

If I understand your test correctly, the above would suggest that the issue is not tied to the throttling of the second hub.

That would be my assumption as well, but I don't know what alarm_control_panel does under the hood to intelligently comment.

Is this something that ALWAYS happens with the second hub, or is it intermittent? Also, when it happens, have you tried refreshing the front-end in your browser (or on your phone, or whatever) to make sure it wasn't the frontend reporting an old cached state? Worth a shot.

If it's not related to the throttling in blink, I don't know what could be wrong with the component, especially if the API calls are succeeding.

EDIT - wait, if this worked in the previous release, it HAS to be related to throttling (it's literally the only thing that changed with the arm/disarm routines). I really encourage you to try my solution once (and then remove it). It's really the only way to confirm or disprove my suspicion here.

[TazUk]

Is this something that ALWAYS happens with the second hub, or is it intermittent?
I first noticed it last week but based on testing today it always seems to be the second hub only.

Also, when it happens, have you tried refreshing the front-end in your browser (or on your phone, or whatever) to make sure it wasn't the frontend reporting an old cached state?
Checked on multiple clients using both the lovelace front end and the states tools

I really encourage you to try my solution once

I've tested this manually as follows:

1. Armed second hub manually (using lovelace entity-button card that calls the arming service)
   
   
   
   • hub arms as expected
   
   
   • state immediately shows as armed_away in UI and states tools
   
   
   • system shows as armed in blink android app
   
   
2. Waited for 7 minutes
   
   
   
   • hub still shows as armed in states, android app
   
   
3. Called service blink.blink_update from services tool
   
   
   
   • hub reverted to disarmed in UI and states
   
   
   • system showed as armed in the Blink android App
   
   
4. Waited 2 minutes
   
   
   
   • hub still shows as disarmed in states, UI
   
   
   • system still shows as armed in Blink android app
   
   
5. Called service blink.blink_update from services tool
   
   
   
   • hub still shows as disarmed in states, UI
   
   
   • system still shows as armed in Blink android app
   
   

So it looks as though the hub state moves to disarmed on the first blink.blink_update call and then remains there regardless of how many subsequent update calls are made.
No errors in logs.

[fronzbot]

hmm... that's really bizarre. I'll have to find where my old sync module is and get it up and running to test on my end to hopefully narrow down the problem.

Thanks for the detailed debug steps, it helps a lot 👍

[TazUk]

No worries - I'll leave it with you.
And thanks for all your hard work with the component.

[TazUk]

Have you had the opportunity to investigate this @fronzbot - is there any more info I could provide to help?

[fronzbot]

Unfortunately, I have not had much time to work on this due to real life interruptions :smile:

I have not forgotten about it, though, I'm hoping to get to it in the next couple weeks. At this point, I don't really need more info until I can get start debugging on my end. I'll let you know though!

[RodneyCapron]

Hello @fronzbot,

I am having a similar issue running version 0.91.4 of Home Assistant.

I have 2 Blink Sync Modules. The first Sync Module (Home Perimeter Cameras) works perfectly fine but the second Sync Module (Home Interior Cameras) disarms itself 4 minutes after arming it. It shows as disarmed in Home Assistant (and HomeKit) but it does not actually disarm the Sync Module (as shown in the Blink iOS App). I too, do not see any errors or issues in the log files.

Both Sync Modules were working perfectly in version 0.87.1 of Home Assistant.

Thank you for your assistance on this. Really appreciate all your hard work on this component. It's awesome being able to Arm/Disarm the Blink Cameras in conjunction with all my other HomeKit Automations. :)

[fronzbot]

Just as a quick update: I'm planning on carving out some time next week (FINALLY) to work on this. So, fingers crossed 🤞

@RodneyCapron thank you for the additional information! Every bit helps 👍

[RodneyCapron]

Hello @fronzbot - thank you very much for the update. 👍

Just a quick status update from this end: After running everything for a couple of weeks (and now on version 0.92.1) both hubs are behaving the same way. The difference is that the first (primary) hub keeps the state for longer periods of time (random for how long the state sticks).

If I can I help with any additional information, please let me know.

[fronzbot]

Sorry for the delay guys. FINALLY got around to looking at this and it appears like it's a very easy fix. My initial hunch (aggressive throttling) was correct, but there was no user-facing way to workaround that. The first sync module is updated immediately followed by the second which ALWAYS gets throttled. I just removed the throttling and it's working again.

I still need to do some testing (and implement other fixes) before the next release, but wanted to let you guys know this is almost finished. I'll link to this issue when I open the home assistant PR and will try my best to get it included in a bug-fix release 👍

[TazUk]

Great news, thanks for the update @fronzbot

[RodneyCapron]

Woohoo! This is really good news @fronzbot. Thank you very much for taking the time to research and correct the issue. 👍🏻