Core: Yale Smart Alarm integration unresponsive

[petheridge]

Home Assistant release with the issue:
0.80.0

Last working Home Assistant release (if known):
0.79.0

Operating environment (Hass.io/Docker/Windows/etc.):
Hassbian - PI B+

Component/platform:

https://www.home-assistant.io/components/alarm_control_panel.yale_smart_alarm/

Description of problem:
After upgrading to 0.80.0 on 12 Oct the Yale Smart Alarm no longer is responding. The alarm is still working via the native Yale app.

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):

Traceback (if applicable):
```
Log Details (ERROR)
Sun Oct 14 2018 09:16:42 GMT+0100 (BST)

Update for alarm_control_panel.yale_smart_alarm fails
Traceback (most recent call last):
File "/srv/homeassistant/lib/python3.5/site-packages/urllib3/connectionpool.py", line 346, in _make_request
self._validate_conn(conn)
File "/srv/homeassistant/lib/python3.5/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
conn.connect()
File "/srv/homeassistant/lib/python3.5/site-packages/urllib3/connection.py", line 326, in connect
ssl_context=context)
File "/srv/homeassistant/lib/python3.5/site-packages/urllib3/util/ssl_.py", line 329, in ssl_wrap_socket
return context.wrap_socket(sock, server_hostname=server_hostname)
File "/usr/lib/python3.5/ssl.py", line 385, in wrap_socket
_context=self)
File "/usr/lib/python3.5/ssl.py", line 760, in __init__
self.do_handshake()
File "/usr/lib/python3.5/ssl.py", line 996, in do_handshake
self._sslobj.do_handshake()
File "/usr/lib/python3.5/ssl.py", line 641, in do_handshake
self._sslobj.do_handshake()
socket.timeout: _ssl.c:704: The handshake operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/srv/homeassistant/lib/python3.5/site-packages/requests/adapters.py", line 445, in send
timeout=timeout
File "/srv/homeassistant/lib/python3.5/site-packages/urllib3/connectionpool.py", line 639, in urlopen
_stacktrace=sys.exc_info()[2])
File "/srv/homeassistant/lib/python3.5/site-packages/urllib3/util/retry.py", line 357, in increment
raise six.reraise(type(error), error, _stacktrace)
File "/srv/homeassistant/lib/python3.5/site-packages/urllib3/packages/six.py", line 686, in reraise
raise value
File "/srv/homeassistant/lib/python3.5/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)
File "/srv/homeassistant/lib/python3.5/site-packages/urllib3/connectionpool.py", line 349, in _make_request
self._raise_timeout(err=e, url=url, timeout_value=conn.timeout)
File "/srv/homeassistant/lib/python3.5/site-packages/urllib3/connectionpool.py", line 309, in _raise_timeout
raise ReadTimeoutError(self, url, "Read timed out. (read timeout=%s)" % timeout_value)
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='www.yalehomesystem.co.uk', port=443): Read timed out. (read timeout=5)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/helpers/entity.py", line 221, in async_update_ha_state
await self.async_device_update()
File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/helpers/entity.py", line 349, in async_device_update
await self.hass.async_add_job(self.update)
File "/usr/lib/python3.5/asyncio/futures.py", line 380, in __iter__
yield self # This tells Task to wait for completion.
File "/usr/lib/python3.5/asyncio/tasks.py", line 304, in _wakeup
future.result()
File "/usr/lib/python3.5/asyncio/futures.py", line 293, in result
raise self._exception
File "/usr/lib/python3.5/concurrent/futures/thread.py", line 55, in run
result = self.fn(self.args, *self.kwargs)
File "/srv/homeassistant/lib/python3.5/site-packages/homeassistant/components/alarm_control_panel/yale_smart_alarm.py", line 84, in update
armed_status = self._client.get_armed_status()
File "/srv/homeassistant/lib/python3.5/site-packages/yalesmartalarmclient/client.py", line 54, in get_armed_status
alarm_state = self._post_authenticated(self._ENDPOINT_GET_MODE, params=params)
File "/srv/homeassistant/lib/python3.5/site-packages/yalesmartalarmclient/client.py", line 86, in _post_authenticated
response = requests.post(endpoint, params=params, cookies=self.cookies, timeout=self._DEFAULT_REQUEST_TIMEOUT)
File "/srv/homeassistant/lib/python3.5/site-packages/requests/api.py", line 112, in post
return request('post', url, data=data, json=json, *kwargs)
File "/srv/homeassistant/lib/python3.5/site-packages/requests/api.py", line 58, in request
return session.request(method=method, url=url, *kwargs)
File "/srv/homeassistant/lib/python3.5/site-packages/requests/sessions.py", line 512, in request
resp = self.send(prep, *send_kwargs)
File "/srv/homeassistant/lib/python3.5/site-packages/requests/sessions.py", line 622, in send
r = adapter.send(request, *kwargs)
File "/srv/homeassistant/lib/python3.5/site-packages/requests/adapters.py", line 526, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='www.yalehomesystem.co.uk', port=443): Read timed out. (read timeout=5)```

Additional information:
Not sure if this is related to the Yale server upgrade/ system outage on 12 October. Perhaps the Yale API being used by the HA integration has changed...

[petheridge]

@domwillcode are you aware of any API issue?

[domwillcode]

It unfortunately seems as though the API is redirecting to the Yale home page. As you say, it could be due to the upgrade but I'm concerned the API may have been discontinued.
I've sent an email to Yale to discuss an alternative solution and should hopefully hear back this week. I'll look into it, it could be worth disabling/removing the component within HA until I can find a solution. @balloob, thoughts?

[atcapollo]

Think I have same issue. Hass says:

- Platform not found: alarm_control_panel.yale_smart_alarm

[phairplay]

Hi,
Is there any further update on this. My has been working on .80 until yesterday morning.

[itshusi]

Mine is also not working .80.3

[domwillcode]

Oddly mine hasn't been working for a week or so now. I haven't had a response from Yale yet. I asked them and ASSA ABLOY customer services again yesterday but I suspect they are probably quite busy with their recent down time. I'll keep pushing for it (and eventually try their social media if I don't have any luck) so watch this space.

I can still log in to the API and make all of the arm/disarm calls as expected. The API is just giving a "panel offline" error when I try to get or set status.

[evoio]

I do not know when exactly it was implemented - I would imagine it was the cause of the app being unusable a week or so ago though - but the newest version of the app is using a different API to the one being used by Home Assistant.

The new API is "https://mob.yalehomesystem.co.uk/yapi/api". It is structured differently and when I tried to log into it without copying all of the headers (including a pre-existing auth token) it failed saying "invalid client".

From what I have seen, I would suggest that there is no immediate desire by Yale to make this a publicly accessible / documented API.

[evoio]

The following is a packet capture of logging in through the app:

• The first blurred out block is a token that may or may not be embedded in the app - I wasn't sure so I blurred mine out
• Username and password are passed as form data
• The server responds with an access_token and refresh_token. I am not certain how the refresh is used, however the access token is passed as the Authorisation for subsequent requests - see below.

The following is a packet capture of setting the alarm through the app:

• The bearer token is the same that was returned from the login request
• I'm not sure what value the returned token has, but I blurred it out nontheless

I have been able to replicate logging in using postman by copying the headers from the first request. I can also do get requests, e.g. get the alarm mode. However, all attempts to make updates have not worked; they all timeout after 30s. It is just cut off the bottom of the capture from the app, but it returns a field called "time" and a value of a little over 1, so my assumption is it should only take 1s to execute. I am not sure what is missing.

[phairplay]

Hi guys,
While I can’t help get this working again I’m more than happy to be a tester if needed. I would love to have this working again.

I don’t mind not being able to turn the Alarm off or on, it’s the status that’s most useful for my automations

[domwillcode]

Awesome, thanks for the PCAP. I've been meaning to grab one myself. I'll try and experiment with it and see if I can get this working again using the new API. I've still had nothing back from Yale and very much doubt I will get any official support, I'll keep trying.

[evoio]

@domwillcode what would be interesting is if the first token used to login is the same for everyone. As I said I don't know if I should post the whole thing here. However mine starts with "VnVWWDZYVjlXSUNz" and ends with "GR0bmNndQ==" do others have the same?

[phairplay]

@evoio
I'm happy to check
How are you able to obtain the token?

[domwillcode]

Yeah, it's oauth2 so it's most likely the client application auth token assigned generated by the provider and specific to the yale app 'client' and should be common across everyone's instance. What did you use to capture the traffic on android? From what I understand, if that token is the client token and not some generated token as a result, it should be transmitted over SSL as it should be private to the application and server. Odd.

[domwillcode]

Looks like SSL Capture did the trick and intercepts the SSL traffic with a vpn proxy, awesome (who knew android was this insecure...). It's the same key. I've got a 55eGhrc0U in the middle. It must be the client application auth token we can use to auth. I'm mobile at the moment but I'll experiment with this once I'm near a computer.

[domwillcode]

So it seems the mode setting issue is due to the proxy on the main DNS timing out. However, when you make the original request it returns a host:port combo (6013 in my case) which you can then use for future requests. This allows you to use your generated token to set the mode with a POST to ...:6013/yapi/api/panel/mode/.

A GET to the same endpoint will return the correct result. Oddly, a GET to the proxy will return an incorrect state of the alarm (which happens to be the same state the old API would return after it stopped working).

We're getting somewhere. It'd just be nice if Yale/ASSA supplied an official app token for the HA client...

[evoio]

Great spot @domwillcode. I added the port and I can now set my alarm using it.

Yeah some sort of support for third party integrations would be great. Considering this is the revamp - and after the experience of this update I imagine they won't want any major updates any time soon - it seems pretty closed.

[evoio]

Using the refresh token is pretty straight forward it turns out. You hit the same endpoint as login but you use a "grant_type" of "refresh_token" and replace username / password with refresh_token=. This gives a new refresh token that can be used to refresh again. This would need to be done less than every 10 hours.

[phairplay]

Hi guys,
Have you had any more success with this?
:)

[TusharaFernando]

Managed to get this to work but also had to snoop the initial token. Unsure if that expires

[evoio]

I have something working, but unfortunately its in Node.js...

Without trying to reverse engineer the app - which I initially tried before I realised I don't really know Android - we can't be sure of the life of the token. My token has stayed the same since I took the captures a week ago. However we can't say that it is either not time-based - with a relatively long duration - or that Yale might decide to change it in the future - likely without warning because well, its not exactly a supported API.

[phairplay]

If this new method involves a few unofficial steps. Could it be created as a custom component instead of an official HA component?

[evoio]

I'd argue that it is no less supported than the old API. It too is based on an undocumented API, likely reverse engineered as the new one has been. As previously discussed what we need is Yale to deliver a properly supported API like every respectable smart device manufacturer should.

However it is done, the component in its current form is defunct. The API it is based on has been decommissioned so either it needs to be updated to the new API or removed. Creating a new component is not strictly necessary. Replacing it with one based on the new API would not be a breaking change.

[domwillcode]

Indeed, the only issue with this new API is the client key which isn't exactly publicised...

Changing the component to use the new API calls shouldn't be an issue and would probably benefit from the refresh token rather than constant logins, too. Although that'd require a slight modification to the HA component rather than just the client.
I haven't had much time to look into updating the client over at https://github.com/domwillcode/yale-smart-alarm-client yet but I had issues getting the 'requests' library to play nice. Bare with me (or feel free to PR), I'll try and find some time in the coming weeks for an official/unofficial component :)

The real solution would be for Yale to provide a client application key specifically for HA but they won't talk to me. So unless someone fancies taking to Twitter/Social Media to get their attention...

[phairplay]

Hi guys
I contacted yale and all I got was "we'll pass this on to our tech team" and they tried to push the new version which links with alexa.

Have you guys had anymore success?

[vortex-uk]

Thanks for all your hard work trying to get this working again. Got my yale system and it worked initially for me on 0.81.6 but then just stopped working. As with phairplay - the alarm status is the most important for me as I had automations based on it and a status indicator integrated on the Lovelace floorplan which are now all missing :(
Thanks again.

[laf]

I have something working, but unfortunately its in Node.js...

would you be able to share this code so someone else might be able to port it to python?

[sparsons99]

Not sure if I'm just incredibly lucky, or misreading the above but mine seems to still be working. I am running 82.1 and getting errors in the log, but it still seems to be reading the state accurately. I have automations which use the state as a condition so I am very pleased it's working but now worried it's going to break at some point!

[crzg]

Are you still able to arm it? I am between ordering Yale and Honeywell Evohome. I currently have PhoneWatch (also called Sector Alarm across Europe https://www.sectoralarm.com/group-overview/sector-alarm-in-europe/) - if anyone has experience with either system I welcome your advice, and will in the meantime patiently await the hope that @evoio will share his code and save us all 😉

[evoio]

Sorry all, been very busy. I have pushed my code here:
https://github.com/evoio/bridge-yale

I've just tested it and it still works with the same tokens that were extracted way back when the switch happened. Couple things, 1) sorry it was originally only written with my consumption in mind, I'll try and add more docs, 2) I want to use MQTT + NodeRED for my logic layer and HA only for presentation so it includes some MQTT stuff.

To use it:
1) run "npm install"
2) set the following environment variables:

• USER_ID - yale email address
• PASSWORD - yale password
• MQTT_BROKER - can use mqtt://test.mosquitto.org for testing purposes, but do not use for prolonged periods since it is completely unsecured
  3) run "node app.js"

If all goes well you should see output something like this:

Logging in...
Logged in successfully!
Syncing state: 1543775936962
Running on http://0.0.0.0:3000
Connected to MQTT broker
Syncing state: 1543775952168
Syncing state: 1543775967171
(...)

This exposes an API on port 3000 and MQTT.

GET /alarm/mode
PUT /alarm/mode
{
"mode": "arm / disarm / home"
}

[laf]

@evoio Thank you so much.

I'm not much of a python coder but I'll start converting the existing hass plugin over to this. I'll keep everyone up to date here unless someone else comes in and does it before me :)

[vortex-uk]

@evoio thanks for your work - it’s much appreciated :)
Thanks @laf for converting - I’d be lost with trying to set it up manually!

[laf]

@vortex-uk Fancy giving my conversion a try? For now you need to replace the client.py file in your install, if it works for you I will create a pypi repo for it and submit a change to hass.

Firstly you need to find the existing file:

find / -wholename '*yalesmartalarmclient/client.py'

For me that was:

/srv/homeassistant/lib/python3.5/site-packages/yalesmartalarmclient/client.py

Now download the updated file, update the path in this command to match the output from your find:

wget https://gist.githubusercontent.com/laf/3e7fb81ba3ca348c4a307203834d418c/raw/aa2d5d81acd9c11b755f974909989c76051517c7/client.py -O /srv/homeassistant/lib/python3.5/site-packages/yalesmartalarmclient/client.py

Now restart hass and it should work before as per your previous config.

Let me know how you get on.

[phairplay]

@laf
could this be tested on hass.io i would like to try too :)

[laf]

@phairplay I’ve never used or looked into hass.io so I couldn’t tell you :(

[itshusi]

@laf I have tested your client.py update and it is now working great! Cheers for your hard work on fixing this!

[laf]

@itshusi thanks for testing. To be honest it was all @evoio really

@evoio, can I buy you a beer or two? If so, my email is in my profile, drop me a message and I’ll send some money you’re way.

[itshusi]

@laf In that case, thank you to you and @evoio 👍

[domwillcode]

@laf, great work! Apologies for dropping off the face of the earth, life took over. If you want to use the existing pypi repo and drop a pull request in, I'm happy to push it up/throw the permissions your way?

[laf]

@domwillcode PR submitted. Check it over though, whilst a couple of people have tested it - don't assume it's all correct :)

[domwillcode]

@laf That's awesome, thanks! It just so happens I had an hour spare, so I've poached what you've done (move the get into a new function), thrown it into the repo (https://github.com/domwillcode/yale-smart-alarm-client/tree/dev/update-new-yale-api) and run some tests on it. It works a charm :)

[laf]

@domwillcode Awesome :)

You doing the pull request to hass or do you want me to?

[domwillcode]

I'll bump it into hass in a mo! Would you mind having a butchers at https://github.com/domwillcode/yale-smart-alarm-client/pull/4?

[laf]

I'll bump it into hass in a mo! Would you mind having a butchers at domwillcode/yale-smart-alarm-client#4?

I'll just test it, give me 5 minutes.

[laf]

Looks good and works fine @domwillcode :)

[laf]

p.s I owe you a couple of beers as well, feel free to drop me a mail and I'll send you some beer money.

[domwillcode]

Nice one, thank you! I believe it is I who owe you the beers for your help, so same goes ;)
I've bumped it up on PyPi as 0.1.5 and I'll get a PR into hass shortly.

I'll probably look into making better use of the refresh token at some point (perhaps over the holidays), but for now, it's working well. Cheers.

[mcgurdan]

I updated the code for client.py having previously installed via pip. However when attempting to use a very simple test script, calling get_armed_status() it bums out with a JSON error - any ideas?:

python alarm_part_arm.py

Traceback (most recent call last):
  File "alarm_part_arm.py", line 6, in <module>
    client.get_armed_status()
  File "/usr/local/lib/python2.7/dist-packages/yalesmartalarmclient/client.py", line 43, in get_armed_status
    alarm_state = self._post_authenticated('/api/panel/mode/')
  File "/usr/local/lib/python2.7/dist-packages/yalesmartalarmclient/client.py", line 93, in _post_authenticated
    data = response.json()
  File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 892, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/__init__.py", line 518, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 370, in decode
    obj, end = self.raw_decode(s)
  File "/usr/local/lib/python2.7/dist-packages/simplejson/decoder.py", line 400, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.errors.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

[laf]

@mcgurdan I've posted in the issue you created with a debug to try

[mcgurdan]

So, my api request is resulting in a null data field. I am unsure what I am missing, sounds like a stupid question but which app are people using? iOS or Android? I am on iOS and I am wondering if it does something different than what has been found here.

[charlesastaylor]

Hi, I've been following along with this issue since Yale changed their API. Was going to help myself but never got round to it and was all to happy to see you guys had come up with a fix! I've tested version 0.1.5 of the python client locally and it all works great. Pending the now merged PR being released I've manually been using the updated version 0.1.5 with home assistant. When I first set it up it all worked fine but I noticed after coming back to it after some time it was completely unresponsive again. I believe it's down to access codes expiring and the client not noticing as the error response has changed. I'm working on a PR to fix it but am just curious that noone has had this happen to them?