Core: Honeywell/somecomfort broken. Developer agrees. Let's solve this! Potential solution avenue included.

[techwood111]

Introduction

As it stands, Home Assistant is not properly working with Honeywell's Total Connect Comfort line of thermostats. This is Honeywell's oldest WiFi thermostat line, and is still being sold today, alongside more modern models. Readily available from Home Depot and Lowes, with a price tag of $99, they are a very popular thermostat. Because of the popularity of these thermostats, coupled with the relative ease of doing so, it should be a priority to update Home Assistant's ability to properly control these thermostats.

The Origin of the Problem

These thermostats are only modifiable through Honeywell's website at https://mytotalconnectcomfort.com, or by way of a smartphone app. Earlier this year, Honeywell made a modification to their website that wreaked havoc on many developed solutions for interacting with thermostats. Brad Goodman's code is a great example of one that suffered from the change.

You will find numerous references to this script all over the internet; frankly, I found it well before I found Home Assistant. It _was_ a very popular way to control Honeywell thermostats. On or about March of this year, Honeywell began adding a timestamp cookie, then sought to see that for subsequent http requests. I _believe_ this is what originally broke Brad Goodman's script, as well as somecomfort, and as a direct result, Home Assistant.

A Solution Has Been Proven

I modified Brad's script, and got it running. Let it be known that I am far from being a programmer, and only learned enough Python to get a working solution to the problem.

During this process, I met people on Reddit and other places who were also experiencing the Honeywell problem. "Our own" @sgauche was one of them. With some help from others, I got a much more "clean" script running to control the Total Connect Comfort thermostats. It uses the python "requests" library to _greatly_ simplify the process.

Note that the script is checking and displaying data for two "hard wired" thermostats, and that the lion's share of the script is merely printing out that data. Very slight modification of the script would allow it to be sent arguments, similar to the manner in which somecomfort (or most anything else) operates. (An important observation is that somecomfort no longer supports the --devices flag; that as well could be part of the reason it no longer operates as it once did within Home Assistant.)

Diagnosing the Problem(s)

The easiest problem to see is that Home Assistant can not modify a thermostat's setpoint using the slider, unless the thermostat is currently under a temporary hold. That is, someone must have pressed the temperature up/down button on the thermostat, either physically on the device, by using the phone app, or by using the mytotalconnectcomfort.com interface. Once "the seal has been broken," the Home Assistant sliders WILL be able to change the setpoint.

Another problem is that it is apparently not possible to control the thermostats if they are in any mode other than "cool." The other options are "heat," "auto," and "off." Of course, it is perfectly acceptable to not allow changes while "off" (even that is forbidden through Honeywell's interfaces), but most users would need control during the "auto" or "heat" settings.

Other problems exist, but are harder to concretely identify. For instance, on occasion, I find it impossible to adjust a cool setpoint, even if the temperature is being overridden via a temporary hold (see the first paragraph of this section.) However, I believe that if the root cause of these problems is solved, then ALL of the problems will go away.

In most (all?) cases, Honeywell is responding with a 500 (internal server error) error.

Where to Go from Here

First and foremost, I want to state that I believe this is an easy problem to solve, in the light of all the technical difficulties facing the community of developers. I really am blown away by how impressive the whole Home Assistant "system" is. You folks deserve some serious accolades.

While this problem does affect me, I have seen that it also is affecting others. One person said to me that they have abandoned any hope of Home Assistant handling their climate control altogether. Being that the Honeywell TCC platform is the most widely used internet-connected thermostat platform out there (sources available), and being that it has attracted people (at least "person") to using Home Assistant in the first place, I think it is critically important to get this component functioning properly.

I have already enlisted three individuals to personally help me with my scripting efforts to talk to these thermostats, both inside of Home Assistant and as stand-alone scripts. I have also spoken to Brad Goodman, who created the interesting push-button control mentioned above. Best of all, perhaps, is that Dan Smith, the developer of somecomfort, says:

I'm happy to fix, I just have to have a heads up about it :)

Between Dan, the three developers who have helped me try to "skin this cat" (shout out to Adam Garcia, Scott Griffin, and Karl Thornton), and the support of the Home Assistant leadership team, there is no doubt that the current problems interfacing with Honeywell can be quickly solved. As for me, I'll be glad to help in any way that I can, but I'll be the first to say that my technical skillset is quite small. With that having been said, I am happy to allow any of you access to my thermostats for testing. Honeywell makes that very easy; I just need to send you an invite through Honeywell, using your email address. (You then need to quickly and easily set up a Honeywell account, and accept the invite.)

Conclusion... and a REWARD?!?

I would be very appreciative if one of you fine fellows would volunteer to address this problem, leading the loose crew I've mustered who already have time invested in trying to solve it. I promise, to each person named above, and to the Home Assistant devs who aid in the solution, a pretty-darned-cool, one-of-a-kind commemorative token of my appreciation... or a six-pack of Charlotte, North Carolina's finest, if that is what you prefer. I'm leaving town tomorrow to go harvest honey on the family farm, and if you'd rather have some of that than anything else, that would work, too!

Thank you very much!

scott at sterling hyphen gaming d0t "kham"

---

Attachments

Aside from the above links, the following may also be helpful:
A very typical, pertinent Home Assistant error log
Another relevant error

[techwood111]

I should have mentioned that I am using the latest (well, downloaded 3 days ago, via the all-in-one) version of Home Assistant, but upgraded somecomfort to 0.3.1 from 0.2.1 in an attempt to fix the problems that exist; both act the same. Perhaps error line numbers shift, though, between the versions.

[jzwack]

OK. So @techwood111 rang.

Hi, I'm Joe, the API Evangelist for Honeywell's Connected Home products.

A few thoughts here:

• Looks like most of this is scraped off the mobile app/website
• We have an official API that supports OAuth2, instead of this sessions auth that the site uses
  
  
  
  • Understand that the portal for this isn't public yet, but I'm able to give access anyway
  
  
  • Development is done on a sandbox environment (not production) so someone will have to have test devices that aren't hooked up to real equipment. I have access to a few.
  
  
• The API supports Microsoft SignalR as a way of streaming real time updates and avoiding polling. I actually have a working version of this in Python.
• Last, the API structure for EU and US devices is quite different. So we have to better account for that.

[jzwack]

Oh, and let me add that some of this is just logic understanding. Especially between temporary hold/permanent hold/schedule etc. I can easily explain that, I have quite a bit of experience from our Amazon work.

Basically there are ways to make this more dynamic as well (valid modes, etc).

[techwood111]

Joe,

You are my new best friend! Solving this will be a HUGE help to a great number of people.

How do we begin? You have my email address. I am happy to add you as a controller of my thermostats, if that will help. Please let me know how best to proceed; I would imagine some of the other people who have sought to tackle this in the past would like to be involved.

[kk7ds]

I'm excited to hear about the new API, but I think it's probably better to keep using the browser API in somecomfort until it's generally available to everyone.

If you can tell me what I should be doing differently to avoid the rash of 500s from the honeywell servers I'd be glad to make the change. I can also try to sniff it out myself by watching the interaction of a current browser client, but it might be quicker to just tell me :)

[jzwack]

How often are you getting that 500 error? Right away? I don't see you doing any periodic polling in somecomfort, right?

Is the 500 only in the /CheckDataSession resource or every resource?

[kk7ds]

DEBUG:requests.packages.urllib3.connectionpool:"GET /portal HTTP/1.1" 200 4174
DEBUG:requests.packages.urllib3.connectionpool:"POST /portal?RememberMe=false&timeOffset=480 HTTP/1.1" 200 4300
DEBUG:requests.packages.urllib3.connectionpool:"GET /portal/Account/KeepAlive HTTP/1.1" 500 10889

Right after login, I call /KeepAlive and get a 500.

[jzwack]

OK, well, I can't provide much support for the non-official API.. but

I don't believe /KeepAlive is used anymore. The session is kept alive by the site polling in the background when it's open. I believe the session dies after ~3-5 minutes of inactivity. Keep in mind the site is basically a proxy at this point.

[kk7ds]

@techwood111 I just pushed a fix (hack) to somecomfort that fixes interaction for me, and merged a pending PR for the mode selection. The command line tool should let you poke some things, but you have to provide credentials on every run now, so:

somecomfort --user FOO --password BAR --get_current_temperature

Can you try the latest from there and see if you can at least do that? Then trying to get HA to run against that to see if it's any better would be good.

[texnofobix]

As someone who helps enable APIs for enterprises, thank you @jzwack for your input! I am curious which grant_types is planned with OAuth2?

[techwood111]

@kk7ds: I have two thermostats. Checking via Honeywell, the current temps are 80 and 77. Here is what I experienced, using 0.3.1:

pi@raspberrypi:~ $ somecomfort --user me --password mypass --get_current_temperature
Failed to load data store: /home/pi/.somecomfort
77.0
pi@raspberrypi:~ $ somecomfort --user me --password mypass --get_current_temperature --device 3xxx25
77.0
pi@raspberrypi:~ $ somecomfort --user me --password mypass --get_current_temperature
77.0
pi@raspberrypi:~ $ somecomfort --user me --password mypass --get_current_temperature --device 3xxx95
77.0
pi@raspberrypi:~ $ 

Now, not knowing any better way of doing it remotely (I left town today), I copied the text from the new main.py and client.py at https://github.com/kk7ds/somecomfort/tree/master/somecomfort, then edited the old files, deleting all the text and pasting the new. Here is that output, for the same calls:

pi@raspberrypi:/home/hass/.homeassistant/deps/somecomfort $ somecomfort --user me --password mypass --get_current_temperature
No handlers could be found for logger "somecomfort"
Login failed
pi@raspberrypi:/home/hass/.homeassistant/deps/somecomfort $ somecomfort --user me --password mypass --get_current_temperature --device 3xxx25
77.0
pi@raspberrypi:/home/hass/.homeassistant/deps/somecomfort $ somecomfort --user me --password mypass --get_current_temperature --device 3xxx95
77.0
pi@raspberrypi:/home/hass/.homeassistant/deps/somecomfort $ 

Note that in all cases, we are getting the temperature of one thermostat, and not the other.

Also, you are aware that your flag --devices, which is documented in the readme, no longer exists, right?

I'm not sure if I have done what you asked; I'm not very experienced in this arena! As I mentioned, if you would like to have access to my thermostats for testing (you or anyone else), I'll be glad to grant you that. Just let me know, and I'll fire off an invitation.

[techwood111]

@kk7ds : Making those changes hosed the Climate panel in HA; it no longer displays. Here is the HA log:

EDIT: Resolved. Silly indent error from copy/paste.

16-09-21 22:25:08 homeassistant.components.climate: Error while setting up platform honeywell
Traceback (most recent call last):
  File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/helpers/entity_component.py", line 107, in _setup_platform
    discovery_info)
  File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/components/climate/honeywell.py", line 92, in setup_platform
    return _setup_us(username, password, config, add_devices)
  File "/srv/hass/hass_venv/lib/python3.4/site-packages/homeassistant/components/climate/honeywell.py", line 55, in _setup_us
    import somecomfort
  File "/home/hass/.homeassistant/deps/somecomfort/__init__.py", line 1, in <module>
    from somecomfort.client import *  # NOQA
  File "/home/hass/.homeassistant/deps/somecomfort/client.py", line 491
    return None
              ^
SyntaxError: 'return' outside function

[jzwack]

I'll install this locally tomorrow and take a closer look if I have some spare time.

[jzwack]

@texnofobix Well that depends. Right now we have two API sets, one is completely public via developer.honeywell.com. That covers our Lyric line of products.

We are working on getting the Total Connect Comfort (TCC) line on that site as well, problem is the process of handing out API keys isn't automated yet (it's all on me, manually).

For the Lyric API we support Auth Code only right now.

For the TCC API we support all three flows (Auth Code, Client Credentials, and ROPC).

[techwood111]

@kk7ds - I don't know if I should post my errors here as I get them or what. But, here are some:

16-09-22 02:02:43 homeassistant.components.climate.honeywell: Failed to initialize honeywell client: 
16-09-22 02:11:40 somecomfort: Login as [email protected] failed
16-09-22 02:11:40 homeassistant.components.climate.honeywell: Failed to login to honeywell account [email protected]

[kk7ds]

@techwood111 The error you posted above about the syntax error is almost definitely a result of you hand-applying the changes. Try cloning the repository and running the code exactly as it is:

$ git clone http://....
$ cd somecomfort
$ python -msomecomfort --user ...

And yeah, I saw the bug about the lack of --devices. I'll try to get that fixed. You can use --device if you know the id of the thermostat you're interested in.

[kk7ds]

@techwood111 I just pushed kk7ds/somecomfort@fb01d038f8ba9203608051b1dfc93762731d7c64 Which fixes --devices and --device.

[jzwack]

OK So I lied. /KeepAlive is for sure still a thing, it was just my account doesn't use it because I'm marked as a developer so I don't get force signed-out.

That being said I'm still getting 401 errors with anything past login and when you're doing /CheckDataSession requests.

I'll keep looking.

[kk7ds]

@jzwack Just to be clear, this new API you're talking about is something that would require any user to get an API key for, right? If the API key is per project (i.e. somecomfort would have a key and hardcode that for anyone to use) then we could definitely talk about moving to that. But it sounded like you meant any user of somecomfort would have to get their own key to use it -- is that right?

[jzwack]

@kk7ds Well.. the official API uses OAuth2, as a result to get tokens you need a key and a secret. Generally you wouldn't want those exposed because if some 3rd party got a hold of the refresh token as well we run into big problems.

So yeah my preference would be to have everyone with their own key. Which to your earlier point is hard if it's not self-service signup yet.

EDIT: Obviously the key being exposed isn't a huge deal (you can see that in our own app today) it's more the secret.

[kk7ds]

Ah, sorry I missed the oauth bit in what you said earlier. So, yeah. Sounds like a better long-term solution for sure though :)

[jzwack]

Yeah. Pesky security. :-)

Either way I can give you (or someone else) an API key just for development sake. I hope to have things more self-service by end of year.. but who knows.

Anyway, you're getting cookies right, and I see them in the cookiejar of requests. But still for some reason I get 401 on /portal/Location/GetLocationListData everytime. Odd.

[texnofobix]

OAuth is the right way to go. The only slight challenge is refreshing tokens periodically. :)

[techwood111]

Thank you one and all!

[vapuri]

@jzwack I'm interested in trying out the official Honeywell APIs for RTH9580WF that you mentioned. I registered as a developer. What do I need to do to get the API key?

[arsaboo]

@kk7ds Does the somecomfort package support the Lyric thermostats?

[scholy]

@jzwack are you able to provide any info on how the US and EU API's differ? I'm experiencing the same problem described here but through the evohomeclient and the EU API.