Core: iAquaLink integration goes offline with SSL error

[derekatkins]

The problem

The iAquaLink integration keeps going offline. The logs show it's complaining about an SSL error.

Environment

• Home Assistant Core release with the issue: 0.114.3
• Last working Home Assistant Core release (if known): It MAY have been working on 0.114.2
• Operating environment (OS/Container/Supervised/Core): HassOS 4.12 / HA 0.114.3
• Integration causing this issue: iAquaLink
• Link to integration documentation on our website: https://www.home-assistant.io/integrations/iaqualink/

Problem-relevant configuration.yaml

Integrated through the configuration UI -- no additional yaml provided.

Traceback/Error logs

It started with this error:

2020-08-24 19:23:02 WARNING (MainThread) [iaqualink] <- 504 Gateway Time-out - https://iaqualink-api.realtime.io/v1/mobile/session.json?actionID=command&command=get_home&serial=[REDACTED]&sessionID=[REDACTED]
2020-08-24 19:23:03 ERROR (MainThread) [iaqualink] Unhandled exception: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://iaqualink-api.realtime.io/v1/mobile/session.json?actionID=command&command=get_home&serial=[REDACTED]&sessionID=[REDACTED]')
2020-08-24 19:23:03 ERROR (MainThread) [iaqualink] Traceback (most recent call last):
2020-08-24 19:23:03 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/system.py", line 77, in update
2020-08-24 19:23:03 ERROR (MainThread) [iaqualink]     await self._parse_home_response(r1)
2020-08-24 19:23:03 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/system.py", line 96, in _parse_home_response
2020-08-24 19:23:03 ERROR (MainThread) [iaqualink]     data = await response.json()
2020-08-24 19:23:03 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 1026, in json
2020-08-24 19:23:03 ERROR (MainThread) [iaqualink]     raise ContentTypeError(
2020-08-24 19:23:03 ERROR (MainThread) [iaqualink] aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://iaqualink-api.realtime.io/v1/mobile/session.json?actionID=command&command=get_home&serial=[REDACTED]&sessionID=[REDACTED]')
2020-08-24 19:23:03 ERROR (MainThread) [iaqualink] 
2020-08-24 19:23:03 WARNING (MainThread) [homeassistant.components.iaqualink] Failed to refresh iAqualink state

But then I repeatedly got THIS error (this is the most recent one of the dozens in the log over the past 24 hours):

2020-08-26 08:08:38 ERROR (MainThread) [iaqualink] Unhandled exception: Cannot connect to host iaqualink-api.realtime.io:443 ssl:default [[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1108)]
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink] Traceback (most recent call last):
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 936, in _wrap_create_connection
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     transport, protocol = await self._create_connection_transport(
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1080, in _create_connection_transport
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     await waiter
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/asyncio/sslproto.py", line 529, in data_received
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     ssldata, appdata = self._sslpipe.feed_ssldata(data)
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/asyncio/sslproto.py", line 189, in feed_ssldata
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     self._sslobj.do_handshake()
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/ssl.py", line 944, in do_handshake
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     self._sslobj.do_handshake()
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink] ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1108)
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink] 
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink] The above exception was the direct cause of the following exception:
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink] 
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink] Traceback (most recent call last):
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/system.py", line 75, in update
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     r1 = await self.aqualink.send_home_screen_request(self.serial)
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/client.py", line 157, in send_home_screen_request
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     r = await self._send_session_request(
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/client.py", line 152, in _send_session_request
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     return await self._send_request(url)
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/client.py", line 79, in _send_request
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     r = await self.session.request(
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/client.py", line 480, in _request
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     conn = await self._connector.connect(
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 523, in connect
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     proto = await self._create_connection(req, traces, timeout)
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 858, in _create_connection
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     _, proto = await self._create_direct_connection(
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 1004, in _create_direct_connection
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     raise last_exc
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 980, in _create_direct_connection
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     transp, proto = await self._wrap_create_connection(
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 941, in _wrap_create_connection
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink]     raise ClientConnectorSSLError(req.connection_key, exc) from exc
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink] aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host iaqualink-api.realtime.io:443 ssl:default [[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1108)]
2020-08-26 08:08:38 ERROR (MainThread) [iaqualink] 
2020-08-26 08:08:38 WARNING (MainThread) [homeassistant.components.iaqualink] Failed to refresh iAqualink state
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink] Unhandled exception: Cannot connect to host iaqualink-api.realtime.io:443 ssl:default [[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1108)]
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink] Traceback (most recent call last):
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 936, in _wrap_create_connection
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     return await self._loop.create_connection(*args, **kwargs)  # type: ignore  # noqa
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1050, in create_connection
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     transport, protocol = await self._create_connection_transport(
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1080, in _create_connection_transport
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     await waiter
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/asyncio/sslproto.py", line 529, in data_received
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     ssldata, appdata = self._sslpipe.feed_ssldata(data)
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/asyncio/sslproto.py", line 189, in feed_ssldata
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     self._sslobj.do_handshake()
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/ssl.py", line 944, in do_handshake
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     self._sslobj.do_handshake()
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink] ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1108)
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink] 
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink] The above exception was the direct cause of the following exception:
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink] 
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink] Traceback (most recent call last):
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/system.py", line 75, in update
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     r1 = await self.aqualink.send_home_screen_request(self.serial)
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/client.py", line 157, in send_home_screen_request
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     r = await self._send_session_request(
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/client.py", line 152, in _send_session_request
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     return await self._send_request(url)
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/client.py", line 79, in _send_request
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     r = await self.session.request(
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/client.py", line 480, in _request
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     conn = await self._connector.connect(
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 523, in connect
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     proto = await self._create_connection(req, traces, timeout)
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 858, in _create_connection
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     _, proto = await self._create_direct_connection(
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 1004, in _create_direct_connection
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     raise last_exc
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 980, in _create_direct_connection
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     transp, proto = await self._wrap_create_connection(
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 941, in _wrap_create_connection
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink]     raise ClientConnectorSSLError(req.connection_key, exc) from exc
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink] aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host iaqualink-api.realtime.io:443 ssl:default [[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1108)]
2020-08-26 08:09:08 ERROR (MainThread) [iaqualink] 

Additional information

This SSL error only started yesterday, well after I upgraded to 0.114.3. It is POSSIBLE that it is a server issue, but hard to tell from here.

[probot-home-assistant[bot]]

iaqualink documentation
iaqualink source
_{^{(message by IssueLinks)}}

[probot-home-assistant[bot]]

Hey there @flz, mind taking a look at this issue as its been labeled with an integration (iaqualink) you are listed as a codeowner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[rmertz3282]

I'm seeing the same errors. The integration then becomes unavailable. Multiple times per hour.

[gregkinney]

I had the same error with iaqualink's webpage, so it's on their end. I contacted aqualink tech support today 8/28/20 2:30pm CST and he said they were unaware of it but are aware of it now and are working on it.

[markiper]

I am seeing the same errors, only started noticing this issue after upgrading HA to 0.114.4 (not sure if it was there before). Haven't had any issues with their app or web portal (iAqualink); but I will also keep an eye on it just in case.

---

I just tried using full system backups (HassOS in Proxmox) as far back as 0.114.0; and the same error persists (see below), but no issues at all using the iAqualink app or web portal.

[iaqualink] Unhandled exception: Cannot connect to host iaqualink-api.realtime.io:443 ssl:default [[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1108)]

[flz]

Does the issue come and go? I'm not seeing anything in the logs for my install. I tried running a Docker container of the dev branch and I'm not seeing any errors during setup either.

[markiper]

@flz , yes; the issue is intermittent. For me specifically; errors started happening yesterday (September 2) at around 9PM EST. Every 10 - 15 minutes, connectivity will be lost for a random amount of time (anywhere from 2 to 10 minutes). since 6:33AM this morning (September 3) until right now (7:16AM) there was no connection. I had to restart my HA instance, and that seemed to have forced a resync and the connection was establish. Let me know if there is anything specific you will want me to provide to allow you to troubleshoot this problem.

[derekatkins]

@flz -- yes, as @markiper says, the issue is intermittent. I suspect the issue is really on their end, where one of their servers is "bad" and when we try to connect to that server specifically we get the error. I don't think the debug logs provide enough information to know which server is being used when we get the failure (printing the IP address would be useful, I think, but I don't know if we have access to that information).

I had an automation to notify me when it went offline, which is how I noticed the problem (I did it to detect a breaker issue). But it was going off way too much so I backed it out to a 1m outage, and now to a 5m outage before notifying me. I haven't seen MANY 5m outages, but I still see the errors in my logs implying shorter outages.

I'll note that I have NOT seen issues on the iAquaLink web interface. So even when HA goes unavailable, I can still see the equipment from the weblink interface. However it might be using a different server.

[markiper]

Did a little more troubleshooting this morning, and I can concur with @derekatkins ; the web portal is solid 100% of the time. In my specific case, the iAqualink cloud server that my device is register to is on Amazon AWS @ IP address: 3.221.133.27. They are using TLSv1.2 for authentication, and no matter how I try to break it (other than forcing older TLS version) it communicates properly at all times.

HA on the other hand, reports the system being offline and the error as listed in my previous post

[iaqualink] Unhandled exception: Cannot connect to host iaqualink-api.realtime.io:443 ssl:default [[SSL: WRONG_VERSION_NUMBER] wrong version number

Network connectivity on my end, just to be 100% that the issue was not on my end, I setup a capture to log packet drops, and in 24 hours, I had 2 packets lost, so I can say that network connectivity on my end has been ruled out. I have not been logging internet outages, but I could almost rule that out as well (that is my next task to make sure that the issue is not local or caused by something on my end)

[derekatkins]

Strangely, the aiohttp package had a bug about this, https://github.com/aio-libs/aiohttp/issues/3829 which is closed claiming it's a server configuration issue. However, unless realtime.io is using anycast (which is certainly a possibility) I cannot reproduce the issue here. I am really thinking it's a realtime.io issue here.

[markiper]

Just to give you an idea; in a 12 hour period, this is how many disconnects (grey areas) I have experienced (from HA). Most disconnects are 1min - 3min in length. Today, September 3 (starting at 00:00:00); has been quite unstable compared to yesterday (very few disconnects); so the issue is intermittent and completely random.

[jasonbwilliams]

I'm seeing the same error in the logs and the integration is failing. On my end, it doesn't happen intermittently, it seems to be down consistently, even persisting after full reboot.

[rmertz3282]

I'm seeing the exact same thing today as Markiper. Worse than it has been.

Looking at yesterday it was fairly stable.

[derekatkins]

OOOH! I was just able to reproduce this manually:

curl -1 -0 -v 'https://iaqualink-api.realtime.io/'
*   Trying 99.86.100.129...
* TCP_NODELAY set
* Connected to iaqualink-api.realtime.io (99.86.100.129) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Compared to a successful output:

curl -1 -0 -v 'https://iaqualink-api.realtime.io/'
*   Trying 54.174.124.202...
* TCP_NODELAY set
* Connected to iaqualink-api.realtime.io (54.174.124.202) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=*.realtime.io
*  start date: Feb 21 00:00:00 2020 GMT
*  expire date: Mar 21 12:00:00 2021 GMT
*  subjectAltName: host "iaqualink-api.realtime.io" matched cert's "*.realtime.io"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
> GET / HTTP/1.0
> Host: iaqualink-api.realtime.io
> User-Agent: curl/7.61.1
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Thu, 03 Sep 2020 16:13:29 GMT
< Content-Type: text/html; charset=utf-8
< Connection: close
< Server: Apache/2.4.34 (Unix) mod_perl/2.0.10 Perl/v5.20.2
< 
* TLSv1.2 (IN), TLS alert, close notify (256):
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, close notify (256):
<html><body><h1>It works!</h1><div id='upstreamID'>HA-0A</div></body></html>

I think we've found their bad server!! Their DNS info does change periodically. When I got the bad curl output I saw:

host iaqualink-api.realtime.io
iaqualink-api.realtime.io is an alias for iaqualink-api.iaqualink.net.
iaqualink-api.iaqualink.net is an alias for p-api.iaqualink.net.
p-api.iaqualink.net has address 99.86.100.128
p-api.iaqualink.net has address 99.86.100.101
p-api.iaqualink.net has address 99.86.100.129
p-api.iaqualink.net has address 99.86.100.25

Whereas during good times I see:

host iaqualink-api.realtime.io
iaqualink-api.realtime.io is an alias for iaqualink-api.iaqualink.net.
iaqualink-api.iaqualink.net is an alias for p-api.iaqualink.net.
p-api.iaqualink.net has address 34.224.76.141
p-api.iaqualink.net has address 54.174.124.202

Does anyone have a contact at realtime.io? I suspect this is an error on one of their servers, at least 99.86.100.129 is being problematic.

[flz]

It looks like one of their AWS instances doesn't support TLS downgrade. The other option (besides a server fix) would be ensuring the client supports TLS 1.3. Right now, the iaqualink integration reuses the default HA client session.

[derekatkins]

It's actually all four of the 99.86.100.x servers that I posted above.. I tested all four of them and all four of them fail. The client is offering TLS 1.3 and the server is erroring out!

I've emailed realtime.io with the data I've found. It's unclear if this is a realtime issue, AWS issue, or cloudfront issue.

[derekatkins]

And it's even worse. It doesn't matter WHAT version of TLS I propose -- the server fails to respond to the client hello properly:

$ curl --tls-max 1.2 -v https://99.86.100.101/
*   Trying 99.86.100.101...
* TCP_NODELAY set
* Connected to 99.86.100.101 (99.86.100.101) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number
$ curl --tls-max 1.1 -v https://99.86.100.101/
*   Trying 99.86.100.101...
* TCP_NODELAY set
* Connected to 99.86.100.101 (99.86.100.101) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.1 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number
$ curl --tls-max 1.0 -v https://99.86.100.101/
*   Trying 99.86.100.101...
* TCP_NODELAY set
* Connected to 99.86.100.101 (99.86.100.101) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* TLSv1.0 (OUT), TLS handshake, Client hello (1):
* error:1408F10B:SSL routines:ssl3_get_record:wrong version number
* Closing connection 0
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

Even more interesting, if I connect to port 443 and talk HTTP, I get a response!

telnet 99.86.100.101 443
Trying 99.86.100.101...
Connected to 99.86.100.101.
Escape character is '^]'.
GET / HTTP/1.0

HTTP/1.1 400 Bad Request
Server: CloudFront
Date: Thu, 03 Sep 2020 16:44:55 GMT
Content-Type: text/html
Content-Length: 915
Connection: close
X-Cache: Error from cloudfront
Via: 1.1 fdcba9ad44214c814bcd3af570cf671b.cloudfront.net (CloudFront)
X-Amz-Cf-Pop: IAH50-C3
X-Amz-Cf-Id: MwQrFfET60mqaKeAuKBdsTqJGQSDB1tCGrXyfb6M1AgOnVnTnvm7aA==

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
[rest removed]

SO.... Looks like a CloudFront error!!

[markiper]

@derekatkins you might be absolutely right; a report on that server throws an error right away (all 4 servers give the same report):

https://www.ssllabs.com/ssltest/analyze.html?d=server-99-86-100-101.iah50.r.cloudfront.net

I also run reports on the iaqualink servers; those servers don't have TLS1.3 support (at least not setup)

https://www.ssllabs.com/ssltest/analyze.html?d=iaqualink-api.realtime.io

[derekatkins]

@markiper I think what we're seeing is that normally iaqualink-api.realtime.io reports two IP addresses (and those work fine), but every once in a while it swaps over to the 99.86.100 set of 4, and those are broken -- so when you are unlucky and DNS reports the 99.86 addresses, you get this error until that DNS times out and the next query returns the pair of working servers.

[markiper]

seems like right now there was a change:

Name: p-api.iaqualink.net Addresses: 13.225.198.10 13.225.198.100 13.225.198.72 13.225.198.88

_p-api.iaqualink.net_ covers both aliases (_iaqualink-api.realtime.io_ and _iaqualink-api.iaqualink.net_); and it goes back and forth from the addresses listed above and the ones listed below:

Name: p-api.iaqualink.net Addresses: 34.224.76.141 54.174.124.202

---

and I just confirmed when the alias resolves to the 13.225.xxx.xxx set of IP addresses is when the iAqualik integration in HA reports unavailable, so those must be using the 99.86 DNS that is down

[derekatkins]

Just tested 13.225.198.100 and it has similar issues to the 99.86 hosts. :-(

[markiper]

as long as the _p-api.iaqualink.net_ point to the 34.224 and 54.174 IP addresses, everything works properly; when it changes to the 13.225 set of IP addresses is when the problem happens. Why the change on their end is what I don't understand (load balancing?); luckily for the past 30 minutes there hasn't been any changes, so HA has also been stable.

[derekatkins]

Yes, that is my observation as well. So long as it has the 34.224 and 54.174 addresses it all works. When it changes to something else (13.225 for you, 99.86 for me) it throws errors.

[derekatkins]

Update: I just heard back from my question to realtime.io; they are in the process of migrating their backends, and were load-balancing between the old and new. They have reverted to just their legacy for now, so I suspect you wont see any failures in the immediate future.

(They tried calling me but I couldn't take their call -- I'll call them back shortly).

[markiper]

That could explain this issue. It is almost 3 hours since I started monitoring the _p-api.iqualink.net_ for any changes from the 34.x.x.x or 54.x.x.x IP addresses, and they have not changed at all. Looking at HA logs, the iAqualink integration has not lost connectivity for the same amount of time.

[derekatkins]

@markiper I'm having a zoom call with them in 5 minutes to talk about it. I'll give you more details after we're done.

[flz]

Looks like most issues on my end disappeared after 10.29am PT. For some reason, it shows as UNKNOWN_PROTOCOL on my side, not WRONG_VERSION_NUMBER.

I'm glad you managed to get a hold of someone at realtime.io. It would be nice to see if they could release the full API for iaqualink.

[derekatkins]

So the issue is that they are in the process of moving their APIs to a new frontend using CloudFront. However, they were unaware that people were using the realtime.io URL and not the aqualink.net URL. Cloudfront was not configured for realtime.io, and therefore didn't know what to do with that request when it came in -- so it didn't forward it to their backend and didn't respond to TLS. They had their DNS load balancer resolving 70% to their legacy infrastructure and 30% to CloudFront, but when we hit that 30% we'd lose. This is why it was relatively random.

For now they've got it pointing at Legacy 100% of the time. They know what they need to do for CloudFront so we're going to plan a test tomorrow to make sure it's working.

I can ask them for an API doc, but let me get this tested first for them. ;-)

[markiper]

Thanks @derekatkins ; if you need anything from my side when you are testing, let me know; and hopefully they can give you an API document as well. Last registered change on IP addresses on my end was at 13:29 EST; there was some packet loss at around 16:19 EST; but basically everything working properly since 13:30 EST today.

[derekatkins]

Just a quick update for everyone -- they reached out to me early this week to say they found another problem so they were going to delay (further) the switchover to the new infrastructure. I plan to reach out to them tomorrow or Monday, but so long as they keep it pointed to the legacy infrastructure out integrations will continue to work.

Apparently another option would be to change the URL from iaqualink-api.realtime.io to iaqualink-api.iaqualink.net.

[derekatkins]

HI. Another update. I have been informed that there is now a 10% chance you will get pointed to the new servers. Please let me know if you see any iAqualink cut-outs! Right now all I am seeing in the logs are some 401 errors:

2020-09-21 13:19:15 WARNING (MainThread) [iaqualink] <- 401 Unauthorized - https://iaqualink-api.realtime.io/v1/mobile/session.json?actionID=command&command=get_home&serial=<serial>&sessionID=<sessionID>
2020-09-21 13:19:15 WARNING (MainThread) [iaqualink] <- 401 Unauthorized - https://iaqualink-api.realtime.io/v1/mobile/session.json?actionID=command&command=get_devices&serial=<serial>&sessionID=<sessionID>
2020-09-21 13:19:15 ERROR (MainThread) [iaqualink] Unhandled exception: 0, message='Attempt to decode JSON with unexpected mimetype: text/html; charset=iso-8859-1', url=URL('https://iaqualink-api.realtime.io/v1/mobile/session.json?actionID=command&command=get_home&serial=<serial>&sessionID=<sessionID>')

followed by a traceback showing where in the code we get the error.

I need to hit the 10% state to see if HTTPS is fixed, and then I can try to debug the 401.

[markiper]

Haven't experience any disconnects; but I do see the same 401 error, especially noticeable today between 12:00 and14:00 EST (10 to 12 occurrences lasting a few seconds each). Outside of that range, system has been stable. I will keep an eye an let you know if I see anything on my end.

[derekatkins]

Hi All,
It appears that they have fixed the issues. They have turned on the new servers (with a 10% chance of hitting them). Please let me know if you see any errors in your logs. I did see ONE error at 12:44pm US/ET on Oct 6th, but I've not seen any issues since then, and if I try to manually hit their new server (FTR: curl --resolve iaqualink-api.realtime.io:443:13.225.230.58) with a valid URL I get a valid response!
So, again, please let me know if you see any issues..
I've suggested that they turn the frequency up.

[rmertz3282]

For the most part my system is stable but I still see the occasional error. The latest was at 12am today.

[derekatkins]

For the most part my system is stable but I still see the occasional error. The latest was at 12am today.

This error is similar to the 1 error I saw, but I received it at noon-ish on Oct 6th. I wish it would print out the HTML received on this error. :(

[derekatkins]

Another quick update; I still periodically get an error back:

2020-10-20 03:45:22 WARNING (MainThread) [iaqualink] <- 504 Gateway Time-out - https://iaqualink-api.realtime.io/v1/mobile/session.json?actionID=command&command=get_devices&serial=[serial]&sessionID=[sessionID]
2020-10-20 03:45:22 ERROR (MainThread) [iaqualink] Unhandled exception: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://iaqualink-api.realtime.io/v1/mobile/session.json?actionID=command&command=get_devices&serial=[serial]&sessionID=[sessionID]')
2020-10-20 03:45:22 ERROR (MainThread) [iaqualink] Traceback (most recent call last):
2020-10-20 03:45:22 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/system.py", line 78, in update
2020-10-20 03:45:22 ERROR (MainThread) [iaqualink]     await self._parse_devices_response(r2)
2020-10-20 03:45:22 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/system.py", line 131, in _parse_devices_response
2020-10-20 03:45:22 ERROR (MainThread) [iaqualink]     data = await response.json()
2020-10-20 03:45:22 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/aiohttp/client_reqrep.py", line 1026, in json
2020-10-20 03:45:22 ERROR (MainThread) [iaqualink]     raise ContentTypeError(
2020-10-20 03:45:22 ERROR (MainThread) [iaqualink] aiohttp.client_exceptions.ContentTypeError: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://iaqualink-api.realtime.io/v1/mobile/session.json?actionID=command&command=get_devices&serial=[serial]&sessionID=[sessionID]')
2020-10-20 03:45:22 ERROR (MainThread) [iaqualink] 
2020-10-20 03:45:22 WARNING (MainThread) [homeassistant.components.iaqualink] Failed to refresh iAqualink state

However I haven't seen the aqualink actually go offline...

NB: it would be nice if HA was able to log the message received when it gets a text/html instead of just reporting the error that it's not JSON. Anyone have any idea where that would need to be changed, handling the ContentTypeError?

[rmertz3282]

I'm seeing this error message maybe once per day

Logger: iaqualink
Source: /usr/local/lib/python3.8/site-packages/iaqualink/system.py:85
First occurred: October 25, 2020, 2:19:40 PM (5 occurrences)
Last logged: 7:19:08 AM

Unhandled exception: Cannot connect to host iaqualink-api.realtime.io:443 ssl:default [Try again]
Unhandled exception: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://iaqualink-api.realtime.io/v1/mobile/session.json?actionID=command&command=get_devices&serial=xxxxxxxxxxxx&sessionID=yyyyyyyyy')
Unhandled exception: 0, message='Attempt to decode JSON with unexpected mimetype: text/html', url=URL('https://iaqualink-api.realtime.io/v1/mobile/session.json?actionID=command&command=get_home&serial=xxxxxxxxxxxxxx&sessionID=yyyyyyyy')

[derekatkins]

I'm seeing this error message maybe once per day

I'm not sure I see it once per day, but I'm certainly seeing it. Is there any (easy) way to get HA to log the text/html response so we can see what's ACTUALLY being returned? Knowing the actual IP address used to connect would be useful, too, but that might be harder to acquire.

[markiper]

I hadn't check mine for some time, since it has been mostly stable, but just went through my logs; and I see the same behavior as described by @rmertz3282 . Digging a little deeper, when the error is reported, the first line of of the logs shows the following:

2020-10-25 23:30:03 WARNING (MainThread) [iaqualink] <- 504 Gateway Time-out - https://iaqualink-api.realtime.io/v1/mobile/session.json?actionID=command&command=get_devices&serial=XXXXXXXXXXX&sessionID=YYYYYYYYYYYYY

For the most part the integration has been working correctly. I had an instance in which the entire controller went offline (seems like there was a DNS resolve issue), so I had to power cycle the unit to get it to sync with the DNS (still keeping an eye on this just in case it repeats itself)

[derekatkins]

@markiper when you say:

I had an instance in which the entire controller went offline ... so I had to power cycle the unit

You mean that you had to power-cycle the iAqualink itself? I had to do that, too! I thought it was a local network hiccup. Interesting to find out others had that issue. I don't recall when it was, but it was a couple weeks ago. FTR, I keep an iAqualink web-UI window open at all times, and that was reporting offline, too, so clearly it was an iAqualink -> Cloud problem and not a cloud issue in and of itself. Repeating myself, I thought it was a local-to-me issue because I had a couple other network hiccups at exactly the same time, so I didn't think anything about it.

And also FTR, yes, I have those 504 errors along with mine. I'd still like to see the actual text/html that gets returned, which may give a hint as to who is erroring out.

[markiper]

@derekatkins

"You mean that you had to power-cycle the iAqualink itself?"

Yes. It would have been easier to restart my router (most likely would have also fixed the issue), but due to work from home needs I couldn't, so I went to the iAqualink panel and disconnected/connected the module itself to force a resync. It was about 2-3 weeks ago, and I didn't make anything out of it (figure something happened with my internet connection that drove the iAqualink crazy). Indeed, interesting to see that you also had the same problem; connection was completely lost, mobile app, web app, and Home Assistant all reported the iAqualink module as not present.

Just in case, I use pfSense as my router/firewall, and all my network gear is Ubiquiti (4 different VLANS to separate home, work, guest, and IoT devices).

[derekatkins]

Interesting. It does sound like a "global" glitch that required all iAqualink devices to restart. Yeah, that's around the same timeframe I had my issue. I just literally cycled the whole unit. I have an HA automation to reset it when it comes back. :)

[derekatkins]

Two days ago I had an iAqualink/HA outage for a bit (not exactly sure how long). In the logs I saw this:

2020-11-16 12:21:11 WARNING (MainThread) [iaqualink] <- 403 Forbidden - https://iaqualink-api.realtime.io/v1/mobile/session.json?actionID=command&command=get_home&serial=[serial#]&sessionID=[sessionID]
2020-11-16 12:21:11 WARNING (MainThread) [iaqualink] <- 403 Forbidden - https://iaqualink-api.realtime.io/v1/mobile/session.json?actionID=command&command=get_devices&serial=[serial#]&sessionID=[sessionID]
2020-11-16 12:21:11 ERROR (MainThread) [iaqualink] Unhandled exception: 'home_screen'
2020-11-16 12:21:11 ERROR (MainThread) [iaqualink] Traceback (most recent call last):
2020-11-16 12:21:11 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/system.py", line 77, in update
2020-11-16 12:21:11 ERROR (MainThread) [iaqualink]     await self._parse_home_response(r1)
2020-11-16 12:21:11 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/system.py", line 100, in _parse_home_response
2020-11-16 12:21:11 ERROR (MainThread) [iaqualink]     if data["home_screen"][0]["status"] == "Offline":
2020-11-16 12:21:11 ERROR (MainThread) [iaqualink] KeyError: 'home_screen'
2020-11-16 12:21:11 ERROR (MainThread) [iaqualink] 

Anyone else see this?

[markiper]

Same issue here, 12:15 to 12:54 EST. iAqualink Web Portal was working properly, mobile app will just spit out an error, HA lost its communication as well. I wasn't home when that happened, and it recovered "relatively" quickly, so I didn't think much of it.

[markiper]

Lots of disconnects today (happen to be home and noticed this issue).

2020-11-24 11:55:26 WARNING (MainThread) [iaqualink] <- 500 Internal Server Error - https://iaqualink-api.realtime.io/v1/mobile/session.json?actionID=command&command=get_home&serial=<my_serial>&sessionID=<my_session_ID>

I see this message almost every 1 - 2 minutes (hasn't stop since it started at 11:55 EST). Web portal seems to be accessible; mobile app it is slow to respond but connects. Anyone else experiencing the same thing?

[derekatkins]

Yes, I'm seeing it too. Followed by a "devices_screen" KeyError.

[derekatkins]

I'm seeing this error:

2020-11-24 13:29:41 ERROR (MainThread) [iaqualink] Unhandled exception: 'devices_screen'
2020-11-24 13:29:41 ERROR (MainThread) [iaqualink] Traceback (most recent call last):
2020-11-24 13:29:41 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/system.py", line 78, in update
2020-11-24 13:29:41 ERROR (MainThread) [iaqualink]     await self._parse_devices_response(r2)
2020-11-24 13:29:41 ERROR (MainThread) [iaqualink]   File "/usr/local/lib/python3.8/site-packages/iaqualink/system.py", line 135, in _parse_devices_response
2020-11-24 13:29:41 ERROR (MainThread) [iaqualink]     if data["devices_screen"][0]["status"] == "Offline":
2020-11-24 13:29:41 ERROR (MainThread) [iaqualink] KeyError: 'devices_screen'
2020-11-24 13:29:41 ERROR (MainThread) [iaqualink] 

But I AM seeing data... Most of the time... when I run curl on that URL. Right now I see 4 hosts connected to the hostname:

$ host iaqualink-api.realtime.io
iaqualink-api.realtime.io is an alias for iaqualink-api.iaqualink.net.
iaqualink-api.iaqualink.net is an alias for p-api.iaqualink.net.
p-api.iaqualink.net has address 13.226.94.41
p-api.iaqualink.net has address 13.226.94.18
p-api.iaqualink.net has address 13.226.94.52
p-api.iaqualink.net has address 13.226.94.108

I get valid data from .41, .52, .108 -- but then .52 resulted in the Internal Error:

{"message": "Internal server error"}

So this is definitely an issue on their backend service, and it doesn't depend on the frontend you hit.
The WebUI, AFAIK, is "different".

[markiper]

Mobile has been completely unreachable for the past 10 - 15 minutes; same period as HA without any data from iAqualink; but their WebUI is up and running just fine (certainly data flows differently to mobile vs Web portal). Has to be something on their end, hopefully something that will recover "shortly"