Core: Sudden neato connection error no Vacuum entity created

[faspina]

Home Assistant release with the issue:

arch | x86_64

---

dev | false
docker | true
hassio | false
os_name | Linux
python_version | 3.7.4
timezone | America/Chicago
version | 0.102.2
virtualenv | false

Last working Home Assistant release (if known):
0.102.3

Operating environment (Hass.io/Docker/Windows/etc.):
home-assistant docker (not hass) running on Unraid

Integration:
https://www.home-assistant.io/integrations/neato/

Description of problem:
After restart yesterday HA will not create the vacuum entity for neato, it creates the battery, floor map and schedule status switch fine.

Problem-relevant configuration.yaml entries and (fill out even if it seems unimportant):
Using the integration from configuration>> integration. After deleting and reconnecting it connects fine just does not show my robot. the Neato IOS app work fine.

Traceback (if applicable):
2019-11-29 17:36:57 WARNING (MainThread) [homeassistant.components.vacuum] Setup of platform neato is taking over 10 seconds.
2019-11-29 17:36:57 ERROR (SyncWorker_12) [homeassistant.components.neato.vacuum] Neato vacuum connection error: Unable to communicate with robot
2019-11-29 17:36:57 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 316, in _async_add_entity
device_info = entity.device_info
File "/usr/src/homeassistant/homeassistant/components/neato/vacuum.py", line 329, in device_info
"manufacturer": self._robot_stats["data"]["mfg_name"],
TypeError: 'NoneType' object is not subscriptable

Additional information:
Been working great last 2-3 month I have had the robot. Suddenly its now not creating the vacuum entity.

[dshokouhi]

Try restarting again later on. There were no changes in 0.102 for neato. It looks like you just hit a random API error.

[faspina]

I have deleted the integration , added it back several teams , several reboots, even reset the vacuum and reconnected the wifi, it seems ist just not return a list of vacuum when the integration connects. I can login to neatorobotics.com and see the robot there as well.

[dshokouhi]

This is probably an API error, I suggest waiting and seeing if it works again later on. You can see the API information here that all botvacs have. As the API is in the cloud its probably out of our control for now.

https://developers.neatorobotics.com/api/robot-remote-protocol/common-messages

[faspina]

That make since, I was just wondering if they change it recently and it broke the integration.

[newAM]

I have the identical problem.

2019-11-29 18:49:19 WARNING (MainThread) [homeassistant.components.vacuum] Setup of platform neato is taking over 10 seconds.
2019-11-29 18:49:19 ERROR (SyncWorker_14) [homeassistant.components.neato.vacuum] Neato vacuum connection error: Unable to communicate with robot

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 316, in _async_add_entity
    device_info = entity.device_info
  File "/usr/src/homeassistant/homeassistant/components/neato/vacuum.py", line 329, in device_info
    "manufacturer": self._robot_stats["data"]["mfg_name"],
TypeError: 'NoneType' object is not subscriptable

I'll look into it further - I'm guessing this may also be non-graceful handling of a cloud outage?
Edit: Problem not an outage since the camera sensor works.

[newAM]

Did some debug, looks like neato is having some problems after all.

from pybotvac import Account
import json

for robot in Account("REDACTED", "REDACTED").robots:
    print(f"{robot=}")
    print(f"{robot.get_robot_state()=}")
    print(f"{robot.get_robot_info()=}")
    print(f"{json.dumps(robot.state, indent=4)=}")

robot=<pybotvac.robot.Robot object at 0x7fe15e63be50>
robot.get_robot_state()=<Response [200]>
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/pybotvac/robot.py", line 66, in _message
    response.raise_for_status()
  File "/usr/lib/python3/dist-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Timeout for url: https://nucleo.neatocloud.com/vendors/neato/robots/REDACTED/messages

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./test.py", line 8, in <module>
    print(f"{robot.get_robot_info()=}")
  File "/usr/local/lib/python3.8/dist-packages/pybotvac/robot.py", line 216, in get_robot_info
    return self._message({'reqId': "1", 'cmd': "getRobotInfo"})
  File "/usr/local/lib/python3.8/dist-packages/pybotvac/robot.py", line 69, in _message
    raise NeatoRobotException("Unable to communicate with robot")
pybotvac.exceptions.NeatoRobotException: Unable to communicate with robot

[dshokouhi]

Thanks @newAM I suspected this was the case since no code changes or firmware updates were done recently. I am sure it will resolve itself soon enough.

[probot-home-assistant[bot]]

Hey there @Santobert, mind taking a look at this issue as its been labeled with a integration (neato) you are listed as a codeowner for? Thanks!

[Santobert]

Thanks @newAM you are right. robot.get_robot_info() is the one that fails. Looks like the nucleo servers can't communicate with the robot.

status_code:504
text:'{"message":"Timeout waiting a response from robot"}'
url:'https://nucleo.neatocloud.com/vendors/neato/robots/<serial>/messages'

I think there is not much we can do about it.

[Santobert]

29156 adds a workaround which allows HA to create the vacuum device even if robot.get_robot_info() failed. Can we then close this issue after #29156 has been merged?

[newAM]

Does neato have a status page or similar communication to their customers about this? Seems odd that this has been ongoing for the past 3 days.

[Santobert]

Neato changed their API. We have to fix this in pybotvac. getRobotInfo ist now called getGeneralInfo

[dshokouhi]

Actually @Santobert those are 2 different APIs that give out different data. getGeneralInfo does not provide the info we use for the device registry. I personally do not have this issue when I restart HA so maybe its out of our control? I think the fix you put in place should be good enough until Neato resolves their internal issue.

https://developers.neatorobotics.com/api/robot-remote-protocol/generalinfo

https://developers.neatorobotics.com/api/robot-remote-protocol/common-messages (last API mentioned)

[Santobert]

@dshokouhi maybe you're right. I just looked it up before the first coffee this morning 🙈 so when you're sure that getRobotInfo still exist, there is nothing we can do 😞

[Santobert]

In fact we only use manufacturer and model. Both are part of getGeneralInfo as well. This would work, wouldn't it?

https://github.com/Santobert/home-assistant/blob/aa080508d11363790434b66da90f600978227ff3/homeassistant/components/neato/vacuum.py#L321

The Standard Response. The data element contains:

{
  "productNumber": "905-0321",
  "serial": "ZZZ99999-00000000000002",
  "model": "BotvacConnected",
  "firmware": "2.0.0-861",
  "battery": {
    "level": 3,
    "timeToEmpty": 14320,
    "timeToFullCharge": 1230,
    "totalCharges": 143,
    "manufacturingDate": "2015-02-20",
    "authorizationStatus": 0,
    "vendor": "Vendor Name"
  }
}

[dshokouhi]

@Santobert there is no manufacturer name in getGeneralInfo The vendor listed is actually different than whats in getRobotInfo I believe all the vacuums are made by panasonic? I know it doesn't mean much here as its only for the config entry but the real question is what do we do when this API goes down? I think handling the failure is the best approach but I am ok to switch it as its not critical for functionality.

[faspina]

So what is causing it. I am confused. Is it something on their side or HA. Why is it suddenly not working.

[dshokouhi]

@faspina its something from neato's side. One of their documented API is returning this error. As per their docs all robots support this call. I do not have this issue personally so seems it may be impacting some of neatos servers but not all. We are just discussing how we can bypass this error.

[newAM]

@faspina I do not have this issue personally so seems it may be impacting some of neatos servers but not all.

Perhaps neato is not aware of this then? I found their contact email in the footer of the API reference and sent them an email asking if they are aware out the outage, and if there is an ETA on resolution from their side. I'll post another comment if I hear back from them.

[Santobert]

Thanks @dshokouhi for clarification. I'd prefer to use getGeneralInfo because getRobotInfo is documented as debug endpoint. Maybe debug endpoints are poorly maintained.

I can't proof it but it feels like it's the robot that is not responding and not neatos server. But that makes no difference to us, thought.

[ostinelli]

All,
We've received the request from @newAM. Thank you Alex for contacting us, let me reply here.

We've investigated what you are experiencing, and saw that with some firmware and under certain conditions it is possible that data sent from the robot gets corrupted. When this happens, an API call results in a 504 Gateway Timeout error as the data is filtered on servers and the client never receives a response.

We've found a way to mitigate this server-side, and have just deployed a fix. If what you are experiencing is linked to our findings, this should fix it. Please tell us if this solves your issues.

Also, even though this has nothing to do with the issue at hand, good call on _not_ using getRobotInfo ^^_

Thank you for your input!

Best,
r.

[faspina]

It looks like HA has the fix in the next release anyway. The Neato API service side fix above did not fix my issue when I restart HA. When I get home I will reboot the robot, reset HA and see what happens.