Core: Uncaught thread exception

[gdiciancia]

The problem

Recurring error multiple times a minute showing up in the logs.

Environment

• Home Assistant Core release with the issue: 118.0
• Last working Home Assistant Core release (if known): 117.3
• Operating environment (OS/Container/Supervised/Core): OS
• Integration causing this issue: roomba
• Link to integration documentation on our website:

Problem-relevant configuration.yaml

Traceback/Error logs

Logger: root
Source: /usr/src/homeassistant/homeassistant/bootstrap.py:318
First occurred: 12:03:37 PM (38 occurrences)
Last logged: 12:11:05 PM
Uncaught thread exception

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.8/site-packages/paho/mqtt/client.py", line 3452, in _thread_main
    self.loop_forever(retry_first_connection=True)
  File "/usr/local/lib/python3.8/site-packages/paho/mqtt/client.py", line 1779, in loop_forever
    rc = self.loop(timeout, max_packets)
  File "/usr/local/lib/python3.8/site-packages/paho/mqtt/client.py", line 1197, in loop
    rc = self.loop_write(max_packets)
  File "/usr/local/lib/python3.8/site-packages/paho/mqtt/client.py", line 1598, in loop_write
    rc = self._packet_write()
  File "/usr/local/lib/python3.8/site-packages/paho/mqtt/client.py", line 2374, in _packet_write
    self._do_on_disconnect(0)
  File "/usr/local/lib/python3.8/site-packages/paho/mqtt/client.py", line 3360, in _do_on_disconnect
    self.on_disconnect(self, self._userdata, rc)
TypeError: _internal_on_disconnect() missing 1 required positional argument: 'rc'

Additional information

For MQTT I use:

• Home Assistant Add-on: Mosquitto broker, version 5.1

This are the only custom things I have (I don't use HACS):

• /local/community/auto-entities.js
• /local/community/slider-entity-row.js

[gdiciancia]

Edit: I removed below customization, the issue persists.
/local/community/auto-entities.js
/local/community/slider-entity-row.js

[elupus]

This is a reopen of: https://github.com/home-assistant/core/pull/41479 that seem to have been incorrectly closed.

[elupus]

This is a guess, but our signature for the on disconnect callback does not match v5 semantics here: https://github.com/eclipse/paho.mqtt.python/blob/5a9ca61a6c3071972caddf554490bf9ee5316491/src/paho/mqtt/client.py#L2027

We don't expect a properties parameter in our mqtt integrations here:
https://github.com/home-assistant/core/blob/7f640c4a2e41fdea4bafb5d43bb5dae0ef3314f8/homeassistant/components/mqtt/__init__.py#L1013

[elupus]

Ps. Still does not explain how the exception is logged.

I wonder if /usr/local is really where hassos installs deps.

[probot-home-assistant[bot]]

Hey there @home-assistant/core, @emontnemery, mind taking a look at this issue as its been labeled with an integration (mqtt) you are listed as a codeowner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[emontnemery]

Opening #41479 was my fault, we should not have rolled back PAHO MQTT to 1.5.0 and it was re-bumped back to 1.5.1 shortly after.

The link to paho code is not correct, Home Asssistant core pins v1.5.1, so this is the right version to link to: https://github.com/eclipse/paho.mqtt.python/blob/v1.5.1/src/paho/mqtt/client.py#L3360

The line numbers in the stack trace matches v.1.5.1precisely, so the issue shouldn't be Home Assistant picking up the wrong version of PAHO MQTT.

Our on_disconnect has the right signature for sure: https://github.com/home-assistant/core/blob/aec17d25fa2282423d6294b6f18f2cfd6e1f9b1c/homeassistant/components/mqtt/__init__.py#L1013

It's however not named _internal_on_disconnect. There's in fact no function _internal_on_disconnect neither in PAHO MQTT nor in Home Assistant.

@gdiciancia have you added some other customization or modification which is interfacing with MQTT?

[gdiciancia]

@emontnemery, Not that we can think of. I use the standard mqtt add-on. If you can be more specific, I can look more in detail. I had either wemos connected via mqtt or 1 Shelly 3EM with auto discovery. No customization on Home assistant

[emontnemery]

Try searching for any python file including _internal_on_disconnect

[elupus]

The signature is not really incorrect per say. It's just not compatible with v5 of mqtt: https://github.com/eclipse/paho.mqtt.python/blob/42f0b13001cb39aee97c2b60a3b4807314dfcb4d/src/paho/mqtt/client.py#L467

It can send an additional properties field. That said, i agree the error would seem to be from another version of paho.

@gdiciancia do you think you can uninstall (pip uninstall paho-mqtt) inside the home assistant core container. Then let restart home assistant so it reinstalls it as a dependency.

[gdiciancia]

@elupus, I’ll give that a try a soon as possible

[emontnemery]

@elupus Home Assistant is not using MQTT version 5, and as I mentioned before the callback is named _mqtt_on_disconnect, not _internal_on_disconnect.

Hence, try to find a python package interfacing PAHO, with a function named _internal_on_disconnect.

[gdiciancia]

Found it. I'm using the Roomba integration (added via discovery).

This is what the search found:
Binary file /usr/local/lib/python3.8/site-packages/roomba/__pycache__/mqttclient.cpython-38.pyc matches
/usr/local/lib/python3.8/site-packages/roomba/mqttclient.py:63: mqtt_client.on_disconnect = self._internal_on_disconnect
/usr/local/lib/python3.8/site-packages/roomba/mqttclient.py:89: def _internal_on_disconnect(self, client, userdata, flags, rc):

[emontnemery]

I opened a PR on roombapy to fix this.

[probot-home-assistant[bot]]

roomba documentation
roomba source
_{^{(message by IssueLinks)}}

[probot-home-assistant[bot]]

Hey there @pschmitt, @cyr-ius, @shenxn, mind taking a look at this issue as its been labeled with an integration (roomba) you are listed as a codeowner for? Thanks!
_{^{(message by CodeOwnersMention)}}

[emontnemery]

This should be fixed by #43594 which bumps roombapy to 1.6.2 which includes pschmitt/roombapy#45