Core: Multiple copies of Home Assistant running on docker image since 0.107.0

[aptonline]

The problem

HomeKit integration not starting.

Environment

• Home Assistant release with the issue: 0.107.0
• Last working Home Assistant release (if known): 0.106.6
• Operating environment (Hass.io/Docker/Windows/etc.): docker (synology)
• Integration causing this issue: HomeKit
• Link to integration documentation on our website:

Problem-relevant configuration.yaml

homekit:
  auto_start: true

Traceback/Error logs

Logger: homeassistant.setup
Source: components/homekit/accessories.py:246 
First occurred: 12:56:11 PM (1 occurrences) 
Last logged: 12:56:11 PM

Error during setup of component homekit
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/socketserver.py", line 452, in __init__
    self.server_bind()
  File "/usr/local/lib/python3.7/http/server.py", line 137, in server_bind
    socketserver.TCPServer.server_bind(self)
  File "/usr/local/lib/python3.7/socketserver.py", line 466, in server_bind
    self.socket.bind(self.server_address)
OSError: [Errno 98] Address in use

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/setup.py", line 170, in _async_setup_component
    hass, processed_config
  File "/usr/src/homeassistant/homeassistant/components/homekit/__init__.py", line 135, in async_setup
    await hass.async_add_executor_job(homekit.setup)
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/homekit/__init__.py", line 318, in setup
    advertised_address=self._advertise_ip,
  File "/usr/src/homeassistant/homeassistant/components/homekit/accessories.py", line 246, in __init__
    super().__init__(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/pyhap/accessory_driver.py", line 212, in __init__
    self.http_server = HAPServer(network_tuple, self)
  File "/usr/local/lib/python3.7/site-packages/pyhap/hap_server.py", line 828, in __init__
    super(HAPServer, self).__init__(addr_port, handler_type)
  File "/usr/local/lib/python3.7/socketserver.py", line 455, in __init__
    self.server_close()
  File "/usr/local/lib/python3.7/site-packages/pyhap/hap_server.py", line 899, in server_close
    for sock in list(self.connections.values()):
AttributeError: 'HAPServer' object has no attribute 'connections'

Additional information

[Jc2k]

The bit that stands out is:

    self.socket.bind(self.server_address)
OSError: [Errno 98] Address in use

0.107.0 can't use the port it wants to because something else is. Most likely cause is a stale HA process that hasn't let go of the socket yet. Do you know what port it is trying to use?

[aptonline]

I’m afraid I don’t, is it worth manual specifying another port in then HomeKit config for now.

EDIT: I changed the default port to 51828 and restarted and the error was gone and devices seem to be working again in home app. I changed back to the default port (51827) and the issue returned, looks like something is hogging that port for some reason.

[foreign-sub]

Hi,
try:
lsof -i :51827
this will tell you who's using the port.

[Dycelll]

I've had the exact same problem.
I'm running HA in a docker container and the setup has worked without problems until 0.107 Changing the port to 51828 makes homekit start again.

I then updated to 0.107.1 and set the port to 51827 in yaml. Homekit is now working again.
Also restarted the container a few times. The problem seems to be resolved in 107.1?

Logger: homeassistant.setup
Source: components/homekit/accessories.py:246
First occurred: 8:05:46 PM (1 occurrences)
Last logged: 8:05:46 PM

Error during setup of component homekit
Traceback (most recent call last):
File "/usr/local/lib/python3.7/socketserver.py", line 452, in __init__
self.server_bind()
File "/usr/local/lib/python3.7/http/server.py", line 137, in server_bind
socketserver.TCPServer.server_bind(self)
File "/usr/local/lib/python3.7/socketserver.py", line 466, in server_bind
self.socket.bind(self.server_address)
OSError: [Errno 98] Address in use

[stenehall]

I had the same problem, changing the port for homekit did not solve it for me. Since I didn't have time to investigate this more I tried to revert to 0.106.6 and it worked again.

[Jc2k]

This is very weird - there haven't really been many changes to the homekit code itself in a while. Reviewing the changes since 0.106.0 and its just really minor cleanups like using constants for units instead of strings. Nothing that could cause this. Are you all using Docker? What about Synology? I guess you are using host mode networking?

Has anyone tried using lsof to find what process is stealing the port?

[stenehall]

@Jc2k I did a quick grep and it came up empty. Tried to change the port (I had it set to 51827 before) to 51828 as well as 51826 with the same result.

[Jc2k]

@stenehall and you've tried 0.107.1 as well? And what about the other questions i asked?

[jwelter1971]

This is very weird - there haven't really been many changes to the homekit code itself in a while. Reviewing the changes since 0.106.0 and its just really minor cleanups like using constants for units instead of strings. Nothing that could cause this. Are you all using Docker? What about Synology? I guess you are using host mode networking?

Has anyone tried using lsof to find what process is stealing the port?

It shows it as python3 using that port with the same PID that is the HA instance. It was working without specifying a port in 106.6, but now I need to specify 51829 manually to keep the error from occurring.

[bdraco]

Seeing this as well

[bdraco]

Its like its being started twice

[bdraco]

I'm also seeing http giving the same error

2020-03-19 01:17:11 ERROR (MainThread) [homeassistant.components.http] Failed to create HTTP server at port 8125: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8125): address in use

[jwelter1971]

I'm also seeing http giving the same error

2020-03-19 01:17:11 ERROR (MainThread) [homeassistant.components.http] Failed to create HTTP server at port 8125: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8125): address in use

Are you running in docker under portainer? What do you have your entrypoint set to?

I was told to use /init but that leads to dual instances running. If you leave it blank it seems to be ok.

Prior versions were different, this has been changed in 107 but I dont see it anywhere in the release notes or breaking changes.

[bdraco]

I'm also seeing http giving the same error

2020-03-19 01:17:11 ERROR (MainThread) [homeassistant.components.http] Failed to create HTTP server at port 8125: [Errno 98] error while attempting to bind on address ('0.0.0.0', 8125): address in use

Are you running in docker under portainer? What do you have your entrypoint set to?

Its running on docker on a synology

[bdraco]

There are two copies running in the container

  198   196 root     S     223m   3%   1   1% python3 -m homeassistant --config /config
  225    45 root     S     225m   3%   3   1% python3 -m homeassistant --config /config

[bdraco]

I recreated my synology docker instances from scratch and the problem went away

[stenehall]

@Jc2k sorry, it was late and had to get to bed yesterday.

• Only tried 0.107.1 never 0.107.0.
• Running in docker.
• Giving it its own IP using macvlan.
• Running on a small Nuc on Proxmox in Ubuntu Core (just to make things as complex as possible :))

It's somewhat of a weird setup since I use the backend network to access influxdb, the web network for a reverse proxy and the vlan to be able to broadcast things like homekit. That being said I shouldn't need the external_links, I think.

services:
  home:
    container_name: home
    networks:
      backend:
      web:
      vlan:
        ipv4_address: <ip>
    ports:
      - 1883:1883
      - 5353:5353
      - 8123:8123
      - 51828:51828
    environment:
      - TZ=Europe/Stockholm
      - LANG=C.UTF-8
      - /etc/localtime:/etc/localtime:ro
    external_links:
      - influxdb
      - mariadb
    volumes:
      - './config:/config'
    restart: always
    image: homeassistant/home-assistant:latest
    command: python -m homeassistant --config /config

[stenehall]

I recreated my synology docker instances from scratch and the problem went away

Not sure what you mean with this?. I.e. deleting all volumes?

[leonardpitzu]

issue solved by deleting the container and starting from scratch (of course reusing the configs)
issue was present on synolgy

[stenehall]

I'm not sure how deleting an instance could possibly resolve this problem. Are home assistant persisting anything in the instance outside of the config volume? Wouldn't downloading an updated version of Home assistant more or less be exactly the same as downloading it the first time?

I tried to delete the instance as well as the image and redownload/start but the problem still remains for me.

[frenck]

From the config by @stenehall:

command: python -m homeassistant --config /config

This is what causes it. That manually overrides the CMD to be executed inside the container. In the latest version of Home Assistant, our containers are using a init system. So the init systems start Home Assistant, however, since you force a command, it also starts Home Assistant. Hence twice.

In general: Don't override commands or entry points on a Docker container, unless the default doesn't satisfy your needs. (The above example given by @stenehall was the old default and therefore not needed).

Reasoning for not overriding those cmd/entry points: It will backfire when the container you are using changes...

[stenehall]

@frenck removing that indeed seems to resolve it.

However, why are Home assistant breaking the core priniples of docker? Using a custom init script that cannot be overridden by a command is really against how docker is supposed to work, unless I'm completely missing something here?

Unless I missed it it's also not mentioned in breaking changes, might be good to add it there?

[frenck]

This is perfectly how Docker is supposed to work.
Docker has 2: entry point & command.

We leverage the entry point, you can override the entry point to be something else (e.g., /bin/bash) and use your CMD to your liking.

So it can be overridden...

It is only a breaking change if you did custom overrides. We don't support custom things, hence not a breaking change.

[stenehall]

@frenck ok, thanks for the info.

[aptonline]

Wow this post really exploded, I can confirm that removing command: python -m homeassistant —config /config resolved the issues for me too. Thanks @frenck :)

[olijouve]

I don’t have any issue with homekit nor Zwave with 0.107.x, but homekit doesn’t start automatically on my system, i have an automation that wait for 2 min, after HA restart , before start homekit. It is to be sure Zwave has fully restarted before to load homekit, as stated in documentation.
Don’t know if it helps thus...

[Jc2k]

@olijouve thanks for suggestion but as @frenck said this was an issue with Docker configuration some users had. I'm going to close this as it's solved.

BTW, docs haven't been updated as homekit doesn't have a maintainer atm to test and update, but a couple of releases ago we started storing enough information in the entity registry to be able to configure things like Alexa and HomeKit integrations even if your other integrations are offline. So your delay /might/ not be needed any more. Thats the direction things are headed in, anyway.

[wittypluck]

Hello,
this is indeed an issue as it breaks a working standard installation, at least on Synology docker.
I upgraded to 0.107.7 and ZWave didn't start anymore ("Unavailable" nodes). Main error in the logs was "Failed to create HTTP server at port XXXX". Finally saw that two python processes were running in the docker container.
The solution was to completely delete the docker container and recreate it from scratch, copying over the environment variables etc. Clearly not a painless upgrade.
Again the initial docker configuration was standard, I had not changed the execution path.
This should be at least mentioned in the release notes as a breaking change.
Thanks !

[carlos-sarmiento]

Although I understand why the HA team might not consider this a breaking change I think it should have been mentioned on the release notes as a possible breaking change if you do custom things. I've been going absolutely mad trying to figure out a bunch of issues and the end result was that I was using a custom command for HA.

This situation caused me to file #33647, #33651 and #33648 which has taken time from other contributors to chase down issues that seemed unrelated but are related.

Once again, I don't expect the HA team to work around my custom craziness, but I would appreciate a warning. Most users might be running vanilla HA, but for those of us who have customized our setup, a couple extra lines on the release notes would have made understanding why suddenly things are not working much easier.

[jwelter1971]

Although I understand why the HA team might not consider this a breaking change I think it should have been mentioned on the release notes as a possible breaking change if you do custom things. I've been going absolutely mad trying to figure out a bunch of issues and the end result was that I was using a custom command for HA.

This situation caused me to file #33647, #33651 and #33648 which has taken time from other contributors to chase down issues that seemed unrelated but are related.

Once again, I don't expect the HA team to work around my custom craziness, but I would appreciate a warning. Most users might be running vanilla HA, but for those of us who have customized our setup, a couple extra lines on the release notes would have made understanding why suddenly things are not working much easier.

And here is an example of the troubles this ha caused. The forums are littered with people having issues and not of the tech level to understand they have two instances running. I've see 5 or 6 of such threads already around Zwave, Homekit, and other problems because of multiple instances.

https://community.home-assistant.io/t/upgraded-to-0-108-3-zwave-is-borked/186248